It's fun to work in a company where people truly BELIEVE in what they're doing!
We're committed to bringing passion and customer focus to the business.
Public Partnerships LLC supports individuals with disabilities or chronic illnesses and aging adults, to remain in their homes and communities and “self” direct their own long-term home care. Our role as the nation’s largest and most experienced Financial Management Service provider is to assist those eligible Medicaid recipients to choose and pay for their own support workers and services within their state-approved personalized budget. We are appointed by states and managed healthcare organizations to better serve more of their residents and members requiring long-term care and ensure the efficient use of taxpayer funded services.
Job Summary
We are seeking an experienced Azure Data Engineer to design, build, and optimize scalable data pipelines and lake house architectures within the Azure ecosystem. This role is responsible for developing end-to-end data ingestion, transformation, and governance frameworks using Azure Data Factory, Databricks, and ADLS Gen2 to support enterprise analytics and reporting needs.
The ideal candidate will have hands-on experience managing batch and near real-time data ingestion from diverse structured and unstructured sources, implementing Spark-based transformation processes across Bronze, Silver, and Gold data layers, and establishing robust data quality, security, and monitoring frameworks. This position also requires expertise in data governance, automation, and cost optimization to ensure high-performing, secure, and reliable data platforms aligned with organizational SLAs..
Key Responsibilities
Azure Data Engineering & Lakehouse Architecture
Design and manage scalable data ingestion pipelines using Azure Data Factory and Azure Service Bus for batch and near real-time processing
Implement Bronze, Silver, and Gold data layer transformations using Databricks (PySpark, SparkSQL) within a lakehouse architecture
Integrate structured, semi-structured, and unstructured data sources using formats such as Parquet, Avro, and JSON into ADLS Gen2
Data Governance, Security & Quality Management
Establish data quality frameworks and validation processes using Databricks and Azure Data Factory
Configure RBAC and ACL-based access controls to secure sensitive datasets and ensure compliance
Manage credential security with Azure Key Vault and enforce governance standards across data platforms.
Automation, Monitoring & Cloud Optimization
Automate pipeline orchestration using Databricks Workflows and CI/CD tools such as Azure DevOps, GitHub Actions, and Terraform
Monitor pipeline health and performance using Azure Monitor, logging, and alerting to meet SLA requirements
Optimize ADLS storage performance and manage cloud costs using Azure Cost Management best practices
All other duties assigned
Required Skills:
Azure Data Factory for ETL/ELT operations.
Build and support self-service BI solutions using Azure Databricks and the Lakehouse architecture
Develop data pipelines and transformations using PySpark and Spark SQL
Enable analytics using Databricks AI/BI features and Genie for business users
Apply ML techniques for data insights, predictions, and feature engineering
Collaborate with analytics and business teams to deliver scalable, governed BI and ML solutions
Databricks (PySpark, SparkSQL) for data transformations.
Azure Data Lake Storage (ADLS Gen2).
SQL Server, PostgreSQL, Cosmos DB, CRM, and ERP systems.
Data Governance, RBAC, and ACLs for managing permissions.
CI/CD tools: Azure DevOps, GitHub Actions, Terraform.
Azure Monitoring, Alerting, and Logging tools.
Qualifications:
Education:
Bachelor’s degree in Computer Science, Information Systems, Data Science, or a related field (Master’s preferred).
Experience:
Minimum 7–10 years experience
Substantial professional experience in a related field may be considered in lieu of formal degree.
Preferred Attributes:
Must have AI/ML Skill
Compensation: $90,000-$120,000/annually
Working Conditions:
Remote work.
The above is intended to describe the general contents and requirements of work being performed by people assigned to this classification. It is not intended to be construed as an exhaustive statement of all duties, responsibilities, or skills of personnel so classified
PPL is an Equal Opportunity Employer dedicated to celebrating diversity and intentionally creating a culture of inclusion. We believe that we work best when our employees feel empowered and accepted, and that starts by honoring each of our unique life experiences. At PPL, all aspects of employment regarding recruitment, hiring, training, promotion, compensation, benefits, transfers, layoffs, return from layoff, company-sponsored training, education, and social and recreational programs are based on merit, business needs, job requirements, and individual qualifications. We do not discriminate on the basis of race, color, religion or belief, national, social, or ethnic origin, sex, gender identity and/or expression, age, physical, mental, or sensory disability, sexual orientation, marital, civil union, or domestic partnership status, past or present military service, citizenship status, family medical history or genetic information, family or parental status, or any other status protected under federal, state, or local law. PPL will not tolerate discrimination or harassment based on any of these characteristics.
If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us!