Note: The job is a remote job and is open to candidates in USA. ThunderYard Solutions is seeking a Data Engineer to support the U.S. Department of Veterans Affairs in designing, developing, and maintaining scalable data solutions. The role involves collaborating with cross-functional teams to optimize data pipelines and ensure compliance with federal standards.
Responsibilities
- Design, develop, and maintain ETL/ELT pipelines to ingest, transform, and load data from multiple sources such as APIs, relational databases, cloud storage, and streaming platforms
- Build scalable batch and near real time data pipelines using Databricks and Apache Spark (PySpark / SQL)
- Implement data transformation logic following best practices for performance, reliability, and reusability
- Support schema evolution, data validation, deduplication, and error handling in ETL workflows
- Develop and optimize pipelines using Delta Lake and medallion (Bronze / Silver / Gold) architecture patterns
- Use Databricks Workflows / Jobs or similar orchestration tools to schedule and monitor pipelines
- Optimize Spark jobs for performance and cost (partitioning, caching, file sizing, query tuning)
- Collaborate on data governance initiatives using Unity Catalog, access controls, and lineage where applicable
- Work closely with data architects, analytics teams, and downstream consumers to define data requirements
- Troubleshoot pipeline failures and data quality issues and implement long term fixes
- Produce documentation for pipelines, datasets, and operational runbooks
- Participate in CI/CD practices using Git based version control for notebooks and code deployments
Skills
- 3+ years of experience as a Data Engineer or in a similar data focused role
- Hands on experience with Databricks
- Strong experience building ETL/ELT pipelines
- Proficiency in Python and SQL
- Experience with Apache Spark / PySpark
- Familiarity with cloud platforms such as Azure
- Solid understanding of data modeling, data warehousing, and analytics use cases
- Design, develop, and maintain ETL/ELT pipelines to ingest, transform, and load data from multiple sources such as APIs, relational databases, cloud storage, and streaming platforms
- Build scalable batch and near real time data pipelines using Databricks and Apache Spark (PySpark / SQL)
- Implement data transformation logic following best practices for performance, reliability, and reusability
- Support schema evolution, data validation, deduplication, and error handling in ETL workflows
- Develop and optimize pipelines using Delta Lake and medallion (Bronze / Silver / Gold) architecture patterns
- Use Databricks Workflows / Jobs or similar orchestration tools to schedule and monitor pipelines
- Optimize Spark jobs for performance and cost (partitioning, caching, file sizing, query tuning)
- Collaborate on data governance initiatives using Unity Catalog, access controls, and lineage where applicable
- Work closely with data architects, analytics teams, and downstream consumers to define data requirements
- Troubleshoot pipeline failures and data quality issues and implement long term fixes
- Produce documentation for pipelines, datasets, and operational runbooks
- Participate in CI/CD practices using Git based version control for notebooks and code deployments
- Experience with Delta Live Tables (DLT) or Databricks Auto Loader
- Experience with orchestration tools such as Airflow
- Familiarity with streaming data technologies (Kafka, Event Hubs, Kinesis)
- Experience supporting analytics tools (Power BI, Tableau, Looker) connected to Databricks
- Databricks certification (Associate or Professional)
Benefits
- Medical, dental and vision insurance
- 401k matching
- PTO
- Certification reimbursement
Company Overview