This is a remote position.
Role Objective
Lead the architectural design and automation of a Databricks 2.0 ecosystem, focusing on scalable Infrastructure-as-Code (IaaC), multi-environment governance, and decentralized CI/CD orchestration.
Core Responsibilities
- Infrastructure Design: Architect Databricks 2.0 environments (Dev/UAT/Prod) using Terraform, managing networking, compute quotas, and cluster policies.
- Security & Governance: Define RBAC frameworks and Unity Catalog strategies. Manage Microsoft Entra ID integrations, Service Principals, and automated key rotations.
- CI/CD Orchestration: Build automated pipelines for promoting jobs, notebooks, and workflows, enabling independent team deployments via approval-gate processes.
- Environment Parity: Design automated data refresh workflows to sync production data to lower environments for consistent testing.
Technical Stack
- Platform: Databricks 2.0 (Unity Catalog, Workflows, DLT).
- IaC/DevOps: Terraform, GitHub Actions/Azure DevOps, Git.
- Identity: Microsoft Entra ID (Service Principals/RBAC).
- Data: Delta Lake, SQL, Python.
Key Requirements
- Architectural Expertise: Proven track record in designing secure, scalable Data Lakehouses.
- Automation-First Mindset: Experience replacing manual setups with robust, version-controlled infrastructure.
- Leadership: Ability to bridge the gap between Infrastructure (Infra) and Data Engineering (DE) teams.