Tech Lead Databricks Data Platform

About the opportunity

Role: Tech Lead – Databricks / Data Platform Location: Remote Employment Type: Full-Time Experience: 10–12 Years JOB SUMMARY We are looking for an experienced Tech Lead – Databricks / Data Platform to lead the design, development, and optimization of a scalable, secure, and reliable cloud-based data platform. The role will provide technical leadership across Databricks, Azure, CI/CD, Infrastructure as Code, orchestration, data security, FinOps, and platform automation. The ideal candidate will be hands-on, technically strong, and experienced in leading engineering teams while partnering closely with Data Engineers, Analytics Engineers, Architects, Security, and Cloud teams to deliver enterprise-grade data solutions. KEY RESPONSIBILITIES Lead the architecture, design, and implementation of Databricks-based data platforms and cloud-native data pipelines. Design and implement robust CI/CD pipelines for data ingestion, transformation, dbt, SQL, notebooks, and Databricks workloads. Lead end-to-end orchestration of Databricks Jobs and Workflows, including ingestion, transformation, data quality checks, and dependencies. Define and implement automated testing and quality gates within CI/CD pipelines, including schema, data model, and contract validation. Drive Infrastructure as Code (IaC) practices using Terraform and reusable modules for Databricks and Azure infrastructure. Automate Databricks platform operations, including workspace and cluster provisioning, runtime and library management, job deployment, configuration, and environment management. Establish and maintain Identity and Access Management (IAM) across Databricks and Azure, including RBAC, TBAC, service principals, groups, roles, and workspace/cluster/table-level access controls. Work closely with the EDP Architect and Security teams to implement secure and scalable Unity Catalog and data governance patterns. Develop reusable Terraform modules, Databricks job templates, Airflow DAG patterns, and engineering frameworks to accelerate onboarding of new projects. Define and govern the code promotion and release management process across development, QA, and production environments. Lead end-to-end orchestration using managed Airflow, ensuring reliable scheduling, dependency management, monitoring, and recovery. Define and track platform SLAs, SLOs, and SLIs, and lead incident triage, root cause analysis, and corrective actions. Implement FinOps best practices for monitoring, optimizing, and allocating Databricks and cloud infrastructure costs. Continuously evaluate and improve platform tooling, architecture, reliability, security, performance, and developer productivity. Provide technical guidance and mentorship to engineers and establish engineering best practices, standards, and reusable patterns. Partner with Data Engineering, Analytics, Architecture, Security, and Infrastructure teams to ensure seamless delivery of enterprise data solutions. Contribute to cloud infrastructure, cybersecurity, disaster recovery, monitoring, logging, and operational readiness. Drive performance optimization and reliability improvements for production data workloads. REQUIRED QUALIFICATIONS 10–12 years of experience in Data Engineering, Cloud Engineering, DevOps, Platform Engineering, SRE, or related technical disciplines. Strong hands-on experience with Databricks in production environments, including workspace and cluster management, Jobs/Workflows, Unity Catalog, and integration with orchestration platforms. Strong experience with Microsoft Azure and cloud-native data platforms. Strong knowledge of CI/CD and Git-based development workflows, using tools such as Azure DevOps, GitHub Actions, GitLab CI, or similar. Strong experience with Terraform / Infrastructure as Code and cloud infrastructure automation. Hands-on experience with Apache Airflow or managed Airflow for data pipeline orchestration. Strong programming/scripting skills in Python, Bash, or PowerShell. Experience implementing automated testing, validation, and quality gates for data pipelines and data models. Experience with production workload management, including monitoring, logging, troubleshooting, performance tuning, and incident management. Strong understanding of cloud security, IAM, RBAC, data access controls, and governance. Experience with FinOps, cloud cost optimization, and resource utilization monitoring. Ability to provide technical leadership, mentor engineers, and drive engineering standards across teams. Strong communication and stakeholder management skills with the ability to work effectively with technical and business stakeholders. PREFERRED QUALIFICATIONS Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent professional experience. Experience with dbt, SQL, Delta Lake, and modern data engineering practices. Experience with Unity Catalog and enterprise data governance. Experience designing reusable platform frameworks and accelerators. Experience in CPG, retail, manufacturing, or distribution environments. Experience with disaster recovery, business continuity, and highly available cloud architectures. Databricks or Azure certifications are a plus.

Responsibilities

• Lead the architecture, design, and implementation of Databricks-based data platforms and cloud-native data pipelines.
• Design and implement robust CI/CD pipelines for data ingestion, transformation, dbt, SQL, notebooks, and Databricks workloads.
• Lead end-to-end orchestration of Databricks Jobs and Workflows, including ingestion, transformation, data quality checks, and dependencies.
• Define and implement automated testing and quality gates within CI/CD pipelines, including schema, data model, and contract validation.
• Drive Infrastructure as Code (IaC) practices using Terraform and reusable modules for Databricks and Azure infrastructure.
• Automate Databricks platform operations, including workspace and cluster provisioning, runtime and library management, job deployment, configuration, and environment management.
• Establish and maintain Identity and Access Management (IAM) across Databricks and Azure, including RBAC, TBAC, service principals, groups, roles, and workspace/cluster/table-level access controls.
• Work closely with the EDP Architect and Security teams to implement secure and scalable Unity Catalog and data governance patterns.

Qualifications

• 10–12 years of experience in Data Engineering, Cloud Engineering, DevOps, Platform Engineering, SRE, or related technical disciplines.
• Strong hands-on experience with Databricks in production environments, including workspace and cluster management, Jobs/Workflows, Unity Catalog, and integration with orchestration platforms.
• Strong experience with Microsoft Azure and cloud-native data platforms.
• Strong knowledge of CI/CD and Git-based development workflows, using tools such as Azure DevOps, GitHub Actions, GitLab CI, or similar.
• Strong experience with Terraform / Infrastructure as Code and cloud infrastructure automation.
• Hands-on experience with Apache Airflow or managed Airflow for data pipeline orchestration.
• Strong programming/scripting skills in Python, Bash, or PowerShell.
• Experience implementing automated testing, validation, and quality gates for data pipelines and data models.