Data Engineer

About the opportunity

Role: Data Engineer Location: Remote Employment Type: Full-Time Experience: 4-7 years KEY RESPONSIBILITIES · Design and implement CI/CD pipelines for data pipelines (ingestion jobs) and transformation projects (e.g., dbt on Databricks, SQL, notebooks) · Orchestrate Databricks jobs and workflows end-to-end (ingestion, transformation, quality checks) · Integrate automated testing into CI/CD, including schema and contract checks for data models and tables · Implement FinOps best practices to support cost monitoring and allocation across the EDP · Automate platform operations for Databricks and related services, such as workspace and cluster provisioning, library and runtime mgmt. and job deployment/config · Implement and maintain identity and access management for Databricks and supporting cloud resources, including workspace- and cluster-level permissions, table- and view-level access controls (e.g., Unity Catalog or equivalent), service principals, groups, and roles for automated workloads, and RBAC & TBAC models in collaboration with EDP Architect · Provide patterns, templates, and reusable modules (Terraform modules, Airflow DAG patterns, Databricks job templates) to accelerate onboarding of new projects · Continuously evaluate and improve tooling, pipelines, and platform architecture to increase reliability, security, and developer productivity on Databricks · Define the code promotion process to minimize impacts across domains as code is promoted to production · Manage end-to-end orchestration using managed Airflow. Contribute to defining and tracking SLA/SLO/SLIs for the platform and participate in incident response (triage, root cause analysis) · Practical knowledge of IT Infrastructure technologies, cloud computing Azure), cybersecurity, and disaster recovery. · Working knowledge of Azure ecosystem, including hands-on experience designing, building, and optimizing scalable data pipelines within cloud-native environments. QUALIFICATIONS · Hands-on experience with Databricks in production environment, including workspace and cluster management, jobs/workflows and integrations with orchestration tools · Strong experience with CI/CD pipelines (e.g., GitHub Actions, GitLab CI, Azure DevOps, or similar) and Git-based workflows. · Strong experience with Infrastructure as Code (IaC) and orchestration tools for provisioning and managing Databricks and cloud infrastructure. · Experience implementing automated tests and quality gates in CI/CD pipelines. · Ability to partner effectively with data engineers, analytics engineers, architects, and security teams. MINIMUM EXPERIENCE & EDUCATION · Bachelor’s degree in Computer Science, Information Technology, or related field preferred, or equivalent work experience. · 4–7+ years in DevOps, Cloud Engineering, Site Reliability Engineering, or Platform Engineering, with at least 2+ years supporting data/analytics platforms. · Experience operating production data workloads, including monitoring, logging, performance tuning, and incident response. · Scripting skills (e.g., Python, Bash, PowerShell) for automation and integration. · Experience in CPG, retail, manufacturing, or distribution environments preferred.

Responsibilities

• · Design and implement CI/CD pipelines for data pipelines (ingestion jobs) and transformation projects (e.
• , dbt on Databricks, SQL, notebooks) · Orchestrate Databricks jobs and workflows end-to-end (ingestion, transformation, quality checks) · Integrate automated testing into CI/CD, including schema and contract checks for data models and tables · Implement FinOps best practices to support cost monitoring and allocation across the EDP · Automate platform operations for Databricks and related services, such as workspace and cluster provisioning, library and runtime mgmt.
• and job deployment/config · Implement and maintain identity and access management for Databricks and supporting cloud resources, including workspace- and cluster-level permissions, table- and view-level access controls (e.
• , Unity Catalog or equivalent), service principals, groups, and roles for automated workloads, and RBAC & TBAC models in collaboration with EDP Architect · Provide patterns, templates, and reusable modules (Terraform modules, Airflow DAG patterns, Databricks job templates) to accelerate onboarding of new projects · Continuously evaluate and improve tooling, pipelines, and platform architecture to increase reliability, security, and developer productivity on Databricks · Define the code promotion process to minimize impacts across domains as code is promoted to production · Manage end-to-end orchestration using managed Airflow.
• Contribute to defining and tracking SLA/SLO/SLIs for the platform and participate in incident response (triage, root cause analysis) · Practical knowledge of IT Infrastructure technologies, cloud computing Azure), cybersecurity, and disaster recovery.

Qualifications

• · Hands-on experience with Databricks in production environment, including workspace and cluster management, jobs/workflows and integrations with orchestration tools · Strong experience with CI/CD pipelines (e.
• , GitHub Actions, GitLab CI, Azure DevOps, or similar) and Git-based workflows.
• · Strong experience with Infrastructure as Code (IaC) and orchestration tools for provisioning and managing Databricks and cloud infrastructure.
• · Experience implementing automated tests and quality gates in CI/CD pipelines.
• · Ability to partner effectively with data engineers, analytics engineers, architects, and security teams.
• MINIMUM EXPERIENCE & EDUCATION · Bachelor’s degree in Computer Science, Information Technology, or related field preferred, or equivalent work experience.
• · 4–7+ years in DevOps, Cloud Engineering, Site Reliability Engineering, or Platform Engineering, with at least 2+ years supporting data/analytics platforms.
• · Experience operating production data workloads, including monitoring, logging, performance tuning, and incident response.