Databricks Administrator

We are seeking a Senior/Lead DevOps Engineer – Databricks with strong experience in Azure Databricks to design, implement, and optimize Databricks infrastructure, CI/CD pipelines, and ML model deployment. The ideal candidate will be responsible for Databricks environment setup, networking, cluster management, access control, CI/CD automation, model deployment, asset bundle management, and monitoring. This role requires hands-on experience with DevOps best practices, infrastructure automation, and cloud-native architectures.

Required Skills & Experience

• 4 to 10 years of experience in DevOps with a strong focus on Azure Databricks.

• Hands-on experience with Azure networking, VNET integration, and firewall rules.

• Strong knowledge of Databricks cluster management, job scheduling, and optimization.

• Expertise in CI/CD pipeline development for Databricks and ML models using Azure DevOps, Terraform, or GitHub Actions.

• Experience with Databricks Asset Bundles (DAB) for packaging and deployment.

• Proficiency in RBAC, Unity Catalog, and workspace access control.

• Experience with Infrastructure as Code (IaC) tools like Terraform, ARM Templates, or Bicep.

• Strong scripting skills in Python, Bash, or PowerShell.

• Familiarity with monitoring tools (Azure Monitor, Prometheus, or Datadog).

Preferred Qualifications

• Databricks Certified Associate/Professional Administrator or equivalent certification.

• Experience with AWS or GCP Databricks in addition to Azure.

• Knowledge of Delta Live Tables (DLT), Databricks SQL, and MLflow.

• Exposure to Kubernetes (AKS, EKS, or GKE) for ML model deployment.

Roles & Responsibilities

Key Responsibilities

1. Databricks Infrastructure Setup & Management

• Configure and manage Azure Databricks workspaces, networking, and security.

• Set up networking components like VNET integration, private endpoints, and firewall configurations.

• Implement scalability strategies for efficient resource utilization.

• Ensure high availability, resilience, and security of Databricks infrastructure.

2. Cluster & Capacity Management

• Manage Databricks clusters, including autoscaling, instance selection, and performance tuning.

• Optimize compute resources to minimize costs while maintaining performance.

• Implement cluster policies and governance controls.

3. User & Access Management

• Implement RBAC (Role-Based Access Control) and IAM (Identity and Access Management) for users and services.

• Manage Databricks Unity Catalog and enforce workspace-level access controls.

• Define and enforce security policies across Databricks workspaces.

4. CI/CD Automation for Databricks & ML Models

• Develop and manage CI/CD pipelines for Databricks Notebooks, Jobs, and ML models using Azure DevOps, GitHub Actions, or Jenkins.

• Automate Databricks infrastructure deployment using Terraform, ARM Templates, or Bicep.

• Implement automated testing, version control, and rollback strategies for Databricks workloads.

• Integrate Databricks Asset Bundles (DAB) for standardized and repeatable Databricks deployments.

5. Databricks Asset Bundle Management

• Implement Databricks Asset Bundles (DAB) to package, version, and deploy Databricks workflows efficiently.

• Automate workspace configuration, job definitions, and dependencies using DAB.

• Ensure traceability, rollback, and version control of deployed assets.

• Integrate DAB with CI/CD pipelines for seamless deployment.

6. ML Model Deployment & Monitoring

• Deploy ML models using Databricks MLflow, Azure Machine Learning, or Kubernetes (AKS).

• Optimize model performance and enable real-time inference.

• Implement model monitoring, drift detection, and automated retraining pipelines.

7. Monitoring, Troubleshooting & Performance Optimization

• Set up Databricks monitoring and logging using Azure Monitor, Datadog, or Prometheus.

• Analyze cluster performance metrics, audit logs, and cost insights to optimize workloads.

• Troubleshoot Databricks infrastructure, pipelines, and deployment issues.