Lead Azure Data Engineer

Job Description

Key Responsibilities • Design and implement streaming data pipelines integrating Kafka with Databricks using Structured Streaming. • Architect and maintain Medallion Architecture with well-defined Bronze, Silver, and Gold layers. • Implement efficient ingestion using Databricks Autoloader for high-throughput data loads. • Work with large volumes of structured and unstructured data, ensuring high availability and performance. • Apply performance tuning techniques such as partitioning, caching, and cluster resource optimization. • Collaborate with cross-functional teams (data scientists, analysts, business users) to build robust data solutions. • Establish best practices for code versioning, deployment automation, and data governance. Required Technical Skills: • Strong expertise in Azure Databricks and Spark Structured Streaming • 7+ Years’ experience in Data Engineering • Processing modes (append, update, complete) • Output modes (append, complete, update) • Checkpointing and state management • Experience with Kafka integration for real-time data pipelines • Deep understanding of Medallion Architecture • Proficiency with Databricks Autoloader and schema evolution • Deep understanding of Unity Catalog and Foreign catalog • Strong knowledge of Spark SQL, Delta Lake, and DataFrames • Expertise in performance tuning (query optimization, cluster configuration, caching strategies) • Must have Data management strategies • Excellent with Governance and Access management • Strong with Data modelling, Data warehousing concepts, Databricks as a platform • Solid understanding of Window functions Show more Show less