Azure DataBricks Data Engineering Manager

Responsibilities

• An Azure Data Engineer is responsible for designing, building, and maintaining the data infrastructure for an organization using Azure cloud services.
• This includes creating data pipelines, integrating data from various sources, and implementing data security and privacy measures.
• The Azure Data Engineer will also be responsible for monitoring and troubleshooting data flows and optimizing data storage and processing for performance and cost efficiency.

Mandatory Skills

• Bachelor's or higher degree in Computer Science, Data Engineering, or a related discipline, or equivalent (minimum 10+ years overall work experience).
• Expertise in designing and implementing high-performance data ingestion pipelines using Databricks and other relevant technologies.
• Proven track record of delivering and presenting proofs of concept for key technology components to project stakeholders.
• Development of scalable and reusable frameworks for ingesting, processing, and enriching datasets using Databricks.
• Integration of end-to-end data pipelines, ensuring data is efficiently transported from various source systems to target data stores.
• At least 7+ years of experience in developing data ingestion, data processing, and analytical pipelines using big data technologies, relational databases, and data warehouse solutions.
• Extensive hands-on experience implementing data ingestion and ETL processes on Databricks, leveraging technologies like Spark, Delta Lake, and MLflow.
• Expertise in Spark, SQL, Python, and Scala for data processing and analytics on Databricks.
• Strong knowledge of cloud migration strategies and tools, including Databricks-specific solutions.
• Familiarity with Big Data file formats and compression techniques.
• Proficiency in cloud architectures and understanding the pros and cons of different migration approaches.
• Proven ability to work independently under pressure while managing multiple priorities.
• Excellent communication skills, both written and verbal, with the ability to collaborate in a team environment.
• Experience working with engineering teams to define best practices in data engineering, including agile methodologies, coding standards, and CI/CD processes.