Azure Data Engineer

Role & responsibilities

Key Responsibilities:

• Lead Data Engineering Projects:

  Oversee the development and deployment of end-to-end data ingestion pipelines using Azure Databricks, Apache Spark, and related technologies, ensuring scalability, performance, and efficiency.

• Design & Architecture:

  Design high-performance, resilient, and scalable data architectures for data ingestion and processing using best practices for Azure Databricks and Spark.

• Team Leadership:

  Provide technical guidance and mentorship to a team of data engineers, fostering a culture of collaboration, continuous learning, and innovation.

• Collaboration:

  Work closely with data scientists, business analysts, and other stakeholders to understand data requirements and ensure smooth integration of various data sources into the data lake/warehouse.

• Optimization & Performance Tuning:

  Ensure data pipelines are optimized for speed, reliability, and cost efficiency in an Azure environment. Conduct performance tuning, troubleshooting, and debugging of Spark jobs and Databricks clusters.

• Code Quality & Best Practices:

  Enforce and advocate for best practices in coding standards, version control, testing, and documentation.

• Integration with Azure Services:

  Work with other Azure services such as Azure Data Lake Storage, Azure SQL Data Warehouse, Azure Synapse Analytics, and Azure Blob Storage to integrate data seamlessly.

• Continuous Improvement:

  Stay current with industry trends and emerging technologies in the field of data engineering and make recommendations for improvements to the teams tools and processes.

• Ensure Data Quality:

  Implement data validation and data quality checks as part of the data ingestion process to ensure consistency, accuracy, and integrity of ingested data.

• Risk & Issue Management:

  Proactively identify risks and blockers and resolve complex technical issues in a timely and effective manner.

• Qualifications:

• Education:

  Bachelors or masters degree in computer science, Information Technology, Engineering, or a related field.

Experience:

• 8+ years of experience in data engineering or a related field.
• Strong hands-on experience with

  Azure Databricks

  ,

  Spark

  ,

  Python/Scala, CICD, Scripting

  for data processing.
• Experience working in multiple file formats like

  Parquet

  ,

  Delta

  , and

  Iceberg

  .
• Knowledge of Kafka or similar streaming technologies for real-time data ingestion.
• Experience with data governance and data security in Azure.
• Proven track record of building large-scale data ingestion and ETL pipelines in cloud environments, specifically Azure.
• Deep understanding of

  Azure Data Services

  (e.g., Azure Blob Storage, Azure Data Lake, Azure SQL Data Warehouse, Event Hubs, Functions etc.).
• Familiarity with

  data lakes

  ,

  data warehouses

  , and modern data architectures.
• Experience with

  CI/CD pipelines

  , version control (Git), Jenkins and agile methodologies.
• Understanding of cloud infrastructure and architecture principles (especially within

  Azure

  ).

Technical Skills:

• Expert-level proficiency in

  Spark, SPARK Streaming

  , including optimization, debugging, and troubleshooting Spark jobs.
• Solid knowledge of

  Azure Databricks

  for scalable, distributed data processing.
• Strong coding skills in

  Python

  and

  Scala

  for data processing.
• Experience working with

  SQL

  , especially for large datasets.
• Knowledge of data formats like

  Iceberg

  ,

  Parquet

  ,

  ORC

  , and

  Delta Lake

  .

Leadership Skills:

• Proven ability to lead and mentor a team of data engineers, ensuring adherence to best practices.
• Excellent communication skills, capable of interacting with both technical and non-technical stakeholders.
• Strong problem-solving, analytical, and troubleshooting abilities.