Lead Data Engineer

Role & responsibilities

• 8+ years

  of experience as a Data Engineer, with a focus on Databricks and cloud-based data platforms, with a minimum of

  4 years

  of experience in writing unit/end-to-end tests for data pipelines and ETL processes on Databricks.
• Hands-on experience in

  PySpark

  programming for data manipulation, transformation, and analysis.
• Strong experience in 

  SQL

  and writing complex queries for data retrieval and manipulation.
• Experience in Docker for containerising and deploying data engineering applications is good to have.
• Strong knowledge of the Databricks platform and its components, including Databricks notebooks, clusters, and jobs.
• Experience in designing and implementing data models to support analytical and reporting needs will be an added advantage.

Preferred candidate profile

you will lead, design, implement, and maintain data processing pipelines and workflows using Databricks on the Azure platform. Your expertise in PySpark, SQL, Databricks, test-driven development, and Docker will be essential to the success of our data engineering initiatives.