Senior Data Engineer -Azure (Databricks, Data Factory, Data Lake Storage, SQL).

Avant de postuler à un emploi, sélectionnez votre langue de préférence parmi les options disponibles en haut à droite de cette page.

Fiche De Poste

Job Summary

Responsibilities

• Generates application documentation.
• Contributes to systems analysis and design.
• Designs and develops moderately complex applications.
• Contributes to integration builds.
• Contributes to maintenance and support.
• Monitors emerging technologies and products.
  

Technical Skills

• Cloud Platforms: Azure (Databricks, Data Factory, Data Lake Storage, Synapse Analytics).
• Data Processing: Databricks (PySpark, Spark SQL), Apache Spark.
• Programming Languages: Python, SQL
• Data Engineering Tools: Delta Lake, Azure Data Factory, Apache Airflow
• Other: Git, CI/CD
  

Professional Experience

• Design and implementation of a scalable data lakehouse on Azure Databricks, optimizing data ingestion, processing, and analysis for improved business insights.
• Develop and maintain efficient data pipelines using PySpark and Spark SQL for extracting, transforming, and loading (ETL) data from diverse sources.(Azure and GCP).
• Strong Proficiency in SQL & Develop SQL stored procedures for data integrity. Ensure data accuracy and consistency across all layers.
• Implement Delta Lake for ACID transactions and data versioning, ensuring data quality and reliability.
• Create frameworks using Databricks and Data Factory to process incremental data for external vendors and applications.
• Implement Azure functions to trigger and manage data processing workflows.
• Design and implement data pipelines to integrate various data sources and manage Databricks workflows for efficient data processing.
• Conduct performance tuning and optimization of data processing workflows.
• Provide technical support and troubleshooting for data processing issues.
• Experience with successful migrations from legacy data infrastructure to Azure Databricks, improving scalability and cost savings.
• Collaborate with data scientists and analysts to build interactive dashboards and visualizations on Databricks for data exploration and analysis.
  

Qualifications

• Minimum 8 years of Relevant experience
• Bachelor’s Degree or International equivalent
• Bachelor's Degree or International equivalent in Computer Science, Information Systems, Mathematics, Statistics or related field
  

Type De Contrat