Data Engineer

Job Title:

Experience:

Role Summary:

We are looking for a skilled Data Engineer with solid experience in the Azure data ecosystem and hands-on expertise in Azure Databricks. The ideal candidate will have worked on designing and developing scalable, cloud-native data pipelines and solutions using Azure data services. You will contribute to building and optimizing data lakehouse and warehouse architectures, enabling efficient data ingestion, transformation, and analytics in a collaborative team environment.

Roles and Responsibilities:

• Design, develop, and maintain scalable data pipelines and workflows using Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Azure Data Lake Storage (ADLS).
• Implement ETL/ELT processes using Azure Data Factory and Databricks to enable efficient data ingestion and transformation.
• Work on building data solutions leveraging Delta Lake and implement best practices such as the Medallion Architecture.
• Collaborate with data scientists, analysts, and business teams to understand data requirements and deliver effective Azure-based data solutions.
• Participate in code reviews, testing, deployment, and documentation to ensure high-quality data engineering deliverables.
• Follow data governance, security, and compliance standards while handling data pipelines and sensitive information.
• Implement and manage secure authentication and authorization using Azure Active Directory (AAD) and Azure Key Vault for secrets management.
• Support CI/CD automation and monitoring of data pipelines using Azure DevOps.
• Stay current with Azure data platform developments and contribute ideas to improve existing data infrastructure and processes.

Must-Have Skills:

• 4+ years of experience in data engineering with a strong focus on Azure data services.
• Hands-on experience with Azure Databricks, Spark, PySpark, Delta Lake, and implementing Medallion Architecture is highly desirable.
• Proficiency with Azure Data Factory for building ETL/ELT pipelines.
• Working knowledge of Azure Synapse Analytics and Azure Data Lake Storage (ADLS).
• Strong programming skills in Python and SQL (T-SQL). PowerShell experience is a plus.
• Experience with Azure Active Directory (AAD) for authentication and role-based access control.
• Familiarity with Azure Key Vault for secrets and certificate management.
• Understanding of modern data lakehouse architectures and cloud-native data platform principles on Azure.
• Basic knowledge of CI/CD and DevOps processes, preferably using Azure DevOps.
• Good communication skills with the ability to work collaboratively in cross-functional teams.

Good-to-Have Skills:

• Experience with Delta Live Tables, MLflow, or Unity Catalog on Azure Databricks.
• Familiarity with Azure Event Hubs, Azure Stream Analytics, or other streaming data services.
• Awareness of Azure Purview or other data governance tools.
• Basic exposure to Snowflake in an Azure environment, including data modeling and performance tuning.
• Experience with Infrastructure as Code (IaC) tools like ARM templates or Terraform on Azure.
• Understanding of machine learning workflows and MLOps integration within Azure.