Azure Data Engineer

Share updated CV to salveen.shaik@covasant.com

Job Title: Azure Data Engineer with Databricks

Location: Hyderabad (WFO)

Experience: 5 8 years

Job Type: Full-time

Client Industry: [e.g., Healthcare, BFSI, ]

Job Level: Senior Engineer / Lead / Architect

Role Overview:

Azure Data Engineer with expertise in Databricks

You will work closely with cross-functional teams including data scientists, architects, business analysts, and client stakeholders to design and implement end-to-end data solutions in a cloud-native environment.

Key Responsibilities:

• Design, develop, and deploy

  end-to-end data pipelines

  using

  Azure Databricks

  ,

  Azure Data Factory

  , and

  Azure Synapse Analytics

  .
• Perform

  data ingestion

  ,

  data wrangling

  , and

  ETL/ELT

  processes from various structured and unstructured data sources (e.g., APIs, on-prem databases, flat files).
• Optimize and tune Spark-based jobs and Databricks notebooks for performance and scalability.
• Implement best practices for

  CI/CD

  ,

  code versioning

  , and

  testing

  in a Databricks environment using

  DevOps pipelines

  .
• Design

  data lake

  and

  data warehouse

  solutions using

  Delta Lake

  and

  Synapse Analytics

  .
• Ensure

  data security

  ,

  governance

  , and

  compliance

  using Azure-native tools (e.g.,

  Azure Purview

  ,

  Key Vault

  ,

  RBAC

  ).
• Collaborate with data scientists to enable

  feature engineering

  and

  model training

  within Databricks.
• Write efficient SQL and PySpark code for data transformation and analytics.
• Monitor and maintain existing data pipelines and troubleshoot issues in a production environment.
• Document technical solutions, architecture diagrams, and data lineage as part of delivery.

Mandatory Skills & Technologies:

• Azure Cloud Services

  : Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage (Gen2), Azure Key Vault, Azure Functions, Azure Monitor

• Databricks Platform

  : Delta Lake, Databricks Notebooks, Job Clusters, MLFlow (optional), Unity Catalog

• Programming Languages

  : PySpark, SQL, Python

• Data Pipelines

  : ETL/ELT pipeline design and orchestration

• Version Control & DevOps

  : Git, Azure DevOps, CI/CD pipelines

• Data Modeling

  : Star/Snowflake schema, Dimensional modeling

• Performance Tuning

  : Spark job optimization, Data partitioning strategies

• Data Governance & Security

  : Azure Purview, RBAC, Data Masking

Nice to Have:

• Experience with

  Kafka

  ,

  Event Hub

  , or other real-time streaming platforms
• Exposure to

  Power BI

  or other visualization tools
• Knowledge of

  Terraform

  or

  ARM templates

  for infrastructure as code
• Experience in

  MLOps

  and integration with

  MLFlow

  for model lifecycle management

Certifications (Good to Have):

• Microsoft Certified: Azure Data Engineer Associate

• Databricks Certified Data Engineer Associate / Professional

• DP-203: Data Engineering on Microsoft Azure

Soft Skills:

• Strong communication and client interaction skills
• Analytical thinking and problem-solving
• Agile mindset with familiarity in

  Scrum/Kanban

• Team player with mentoring ability for junior engineers