AZURE DATA ENGINEER

"Key Responsibilities:
Data Pipeline Development
Design and implement robust ETL/ELT pipelines using Azure Data Factory and Databricks (PySpark, SQL, Delta Live Tables).
Ingest and transform data from diverse sources including APIs, databases, file systems, and third-party platforms.
Apply Medallion Architecture principles to organize data into bronze, silver, and gold layers for analytics and ML.
Cloud Integration & Optimization
Leverage Azure services such as ADLS Gen2, Azure Synapse, Azure Functions, and Event Hub for scalable data solutions.
Optimize Spark jobs and data workflows for performance, cost-efficiency, and reliability.
Integrate pipelines into CI/CD workflows using Azure DevOps or GitHub Actions.
Governance, Security & Monitoring
Implement data governance practices including metadata management, lineage tracking, and access control using Unity Catalog, Azure Purview, or similar tools.
Ensure compliance with data privacy and security standards (e.g., GDPR, HIPAA).
Monitor pipeline health and data quality using Azure Monitor, Log Analytics, and custom alerting.
Collaboration & Leadership
Collaborate with data scientists, analysts, and business stakeholders to deliver curated, analytics-ready datasets.
Participate in architecture reviews, code reviews, and performance tuning sessions.
Mentor junior data engineers and contribute to best practices in data engineering and DevOps.
Required Skills & Experience:
6+ years of experience in data engineering, with 3+ years working with Azure and Databricks.
Strong hands-on experience with:
Azure Data Factory, ADLS Gen2, Azure Functions
Databricks (PySpark, SQL, Delta Lake, Workflows)
Medallion Architecture and lakehouse design
CI/CD tools (Azure DevOps, GitHub Actions)
Data quality, observability, and monitoring tools
Preferred Qualifications:
Databricks Certified Data Engineer Associate or Professional
Microsoft Certified: Azure Data Engineer Associate
Experience with Unity Catalog, Azure Purview, or Informatica
Familiarity with MLflow, Feature Store, or MLOps practices
Knowledge of data modeling, schema design, and performance tuning
Expected Outcomes:
Scalable, secure, and high-performance data pipelines supporting analytics and AI
Well-governed and discoverable data assets across the enterprise
Improved data quality, observability, and operational efficiency
Strong collaboration between data engineering, analytics, and platform teams"