Data Engineer With Microsoft Fabric

Microsoft Fabric

Key Responsibilities

• Design and Development:

  Architect, design, develop, and implement end-to-end data solutions within the Microsoft Fabric ecosystem, including

  Lakehouse, Data Warehouse, and Real-Time Analytics

  components.

• Data Pipeline Construction:

  Build, test, and maintain robust and scalable data pipelines for data ingestion, transformation, and curation from diverse sources using

  Microsoft Fabric Data Factory (Pipelines and Dataflows Gen2)

  and

  Azure Databricks (PySpark/Scala)

  .

• Data Modeling & Optimization:

  Develop and optimize data models within Microsoft Fabric, adhering to best practices for performance, scalability, and data integrity (e.g., dimensional modeling).

• ETL/ELT Processes:

  Implement efficient ETL/ELT processes to extract data from various sources, transform it into suitable formats, and load it into the data lakehouse or analytical systems.

• Performance Tuning:

  Continuously monitor and fine-tune data pipelines and processing workflows to enhance overall performance and efficiency, especially for large-scale datasets.

• Data Quality & Governance:

  Design and implement data quality, validation, and reconciliation processes to ensure data accuracy and reliability. Ensure data security and compliance with data privacy regulations.

• Collaboration:

  Work closely with data architects, data scientists, business intelligence developers, and business stakeholders to understand data requirements and translate them into technical solutions.

• Automation & CI/CD:

  Implement CI/CD pipelines for data solutions within Azure DevOps or similar tools, ensuring automated deployment and version control.

• Troubleshooting:

  Troubleshoot and resolve complex data-related issues and performance bottlenecks.

• Documentation:

  Maintain comprehensive documentation for data architectures, pipelines, data models, and processes.

• Stay Updated:

  Keep abreast of the latest advancements in Microsoft Fabric, Azure data services, and data engineering best practices.

Required Skills & Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related quantitative field, or equivalent practical experience.
• 6+ years of hands-on experience as a Data Engineer or Data Architect.

• Mandatory hands-on experience with Microsoft Fabric

  , including its core components such as

  Lakehouse, Data Warehouse, and Data Factory (Pipelines, Dataflows Gen2), and Spark notebooks.

• Strong expertise in Microsoft Azure data services, including:

• Azure Databricks

  (PySpark/Scala for complex data processing and transformations).

• Azure Data Lake Storage Gen2

  (for scalable data storage).

• Azure Data Factory

  (for ETL/ELT orchestration).
• Proficiency in

  SQL

  for data manipulation and querying.
• Experience with

  Python

  or Scala for data engineering tasks.
• Solid understanding of data warehousing concepts, data modeling (dimensional, relational), and data lakehouse architectures.
• Experience with version control systems (e.g., Git, Azure Repos).
• Strong analytical and problem-solving skills with a keen eye for detail.
• Excellent communication (written and verbal) and interpersonal skills to collaborate effectively with cross-functional teams.