Senior Data Engineer (AWS)

Roles & Responsibilities

• Data Engineering Excellence

  :
• Design and implement data pipelines using formats like JSON, Parquet, CSV, and ORC, utilizing batch and streaming ingestion.

• Cloud Data Migration Leadership

  :
• Lead cloud migration projects, developing scalable Spark pipelines.

• Medallion Architecture

  :
• Implement Bronze, Silver, and gold tables for scalable data systems.

• Spark Code Optimization

  :
• Optimize Spark code to ensure efficient cloud migration.

• Data Modeling

  :
• Develop and maintain data models with strong governance practices.

• Data Cataloging & Quality

  :
• Implement cataloging strategies with Unity Catalog to maintain high-quality data.

• Delta Live Table Leadership

  :
• Lead the design and implementation of Delta Live Tables (DLT) pipelines for secure, tamper-resistant data management.

• Customer Collaboration

  :
• Collaborate with clients to optimize cloud migrations and ensure best practices in design and governance.

Educational Qualifications

• Experience

  :
• Minimum 5 years of hands-on experience in data engineering, with a proven track record in complex pipeline development and cloud-based data migration projects.

• Education

  :
• Bachelors or higher degree in Computer Science, Data Engineering, or a related field.

• Skills

  :
• Proficiency in Spark, SQL, Python, and other relevant data processing technologies.
• Expertise in on-premises to cloud Spark code optimization and Medallion Architecture.
• Strong knowledge of Databricks and its components, including Delta Live Table (DLT) pipeline implementations.
• Familiarity with AWS services (experience with additional cloud platforms like GCP or Azure is a plus).

• Soft Skills

  :
• Excellent communication and collaboration skills, with the ability to work effectively with clients and internal teams.

Good to Have

• Certifications

  :
• AWS/GCP/Azure Data Engineer Certification.