AWS Data Engineering_Manager

EY – Consulting – AWS Data Engineering Manager

The Opportunity

AWS Data Engineering Manager

Key Responsibilities

• Lead the design and implementation of scalable data pipelines using AWS technologies, supporting both batch and real-time data processing.
• Architect robust data lake solutions based on the Medallion Architecture using Amazon S3 and integrate with Redshift and Oracle for downstream analytics.
• Oversee the development of data ingestion frameworks from diverse sources including on-premise databases, batch files, and Kafka streams.
• Guide the development of Spark streaming applications on Amazon EMR and batch processing using AWS Glue and Python.
• Manage workflow orchestration using Apache Airflow and ensure operational excellence through monitoring and optimization.
• Collaborate with cross-functional teams including data scientists, analysts, and DevOps to align data solutions with business goals.
• Provide technical leadership, mentorship, and performance management for a team of data engineers.
• Engage with clients to understand business requirements, define data strategies, and deliver high-quality solutions.
  

Required Skills And Experience

• Proven leadership experience in managing data engineering teams and delivering complex data solutions.
• Deep expertise in AWS services including S3, Redshift, Glue, EMR, and Oracle.
• Strong programming skills in Python and Spark, with a solid understanding of data modeling and ETL frameworks.
• Hands-on experience with Kafka for real-time data ingestion and processing.
• Proficiency in workflow orchestration tools like Apache Airflow.
• Strong understanding of Medallion Architecture and data lake best practices.
  

Preferred / Nice-to-Have Skills

• Experience with Infrastructure as Code (IaC) using Terraform.
• Familiarity with additional AWS services such as SNS, SQS, DynamoDB, DMS, Athena, and Lake Formation.
• Knowledge of monitoring and alerting tools like CloudWatch, Datadog, or Splunk.
• Understanding of data security best practices for data at rest and in transit.
  

Qualifications

• BTech / MTech / MCA / MBA or equivalent.
• AWS certifications (e.g., AWS Certified Data Analytics – Specialty, AWS Certified Solutions Architect) are a plus.
  

EY | Building a better working world