AWS Data Engineer

• Design and implement ETL pipelines using AWS Glue and PySpark to transform raw data into consumable formats.
• Automate data processing workflows using AWS Lambda and Step Functions.

Data Integration & Storage:

• Integrate and ingest data from various sources into Amazon S3 and Redshift.
• Optimize Redshift for query performance and cost efficiency.

Data Processing & Analytics:

• Use AWS EMR and PySpark for large-scale data processing and complex transformations.
• Build and manage data lakes on Amazon S3 for analytics use cases. Monitoring & Optimization:
• Monitor and troubleshoot data pipelines to ensure high availability and performance.
• Implement best practices for cost optimization and performance tuning in Redshift, Glue, and EMR.

Terraform & Git-based Workflows:

• Design and implement Terraform modules to provision cloud infrastructure across AWS/Azure/GCP.
• Manage and optimize CI/CD pipelines using Git-based workflows (e.g., GitHub Actions, GitLab CI, Jenkins, Azure DevOps).
• Collaborate with developers and cloud architects to automate infrastructure provisioning and deployments.
• Write reusable and scalable Terraform modules following best practices and code quality standards.
• Maintain version control, branching strategies, and code promotion processes in Git.

Collaboration:

• Work closely with stakeholders to understand requirements and deliver solutions.
• Document data workflows, designs, and processes for future reference.

Must-Have Skills:

• Strong proficiency in 

  Python

   and 

  PySpark

  for data engineering tasks.
• Hands-on experience with 

  AWS Glue, Redshift, S3, and EMR

  .
• Expertise in building, deploying, and optimizing data pipelines and workflows.
• Solid understanding of 

  SQL

   and databas optimization techniques.
• Strong hands-on experience with 

  Terraform

  , including writing and managing modules, state files, and workspaces.
• Proficient in 

  CI/CD pipeline

   design and maintenance using tools like: 

  GitHub

   Actions / GitLab CI / Jenkins / Azure DevOps Pipelines
• Deep understanding of Git workflows (e.g., GitFlow, trunk-based development).
• Experience in serverless architecture using AWS Lambda for automation and orchestration.
• Knowledge of data modeling, partitioning, and schema design for data lakes and warehouses.