Data Engineer

Job Role: Data Engineer

YOE:8 to 10Y

Locations: Bangalore, Hyderabad, Pune, Mumbai,

We are seeking an experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have over 8 years of experience in designing and implementing large-scale data processing pipelines. The role requires extensive hands-on expertise in Python and SQL scripting, with a strong understanding of modern data frameworks like Apache Airflow and Apache Spark. Exposure to cloud technologies, specifically AWS (S3, Lambda, Glue), is essential.

The successful candidate will be instrumental in building, maintaining, and optimizing data systems that power data-driven decisions across the organization.

Key Responsibilities:

• Design, develop, and optimize data pipelines and workflows to process large-scale datasets efficiently.
• Implement and manage Apache Airflow for orchestrating ETL pipelines and ensuring data availability.
• Use Apache Spark for distributed data processing, optimization, and ensuring high-performance analytics.
• Write clean, efficient, and scalable Python scripts for data extraction, transformation, and loading (ETL).
• Develop and optimize complex SQL queries for data analysis and reporting.
• Manage cloud data storage (primarily AWS S3) and compute services (AWS Lambda, AWS Glue) to support data operations.
• Collaborate with cross-functional teams (Data Analysts, Data Scientists, Business Analysts) to ensure data availability, integrity, and quality.
• Troubleshoot and optimize existing systems to ensure smooth data flow and minimal downtime.
• Lead the technical direction for the development of data engineering practices across the team.

Required Skills and Qualifications:

• 8+ years

  of professional experience in data engineering, data analysis, or related fields.
• Expertise in

  Python

  for building and optimizing data processing workflows.
• Advanced proficiency in

  SQL

  for querying and manipulating large datasets.
• Experience with

  Apache Airflow

  for orchestrating complex data workflows and managing dependencies.
• Solid experience in

  Apache Spark

  for distributed data processing and optimization of performance.
• Exposure to

  AWS

  technologies, specifically

  S3

  (for storage),

  Lambda

  (for serverless compute), and

  Glue

  (for ETL and data cataloging).
• Strong understanding of data modeling, data pipelines, and ETL best practices.
• Experience with version control tools like

  Git

  .
• Ability to work with large, complex datasets and troubleshoot performance bottlenecks.
• Strong problem-solving skills and attention to detail.
• Ability to work independently and in collaborative team environments.

Preferred Skills (Nice to Have):

• Experience with additional cloud services such as

  AWS Redshift

  ,

  AWS Athena

  , or

  AWS EMR

  .
• Familiarity with data visualization tools like

  Tableau

  ,

  Power BI

  , or similar.
• Experience with

  Docker

  or containerization for building deployable environments.
• Knowledge of

  CI/CD

  pipelines for automating data processing workflows.

Thanks & Regards,

Yuvaraj U

Client Engagement Executive | Vy Systems

Mobile: 9150019640 | Email: yuvaraj@vysystems.com

www.vysystems.com