Lead Data Engineer

Candidate,

Hungry Bird IT Consulting Services Pvt Ltd.

Lead Data Engineer

Job Title: Lead Data Engineer

Location:

Role Overview:

Lead Data Engineer

Key Responsibilities:

• Lead the design, development, and optimization of robust ETL pipelines and data workflows
• Architect scalable and secure data solutions using

  Databricks

  and

  AWS

• Manage and optimize data storage using

  AWS S3

  , following data lake and warehouse best practices
• Implement centralized data governance using

  Databricks Unity Catalog

• Develop data APIs using

  FastAPI

  for internal and external integrations
• Use

  Airflow

  for workflow orchestration and automation
• Work with columnar databases like

  Rockset

  ,

  Clickhouse

  , and

  CrateDB

• Ensure data quality, security, and compliance across all systems
• Collaborate with data scientists and analysts to meet analytical and ML needs
• Mentor and guide junior data engineers, promoting code quality and reusability
• Stay updated with new tools and technologies to enhance data infrastructure
• Contribute to cross-functional projects with technical leadership
  

Required Qualifications:

• Bachelors/Masters in Computer Science, Engineering, or a related field
• 5+ years of experience in data engineering, with 2+ years in a lead role
• Strong expertise in

  Python

  ,

  PySpark

  , and

  SQL

• Hands-on experience with

  Databricks

  ,

  AWS (S3, IAM, Lambda, ECR)

• Experience with

  Airflow

  ,

  FastAPI

  , and

  columnar databases

• Deep understanding of

  ETL pipelines

  ,

  data modeling

  , and

  data warehousing

• Familiarity with Git, CI/CD, and Agile methodologies
• Knowledge of data governance and compliance standards
  

Preferred Qualifications:

• Experience with

  real-time data processing/streaming

• Familiarity with

  MLOps

  and

  machine learning workflows

• Certifications in

  Databricks

  and/or

  AWS

• Exposure to

  data mesh

  or

  data fabric architectures

• Understanding of

  metadata management

  and

  data lineage

Tech Stack:

• Languages & Frameworks:

  Python, PySpark, SQL, FastAPI

• Cloud:

  AWS (S3, IAM, Lambda, ECR)

• Big Data:

  Databricks

• Orchestration:

  Airflow

• Databases:

  Rockset, Clickhouse, CrateDB

• Governance:

  Databricks Unity Catalog

aradhana@hungrybird.in

PLEASE MENTION THE RELEVANT POSITION IN THE SUBJECT LINE OF THE EMAIL.

Example: KRISHNA, HR MANAGER, 7 YEARS, 20 20DAYS NOTICE.

Name:

Position applying for:

Total experience:

Notice period:

Current Salary:

Expected Salary:

Thanks and Regards

Aradhana

+919959417171