Senior Data Engineer - AI | ML

Job Summary:

Senior Data Engineer

Key Responsibilities:

• Design and implement

  data pipelines

  for structured, semi-structured and unstructured data from diverse sources (real-time and batch).
• Develop and maintain

  data lakes, data warehouses, and feature stores

  for scalable ML model training and inference.
• Collaborate with AI/ML teams to build

  model-ready datasets

  , automate feature extraction, and streamline

  MLOps

  workflows.
• Optimize data systems for

  performance, scalability, security

  , and

  cost-efficiency

  .
• Build and manage ETL/ELT processes using tools like

  Apache Airflow

  ,

  dbt

  , or custom Python jobs.
• Integrate

  real-time streaming data

  using Kafka, Spark Streaming, or Flink for use cases like fraud detection, recommendation engines, etc.
• Work with

  cloud services

  (AWS/GCP/Azure) to provision infrastructure and deploy production-grade systems.
• Implement

  data governance, quality checks, and monitoring

  to ensure trust in data pipelines.
• Collaborate in

  cross-functional teams

  using Agile methodologies and DevOps best practices.

Required Skills & Qualifications:

Core Data Engineering:

• 5+ years in

  data engineering roles

  , designing complex pipelines and data architectures.
• Strong command of

  Python

  and

  SQL

  for ETL, data Wrangling, and scripting.
• Experience with

  Apache Spark

  ,

  PySpark

  ,

  Kafka

  , or

  Flink

  .
• Hands-on with

  data orchestration

  tools like

  Airflow

  ,

  Dagster

  , or

  Prefect

  .
• Proficient in

  data modeling

  ,

  partitioning

  , and

  schema design

  for analytics and ML.

Cloud & DevOps:

• 3+ years working on

  cloud platforms

  : AWS (S3, Redshift, EMR, Glue), GCP (BigQuery, Dataflow), or Azure (ADF, Synapse).
• Experience with

  Terraform

  ,

  CloudFormation

  , or infrastructure-as-code is a plus.
• Familiar with

  containerization

  using Docker and

  CI/CD

  for data pipelines.

AI/ML Integration:

• Familiar with

  ML lifecycle

  , including feature engineering, model versioning, and deployment.
• Worked with tools like

  MLflow

  ,

  Kubeflow

  ,

  Vertex AI

  , or

  SageMaker

  .
• Exposure to AI frameworks like

  TensorFlow

  ,

  PyTorch

  , or

  scikit-learn

  .
• Experience working with

  vector databases

  (e.g., Pinecone, FAISS, Weaviate) for semantic search is a plus.

Other Skills:

• Excellent understanding of

  data privacy, security, and compliance

• Strong communication skills to collaborate with business and technical stakeholders.
• Problem-solving mindset with attention to performance and reliability.

Preferred Qualifications:

• Bachelor's or Master's in Computer Science, Engineering, or Data-related field.
• Certifications in

  cloud platforms

  (AWS/GCP/Azure) or

  Databricks

  .
• Published work, contributions to open-source, or AI/data blogs are a plus.