Senior Data Engineer + AI

Senior Data Engineer + AI

Job Summary:

Data Engineer

Required Skills & Experience:

• Advanced proficiency in 

  PySpark

  , 

  Apache Spark

  , and 

  Databricks

   for batch and streaming data pipelines.
• Strong experience with 

  SQL

   for data analysis, transformation, and modeling.
• Expertise in 

  data visualization

   and 

  dashboarding tools

   (Power BI, Tableau, Looker).
• Solid understanding of 

  data warehouse design

  , 

  relational databases

   (PostgreSQL, Snowflake, SQL Server), and 

  data lakehouse

   architectures.
• Exposure to 

  Generative AI

  , 

  RAG

  , 

  embedding models

  , and 

  vector databases

   (e.g., FAISS, Pinecone, ChromaDB).
• Experience with 

  Agentic AI frameworks

  : LangChain, Haystack, CrewAI, or similar.
• Familiarity with 

  cloud services

   for data and AI (Azure, AWS, or GCP).
• Excellent problem-solving and collaboration skills with an ability to bridge engineering and business needs.

Preferred Skills:

• Experience with 

  MLflow

  , 

  Delta Live Tables

  , or other Databricks-native AI tools.
• Understanding of 

  prompt engineering

  , 

  LLM deployment

  , and 

  multi-agent orchestration

  .
• Knowledge of 

  CI/CD

  , 

  Git

  , 

  Docker

  , and DevOps pipelines.
• Awareness of 

  Responsible AI

  , 

  data privacy regulations

  , and 

  enterprise data compliance

  .
• Background in consulting, enterprise analytics, or AI/ML product development.

Key Responsibilities:

1. Design, build, and optimize 

   distributed data pipelines

    using 

   PySpark

   , 

   Apache Spark

   , and 

   Databricks

    to support both analytics and AI workloads.
2. Support 

   RAG pipelines

   , 

   embedding generation

   , and 

   data pre-processing

    for LLM applications.
3. Create and maintain 

   interactive dashboards

    and 

   BI reports

    using 

   Power BI

   , 

   Tableau

   , or 

   Looker

    for business stakeholders and consultants.
4. Conduct 

   adhoc data analysis

    to drive data-driven decision making and enable rapid insight generation.
5. Develop and maintain robust 

   data warehouse schemas

   , 

   star/snowflake models

   , and support 

   data lake architecture

   .
6. Integrate with and support 

   LLM agent frameworks

    such as 

   LangChain

   , 

   LlamaIndex

   , 

   Haystack

   , or 

   CrewAI

   for intelligent workflow automation.
7. Ensure data pipeline monitoring, cost optimization, and scalability in 

   cloud environments

    (Azure/AWS/GCP).
8. Collaborate with cross-functional teams including AI scientists, analysts, and business teams to drive use-case delivery.
9. Maintain strong 

   data governance

   , 

   lineage

   , and metadata management practices using tools like 

   Azure Purview

   or 

   DataHub

   .