Senior Data Engineer + AI

5 - 9 years

0 Lacs

Posted:3 days ago| Platform: Shine logo

Apply

Work Mode

On-site

Job Type

Full Time

Job Description

As a Senior Data Engineer + AI, you will play a crucial role in designing and optimizing distributed data pipelines using PySpark, Apache Spark, and Databricks to cater to both analytics and AI workloads. Your expertise in PySpark, Apache Spark, and Databricks for batch and streaming data pipelines will be instrumental in contributing to high-impact programs with clients. Your strong SQL skills for data analysis, transformation, and modeling will enable you to drive data-driven decision-making and facilitate rapid insight generation. Your responsibilities will involve supporting RAG pipelines, embedding generation, and data pre-processing for LLM applications, as well as creating and maintaining interactive dashboards and BI reports using tools such as Power BI, Tableau, or Looker. You will collaborate with cross-functional teams, including AI scientists, analysts, and business teams, to ensure the successful delivery of use cases. In this role, you will need to have a solid understanding of data warehouse design, relational databases such as PostgreSQL, Snowflake, SQL Server, as well as data lakehouse architectures. Your familiarity with cloud services for data and AI, such as Azure, AWS, or GCP, will be essential for ensuring data pipeline monitoring, cost optimization, and scalability in cloud environments. Furthermore, your exposure to Generative AI, RAG, embedding models, and vector databases like FAISS, Pinecone, ChromaDB, as well as experience with Agentic AI frameworks such as LangChain, Haystack, CrewAI, will be beneficial. Your knowledge of MLflow, Delta Live Tables, or other Databricks-native AI tools, CI/CD, Git, Docker, and DevOps pipelines will also be advantageous in this role. If you have a background in consulting, enterprise analytics, or AI/ML product development, it will further enhance your ability to excel in this position. Your excellent problem-solving and collaboration skills, coupled with your ability to bridge engineering and business needs, will be key to your success as a Senior Data Engineer + AI.,

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now

RecommendedJobs for You