Data Engineer Python

Data Engineer - Python

We are looking for a Data Engineer with experience in building data pipelines and implementing AI/ML solutions in Azure. This role involves integrating structured and unstructured data sources for efficient retrieval and processing in support of OpenAI-based RAG pipelines

Responsibilities

• Design and implement data pipelines using Azure Data Factory or equivalent.
• Set up and manage SQL databases and integrate with Azure AI Search.
• Prepare and store embeddings for RAG using Azure Vector Search.
• Ensure data quality, versioning, and security with Azure Blob, Key Vault, and Monitoring.
• Collaborate with prompt engineers and backend teams to optimize data flow.
• Collaborating with various stakeholders to determine software requirements.
• Design and develop logical flows for each business requirement
• Prepare technical documentation for each feature and guide/coach the junior developers during the implementation phase.
• Work closely with the other members of the backend and frontend team to integrate different components into the applications.
• Researching and implementing new technologies for the product
• Troubleshooting and resolving issues with coding or design.
• Testing the final product to ensure it is completely functional and meets requirements.

Requirements

• SQL, Azure Data Factory, Azure Blob, Azure Key Vault
• Experience with vector stores and embeddings
• Familiarity with Azure OpenAI and AI Search
• Data modeling and performance tuning
• REST API integration and scripting
• Experience with building software using Python.
• Experience with building applications with micro services & serverless architecture
• 4+ years of experience in software development roles
• Good communication skills

Technical Skills

• Language/ Framework:

  Python, Azure Data Factory, One Lake, Azure Blob, Azure Key Vault

• Database:

  Postgres, SQL Server

• OS:

  Unix/Linux/Windows/Serverless

• Others(good to have)

  Machine Learning Model Building, NLP – NLTK, Spacy, NumPy etc.

Location

• Bangalore