AI Data Engineer

Job Type:

Experience Level:

About the Role:

We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in Python, Azure, CI/CD, deployment, and writing efficient data pipelines. You will be responsible for designing, developing, and maintaining robust data infrastructure and pipelines that support our data-driven decision-making processes.

Key Responsibilities:

• AI Integration:

  Implement LLM-based workflows, RAG pipelines, and agentic orchestration using frameworks like LangChain, LangGraph, CrewAI, or similar.

• Data Pipeline Development:

  Design, develop, and maintain efficient and scalable data pipelines using Python and Azure tech stack.

• Collaboration:

  Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

• Automation & Deployment:

  Containerize and deploy solutions using Docker, Kubernetes, and cloud-native architectures (Azure/AWS/GCP).

• Monitoring & Governance:

  Ensure data quality, observability, and compliance with security standards.

• Documentation:

  Create and maintain comprehensive documentation for data pipelines, processes, and solutions.

• Troubleshooting:

  Identify and resolve data-related issues and bottlenecks.

Qualifications:

• Education:

  Bachelor's or Master's degree in Computer Science, Information Technology, or equivalent in related field.

• Experience:

  6-8 years of experience in data engineering, with a strong focus on Python, Azure, CI/CD, and deployment.

• Technical Skills:

• Machine Learning Fundamentals: Model training, feature engineering.
• Generative AI & LLMs: Fine-tuning GPT/BERT, prompt engineering, RAG pipelines.
• Data Engineering Expertise: Advanced ETL/ELT, handling structured/unstructured data.
• Cloud & Big Data Tools: Spark, Hadoop, Kafka, AWS/Azure/GCP.
• Programming: Python (dominant for AI), SQL, ML libraries (TensorFlow, PyTorch).
• Proficiency in Python, Apis for data processing and automation.
• Extensive experience with Azure cloud services (Azure Data Factory, Azure Databricks, Azure SQL Database, etc.).
• Understanding of CI/CD principles and tools (e.g., Jenkins, GitLab CI/CD).
• Understandin with containerization and orchestration tools (e.g., Docker, Kubernetes).
• Knowledge of SQL and NoSQL databases.
• Familiarity with data warehousing concepts and technologies.

• Soft Skills:

• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.
• Ability to work independently and as part of a team.
• Detail-oriented with a focus on quality and accuracy.

Preferred Qualifications:

• Experience with big data technologies (e.g., Hadoop, Spark).
• Knowledge of machine learning and data science concepts.
• Certification in Azure or other relevant technologies.

Benefits:

• Competitive salary and performance-based bonuses.
• Health, dental, and vision insurance.
• Retirement savings plan with company match.
• Professional development opportunities.
• Flexible working hours and remote work options.