Senior Quality Engineer (AI Platform Testing)

• The Software Product Engineering (SPE) organization is actively looking for a Senior Quality Engineer with strong hands-on experience in AI platform testing, chatbot testing, AI model validation, agents testing, AI test automation, and API testing.
• This is a highly specialized role focused on validating complex AI and ML systems and ensuring scalable, safe, and effective deployment of AI-based solutions.
• UI automation experience using tools like Selenium, Cypress, or Playwright is desirable as a secondary skill.

What You ll Be Doing:

• You will drive quality engineering initiatives specifically focused on AI-powered platforms and solutions, including LLMs, chatbots, AI agents, and intelligent workflows.
• You ll build robust test strategies and frameworks to validate data pipelines, model inference accuracy, prompt engineering, hallucination control, API contracts, and performance under real-world conditions.
• This role requires strong analytical and problem-solving skills, a deep understanding of AI systems testing, and the ability to collaborate across multidisciplinary teams such as SWE, SRE, ML Engineering, and Product.

• AI Platform & Model Testing (Primary Focus):
  • Validate the behaviour and performance of AI/ML models, including LLMs , RAG pipelines , chatbots , and autonomous agents .
  • Design and execute prompt evaluation , response accuracy , toxicity detection , and hallucination control test scenarios.
  • Implement and enhance automated AI testing frameworks tailored to model versioning, retraining, and feedback loops.
  • Ensure quality in human-in-the-loop (HITL) and continuous learning pipelines.
• API Testing:
  • Conduct thorough API validation using Postman , REST Assured , or GraphQL , with a focus on AI service endpoints, inference APIs, and orchestrators.
  • Build robust integration test suites to ensure seamless functionality between APIs and underlying AI systems.
• AI Test Automation:
  • Build test harnesses to validate AI features through synthetic data, mock services, and model stubs.
  • Integrate test suites into CI/CD pipelines to ensure continuous validation of AI behaviors.
• UI and Functional Test Automation (Secondary Focus):
  • Support end-to-end automation of AI-powered applications using tools such as Selenium , Cypress , Playwright , and WebdriverIO .
  • Automate critical user journeys involving AI-enabled decisions and interactions.
• Collaboration & Test Strategy:
  • Work closely with ML Engineers , SREs , and Product Managers to translate model design into testable components.
  • Monitor AI behavior in production using observability tools and adjust quality strategies based on live insights.
  • Drive discussions on fairness , bias , explainability , and model drift .
• Agile & DevOps Integration:
  • Participate in Agile ceremonies and actively contribute to sprint planning, test case reviews, and retrospectives.
  • Collaborate with DevOps teams to embed AI testing into CI/CD workflows using tools like GitHub , Jenkins , and Azure DevOps .

• Bachelor s or Master s degree in Computer Science, Engineering, AI/ML, or a related field
• 6+ years of experience in Quality Engineering with at least 2 years in AI platform testing or model validation
• Hands-on experience in AI model testing, chatbot testing, prompt tuning , or agent workflows
• Proficiency in AI test automation and API testing tools (Postman, REST Assured, GraphQL)
• Working knowledge of Python , JavaScript , or TypeScript
• Experience integrating tests into CI/CD pipelines using GitHub , Jenkins , or Azure DevOps
• Knowledge of OpenAI , Bedrock , Anthropic , LangChain , RAG , and vector stores
• Understanding of LLM evaluation techniques , including metrics like BLEU , ROUGE , Toxicity Score , and RAGAs

• Experience testing AI applications hosted in multi-geographical and cloud-native environments (e.g., AWS, GCP, Azure)
• Exposure to AI observability platforms such as Weights & Biases , Arize AI , or WhyLabs
• Understanding of prompt engineering , embedding quality , and tokenization behaviour
• Familiarity with security , performance , or accessibility testing
• Experience with AI governance frameworks and regulatory compliance (e.g., FDA, HIPAA in AI contexts)