Staff System Test Engineer

Job Description

Job Area: Engineering Group, Engineering Group > Systems Test Engineering General Summary: We are seeking a Senior Staff AI System-Level Test Engineer to lead end-to-end testing of Retrieval-Augmented Generation (RAG) AI systems for Hybrid, Edge-AI Inference solutions. This role will focus on designing, developing, and executing comprehensive test strategies for evaluating the reliability, accuracy, usability and scalability of large-scale AI models integrated with external knowledge retrieval systems. The ideal candidate needs to have deep expertise in AI testing methodologies, experience with large language models (LLMs), expertise in building test solutions for AI Inference stacks, RAG, search/retrieval architecture, and a strong background in automation frameworks, performance validation, and building E2E automation architecture. Experience testing large-scale generative AI applications, familiarity with LangChain, LlamaIndex, or other RAG-specific frameworks, and knowledge of adversarial testing techniques for AI robustness are preferred qualifications Key Responsibilities: Test Strategy & Planning Define end-to-end test strategies for RAG, retrieval, generation, response coherence, and knowledge correctness Develop test plans & automation frameworks to validate system performance across real-world scenarios. Hands-on experience in benchmarking and optimizing Deep Learning Models on AI Accelerators/GPUs Implement E2E solutions to integrate Inference systems with customer software workflows Identify and implement metrics to measure retrieval accuracy, LLM response quality Test Automation Build automated pipelines for regression, integration, and adversarial testing of RAG workflows. Validate search relevance, document ranking, and context injection into LLMs using rigorous test cases. Collaborate with ML engineers and data scientists to debug model failures and identify areas for improvement. Conduct scalability and latency tests for retrieval-heavy applications. Analyze failure patterns, drift detection, and robustness against hallucinations and misinformation. Collaboration Work closely with AI research, engineering teams & customer teams to align testing with business requirements. Generate test reports, dashboards, and insights to drive model improvements. Stay up to date with the latest AI testing frameworks, LLM evaluation benchmarks, and retrieval models. Required Qualifications: 8+ years of experience in AI/ML system testing, software quality engineering, or related fields. Bachelors or masters degree in computer science engineering/ data science / AI/ML Hands-on experience with test automation frameworks (e.g., PyTest, Robot Framework, JMeter). Proficiency in Python, SQL, API testing, vector databases (e.g., FAISS, Weaviate, Pinecone) and retrieval pipelines. Experience with ML model validation metrics (e.g., BLEU, ROUGE, MRR, NDCG). Expertise in CI/CD pipelines, cloud platforms (AWS/GCP/Azure), and containerization (Docker, Kubernetes). Why Join Us Work on cutting-edge AI retrieval-augmented generation technologies Collaborate with world-class AI researchers and engineers. If you are passionate about AI system testing and ensuring the reliability of next-generation generative models, apply now! Minimum Qualifications: Bachelor's degree in Engineering, Information Systems, Computer Science, or related field and 6+ years of Systems Test Engineering or related work experience. OR Master's degree in Engineering, Information Systems, Computer Science, or related field and 5+ years of Systems Test Engineering or related work experience. OR PhD in Engineering, Information Systems, Computer Science, or related field and 4+ years of Systems Test Engineering or related work experience.