LLM Ops EngineerOpenAI OR Anthropic OR Bedrock) AND (AWS) AND (LLM OR Pattern Technologies

3.0 - 5.0 years

20 - 35 Lacs

Pune

Hybrid

Role Overview :- Monitor, evaluate, and optimize AI/LLM workflows in production environments. Ensure reliable, efficient, and high-quality AI system performance by building out an LLM Ops platform that is self-serve for the engineering and data science departments. Key Responsibilities:- Collaborate with data scientists and software engineers to integrate an LLM Ops platform (Opik by CometML) for existing AI workflows Identify valuable performance metrics (accuracy, quality, etc) for AI workflows and create on-going sampling evaluation processes using the LLM Ops platform that alert when metrics drop below thresholds Cross-team collaboration to create datasets and benchmarks for new AI workflows Run experiments on datasets and optimize performance via model changes and prompt adjustments Debug and troubleshoot AI workflow issues Optimize inference costs and latency while maintaining accuracy and quality Develop automations for LLM Ops platform integration to empower data scientists and software engineers to self-serve integration with the AI workflows they build Requirements:- Strong Python programming skills Experience with generative AI models and tools (OpenAI, Anthropic, Bedrock, etc) Knowledge of fundamental statistical concepts and tools in data science such as: heuristic and non-heuristic measurements in NLP (BLEU, WER, sentiment analysis, LLM-as-judge, etc), standard deviation, sampling rate, and a high level understanding of how modern AI models work (knowledge cutoffs, context windows, temperature, etc) Familiarity with AWS Understanding of prompt engineering concepts People skills: you will be expected to frequently collaborate with other teams to help to perfect their AI workflows Experience Level 3-5 years of experience in LLM/AI Ops, MLOps, Data Science, or MLE

Posted 2 weeks ago

Apply

Senior LLM Engineer Nomiso

7.0 - 12.0 years

4 - 8 Lacs

Bengaluru, Karnataka, India

On-site

Roles and Responsibilities: Model Expertise : Work with transformer models such as GPT , BERT , T5 , RoBERTa , and others for a variety of NLP tasks, including text generation, summarization, classification, and translation. Model Fine-Tuning : Fine-tune pre-trained models on domain-specific datasets to improve performance for specific applications such as summarization, text generation, and question answering. Prompt Engineering : Craft clear, concise, and contextually relevant prompts to guide transformer-based models towards generating desired outputs for specific tasks. Iterate on prompts to optimize model performance. Instruction-Based Prompting : Implement instruction-based prompting to guide the model toward achieving specific goals, ensuring that the outputs are contextually accurate and aligned with task objectives. Zero-shot, Few-shot, Many-shot Learning : Utilize zero-shot , few-shot , and many-shot learning techniques to improve model performance without the need for full retraining. Chain-of-Thought (CoT) Prompting : Implement Chain-of-Thought (CoT) prompting to guide models through complex reasoning tasks, ensuring that the outputs are logically structured and provide step-by-step explanations. Model Evaluation : Use evaluation metrics such as BLEU , ROUGE , and other relevant metrics to assess and improve the performance of models for various NLP tasks. Model Deployment : Support the deployment of trained models into production environments and integrate them into existing systems for real-time applications. Bias Awareness : Be aware of and mitigate issues related to bias , hallucinations , and knowledge cutoffs in LLMs, ensuring high-quality and reliable outputs. Collaboration : Collaborate with cross-functional teams including engineers, data scientists, and product managers to deliver efficient and scalable NLP solutions. Must Have Skill Overall 7 years with at least 5+ years of experience working with transformer-based models and NLP tasks , with a focus on text generation , summarization , question answering , classification , and similar tasks. Expertise in transformer models like GPT (Generative Pre-trained Transformer) , BERT (Bidirectional Encoder Representations from Transformers) , T5 (Text-to-Text Transfer Transformer) , RoBERTa , and similar models. Familiarity with model architectures, attention mechanisms, and self-attention layers that enable LLMs to generate human-like text. Experience in fine-tuning pre-trained models on domain-specific datasets for tasks such as text generation , summarization , question answering , classification , and translation . Familiarity with concepts like attention mechanisms , context windows , tokenization , and embedding layers . Awareness of biases , hallucinations , and knowledge cutoffs that can affect LLM performance and output quality. Expertise in crafting clear, concise, and contextually relevant prompts to guide LLMs towards generating desired outputs. Experience in instruction-based prompting Use of zero-shot , few-shot , and many-shot learning techniques for maximizing model performance without retraining. Experience in iterating on prompts to refine outputs, test model performance, and ensure consistent results. Crafting prompt templates for repetitive tasks, ensuring prompts are adaptable to different contexts and inputs. Expertise in chain-of-thought (CoT) prompting to guide LLMs through complex reasoning tasks by encouraging step-by-step breakdowns. Proficiency in Python and experience with NLP libraries (e.g., Hugging Face, SpaCy, NLTK). Experience with transformer-based models (e.g., GPT, BERT, T5) for text generation tasks. Experience in training, fine-tuning, and deploying machine learning models in an NLP context. Understanding of model evaluation metrics (e.g., BLEU, ROUGE) Qualification: BE/B.Tech or Equivalent degree in Computer Science or related field. Excellent communication skills in English, both verbal and written

Posted 3 weeks ago

Apply

Test Engineer + AI & LLM Evaluation Resourcemojo

3.0 - 5.0 years

9 - 12 Lacs

Bengaluru

Work from Office

Responsibilities: * Collaborate with dev team on API testing using GIT and CI/CD pipeline. * Develop automated tests with Python, PyTest, and frameworks.

Posted 1 month ago

Apply

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.