LLM Evaluator (Model Response Analyst)

About the role

We are seeking a detail‑oriented and analytical LLM Evaluator to assess, analyze, and improve the performance of large language models (LLMs). In this role you will evaluate AI‑generated content for accuracy, coherence, factual reliability, bias, safety, and alignment with defined guidelines.

Key responsibilities

• Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Review multiple model responses to the same prompt, determine human‑preferred output, and provide justification.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Craft adversarial prompts to expose biased, harmful, or insecure model behavior and help patch safety vulnerabilities.
• Collaborate with quality assurance to refine evaluation guidelines and calibrate scoring standards across the global team.
• Analyze underperforming model outputs to hypothesize root causes and flag novel behaviors for the research team.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Deep understanding of prompt engineering and ability to explain why a model output is good or bad.
• Experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation.
• Proven ability to monitor and improve consistency among evaluation teams and conduct calibration sessions.
• Experience sourcing, cleaning, and annotating datasets for LLM fine‑tuning or evaluation.

Required skills

• Reinforcement Learning from Human Feedback (RLHF) data collection
• Dataset sourcing, cleaning, and annotation
• A/B testing concepts applied to AI models
• Experiment design for model comparison