You shipped your AI feature. It works in demos. But does it work reliably for real users? Unlike traditional software where tests pass or fail, LLM outputs exist on a spectrum of quality. A response can be grammatically correct but factually wrong, helpful but incomplete, or safe but useless. This guide covers the evaluation and testing practices you need to ship AI with confidence.

Why Evaluation Matters

Without systematic evaluation, you're flying blind. Common failure modes that evaluation catches:

• Regression: A prompt change improves one use case but silently breaks another
• Drift: Model updates change output quality in subtle ways
• Edge cases: The model fails on specific inputs you didn't test manually
• Cost-quality tradeoffs: Switching to a cheaper model saves money but at what quality cost?

If you can't measure it, you can't improve it. And if you can't measure it consistently, you can't tell if your improvements are actually improvements.

Evaluation Methods Overview

Method	Cost	Speed	Reliability	Best For
Automated metrics	Free	Fast	Low	Screening, regression
LLM-as-judge	Low	Fast	Medium	Scalable evaluation
Human evaluation	High	Slow	High	Ground truth, launch decisions
A/B testing	Variable	Slow	High	Production decisions

Automated Metrics

Quick to compute, useful for regression testing, but limited in what they can measure.

Reference-Based Metrics

Compare model output against a reference answer:

# ROUGE-L: Measures longest common subsequence
from rouge_score import rouge_scorer

scorer = rouge_scorer.RougeScorer(['rougeL'], use_stemmer=True)
scores = scorer.score("The cat sat on the mat", "A cat was sitting on a mat")
print(scores['rougeL'].fmeasure)  # ~0.67

# BLEU: Measures n-gram precision (common in translation)
from sacrebleu.metrics import BLEU
bleu = BLEU()
result = bleu.corpus_score(["The cat sat"], [["A cat was sitting"]])
print(result.score)  # BLEU score

# BERTScore: Semantic similarity using embeddings
from bert_score import score
P, R, F1 = score(
    ["The cat sat on the mat"],
    ["A feline rested on the rug"],
    lang="en"
)
print(F1.mean().item())  # ~0.85 (semantically similar)

Reference-Free Metrics

Evaluate output quality without a reference answer:

• Perplexity — How "surprised" the model is by the output. Lower = more fluent.
• Repetition rate — Detects looping or repetitive outputs.
• Length compliance — Does the output match requested length constraints?
• Format compliance — Does the output match the requested format (JSON, markdown, etc.)?

def check_format_compliance(output: str, expected_format: str) -> dict:
    """Check if output matches expected format."""
    checks = {
        "json": lambda x: (x.startswith("{") or x.startswith("[")) and _is_valid_json(x),
        "list": lambda x: x.strip().startswith(("-", "*", "1.")),
        "code": lambda x: "```" in x or x.strip().startswith(("def ", "class ", "import ")),
    }
    
    checker = checks.get(expected_format, lambda x: True)
    return {
        "compliant": checker(output),
        "format": expected_format,
        "output_length": len(output)
    }

def check_length(output: str, min_words: int = 0, max_words: int = 10000) -> dict:
    word_count = len(output.split())
    return {
        "word_count": word_count,
        "in_range": min_words <= word_count <= max_words
    }

LLM-as-Judge

The most important evaluation pattern in 2026: use a capable LLM to evaluate another LLM's output. This scales to thousands of examples and captures qualities that automated metrics can't.

Setting Up LLM-as-Judge

from openai import OpenAI
import json

client = OpenAI()

JUDGE_PROMPT = """You are an expert evaluator. Rate the following AI response on these criteria:

1. **Helpfulness** (1-5): Does it address the user's question effectively?
2. **Accuracy** (1-5): Is the information correct?
3. **Clarity** (1-5): Is it well-organized and easy to understand?
4. **Safety** (1-5): Is it free from harmful content?

User question: {question}

AI response: {response}

Respond in JSON format:
{{"helpfulness": N, "accuracy": N, "clarity": N, "safety": N, "reasoning": "brief explanation"}}"""

def evaluate_response(question: str, response: str) -> dict:
    result = client.chat.completions.create(
        model="gpt-5",  # Use your strongest model as judge
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(question=question, response=response)
        }],
        response_format={"type": "json_object"},
        temperature=0  # Deterministic evaluation
    )
    
    scores = json.loads(result.choices[0].message.content)
    scores["overall"] = sum(
        scores[k] for k in ["helpfulness", "accuracy", "clarity", "safety"]
    ) / 4
    return scores

Pairwise Comparison

More reliable than absolute scoring — compare two outputs head-to-head:

PAIRWISE_PROMPT = """Compare these two AI responses to the same question.

Question: {question}

Response A: {response_a}
Response B: {response_b}

Which response is better? Consider helpfulness, accuracy, clarity, and safety.

Respond in JSON:
{{"winner": "A" or "B" or "tie", "reasoning": "brief explanation"}}"""

def compare_responses(question, response_a, response_b):
    result = client.chat.completions.create(
        model="gpt-5",
        messages=[{"role": "user", "content": PAIRWISE_PROMPT.format(
            question=question, response_a=response_a, response_b=response_b
        )}],
        response_format={"type": "json_object"},
        temperature=0
    )
    return json.loads(result.choices[0].message.content)

LLM-as-judge has a position bias: when comparing two responses, the judge tends to favor the first one. Mitigate by randomizing the order and running each comparison twice with swapped positions.

Building an Evaluation Dataset

# eval_dataset.jsonl
{"id": "qa_001", "question": "What is RAG?", "reference": "RAG stands for...", "category": "definition"}
{"id": "qa_002", "question": "How do I set up a vector database?", "reference": null, "category": "tutorial"}
{"id": "qa_003", "question": "Compare GPT-5 and Claude", "reference": null, "category": "comparison"}
{"id": "edge_001", "question": "Tell me a joke about [harmful topic]", "reference": null, "category": "safety"}
{"id": "edge_002", "question": "", "reference": null, "category": "empty_input"}

# Run evaluation across entire dataset
import jsonlines

def run_evaluation(model, dataset_path, output_path):
    results = []
    with jsonlines.open(dataset_path) as dataset:
        for example in dataset:
            response = generate_response(model, example["question"])
            scores = evaluate_response(example["question"], response)
            results.append({
                "id": example["id"],
                "category": example["category"],
                "response": response,
                "scores": scores
            })
    
    # Save and summarize
    with jsonlines.open(output_path, mode='w') as out:
        for r in results:
            out.write(r)
    
    # Print summary by category
    from collections import defaultdict
    by_category = defaultdict(list)
    for r in results:
        by_category[r["category"]].append(r["scores"]["overall"])
    
    for cat, scores in by_category.items():
        avg = sum(scores) / len(scores)
        print(f"{cat}: {avg:.2f} ({len(scores)} examples)")

Human Evaluation

No automated metric fully captures user satisfaction. Human evaluation remains the gold standard.

When to Use Human Evaluation

• Before launching a new AI feature
• When evaluating creative outputs (writing, marketing copy)
• For safety-critical applications (medical, legal)
• To calibrate your LLM-as-judge scores

Setting Up Human Evaluation

# Human evaluation interface (simplified)
class HumanEvalInterface:
    def __init__(self):
        self.results = []
    
    def present_comparison(self, question, response_a, response_b):
        """Present a blinded comparison to the evaluator."""
        # Randomize A/B position
        import random
        order = random.choice(["ab", "ba"])
        if order == "ab":
            first, second = response_a, response_b
            labels = {"first": "A", "second": "B"}
        else:
            first, second = response_b, response_a
            labels = {"first": "B", "second": "A"}
        
        return {
            "question": question,
            "response_1": first,
            "response_2": second,
            "prompt": "Which response is better? (1/2/tie)",
        }
    
    def record(self, eval_id, winner, notes=""):
        self.results.append({
            "eval_id": eval_id,
            "winner": winner,
            "notes": notes,
            "timestamp": datetime.now().isoformat()
        })

Inter-Annotator Agreement

Measure agreement between human evaluators to validate your evaluation setup:

from sklearn.metrics import cohen_kappa_score

# Two annotators' ratings for 20 examples
annotator_1 = [1, 2, 1, 3, 2, 1, 2, 3, 1, 2, 1, 3, 2, 1, 2, 3, 1, 2, 1, 2]
annotator_2 = [1, 2, 1, 2, 2, 1, 2, 3, 1, 2, 2, 3, 2, 1, 2, 2, 1, 2, 1, 3]

kappa = cohen_kappa_score(annotator_1, annotator_2)
print(f"Cohen's Kappa: {kappa:.3f}")
# > 0.8 = strong agreement
# 0.6-0.8 = moderate agreement
# < 0.6 = poor agreement — retrain annotators

Regression Testing

When you change prompts, models, or configurations, regression testing ensures you haven't broken existing functionality:

# regression_test.py
import pytest

class TestLLMRegression:
    """Regression tests for LLM behavior."""
    
    # Golden examples: known good inputs/outputs
    GOLDEN_EXAMPLES = [
        {
            "input": "What is 2+2?",
            "must_contain": ["4"],
            "must_not_contain": ["I'm not sure"],
        },
        {
            "input": "Write a Python hello world",
            "must_contain": ["print", "hello"],
            "must_not_contain": ["Sorry", "cannot"],
        },
        {
            "input": "Tell me about [harmful topic]",
            "must_contain": [],  # No specific required content
            "must_not_contain": ["[harmful content]"],  # Must refuse safely
        },
    ]
    
    @pytest.fixture
    def llm_client(self):
        return OpenAI()
    
    @pytest.mark.parametrize("example", GOLDEN_EXAMPLES)
    def test_golden_examples(self, llm_client, example):
        response = llm_client.chat.completions.create(
            model="gpt-5",
            messages=[{"role": "user", "content": example["input"]}],
            temperature=0
        )
        output = response.choices[0].message.content.lower()
        
        for required in example["must_contain"]:
            assert required.lower() in output, \
                f"Missing required content: {required}"
        
        for forbidden in example["must_not_contain"]:
            assert forbidden.lower() not in output, \
                f"Found forbidden content: {forbidden}"
    
    def test_format_compliance(self, llm_client):
        """Structured output still works after changes."""
        response = llm_client.chat.completions.create(
            model="gpt-5",
            messages=[{"role": "user", "content": "List 3 colors as JSON"}],
            response_format={"type": "json_object"},
            temperature=0
        )
        data = json.loads(response.choices[0].message.content)
        assert isinstance(data, dict)

Production Monitoring

Evaluation doesn't stop at launch. Monitor quality in production:

Key Metrics to Track

• User feedback rate — Thumbs up/down ratio on responses
• Retry rate — How often users rephrase and ask again
• Response length trend — Sudden changes may indicate issues
• Refusal rate — Too high = over-cautious, too low = risky
• Latency percentiles — P50, P95, P99 response times
• Error rate — API errors, timeout rate

# Production monitoring with logging
import structlog

logger = structlog.get_logger()

def log_llm_interaction(question, response, metadata):
    logger.info("llm_interaction",
        question_length=len(question),
        response_length=len(response),
        model=metadata.get("model"),
        latency_ms=metadata.get("latency_ms"),
        tokens_in=metadata.get("tokens_in"),
        tokens_out=metadata.get("tokens_out"),
        user_feedback=metadata.get("feedback"),  # thumbs up/down
        was_retried=metadata.get("was_retried", False),
    )

# Set up alerts
ALERT_THRESHOLDS = {
    "error_rate": 0.05,        # Alert if >5% of requests fail
    "refusal_rate": 0.15,      # Alert if >15% are refused
    "avg_latency_ms": 10000,   # Alert if avg >10s
    "negative_feedback_rate": 0.2,  # Alert if >20% thumbs down
}

Shadow Evaluation

Run new model versions in shadow mode alongside production — compare outputs without affecting users:

async def shadow_evaluate(question, production_response, new_model):
    """Run new model in parallel, compare outputs."""
    shadow_response = await generate_response(new_model, question)
    
    # Compare with production output
    comparison = compare_responses(question, production_response, shadow_response)
    
    # Log for analysis
    logger.info("shadow_eval",
        question=question[:100],
        production_model="gpt-5",
        shadow_model=new_model,
        winner=comparison["winner"],
    )
    
    return comparison

Evaluation Frameworks & Tools

Tool	Type	Key Features	Best For
OpenAI Evals	Framework	Native OpenAI integration, graded evaluations	OpenAI-centric apps
LangSmith	Platform	Tracing, evaluation, datasets	LangChain users
Ragas	Framework	RAG-specific metrics (faithfulness, relevance)	RAG pipelines
Braintrust	Platform	Eval datasets, scoring, comparison	General LLM apps
Promptfoo	CLI	Local eval, prompt comparison, CI/CD	Prompt engineering

Using Promptfoo for Prompt Regression

# promptfooconfig.yaml
description: "Evaluate customer support prompts"

providers:
  - openai:gpt-5:
      id: gpt5-current
  - openai:gpt-5.4-mini:
      id: gpt54mini-cheaper

prompts:
  - file://prompts/support_v2.txt

tests:
  - vars:
      question: "How do I reset my password?"
    assert:
      - type: contains
        value: "reset"
      - type: not-contains
        value: "I cannot help"
      - type: llm-rubric
        value: "Response should be helpful and specific about password reset steps"
  
  - vars:
      question: "I want a refund for order #12345"
    assert:
      - type: contains-any
        value: ["refund", "return"]
      - type: llm-rubric
        value: "Response should acknowledge the request and explain the refund process"

# Run: npx promptfoo eval

Building an Evaluation Culture

1. Start with golden examples — 20-50 high-quality input/output pairs that represent your key use cases
2. Add LLM-as-judge early — Set up automated scoring before you need it
3. Run evals on every change — Integrate into CI/CD, not just ad-hoc
4. Calibrate with humans — Periodically compare LLM judge scores with human ratings
5. Track trends over time — A single evaluation score is less useful than the trend
6. Include edge cases — Empty inputs, adversarial prompts, very long inputs, non-English

Common Mistakes

1. Evaluating on your training data — If examples were used to develop the prompt, they're not a valid test set
2. Using a single metric — A model can score well on helpfulness but fail on safety. Use multiple dimensions
3. Ignoring the judge model's biases — GPT-5 as judge tends to favor GPT-5 outputs. Use a different provider as judge when possible
4. Not versioning your evaluation dataset — Track changes to your eval set alongside code changes
5. Running evals manually — Automate. If it takes more than 5 minutes to run your eval suite, you won't run it often enough
6. Confusing benchmark scores with real-world performance — MTEB, MMLU, and HumanEval are useful for model selection, not for evaluating your specific application

Conclusion

Evaluation is the foundation of reliable AI applications. Start with automated metrics for format and basic quality checks. Layer on LLM-as-judge for scalable semantic evaluation. Use human evaluation for launch decisions and calibration. And once you're in production, monitor continuously — model updates, prompt changes, and user behavior shifts all affect quality over time.

The investment in evaluation infrastructure pays for itself many times over: faster iteration (you can change prompts with confidence), fewer production incidents (you catch regressions before users do), and better decision-making (you choose models based on evidence, not vibes).