Building Custom Evaluation Pipelines for Your Specific Agent

Why Generic Evals Fail Production Agents

A customer support agent that handles returns, answers product questions, and escalates complaints has a very specific definition of success: customers get accurate answers quickly, with correct tone, and with appropriate tool calls to the CRM. AgentBench's Linux shell tasks, GAIA's multi-hop reasoning puzzles, and SWE-bench's code patches measure none of these things. When you use generic benchmarks to evaluate a specialized agent, you're measuring a proxy for quality, and proxies mislead.

The consequences are real: teams optimize for benchmark score and inadvertently degrade the specific domain behaviors that matter to their users. A model that scores 5 points higher on GAIA may handle your specific product inquiry tasks significantly worse, you won't know unless you have a custom eval that measures it.

What Custom Evals Measure That Generic Benchmarks Cannot

• Domain-specific correctness: Whether the agent's answer is factually correct according to your product documentation, internal data, and business policies, not general internet knowledge.
• Tone and brand alignment: Whether the agent's communication style matches your company's voice and meets your customers' expectations.
• Integration fidelity: Whether the agent uses your specific tools correctly, with the right parameters, at the right times.
• Business rule compliance: Whether the agent follows your specific business rules (escalation triggers, discount authority limits, compliance requirements) that don't exist in any benchmark.
• User population fit: Whether the agent handles the specific distribution of requests your users actually send, which may be very different from benchmark task distributions.

Custom Eval Pipeline Architecture

A custom eval pipeline has four layers: data (tasks, golden outputs, fixtures), orchestration (task sampling, agent execution, timeout management), scoring (multiple scoring methods), and reporting (dashboards, alerts, trends). Each layer has distinct design requirements.

Data Layer Design

The data layer is the foundation of your eval pipeline. It must be well-organized, version-controlled, and regularly updated. Organize tasks by category and difficulty. Store golden outputs as structured JSON with annotation fields explaining key aspects of each output. Maintain a separate set of fixtures (mock tool responses) that match real tool output schemas.

Orchestration Layer Design

The orchestration layer executes tasks reliably and handles the failure modes that come with running hundreds of agent tasks: model API rate limits, timeouts, transient errors, and partial results. Build with async execution from the start, sequential task execution doesn't scale. Implement per-task timeouts (typically 2–5x the expected task duration) to prevent runaway agents from blocking the eval run.

Scoring Layer Design

The scoring layer applies multiple scoring methods to each completed task. No single method is sufficient: exact match catches factual errors but misses quality issues; semantic similarity catches quality issues but misses specific factual errors; LLM judge scores holistic quality but can be miscalibrated. Using all three provides defense in depth for catching different classes of failures.

Building Your Task Bank

The task bank is the most important component of your custom eval pipeline. A poorly constructed task bank produces misleading results no matter how sophisticated your scoring. Task bank quality determines eval quality.

Task Collection Sources

• Production sampling: Sample 200–500 real tasks from production logs (with PII removed). Stratify by task type, complexity, and outcome (successful vs failed completions) to ensure representative coverage.
• Failure case library: Every agent failure logged in production becomes a test case. Add failures immediately after they're triaged, fresh failures are the most valuable additions to your task bank.
• Expert-crafted challenges: Have domain experts craft tasks that probe specific capability boundaries. These are tasks the agent should handle but that don't naturally appear often in production.
• Synthetic edge cases: Programmatically generate tasks that cover boundary conditions: empty inputs, maximum-length inputs, ambiguous queries, contradictory instructions.
• Adversarial tasks: Craft tasks specifically designed to surface known failure modes: tasks with misleading framing, tasks requiring refusal, tasks with conflicting constraints.

Task Quality Criteria

• Unambiguous success criteria: Every task must have a clear, verifiable definition of success. If two domain experts disagree on whether an output is correct, the task needs a clearer specification.
• Representative of production: Tasks should be representative of what real users actually request, not just what's easy to create as test cases.
• Appropriate difficulty distribution: Include tasks across a difficulty spectrum. If all tasks are easy, your eval won't detect capability regressions. If all tasks are hard, you won't track incremental improvements.
• Stable over time: Tasks should have answers that don't change unless your system changes. Tasks asking about current prices, recent events, or external data that changes daily are unstable and produce noisy eval results.

Designing Task-Specific Metrics

Generic metrics (BLEU score, ROUGE-L, perplexity) were designed for machine translation and summarization. They're largely meaningless for evaluating agent task completion. Design metrics that measure what your agent is actually supposed to do.

Functional Correctness Metrics

• Task completion rate: Did the agent achieve the stated goal? Binary pass/fail. The primary metric for any agent eval, everything else is secondary.
• Sub-goal completion rate: For multi-step tasks, score each sub-goal independently. An agent that completes 3 of 4 sub-goals should score better than one that fails entirely. Tracks progress on complex tasks.
• Tool precision: Of the tool calls made, what fraction were necessary and correct? High precision means the agent doesn't waste calls. Calculate as: correct_tool_calls / total_tool_calls.
• Tool recall: Were all necessary tool calls made? An agent that achieves the right answer without calling the required verification tool has low tool recall. Calculate as: necessary_tool_calls_made / total_necessary_tool_calls.

Quality Metrics

• Entity recall: Fraction of required entities (names, IDs, dates, numbers) present in the output. High entity recall means the agent doesn't omit required information.
• Format compliance: Does the output match the required format specification? Validate against a schema for structured outputs. Check structure for semi-structured outputs (required sections, required headers).
• Constraint adherence: Were all stated constraints (length limits, topic restrictions, tone requirements) met? Score as percentage of constraints satisfied.
• Factual accuracy: Are statements in the output factually correct relative to the provided context? Use an LLM judge with access to the ground truth to score factual accuracy.

The Scoring Rubric Framework

A well-designed scoring rubric converts multi-dimensional eval results into a single actionable score while preserving the detail needed to understand why a score changed. The composite score enables PR-gated decisions; the per-dimension breakdown enables debugging.

Weight Calibration

Weights should reflect business value. For a safety-critical agent (medical, financial), Safety might be weighted at 30% rather than 10%. For a creative writing agent, Answer Quality might be weighted at 50%. Don't use default weights, calibrate to your specific use case by asking: if an agent scores perfectly on everything except this dimension, how bad is that outcome?

Score Thresholds

Define explicit pass/fail thresholds at both the composite and per-dimension level. Example: composite score ≥ 80 to pass. Any safety dimension score < 4 fails the task regardless of composite score. Task completion score < 3 fails regardless of other dimensions. Hard per-dimension floors prevent high scores on easy dimensions masking critical failures.

LLM Judge Design and Calibration

LLM judges are the most powerful and most dangerous component of a custom eval pipeline. Powerful because they can assess quality dimensions (clarity, appropriateness, completeness) that no rule-based metric can. Dangerous because a miscalibrated judge systematically misscores, and you may not realize it.

Judge Prompt Engineering

• Provide rubric before output: Give the judge the scoring criteria before showing it the output to score. This anchors its assessment to the rubric rather than general impressions.
• Require justification before score: Instruct the judge to write a brief justification before assigning a score. Chain-of-thought scoring is significantly more consistent than direct scoring.
• Use dimension-by-dimension scoring: Have the judge score each dimension independently in separate assessments rather than producing a single holistic score. Multi-dimension prompts produce less consistent results.
• Avoid leading language: Don't frame the scoring task in a way that suggests a particular score. 'Rate this terrible response' will score lower than 'Rate this response.'

Judge Calibration Process

Calibrate your LLM judge by having human experts score a set of 50–100 outputs on the same rubric. Compare judge scores to human scores. If inter-rater agreement (measured by Cohen's kappa) is below 0.60, the judge is miscalibrated. Improve calibration by adding examples of correct scoring to the judge prompt (few-shot calibration) until agreement exceeds 0.70.

Running Evals at Scale Efficiently

Eval infrastructure that's too expensive or too slow doesn't get used, which is worse than no eval infrastructure. Design for efficiency from the start.

Efficiency Techniques

• Response caching: Cache LLM responses keyed on (task_id, prompt_version, model_version). PR runs on unchanged tasks use cached responses. 70–80% cache hit rate on typical PR runs.
• Parallel execution: Run 20–50 tasks concurrently using async task execution. A 200-task eval suite that takes 60 minutes sequentially takes 5–8 minutes at 50 parallelism.
• Tiered suites: Maintain a 'smoke' suite (20 highest-discriminative tasks, < 3 minutes), a 'standard' suite (100 tasks, < 15 minutes), and a 'full' suite (200+ tasks, < 30 minutes). Gate selection on change risk.
• Discriminative task selection: Identify which tasks produce the most variance across model versions (high-discrimination tasks). Run these first, they catch most regressions in 20% of the task count.
• Cheap judge models: Use fast, cheap models (haiku, gpt-4o-mini) for LLM judge scoring. Reserve powerful models for human-review escalations, not routine scoring.

Test Coverage Analysis

A test suite that appears comprehensive may have systematic gaps. Test coverage analysis maps your task bank against your agent's full capability space to identify which capabilities are well-tested and which are at risk.

Building the Coverage Map

Tag every task in your bank with: task type (simple Q&A, multi-step, tool chaining, long context, adversarial, edge case), difficulty level (easy/medium/hard), capability tested (retrieval, reasoning, tool use, refusal, formatting), and test type (unit, eval harness, integration, red team, E2E). The coverage matrix shows the intersection of test types and task types. Red cells in the matrix are the highest risk areas.

Coverage Gap Remediation

For each identified coverage gap, either add tasks to fill it or explicitly accept the risk and document why. Some gaps are intentional (very rare task types not worth the eval investment). Others are accidents of test construction that should be filled. Coverage gap analysis turns vague intuitions about test quality into concrete, actionable findings.

Eval-Driven Development Workflow

The highest-value use of a custom eval pipeline is as a development tool, not just a gating mechanism. Eval-driven development (EDD) uses eval results to guide every design and implementation decision.

The EDD Loop

• Start with eval: Before making any change, run the eval suite to establish a baseline. This is your before score.
• Hypothesize: State what improvement you expect and why. 'Adding explicit format instructions should improve format compliance score by at least 5 points.'
• Implement: Make the change, prompt edit, architecture change, tool update, whatever the hypothesis requires.
• Measure: Run the eval suite. Compare against baseline. Did the target metric improve? Did any other metric regress?
• Decide: If the target improved without regressions, merge. If the target improved but caused regressions, iterate. If neither improved, discard the change and try a different hypothesis.

Tracking Improvement Over Time

Maintain a longitudinal record of eval scores. A chart of composite score and per-dimension scores over time is the most useful single view into your agent's quality trajectory. It makes improvement visible, makes regressions visible, and provides the context needed to understand why a score changed.

Production Eval Infrastructure

A mature eval pipeline eventually needs production-grade infrastructure: reliable storage for task banks and results, a web dashboard for browsing results, alerting for score regressions, and APIs for integration with CI systems.

Infrastructure Components

• Task bank storage: Version-controlled repository of tasks, golden outputs, and fixtures. Git works well for small banks; a database with schema versioning works better at scale.
• Results database: Store every eval run result with full metadata: task ID, prompt version, model version, scores, outputs, timestamps. Enable queries like 'all tasks where format_compliance dropped > 5 points in the last week.'
• Dashboard: A web UI for browsing eval results, viewing per-task outputs, comparing runs, and visualizing trend charts. Teams that can see eval results act on them; teams that can only query APIs don't.
• Alerting: Automated alerts when composite score drops below threshold, when any dimension regresses by more than N points, or when a specific test that was passing starts failing.
• CI integration: Webhook or API that CI systems can call to trigger eval runs and query pass/fail results. Gate PR merges on eval suite pass.