AWS

The Agent Evaluation Gap: Why Most Enterprises Can't Tell If Their AI Agents Actually Work

July 23, 2026 · Rajat Jindal

The Question That Ends the Demo

In my presales engagements over the past year, a pattern keeps repeating. A team proudly shows me an AI agent they have built — a customer-support agent, an internal knowledge assistant, a claims-triage agent. The demo always works. Then I ask one question: "How do you know it's still working today?"

The room usually goes quiet.

This is the agent evaluation gap, and I now consider it the single most under-invested part of enterprise agentic AI. Enterprises have learned to build agents in weeks. What most of them still cannot do is answer, with evidence, whether the agent is accurate this week versus last week, whether a prompt change made things better or worse, or how often the agent quietly gives a confident-but-wrong answer. They ship on vibes — a handful of manual test prompts before launch — and then fly blind in production.

This blog is not a case study. It is the argument I make to technology leaders about why evaluation is the real work of production agentic AI, and the reference framework I walk them through for closing the gap on AWS. The numbers used are illustrative — representative of what teams typically encounter — not measurements from a specific deployment. The architecture, code patterns, and failure modes are real engineering, drawn from how these systems actually behave.

Why the Gap Exists: Building Is Easy, Proving Is Hard

The reason so many enterprises land in this gap is structural, not lazy. Three forces push it:

1. The demo bias. An agent that answers ten hand-picked questions convincingly feels done. Demos reward fluency, and modern models are extremely fluent — even when wrong. The gap between "sounds right in a demo" and "is right across ten thousand real questions" is invisible until production makes it visible, usually through an escalation.

2. Free-text has no obvious pass/fail. Traditional software has deterministic tests: given input X, assert output Y. An agent's answer is free text — there is no single correct string. So teams either skip automated testing entirely or fall back to a human eyeballing a few answers before each release. Neither scales, and neither is continuous.

3. Evaluation feels like overhead. Under delivery pressure, teams spend ~90% of their effort on building the agent and almost nothing on knowing whether it keeps working. That ratio is exactly backwards for anything customer-facing or regulated, where an unmeasured agent is an unmanaged liability — but the cost of that imbalance only shows up later.

The consequence is a set of questions enterprises routinely cannot answer:

What is our agent's accuracy on the questions that matter most (pricing, charges, policy, eligibility)?
If we tweak a prompt to fix one wrong answer, did we silently break three others?
How often does the agent give a confident wrong answer, and would we catch it before a customer does?
Can we demonstrate the accuracy of automated responses to a compliance or risk reviewer?

If those questions make a technical leader uncomfortable, that discomfort is the gap.

The Framework: Three Layers That Actually Close the Gap

The approach I recommend — and the one I walk customers through on AWS — is not a single tool. It is three complementary layers, each covering the others' blind spots.

Layer 1: A Golden Dataset Owned by the Business

A curated, versioned set of representative questions with approved reference answers, owned by the people who know what "correct" means — support quality and compliance, not just engineers. Stored in S3, version-controlled, and treated as a first-class asset. Every model or prompt change is scored against the same dataset, which is what turns "did this change help or hurt?" from an argument into a number.

Layer 2: LLM-as-a-Judge for Scaled Grading

A separate, dedicated model (distinct from the agent's own model) grades each answer against the reference answer on defined criteria — factual correctness, completeness, policy compliance, appropriate deferral — returning a structured score with a rationale. This scales grading from "a human eyeballs 20 answers" to "hundreds of answers scored in minutes," with far more consistency than rotating human reviewers.

Layer 3: Deterministic Checks for What You Must Never Trust a Model To Judge

For anything factual or numeric, a model's opinion is not good enough. Hard-coded assertions handle these: does any rupee figure in the answer exist in the source-of-truth policy table? Does the answer leak PII? Does it stay within approved topics? These catch the exact failure class LLM judges are worst at — the fluent, confident, wrong answer.

Why all three, and why in this order:

LLM-as-a-judge scales grading, but is biased toward fluency — it can be fooled by a confident wrong answer.
Deterministic checks catch what the judge misses, but only for things you can express as a rule.
The golden dataset is what makes any of it a regression system — without a stable, versioned baseline, you are measuring against a moving target.
The judge must not be the agent's own model — a model grading its own homework is not evaluation.

Reference Architecture on AWS

Here is the reference architecture I use to illustrate the framework. It runs in two modes: offline (against the golden dataset, on every prompt/model change, wired into CI/CD) and online (sampling live production traffic on a schedule).

The LLM-as-a-Judge Scoring Prompt

The judge does not get a vague "is this good?" instruction. It gets a rubric, the reference answer, and a required structured output. This is what makes scoring consistent and auditable.

# Lambda: LLM-as-a-judge scoring against a reference answer
import boto3
import json

bedrock = boto3.client("bedrock-runtime", region_name="ap-south-1")

# Fast, low-cost model for bounded rubric grading against a reference answer.
# A stronger model can be swapped in for harder domains; keep judge != agent model.
JUDGE_MODEL_ID = "anthropic.claude-3-5-haiku-20241022-v1:0"

JUDGE_RUBRIC = """You are an impartial evaluator of a financial-services support agent.
You are given a customer QUESTION, the agent's ANSWER, and an approved REFERENCE_ANSWER.

Score the ANSWER on each dimension from 0-1 (1 = fully meets):
- factual_correctness: Does the answer match the facts in the reference? Any fabricated
  number, charge, or policy detail scores 0 on this dimension regardless of fluency.
- completeness: Does it cover the key points in the reference the customer needs?
- policy_compliance: Does it avoid giving advice/commitments outside approved policy?
- appropriate_uncertainty: If the reference indicates the agent should defer to a human
  or ask for account details, does the answer do so instead of guessing?

Return ONLY valid JSON:
{"factual_correctness": <0-1>, "completeness": <0-1>, "policy_compliance": <0-1>,
 "appropriate_uncertainty": <0-1>, "overall_pass": <true|false>, "rationale": "<one sentence>"}
Set overall_pass=false if factual_correctness < 1.0. Fluent wrong answers must fail."""


def judge_answer(question: str, agent_answer: str, reference_answer: str) -> dict:
    prompt = (
        f"{JUDGE_RUBRIC}\n\n"
        f"QUESTION:\n{question}\n\n"
        f"ANSWER:\n{agent_answer}\n\n"
        f"REFERENCE_ANSWER:\n{reference_answer}\n"
    )

    response = bedrock.invoke_model(
        modelId=JUDGE_MODEL_ID,
        body=json.dumps({
            "anthropic_version": "bedrock-2023-05-31",
            "max_tokens": 512,
            "temperature": 0,  # deterministic judging
            "messages": [{"role": "user", "content": prompt}],
        }),
    )

    body = json.loads(response["body"].read())
    verdict_text = body["content"][0]["text"]
    return json.loads(verdict_text)

The Deterministic Safety Net

# Lambda: deterministic checks that run alongside the LLM judge
import re

def deterministic_checks(agent_answer: str, policy_charges: dict,
                         approved_topics: list) -> dict:
    """Hard assertions. These catch fluent-but-wrong answers the judge may miss."""
    results = {}

    # 1. PII leakage: agent must never echo a full account number or PAN
    pan_pattern = r"\b[A-Z]{5}[0-9]{4}[A-Z]\b"
    acct_pattern = r"\b\d{11,16}\b"
    results["no_pii_leak"] = not (
        re.search(pan_pattern, agent_answer) or re.search(acct_pattern, agent_answer)
    )

    # 2. Charge grounding: any rupee figure quoted must exist in the policy table
    quoted_amounts = re.findall(r"₹\s?([\d,]+(?:\.\d{1,2})?)", agent_answer)
    valid_amounts = {str(v) for v in policy_charges.values()}
    results["charges_grounded"] = all(
        amt.replace(",", "") in {v.replace(",", "") for v in valid_amounts}
        for amt in quoted_amounts
    ) if quoted_amounts else True

    # 3. Topic scope: answer must stay within approved servicing topics
    results["in_scope"] = any(topic in agent_answer.lower() for topic in approved_topics) \
        or "connect you with a representative" in agent_answer.lower()

    results["all_passed"] = all(results.values())
    return results

The CI Gate

# .github/workflows/agent-eval.yml (excerpt)
name: Agent Evaluation Gate
on:
  pull_request:
    paths: ["agent/prompts/**", "agent/config/**"]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run offline evaluation against golden dataset
        run: |
          aws stepfunctions start-sync-execution \
            --state-machine-arn "${{ secrets.EVAL_SM_ARN }}" \
            --input '{"mode":"offline","dataset":"golden/v7.jsonl"}' \
            > result.json
      - name: Enforce quality threshold
        run: |
          PASS_RATE=$(jq '.output | fromjson | .pass_rate' result.json)
          echo "Pass rate: $PASS_RATE"
          # Block merge if pass rate drops below 0.90
          awk "BEGIN {exit !($PASS_RATE >= 0.90)}"

What to Measure, and What "Good" Looks Like

Once the harness is running, the value is that vague confidence becomes concrete metrics. The figures below are illustrative targets — the kind of numbers I encourage teams to aim for and track — not results from a specific deployment. Your real baselines will differ, and the point of the harness is precisely that you measure your own.

Metric	What it tells you	Illustrative target
Accuracy on high-stakes questions (charges, eligibility, policy)	The number compliance actually cares about	Track weekly; aim for a rising trend toward the low-to-mid 90s%
Wrong-but-confident answers caught before the customer	Whether online sampling is doing its job	The higher the better; the goal is to catch them internally, not via escalations
Regressions caught pre-merge	Whether the CI gate is protecting you	Every regression caught at a pull request is one that never reached production
Judge–human agreement (on an audited sample)	Whether you can trust the judge's scores	Validate periodically; if it drifts, revisit the rubric
Time to run a full evaluation	Whether evaluation is fast enough to be continuous	Minutes, not hours — otherwise teams skip it under pressure

The mindset shift I push for: stop asking "is the agent good?" (unanswerable) and start asking "what is the agent's measured accuracy on the 200 questions that matter most, and is that number going up or down this week?" (answerable, and actionable).

What This Costs: Evaluation Should Be a Small Fraction of the Agent

A common objection is that evaluation doubles your AI bill. It should not. Evaluation is bounded, structured scoring — not open-ended generation — so it belongs on a fast, low-cost model. Here is representative sizing based on current AWS pricing, for a harness scoring roughly ~10,000–12,000 answers per month (offline runs on every change plus a modest daily online sample).

Each scoring is small: ~1,500 input tokens (rubric + question + agent answer + reference answer) and ~150 output tokens (the JSON verdict) — on the order of 17–18M input and ~1.75M output tokens per month at that volume.

Component	Representative Monthly Cost
Bedrock judge inference (Claude 3.5 Haiku, ~17.5M in / 1.75M out tokens)	₹2,600 ($31)
Step Functions + Lambda (orchestration + deterministic checks)	₹1,400 ($17)
S3 (datasets, versioned reports) + CloudWatch metrics	₹1,200 ($14)
QuickSight (compliance dashboard, ~3 authors, annual commitment)	₹4,600 ($55)
Amazon Comprehend (PII redaction on sampled transcripts)	₹1,200 ($14)
Representative total	≈ ₹11,000/month (~$131)

The important property is not the absolute number — it scales with how much traffic you sample — but the ratio: a well-designed evaluation harness typically lands well under 10% of the production agent's own inference cost. That is a modest premium for knowing, with evidence, that the agent works. Budget the judge line as a function of volume sampled, not a fixed figure: raise sampling from ~150/day to ~1,000/day and the judge cost grows proportionally.

The Failure Modes Every Evaluation System Hits

These are the recurring ways evaluation harnesses go wrong. They are patterns I flag to teams up front, because most hit at least one of them.

Failure Mode 1: The Judge Rewards Confident, Fluent, Wrong Answers

The pattern: A holistic "is this a good answer?" rubric produces a high pass rate, while deterministic checks fail a meaningful share of the same answers. The judge and reality disagree, because LLM judges have a well-documented bias toward fluency and length — a confident, professional-sounding wrong answer scores high.

The fix: Make factual_correctness a separate, gating dimension; hard-code "any fabricated number forces overall_pass=false"; set judge temperature=0. Critically, never trust the judge alone for anything numeric — let a deterministic check against a source-of-truth table be the authority on factual accuracy, and let the judge handle the softer dimensions (completeness, tone, appropriate deferral). This is the single most important design decision in the whole harness.

Failure Mode 2: The Golden Dataset Goes Stale and Misses Real Failures

The pattern: The offline pass rate looks healthy while live traffic surfaces failures the offline suite never catches. The agent is "passing the exam" but failing in the field — because the golden dataset reflects the clean questions engineers imagined users ask, not the messy, compound questions real users actually ask.

The fix: Build a feedback loop. Every online failure a human confirms as a genuine miss gets anonymised, given an approved reference answer, and promoted into the golden dataset. Over time the offline suite becomes a growing museum of every real way the agent has failed, so no fixed regression can silently return. A golden dataset should grow from real failures, not imagination.

Failure Mode 3: The Evaluation Pipeline Becomes an Unmanaged PII Store

The pattern: Teams put PII controls on the agent's outputs but overlook that the evaluation pipeline itself now handles raw customer conversations as input data. Sampled live transcripts — names, account numbers, PAN — end up written to an evaluation bucket and sent to the judge model in the clear. The evaluation system has quietly become a new place PII lives.

The fix: Put redaction (e.g., Amazon Comprehend PII detection) at the very front of the online path — before any transcript is stored or sent to the judge — replacing identifiers with typed placeholders while preserving the structure needed to evaluate. Add a strict bucket policy, short lifecycle expiry on evaluation data, and least-privilege IAM scoped to redacted objects. An evaluation system is a data system; it inherits every governance obligation of the thing it evaluates.

A Presales Perspective: Evaluation Is What Sells the Next Three Agents

In my presales engagements, the conversation about AI agents has shifted over the past year. Eighteen months ago the question was "can we build one?" Today most enterprises can. The question that now stalls deals is different: "How do we govern these at scale without a room full of people manually checking outputs?"

Evaluation is the answer, and it is chronically under-invested. The reframe that lands with leaders is moving evaluation from "QA overhead that slows shipping" to "the capability that lets you scale safely." In a regulated business the logic is decisive: the moment you can prove one agent's accuracy with a weekly report, compliance can flip from blocking new agents to sponsoring them — because the same harness evaluates all of them. Evaluation becomes the platform, not the chore.

The moment that shifts the conversation is never the LLM-as-a-judge architecture. It is the opening question: "How do you know your agent is still working today?" Watching a technical leader realise they cannot answer that — for a system already talking to customers — is the moment evaluation stops being optional. The follow-up is always: "How do we retrofit this onto the agents we already shipped?" That question is the entire agentic-AI-operations roadmap in one sentence.

Lessons for Technology Leaders

If you can't measure it, you're not operating it — you're hoping — An agent in production without automated evaluation is not a managed system. It is a demo that happens to be exposed to customers.
LLM-as-a-judge scales grading, but never let it judge numbers — Use the model for soft, subjective dimensions where it is genuinely good. For anything factual or numeric, use a deterministic check against a source of truth. Fluent wrong answers are the failure mode that hurts most, and judges are biased to reward them.
Your golden dataset should be a museum of real failures, not imagined ones — The dataset engineers write reflects the questions they wish users asked. The valuable dataset grows from the messy, real failures caught in production. Build the feedback loop.
The evaluation system is a data system — It inherits every governance obligation of the thing it evaluates. If your agent handles PII, so does your evaluation pipeline — plan redaction, retention, and access control from day one.
Evaluation is what makes scaling safe, not what slows it down — In a regulated business it is the thing that lets compliance say yes to the next three agents. Measurement is the accelerant, not the brake.

Conclusion

The hard part of enterprise agentic AI is no longer building the agent — it is proving, continuously and with evidence, that the agent works. Most enterprises have crossed the building threshold and stalled at the proving one. That gap is where wrong answers hide, where regressions ship silently, and where compliance questions go unanswered.

Closing it does not require a smarter agent. It requires the ability to answer one question — "How do you know it's working today?" — with a number instead of a shrug. A golden dataset owned by the business, an LLM judge that scales grading, and deterministic checks that guard the facts. That is a few weeks of engineering and a cost that lands well under 10% of the agent it protects.

An agent you cannot evaluate is an agent you cannot trust, scale, or govern. Building the evaluation harness is not the unglamorous chore that comes after the real work. In production agentic AI, it is the real work.

About the Author

Rajat Jindal is VP – Presales at AeonX Digital Technology Limited, where he architects winning cloud strategies for enterprise customers and translates modernization into measurable business value. He is a strong advocate of AWS, committed to sharing thought leadership that helps technology leaders make faster, better-informed decisions.

← All insights