AI Agent Evaluation: Quality Gates That Check Output Before It Reaches a Customer
A demo shows an agent at its best. A quality gate checks every output before a customer sees it.
A pilot agent drafts good replies in the demo. Then someone asks who signs off that its output can reach a customer, and nobody can answer. The pilot waits, or a person re-reads every output and the time savings disappear.
AI agent evaluation fixes that. It means agreeing what good output looks like before the build, testing the agent on real cases, and then checking every output with quality gates before it reaches a customer. Evals tell you whether the system is good enough to ship. Gates decide, case by case, whether one output passes, is held or goes to a person.
Below: a test set, four gate types, pass, hold and escalate paths, and a checklist. All of it sits inside the AI agent harness, the control layer around a model.
Why a good demo proves so little
A demo uses hand-picked cases. Failures sit in the odd ones real traffic brings: a customer who quotes the wrong policy, a bank file with a missing column, an upside-down scan.
Agents also compound mistakes, because each step builds on the last. Anthropic's engineers make this point in Demystifying evals for AI agents.
The arithmetic: if an agent gets a case right 3 times out of 4, the chance of 3 right in a row is 0.75 × 0.75 × 0.75, about 42%. Anthropic uses the same example to separate a best run from consistent performance. Customers see the consistent one, so AI agent evaluation has to measure repeat runs.
Three terms:
- Eval: a fixed set of test cases that shows whether the agent, or a change to it, is good enough.
- Quality gate: a check on one live output before it moves on.
- Grader: whatever scores an output: code, a second model or a person.
Start your AI agent evaluation with real cases
Build the test set from what exists: the manual checks your team runs today, last quarter's errors, the complaint queue. Anthropic suggests starting with 20 to 50 simple tasks drawn from real failures. Treat that as a start, not a rule.
A good case passes one test: two experts would reach the same pass or fail verdict. If not, the criteria are unclear and no agent can meet them.
Include cases where the right answer is "send this to a person" or "do nothing." Easy wins teach you nothing about judgment.
| Case input | What good looks like | Must never happen | Owner |
|---|---|---|---|
| Customer asks about a refund outside the window | Reply cites the policy text and offers the escalation path | Promises a refund the policy does not allow | Support lead |
| Bank file and ledger for one month | Matched pairs sum to the ledger total; the rest listed as exceptions | A forced match to make totals agree | Finance operations |
Write these criteria before the build and have an operations owner sign them. In our scoping sprint, they are a deliverable.
Anthropic's tip: if an agent fails every trial of one case, suspect the case first.
Four quality gates for AI agents and what each one misses
No single check catches everything, so Anthropic describes layering methods: what one misses, another catches. Birgitta Böckeler's article Harness engineering for coding agent users adds a useful split: guides that steer the agent before it acts, and sensors that check its work after. She also separates deterministic checks from model-based ones.
| Gate | What it checks | Catches | Misses | Speed and cost |
|---|---|---|---|---|
| Rules and schema | Code tests fields, limits, totals, approved sources | Missing data, wrong totals, forbidden claims | Tone, meaning, anything you did not write a rule for | Fast, low |
| Model reviewer | A second model grades against a written rubric | Policy fit, tone, unsupported statements | Its own biases; results can vary between runs | Fast, per-call cost |
| Sampled human review | A person checks a sample and every held case | Subtle errors; calibrates the model reviewer | Anything outside the sample | Slow, highest |
| Hard stops | Fixed rules that always route to a person | High-value, irreversible or legal cases | Cases nobody thought to list | Instant, near zero |
Order matters: cheap deterministic gates first, the model reviewer second, people last.
Rule and schema checks
Plain code, not a model: required fields present, amounts within limits, totals reconciled, forbidden phrases absent. Repeatable and cheap, but blind to meaning. The OWASP Top 10 for LLM applications lists improper output handling as a risk: treat model output as untrusted input before another system acts on it.
A second model as reviewer
"LLM-as-judge" means a separate model call grades the output against your rubric. It handles free text that rules cannot.
Its limits are documented: Zheng et al. in Judging LLM-as-a-Judge describe position, verbosity and self-enhancement bias. Our judgment: where you can, use a different model from the one that wrote the output.
Anthropic's guidance: grade one criterion per call, give the reviewer an "unknown" exit, compare it with human verdicts and read the transcripts. A generator-and-reviewer loop works when criteria are clear, as Building effective agents notes.
Sampled human review
A person checks a sample on a schedule, plus every case a gate held. It is the most reliable check on quality and calibrates the model reviewer, but it is slow and costly. Spend it where risk is highest.
The sample rate is a decision based on risk and volume. Write it down, and log who set it.
Hard stops
Some cases should always go to a person: an amount above a limit, an irreversible action, complaint or legal wording, low confidence, an out-of-scope request. A hard stop is a fixed rule, not a score. No model can talk its way past it.
Wire the gates into your AI agent evaluation workflow
Pass, hold, escalate
Every gate ends in one of three outcomes:
- Pass: the output is released.
- Hold: the output stops and the reason is recorded. Optionally allow a retry with the reviewer's feedback, under a hard retry limit.
- Escalate: a named person or queue receives the case, the output and the reason.
Nothing fails silently into "sent."
Log every decision
For each gate, record the case ID, gate, rule or rubric version, verdict, reason, time and any override. This is the audit trail from the AI agent harness: it shows a compliance reader what was checked, and by whom.
Re-run the evals when anything changes
Re-run on a new model, prompt, tool, policy text or data source. Keep a regression set of cases that must always pass. Anthropic notes that teams with evals can adopt new models faster.
Who owns the criteria
An operations owner signs the pass criteria, not only engineering. Review held cases weekly, and turn every miss into a new case.
The gates are the same on any platform. How each platform handles logging and tracing is compared in where your agents should run.
Pre-launch checklist for quality gates
- Pass criteria are written and signed by an operations owner.
- The set has 20 or more real cases, including "send to a person" cases.
- Every output type has at least one gate.
- Gates run in order: rules, model reviewer, people.
- Hard stops are listed and tested.
- Hold and escalate paths are tested end to end.
- Every verdict is logged with the rule or rubric version.
- The model reviewer is calibrated against a human sample.
- The regression set runs on every change.
- A named owner and a kill switch exist.
Not sure which gate your pilot is missing? Book an AI scoping call. We review your workflow, your systems and where a person should stay in the loop.
Questions to ask a vendor or your own team
- Where are the pass criteria written, and who signed them?
- How often is the model reviewer compared with human verdicts?
- What re-runs when the model or prompt changes?
- Can I see every held case and the reason?
- Which actions can never be released without a person?
How we check our own agents' work
We run ByteandRise on a team of agents with human oversight. An Editor agent checks every draft against our brand and claims rules and holds drafts that fail. A person on our team approves strategy and what goes out, and can override from an admin panel. Every agent's work is logged.
Same pattern, applied to our own work: how we run ByteandRise.
How we would approach this for your company
- Scoping sprint. We review one workflow, its systems and the cost of a wrong output, and mark where a person stays in the loop. You get a written plan and cost model, usable with or without us. Our AI agent cost calculator shows what reviewer calls and human review add to running cost.
- Pilot build. One agent on real systems (staging or limited production) with an evaluation set and success criteria agreed up front, the gates above, human review where needed and a usage and cost dashboard. We measure it against the sprint's criteria.
- Production build. Security review steps, monitoring, documentation and handover, deployed where your data lives: Azure AI Foundry, Google Vertex AI, AWS Bedrock or Claude Platform.
Details: our AI agents service page, how we work and, for fixed multi-step flows, AI workflows. Gates reduce risk; they do not make an agent infallible, and high-stakes steps keep a person.
Put one workflow through a Pilot Build with the gates built in, so you decide on evidence, not on a demo. Book an AI scoping call or see how we run ByteandRise.
FAQ
What is the difference between an eval and a quality gate?
An eval is a set of test cases you run to see whether the agent, or a change to it, is good enough to ship. A quality gate checks each live output before it moves on. Evals judge the system. Gates judge one result.
Can one AI model check another model's work?
Yes, as one layer. A model reviewer is flexible but has documented biases and can vary between runs. Give it a written rubric, one criterion per call and an "unknown" option, then compare its verdicts with human ones on a sample. For high-stakes steps, keep a person.
How many test cases do we need before launch?
There is no fixed number. Start with a small set drawn from real failures; Anthropic suggests 20 to 50 simple tasks. Include cases that should go to a person. Then add a case whenever something slips through. Quality and coverage matter more than count.
How much human review is enough?
It depends on risk and volume. Review everything that is irreversible or high-value, plus every case a gate held, and sample the rest. Start with a higher sample rate and lower it only when logged review results justify it. Write the rule down and log who decided.
When should we re-run the evaluations?
Re-run them whenever the model, prompt, tools, policy text or data source changes, and on a regular schedule. Keep a regression set of cases that must always pass, so a change that helps one case does not break another. Treat a failed regression case as a release blocker.
Do we need an evaluation framework or tool?
Not first. Write the cases and pass criteria, then pick a tool that fits your stack and the systems you already run. Tool choice is easier to change later than bad criteria, so spend most of the effort on the cases and on who signs them off.