Score-only pipelines
Failed grades without approved alternatives.
Automated eval vs human review
Automate repeatable checks. Ask people to judge difficult cases.
Automated graders can check many replies quickly. Reviewers handle ambiguous cases, check policy decisions, and approve corrections. Choose the mix based on the task and the cost of a wrong answer.
A failed score still needs a correction.
Record what the agent should have said or done, then test whether the proposed fix works on other cases.
Grader blind spots
Automated checks can miss subtle policy mistakes or bad handoff decisions. Review a sample with qualified people.
Synthetic mismatch
Lab scenarios miss production language and tool state.
Unused corrections
Review findings stay in dashboards instead of becoming test cases or examples the agent can use.
Use scores to guide review work
Run automated checks, have reviewers inspect failures and a sample of passes, and save approved corrections. Test whether using those examples helps the agent.
Graders at scale
Run rubrics on high-volume traces and regressions.
Human approval gates
Experts sign off on corrections, scope, and retirement.
Update the tests
Add new real failures to the test set. Keep retrieval examples separate from cases used to measure their effect.
Split the work deliberately
Use machines for throughput. Use humans for judgment, policy, and approval.
Automate
Regression detection, rubric scoring, duplicate failure clustering.
Human review
Ambiguous trajectories, compliance paths, tool escalation decisions.
Store outcomes
Only approved corrections enter the experience library.
Free template
Get the AI agent evaluation rubric
CSV template for task outcome, tools, state, recovery, escalation, safety, cost, and latency.
This template uses made-up examples, not client data. Adapt it to your policies, privacy requirements, and approval process.
Who should read this
- Teams with eval dashboards but flat production quality
- Regulated workflows requiring human approval gates
- ML leads building regression gates before release
Eval plus approved corrections
Review
Write scoring rules based on recurring real mistakes.
Build
Set up grading and approve corrections for the agreed retrieval or training test.
Evaluate
Compare the existing and changed agent on one task.
Test whether corrections help
Compare the agent with and without the approved examples on the task you're scoring.
How we test changes
We review real conversations with your team and agree what the agent should do differently. Approved corrections can become training data, test cases, or examples the agent looks up. We compare each change with your best existing setup on separate test cases, then check for new failures in the software you plan to deploy.
References:
Common questions
What does the work include?
Run automated checks, have reviewers inspect failures and a sample of passes, and save approved corrections. Test whether using those examples helps the agent.
Do you fine-tune our model?
If it's worth testing after we review the problem. We try your best prompt and consider better examples or tool fixes first. We test an adapter on cases it wasn't trained on and check for new failures. The contract sets delivery and usage rights; the base model's license still applies.
What do we need to start?
Logs or conversation exports for one task, examples of good results, and a reviewer who can approve corrections. We agree data handling before you send anything. Remove unnecessary personal details.
How is success measured?
We test the existing and changed agent on the same task. Where the data is available, we compare completed tasks, human edits, handoffs, response time, and cost.
Related
Plan the right mix of automated and human review.
Bring examples of failures. We'll discuss which checks can run automatically and which need a reviewer.