Training latency
By the time a fine-tune ships, the failure cluster and policies may have already changed.
Improve agents without fine-tuning
Try a simpler fix before retraining your agent.
If your agent is missing a fact or following an outdated policy, it may need better information rather than new weights. Test a prompt change, fix a tool, or let the agent look up approved examples for similar requests.
Keep changing information easy to update.
Prices, policies, and customer-specific exceptions can change faster than a model is trained. Store them where you can check, update, or remove them.
Opaque corrections
Individual approved answers disappear inside weights and cannot be audited or retired cleanly.
Overloaded prompts
Adding every exception to the prompt can create conflicting instructions and increase token costs.
Scores without corrections
A failed grade identifies the problem, but the team still needs to decide what the correct reply should be.
Give the agent relevant examples to follow
Review mistakes, save approved corrections in a searchable library, and let the agent retrieve them on matching requests. Test the results before using the library more widely.
Corrections from real cases
Review the messages, tool results, and outcome of a failed task before writing the correction.
An editable example library
Keep the approved reply, review notes, version, and rules for reuse together. Remove examples that no longer apply.
Measured before/after
Compare task success, edits, escalations, latency, and cost on one workflow.
When to test retrieval
Fine-tuning may help with a consistent style or recurring behavior. Retrieval is worth testing when an example needs to stay editable or only applies in specific cases.
Exception-heavy workflows
Support, voice, claims, and compliance paths where humans already rewrite agent output.
Need for auditability
Regulated or high-risk flows that must show which example influenced a decision.
Frequent changes
Teams that need to update examples without running another training job.
Prompt requirements
An adapter trained with a system prompt may still need it. Test any change to the prompt or runtime before deployment.
Store corrections as golden datasets for AI agentsFine-tuning for characters that keep slipping into assistant mode
What a retrieval test includes
Test whether approved examples help without retraining the model.
Recommendation
Which recurring mistakes to test with retrieval and which may need fine-tuning.
Example library
Approved corrections with review notes, versions, where they apply, and when to remove them.
Retrieval prototype
Look up matching examples for one task and add them to the prompt. Use filters and thresholds to exclude unsuitable records.
Comparison report
Completed tasks, edits, handoffs, response time, and cost compared with the existing setup.
Rollout controls
Approval gates, negative retrieval tests, and expand/stop criteria.
Free template
Start from an illustrative golden-dataset record
Inspect fields for source trace, failure, approved correction, critique, scope, version, and approval status.
This template uses made-up examples, not client data. Adapt it to your policies, privacy requirements, and approval process.
Who this is for
- Teams blocked on fine-tune cycles while the same failures keep shipping
- Teams that want to reuse edits their reviewers already make
- Teams that want test results before spending on training infrastructure
- Teams that need to restrict where corrections apply and remove them when policies change
Compare retrieval and training for one task
Review
Group recurring mistakes and decide whether to test retrieval or fine-tuning.
Build
Approved examples and rules for where they can be retrieved.
Evaluate
Compare the existing and changed agent and recommend whether to deploy.
Test whether a simpler change works
We compare the change with your current setup on separate test cases. The results help decide whether fine-tuning is still worth trying. Improvement isn't guaranteed.
Limitations
- Stable, high-volume behaviors may still need fine-tuning later.
- Retrieval without scope filters can transfer the wrong exception.
- Stale approved examples can hurt performance if not retired.
- An improvement on one task may not carry over to other request types.
- An adapter trained with a system prompt may still need that prompt. Test the settings you plan to deploy.
How we test changes
We review real conversations with your team and agree what the agent should do differently. Approved corrections can become training data, test cases, or examples the agent looks up. We compare each change with your best existing setup on separate test cases, then check for new failures in the software you plan to deploy.
References:
Common questions
What does the work include?
Review mistakes, save approved corrections in a searchable library, and let the agent retrieve them on matching requests. Test the results before using the library more widely.
Do you fine-tune our model?
If it's worth testing after we review the problem. We try your best prompt and consider better examples or tool fixes first. We test an adapter on cases it wasn't trained on and check for new failures. The contract sets delivery and usage rights; the base model's license still applies.
What do we need to start?
Logs or conversation exports for one task, examples of good results, and a reviewer who can approve corrections. We agree data handling before you send anything. Remove unnecessary personal details.
How is success measured?
We test the existing and changed agent on the same task. Where the data is available, we compare completed tasks, human edits, handoffs, response time, and cost.
Related
Discuss a fix before your next training run.
Bring examples of recurring mistakes. We'll look at whether the problem needs better information, a prompt change, a tool fix, or training.