Time to train
The policies or recurring mistakes may change while training is underway.
Fine-tuning vs golden datasets
A golden dataset can support fine-tuning or retrieval.
A golden dataset contains examples your reviewers have checked and approved. Use separate sets to train, test, or give the agent examples to look up. The right choice depends on the mistake you're trying to fix.
Test the prompt before changing the weights.
In one internal comparison, adding a persona prompt made the base model's replies much shorter. That doesn't tell us the best fix for every model, but it shows why a strong prompt matters. Fine-tuning has other costs to consider:
Harder to inspect
Model weights don't show which training example caused a reply.
Hard to retire
Removing one bad example from trained behavior is expensive.
Limits to auditing
Training alone doesn't leave a record of which example influenced each decision.
What golden datasets add
Keep the source conversation, reviewer approval, and version history for each example. Decide which records are for training, retrieval, or testing. Keep test conversations out of the training set.
Clear expected replies
Reviewers can point to a reply they approve instead of relying on a vague instruction such as 'be more natural'.
Source records
You can check the original conversation, who approved the correction, and where it applies.
Two ways to apply corrections
Retrieve selected examples at runtime or train an adapter. Keep the source dataset versioned even when it is used to change weights.
Decision guide
Consider what went wrong, what you need to audit, and the results on separate test cases.
Test retrieval
Changing policies, product facts, or narrow exceptions that need to be removable at runtime.
Test fine-tuning
The model keeps using the wrong tone, losing track of the conversation, or breaking character despite a good prompt.
Test the whole setup
An adapter may still need a prompt, tools, and retrieved information. Test them together in your deployment software and check for new failures.
Improve agents without fine-tuning firstFine-tuning for character agents
Free template
Inspect a sample golden-dataset record
See fields for source trace, failure, approved correction, critique, scope, version, and approval status.
This template uses made-up examples, not client data. Adapt it to your policies, privacy requirements, and approval process.
Who should read this
- Teams defaulting to fine-tune for every regression
- Teams that need test results before spending on training infrastructure
- Compliance-sensitive workflows requiring removable corrections
Test the smallest useful change
Audit
Cluster failures, strengthen the prompt baseline, and decide what to test.
Build
Prepare reviewed examples for the agreed retrieval or training experiment.
Evaluate
Compare results on separate test cases, check for new failures, and measure running costs for one task.
Evidence before training spend
We test changes on one task and recommend whether training is worth pursuing. We can't promise improvement before reviewing the data.
How we test changes
We review real conversations with your team and agree what the agent should do differently. Approved corrections can become training data, test cases, or examples the agent looks up. We compare each change with your best existing setup on separate test cases, then check for new failures in the software you plan to deploy.
References:
Common questions
What does the work include?
Keep the source conversation, reviewer approval, and version history for each example. Decide which records are for training, retrieval, or testing. Keep test conversations out of the training set.
Do you fine-tune our model?
If it's worth testing after we review the problem. We try your best prompt and consider better examples or tool fixes first. We test an adapter on cases it wasn't trained on and check for new failures. The contract sets delivery and usage rights; the base model's license still applies.
What do we need to start?
Logs or conversation exports for one task, examples of good results, and a reviewer who can approve corrections. We agree data handling before you send anything. Remove unnecessary personal details.
How is success measured?
We test the existing and changed agent on the same task. Where the data is available, we compare completed tasks, human edits, handoffs, response time, and cost.
Related
Book a model review before your next training run.
Bring examples of the mistakes you want to fix. We'll discuss whether to test prompts, retrieval, or fine-tuning.