Unlabeled failures
Errors without an ideal response cannot train or retrieve well.
Production traces → golden data
Turn conversation logs into examples your agent can learn from.
We review failed conversations and tool calls, record what should have happened, and get your team's approval. The resulting golden dataset includes the original context, corrected reply, and rules for where each example can be used.
Raw traces are not a golden dataset.
Logs show what happened. A useful example also records the correct reply, why it's correct, and who approved it.
Synthetic-only sets
Made-up examples miss your tools, policies, and edge cases.
Exports that rot
CSV dumps lack versioning, scope, and runtime retrieval design.
Training every correction
Some facts and exceptions change too often to put into model weights. Keep those in a source you can update.
Build and test a library of approved examples
Review the logs, approve corrections, and make relevant examples available to the agent. Test whether they help on separate cases.
Ingest what you already have
OpenTelemetry preferred; conversation or observability exports work for a first sprint.
Review with experts
Your reviewers, trained annotators, or both use the rules and approval process you agree on.
Ship a usable library
Each record keeps context, failed output, rewrite, critique, and status.
A record needs enough context to be reused safely
Keep enough context to check why an example was approved and where it applies. A correction for one customer shouldn't silently become a rule for everyone.
Source context
The log ID, user's request, available tools, current policy, and the reply or actions that went wrong.
Approved correction
The replacement response or action path, reviewer critique, approval status, and any escalation note.
Retrieval controls
Workflow, tenant, policy, confidence, and date metadata that limits where an experience can be used.
See the commercial golden-datasets service and its dataset design approachFollow the practical build process step by step
Prepare logs for review
Put records in a consistent format, check the context, and have reviewers approve the correct reply or actions.
Accepted formats
OpenTelemetry spans, conversation exports, ticket threads, and observability dumps with stable identifiers.
Field mapping
Map request, tools, intermediate state, final output, outcome, and version fields into a reviewable record.
Eligibility checklist
Use the trace-readiness checklist to reject incomplete, unsafe, or unresolvable candidates.
Review process
Format the log, have a reviewer approve the correction, and assign the example to a training, retrieval, or test set.
Incomplete logs
Set aside records with missing tool results, conflicting information, or outdated policies. Don't invent the missing context.
Free template
Check whether your production traces are ready
A print-ready checklist for required fields, privacy handling, eligibility, and rejection criteria.
This template uses made-up examples, not client data. Adapt it to your policies, privacy requirements, and approval process.
Who this is for
- Teams with production traffic and repeating request patterns
- Teams that already review or edit agent replies
- Teams that want training and test data based on real mistakes
- Teams that want to test an improvement before changing their architecture
Sprint deliverables
Review
Identify recurring mistakes and which cases your logs cover.
Build
Prepare approved examples with source records for the agreed training, retrieval, or testing use.
Evaluate
Compare the changed and existing agent, then recommend the next step.
Keep a record of why each example exists
Link each correction to the original conversation so reviewers can check it and remove it when the policy or product changes.
Limitations
- If important evidence is missing from a log, reviewers may not be able to approve a correction.
- Exports without version metadata make later auditing unreliable.
- Removing personal information can also remove context reviewers need. Check exports before sharing them.
- A large unlabeled history is not a golden dataset.
How we test changes
We review real conversations with your team and agree what the agent should do differently. Approved corrections can become training data, test cases, or examples the agent looks up. We compare each change with your best existing setup on separate test cases, then check for new failures in the software you plan to deploy.
References:
Common questions
What does the work include?
Review the logs, approve corrections, and make relevant examples available to the agent. Test whether they help on separate cases.
Do you fine-tune our model?
If it's worth testing after we review the problem. We try your best prompt and consider better examples or tool fixes first. We test an adapter on cases it wasn't trained on and check for new failures. The contract sets delivery and usage rights; the base model's license still applies.
What do we need to start?
Logs or conversation exports for one task, examples of good results, and a reviewer who can approve corrections. We agree data handling before you send anything. Remove unnecessary personal details.
How is success measured?
We test the existing and changed agent on the same task. Where the data is available, we compare completed tasks, human edits, handoffs, response time, and cost.
Related
Plan a dataset from your conversation logs.
Book a call to discuss recurring mistakes and the examples needed to address them.