Context
A customer asks whether a travel booking has completed after an agent tool call.
How-to
How to build a golden dataset from production traces.
A practical sequence teams use to turn live agent failures into approved, retrievable examples—then prove whether they help.
Steps
- 01
Collect traces
Export OpenTelemetry spans or conversation logs that include requests, tools, intermediate steps, final responses, and outcome where available.
Start with a bounded workflow and a repeatable time window. Remove or mask unnecessary personal data before review, then retain a trace identifier so every selected example can be audited.
- 02
Cluster repeated failures
Group by intent, policy break, tool error, misunderstanding, or escalation—not by one-off noise.
Prioritize a cluster only when the failure repeats, has a measurable cost, and a reviewer can explain what a good result looks like. A small, coherent sample beats a large undifferentiated export.
- 03
Write the approved replacement
Capture the response or trajectory that should have happened, plus a short critique.
The correction should state what changed and why: missing fact, wrong tool choice, incomplete policy step, unsafe claim, or weak handoff. Use an escalation state rather than inventing a confident answer.
- 04
Store outside weights
Keep context, rewrite, scope, version, and approval status in a library you can filter and disable.
A minimum record includes source trace, workflow, failed output, approved replacement, reviewer, approval timestamp, policy scope, and retrieval restrictions. That provenance lets teams revise or retire an experience safely.
- 05
Retrieve and measure
Inject relevant examples at runtime and compare success, edits, escalations, latency, and cost.
Test against a held-out slice or a controlled workflow. Measure whether retrieved examples help the target scenario without degrading unrelated intents, then expand only if the evidence supports it.
Common mistakes
- Treating raw logs as gold without an approved rewrite
- Building huge synthetic sets before sampling real failures
- Skipping approval gates and letting stale examples into production
- Optimizing a score that is not tied to a business outcome
Example experience record
Observed failure
The agent states that a ticket was purchased without confirming the result or providing a safe next step.
Approved correction
Confirm the completed state only when the trace supports it, state the relevant booking detail, and invite the customer to return if anything changes.
Controls
Tag as travel support, completion confirmation, approved; restrict retrieval to matching workflow and policy scope.
See how this governed record fits a golden-dataset serviceUse a production-trace sprint when you need help applying the method
How less than three runs this
The Agent Improvement Sprint compresses this loop for one workflow: audit, human-corrected dataset, retrieval prototype, and before/after evidence.
Related
Want this done on your traces?
Book a call. Bring a sample of production traces and we will outline a focused golden-dataset sprint.