Unlabeled failures
Errors without an ideal response cannot train or retrieve well.
Production traces → golden data
Golden datasets from production traces—not synthetic guesses.
Your best training signal is already happening: humans fixing agent mistakes. We capture those corrections as an approved experience library.
Raw traces are not a golden dataset.
Production history shows outcomes. Without the chosen rewrite, critique, and approval status, traces cannot teach the next request.
Synthetic-only sets
Made-up examples miss your tools, policies, and edge cases.
Exports that rot
CSV dumps lack versioning, scope, and runtime retrieval design.
Fine-tune pressure
Pushing every fix into weights slows iteration and hides provenance.
The production-to-library loop
Connect traces → human review → approved experiences → runtime retrieval → measured outcome.
Ingest what you already have
OpenTelemetry preferred; conversation or observability exports work for a first sprint.
Correct with experts
Internal reviewers, trained annotators, or both—your rubric, your approvals.
Ship a usable library
Each record keeps context, failed output, rewrite, critique, and status.
A record needs enough context to be reused safely
The artifact is a governed experience record, not a row of text. The fields below make it possible to audit retrieval and avoid applying one customer’s exception to another workflow.
Source context
Trace ID, user intent, relevant tools, policy state, and the failed output or trajectory.
Approved correction
The replacement response or action path, reviewer critique, approval status, and any escalation note.
Retrieval controls
Workflow, tenant, policy, confidence, and date metadata that limits where an experience can be used.
See the commercial golden-datasets service and its dataset design approachFollow the practical build process step by step
Who this is for
- Teams with production traffic and repeating request patterns
- Orgs that already review or edit agent output
- Builders who want golden data aligned to live failures
- Leaders who need measurable lift before architecture changes
Sprint deliverables
A — Audit
Coverage estimate and failure clustering from your trace sample.
B — Dataset
Human-corrected golden set for one workflow plus retrieval prototype.
C — Prove
Baseline vs corrected comparison and next-step recommendation.
Why this converts better than generic data
Every example points back to a real failure your users hit—so retrieval stays relevant and auditable.
Related
Send traces. Get a golden-dataset plan.
Book a call with a sample of production traces—we will map repeated failures and one sprint path.