Training latency
Failure clusters and policies change before a fine-tune ships.
Fine-tuning vs golden datasets
Fine-tuning vs golden datasets: which fixes repeating production failures faster?
A frozen 2x2 decides the first move. Gold feeds retrieve or train: exceptions stay outside weights; a speaker adapter is for identity the prompt cannot install. Disclose the capability tax before you train.
Fine-tuning is not wrong—it is often the wrong first move.
On an internal 2x2, a 760-character persona prompt made a 27B base shorter and more casual. It still had no speaker. Repeating exceptions need retrieval first; speaker identity needs approved gold, then an adapter judged on the same frozen turns.
Opaque corrections
Individual approved answers disappear inside weights.
Hard to retire
Removing one bad example from trained behavior is expensive.
No runtime audit
Compliance teams cannot see which example influenced a decision.
What golden datasets add
A governed library of approved corrections from production traces—versioned, scoped, and removable—retrieved when similar requests arrive.
Faster proof loop
Measure retrieval impact on one workflow this sprint—not next train cycle.
Inspectability
Every record ties to source trace, reviewer, scope, and approval status.
Composable with training
Retrieve sparse exceptions. Train a speaker adapter when a 2x2 shows the prompt ceiling. Disclose IFEval/MMLU cost.
Decision guide
Choose the first move based on failure type, audit needs, and the 2x2—not vendor defaults.
Choose golden datasets first
Exception-heavy workflows where humans already rewrite agent output.
Choose a speaker adapter
Register, ownership, and identity still fail after the best system prompt.
Use both
Retrieval for exceptions; training for speaker when the frozen 2x2 fails. Capability benches can fall.
Improve agents without fine-tuning firstSpeaker gold for character agents
Reusable resource
Inspect a sample golden-dataset record
See fields for source trace, failure, approved correction, critique, scope, version, and approval status.
Illustrative template—not client data. Adapt it to your policies, privacy controls, and approval process.
Who should read this
- Teams defaulting to fine-tune for every regression
- Leaders who need measured lift before training infrastructure spend
- Compliance-sensitive workflows requiring removable corrections
Prove retrieval before you train
A — Audit
Cluster repeating failures and decide retrieval vs train candidates.
B — Dataset
Human-approved experience library + retrieval prototype.
C — Proof
Baseline vs corrected comparison on one workflow.
Evidence before training spend
We do not promise improvement before examining traces. The sprint produces evidence—and a recommendation on whether fine-tuning is still needed.
Methodology by LTTAI LTD
This page describes less than three’s production-trace review method: reconstruct failures, capture human-approved corrections, store them outside model weights, and measure whether scoped retrieval improves a defined workflow.
Primary references used for terminology and risk framing:
Common questions
What does fine-tuning vs golden datasets include?
A governed library of approved corrections from production traces—versioned, scoped, and removable—retrieved when similar requests arrive.
Do you fine-tune our model?
Not first. Approved corrections for exceptions stay outside weights and can be retrieved at runtime. We train a speaker adapter when a frozen 2x2 shows the prompt cannot install identity.
What do we need to start?
A sample of production traces or conversation exports for one workflow, plus a working definition of success for that workflow.
How is success measured?
We compare a fixed baseline against a changed agent on the same workflow using task success, edits, escalations, latency, and cost where available.
Related
Book a trace review before your next training run.
Send production traces. We will map which failures can improve through approved retrieval first.