One fix, many breaks
Patching intent A weakens intent B in the same prompt.
Prompt engineering vs experience library
Prompt engineering vs an experience library: stop packing exceptions into the system prompt.
Longer prompts fix one case and break another. An experience library stores approved corrections outside the prompt—retrieved only when similar failures appear.
Prompt sprawl has a ceiling.
Every exception added to the system prompt increases conflict risk, token cost, and regression surface—without giving reviewers a durable place to store fixes.
No provenance
Reviewers cannot see which prompt clause caused a regression.
No compounding
Ticket rewrites disappear instead of becoming reusable examples.
Eval disconnect
A failed score does not add an approved replacement to the prompt.
What an experience library adds
Scoped, approved corrections from production traces—retrieved dynamically instead of permanently embedded in the system prompt.
Targeted retrieval
Only relevant approved examples enter context for similar requests.
Reviewer ownership
Experts approve, scope, version, and retire records explicitly.
Measured impact
Compare baseline vs retrieval-enabled agent on one workflow.
When prompts still win
Prompts define stable behavior. Libraries capture exceptions. You need both—just not the same mechanism.
Keep in prompts
Core persona, tool policies, safety rails, stable formatting rules.
Move to library
Repeated failure modes humans already rewrite in production.
Measure both
Fix prompt regressions separately from retrieval coverage gaps.
Build golden datasets for AI agentsSee the RAHF method on the homepage
Reusable resource
Get the trace readiness checklist
Confirm you have the fields needed before building an experience library from production traces.
Illustrative template—not client data. Adapt it to your policies, privacy controls, and approval process.
Who should read this
- Teams whose system prompts grow every sprint
- Support and voice agents with repeating exception clusters
- Operators who want reviewer edits to compound
From prompt patches to a library
A — Audit
Map prompt exceptions to recurring trace failures.
B — Dataset
Approved experience records + retrieval prototype.
C — Proof
Before/after on one workflow with prompt held stable.
Hold the prompt stable; prove retrieval lift
The sprint isolates retrieval impact by fixing the baseline prompt and measuring correction library coverage.
Methodology by LTTAI LTD
This page describes less than three’s production-trace review method: reconstruct failures, capture human-approved corrections, store them outside model weights, and measure whether scoped retrieval improves a defined workflow.
Primary references used for terminology and risk framing:
Common questions
What does prompt engineering vs experience library include?
Scoped, approved corrections from production traces—retrieved dynamically instead of permanently embedded in the system prompt.
Do you fine-tune our model?
Not first. Approved corrections for exceptions stay outside weights and can be retrieved at runtime. We train a speaker adapter when a frozen 2x2 shows the prompt cannot install identity.
What do we need to start?
A sample of production traces or conversation exports for one workflow, plus a working definition of success for that workflow.
How is success measured?
We compare a fixed baseline against a changed agent on the same workflow using task success, edits, escalations, latency, and cost where available.
Related
Book a trace review if prompt sprawl is slowing releases.
Send production traces. We will estimate experience coverage without another prompt rewrite.