Conflicting rules
A rule that helps one request can hurt another. Run the other tests after each change.
Prompt engineering vs experience library
Keep general rules in the prompt. Look up relevant examples.
An experience library is a searchable collection of approved examples. Instead of adding every exception to the system prompt, the agent looks up examples that match the current request. Test whether this helps before adding it to your setup.
Long prompts can become hard to maintain.
Extra rules increase token costs and can conflict with existing instructions. Keep a record of why each exception exists and test what changes when you add it.
Unclear reasons for rules
Without source examples and version history, reviewers may not know why an instruction was added.
Lost corrections
Useful edits stay in tickets instead of becoming approved examples.
Eval disconnect
A failed score does not add an approved replacement to the prompt.
What an experience library adds
The agent looks up approved corrections from similar real cases. Filters restrict which examples it can use for the current task and policy.
Targeted retrieval
Only relevant approved examples enter context for similar requests.
Reviewer ownership
Experts approve, scope, version, and retire records explicitly.
Measured impact
Compare baseline vs retrieval-enabled agent on one workflow.
What to keep in the prompt
Use the prompt for instructions that apply to every request. Consider a library when examples apply only to particular cases. Not every agent needs both.
Keep in prompts
Core persona, tool policies, safety rails, stable formatting rules.
Move to library
Repeated failure modes humans already rewrite in production.
Measure both
Fix prompt regressions separately from retrieval coverage gaps.
Build golden datasets for AI agentsSee how we prepare and test model changes
Free template
Get the trace readiness checklist
Confirm you have the fields needed before building an experience library from production traces.
This template uses made-up examples, not client data. Adapt it to your policies, privacy requirements, and approval process.
Who should read this
- Teams whose system prompts grow every sprint
- Support and voice agents with repeating exception clusters
- Teams that want to reuse approved reviewer edits
From prompt patches to a library
Review
Link prompt exceptions to recurring mistakes in the logs.
Build
Prepare approved, versioned examples with rules for where they apply.
Evaluate
Compare results with and without retrieval while keeping the prompt the same.
Keep the prompt the same during the comparison
Compare the agent with and without retrieved examples on separate test cases. Keeping the prompt fixed helps identify what retrieval changes.
How we test changes
We review real conversations with your team and agree what the agent should do differently. Approved corrections can become training data, test cases, or examples the agent looks up. We compare each change with your best existing setup on separate test cases, then check for new failures in the software you plan to deploy.
References:
Common questions
What does the work include?
The agent looks up approved corrections from similar real cases. Filters restrict which examples it can use for the current task and policy.
Do you fine-tune our model?
If it's worth testing after we review the problem. We try your best prompt and consider better examples or tool fixes first. We test an adapter on cases it wasn't trained on and check for new failures. The contract sets delivery and usage rights; the base model's license still applies.
What do we need to start?
Logs or conversation exports for one task, examples of good results, and a reviewer who can approve corrections. We agree data handling before you send anything. Remove unnecessary personal details.
How is success measured?
We test the existing and changed agent on the same task. Where the data is available, we compare completed tasks, human edits, handoffs, response time, and cost.
Related
Discuss a prompt that's getting hard to maintain.
Bring the prompt and examples of recurring mistakes. We'll look at which rules should stay and whether retrieved examples are worth testing.