Benchmarks: agent memory on real Claude agents
Measured results of Retrack on Claude Haiku 5.5 agents: avoidable mistakes −80% on a support agent, tool errors −67% on an ops agent, and the limits.
Updated
These are the measured results of Retrack on real Claude agents, including what did not improve. Every number on this site comes from these runs.
Maple & Main: a real support agent
A Claude Haiku 5.5 support agent for an online store, built with LangGraph, with 11 tools and 5 to 10 steps per request. The store has rules the agent is never told: which cancellation reasons are valid, how gift-card orders are refunded, when a refund needs approval. It learns them only from errors.
The agent handled 16 customer requests twice from the same starting state: once without memory, and once with Retrack's memory from 8 earlier requests. Every request was checked against the store's final state.
| Without memory | With Retrack | Change | |
|---|---|---|---|
| Requests resolved correctly | 16/16 | 16/16 | |
| Avoidable mistakes | 15 | 3 | −80% |
| Steps per request | 7.19 | 6.25 | −13% |
| Tokens per request | 17,450 | 18,398 | +5% |
| Claude cost per request | $0.0344 | $0.0349 | +2% |
| Seconds per request | 8.6 | 8.1 | −6% |
The 13 requests that received memory made no avoidable mistakes. The other 3 received no memory by design: requests in Retrack's holdout group (a random slice gets no memory so the effect stays measurable) and the request carrying an injected instruction, which the firewall quarantined and never learned from. They are counted as they are.
Mistakes by type, without and with Retrack: invalid cancellation reasons 12 → 3, wrong refund methods 3 → 0.
Ops Desk: a second agent
An operations agent handling address changes, cancellations and refunds against APIs with unwritten rules (reason codes, PIN fields, approval limits, shipped orders). 16 learning tasks, then 20 evaluation tasks run with and without memory from the same seed.
| Without memory | With Retrack | |
|---|---|---|
| Success rate | 95% | 100% |
| Tool calls per task | 2.90 | 2.40 |
| Tool errors per task | 0.75 | 0.25 |
| Tokens per task | 5,468 | 5,836 |
Tool errors fell 67% (95% interval: −0.75 to −0.25 errors per task) and tool calls 17%. The success-rate change, +5 points, is not conclusive at this size (interval 0 to +15). Memory was recalled on 19 of 20 tasks. Tokens rose about 7%, within noise.
How the tests are run
- Same tasks, same starting state: each request runs with and without memory from the same seed, so the only difference is memory.
- Hidden rules: the agent is never told the rules it must learn; it discovers them from tool errors, as a production agent does.
- Checked outcomes: success is the system's final state, not the agent's claim.
- One model: Claude Haiku 5.5 as the agent; Retrack's own AI steps also use Claude Haiku 5.5.
The Maple & Main test is open: its agent, store, tasks and raw results ship with the connector's examples, so you can rerun it.
Honest limits
- 16 paired requests show a clear effect, not a statistical proof.
- The support agent resolved every request even without memory, by trial and error. Here memory removes mistakes and wasted steps; it does not turn failures into successes.
- Memory adds a playbook to the prompt, a few percent of tokens. Fewer steps and retries pay for most of it, so cost is about flat. We do not claim token savings.
- Two simulated businesses and one model. Your agent will differ, which is why every project has a holdout group that measures the effect on your own traffic.
How these results changed Retrack
The first real run showed memory helping less than it should: warnings named errors without the fix, playbooks from customer-facing runs could never gain a step, and one kind of task worded three ways became three playbooks that were never recalled. Each cause was fixed in Retrack for every agent and checked on both agents. Avoidable mistakes per 8 requests with memory went from 5 before those fixes to 1.5 after.