Measuring impact with a holdout group
Retrack holds out a slice of runs with no memory and compares success rate, tokens and cost, so the effect of memory is measured, not estimated.
Updated
Most memory products ask you to believe they help. Retrack measures it: a small random slice of your runs gets no memory, and the dashboard compares them with the rest.
How the holdout works
- By default 10% of runs are held out; change it (0 to 50%) in your project's Settings. Which runs is decided deterministically from the project and the run id, so the split is stable and fair.
- For a held-out run, recall returns empty memory and
memory.holdoutisTrue. Your agent runs exactly as it would without Retrack. - Everything else is the same, so the only difference between the two groups is memory.
What is compared
The dashboard's overview (and GET /v1/metrics) compares runs that received memory with held-out runs:
| Metric | Meaning |
|---|---|
| Success rate | Share of runs with a known outcome that succeeded |
| Lift | Memory success rate minus holdout success rate, in percentage points |
| Tokens and cost | Average per run, and the relative change with memory |
| Recall hit rate | Share of recalls that returned at least one playbook |
| Recall latency | Median and 95th percentile |
Success rates use runs with a known outcome only, so send outcomes where you can.
When to trust the numbers
Small samples are noisy. Retrack marks the comparison as having enough evidence once at least 20 held-out runs have a known outcome. Before that, treat the numbers as early signs.
Reading the API
curl -H "Authorization: Bearer $RETRACK_API_KEY" \
"https://api.retrack-ai.online/v1/metrics?days=30"The response includes memory and holdout blocks (runs, success rate, average tokens and cost), lift_points, token_change, cost_change, enough_evidence and recall statistics. See the HTTP API.