Playbooks: procedures your agent learned
What a Retrack playbook contains, how it is learned from verified agent runs, how it gains conditional steps and pitfalls, and when it is recalled.
Updated
A playbook is the procedure that worked for one kind of task, learned from your agent's verified runs. Your agent receives it in its prompt before a similar task, so it follows a known-good path instead of planning from scratch and rediscovering every rule by trial and error.
What a playbook contains
- Title and task pattern: the kind of task it covers, such as "Refund a wrong-item delivery order".
- Preconditions: what must be true before starting.
- Steps: each with the tool it uses. A step can carry a condition (
when) and apply only when that condition holds. - Pitfalls: short, general lessons from errors the agent recovered from.
Here is a playbook Retrack learned in the Ops Desk benchmark, as your agent sees it:
Playbook: Update order delivery address
1. Confirm the order ID and the full new address with PIN code are provided.
2. Call update_address with order_id, address, and pin_code fields. (update_address)
3. Check that the response shows updated true and status 200.
4. If update_address returns status 409 saying the order already shipped:
Call reroute_shipment with order_id, address, and pin_code fields. (reroute_shipment)
Pitfalls: Pass the PIN code as its own field rather than inside the address text.How playbooks are learned
- A run succeeds and is verified (see Verifying runs).
- If it matches an existing playbook's kind of task, it merges into it as more support. A run that followed the same procedure but was worded differently also merges, and the playbook remembers that wording, so it is recalled however the task is phrased.
- Otherwise Retrack writes a new playbook from the run, as a
candidate. - A candidate becomes
activewhen you approve it in the dashboard, or automatically after 3 trusted successes with a hard signal (a test, a business event or user feedback). AI-judge verdicts add support but never promote.
How playbooks improve
When a verified run teaches something the playbook lacks, Retrack proposes a new version:
- A tool the task needed becomes a new step with a condition. Incidental calls, such as an extra lookup, are left out.
- An error the agent recovered from becomes a pitfall that names the tool and the error.
- Every existing step is kept; rewording alone changes nothing.
A trusted run updates the playbook directly. A change suggested by an untrusted run to an active playbook goes to Review as a proposal with the diff. Every version is kept, and you can roll back to any of them.
Quality controls
- Usage tracking: each recall is linked to the run that follows, so every playbook has a real success rate.
- Auto-retire: a playbook whose success rate falls below 50% after 10 or more uses is retired.
- No duplicates: a recall never returns two copies of one procedure.
- Model and tool versions: a playbook validated on your current model and tool versions ranks higher; one whose tool had a breaking version change is not recalled.
Using playbooks without a prompt
memory.as_prompt() is the simplest way to use memory. The raw playbooks are also available as memory.playbooks (a list of dictionaries), and through the API at GET /v1/playbooks.