Documentation menu

Playbooks: procedures your agent learned

What a Retrack playbook contains, how it is learned from verified agent runs, how it gains conditional steps and pitfalls, and when it is recalled.

Updated

A playbook is the procedure that worked for one kind of task, learned from your agent's verified runs. Your agent receives it in its prompt before a similar task, so it follows a known-good path instead of planning from scratch and rediscovering every rule by trial and error.

What a playbook contains

  • Title and task pattern: the kind of task it covers, such as "Refund a wrong-item delivery order".
  • Preconditions: what must be true before starting.
  • Steps: each with the tool it uses. A step can carry a condition (when) and apply only when that condition holds.
  • Pitfalls: short, general lessons from errors the agent recovered from.

Here is a playbook Retrack learned in the Ops Desk benchmark, as your agent sees it:

Playbook: Update order delivery address
1. Confirm the order ID and the full new address with PIN code are provided.
2. Call update_address with order_id, address, and pin_code fields. (update_address)
3. Check that the response shows updated true and status 200.
4. If update_address returns status 409 saying the order already shipped:
   Call reroute_shipment with order_id, address, and pin_code fields. (reroute_shipment)
Pitfalls: Pass the PIN code as its own field rather than inside the address text.

How playbooks are learned

  1. A run succeeds and is verified (see Verifying runs).
  2. If it matches an existing playbook's kind of task, it merges into it as more support. A run that followed the same procedure but was worded differently also merges, and the playbook remembers that wording, so it is recalled however the task is phrased.
  3. Otherwise Retrack writes a new playbook from the run, as a candidate.
  4. A candidate becomes active when you approve it in the dashboard, or automatically after 3 trusted successes with a hard signal (a test, a business event or user feedback). AI-judge verdicts add support but never promote.

How playbooks improve

When a verified run teaches something the playbook lacks, Retrack proposes a new version:

  • A tool the task needed becomes a new step with a condition. Incidental calls, such as an extra lookup, are left out.
  • An error the agent recovered from becomes a pitfall that names the tool and the error.
  • Every existing step is kept; rewording alone changes nothing.

A trusted run updates the playbook directly. A change suggested by an untrusted run to an active playbook goes to Review as a proposal with the diff. Every version is kept, and you can roll back to any of them.

Quality controls

  • Usage tracking: each recall is linked to the run that follows, so every playbook has a real success rate.
  • Auto-retire: a playbook whose success rate falls below 50% after 10 or more uses is retired.
  • No duplicates: a recall never returns two copies of one procedure.
  • Model and tool versions: a playbook validated on your current model and tool versions ranks higher; one whose tool had a breaking version change is not recalled.

Using playbooks without a prompt

memory.as_prompt() is the simplest way to use memory. The raw playbooks are also available as memory.playbooks (a list of dictionaries), and through the API at GET /v1/playbooks.