Regression testing for AI agents, generated
Every playbook and recurring failure becomes a regression test. Run retrack test in CI so prompt and code changes cannot undo what agents learned.
Updated
Agents regress quietly. A prompt tweak, a new tool, a library upgrade or a model change, and a task that worked last week starts failing in production, where you hear about it from a customer. Production failures rarely become tests, because writing agent tests by hand is slow.
Retrack writes them for you, from what your agent already learned.
Every lesson is a test
- When a playbook becomes active, it becomes a regression case: the task it was learned from, with assertions that the agent completes and calls the playbook's tools in order.
- When a failure happens twice, it becomes a critical case asserting that it does not happen again.
- Cases update when the playbook does, and retire with it.
The suite grows by itself as your agent works.
Run it on every pull request
retrack test --entry my_agent:run_task --agent support-agent --threshold 0.9The command runs each case through your own agent, on your machine or CI runner, and fails when the pass rate drops below the threshold or a critical case that passed last time now fails. Test runs never become memory.
- run: pip install -r requirements.txt retrack-ai
- run: retrack test --entry my_agent:run_task --agent support-agent
env:
RETRACK_API_KEY: ${{ secrets.RETRACK_API_KEY }}See what changed
The dashboard's Regression page shows every case, its source playbook or failure, and the pass rate of each test run, so a regression points straight at the procedure that broke.
Assertions that fit agents
completes: the agent finished without raising.tool_sequence: the expected tools were called in order, with other calls allowed in between.no_failure: a known failure did not recur.output_contains: the answer contains the expected text.
Set it up with the regression tests guide. Upgrading models? See model migration.