Model migration testing for LLM agents

Replay your agent's memory on a new model before a forced upgrade and get a per-playbook report of what passes, breaks or needs relearning.

Updated

Model upgrades are not optional. Providers deprecate models on fixed schedules, sometimes with only weeks between the notice and retirement, and a newer model does not always handle your agent's specific tasks the same way. Teams find out what broke after the switch.

Retrack makes the switch a checklist.

Replay your agent's memory on the new model

retrack migrate --to claude-sonnet-5 --from claude-haiku-5-5 \
  --entry my_agent:run_task --agent support-agent

The command replays every regression case, the ones generated from your playbooks and known failures, through your own agent on the new model. It runs on your side with your model keys, so Retrack never calls the model and your data never leaves your setup for this.

A per-playbook report

ResultMeaning
passesThis procedure still works on the new model
needs relearningThe agent finished, but did it differently
breaksThe agent raised an error
fixedA known failure no longer happens
returnsA known failure still happens

Playbooks that pass are marked as validated on the new model, so recall trusts them fully the day you switch. The rest is a short list of exactly what to fix first.

Gate the switch in CI

retrack migrate exits with an error when anything breaks or a known failure returns, so a model change cannot ship until its report is clean.

Why it matters

Memory is matched to the model and tool versions it was validated on. Without a migration check, switching models silently lowers trust in every playbook; with one, you know in advance which procedures carry over and which need new runs.

Set it up with the model migration guide.