Model migration: test a new model first
Replay your agent's memory on a new LLM before a forced model upgrade with retrack migrate, and see per playbook what passes, breaks or needs relearning.
Updated
Model providers retire models on fixed schedules, and a newer model does not always handle your agent's tasks the same way. retrack migrate replays your agent's memory on the new model before you switch, and reports, playbook by playbook, what still works.
What it does
- Opens a migration report in Retrack for your agent and the target model.
- Runs every regression case (or a sample) through your agent on the new model, on your machine, with your own model keys. Retrack does not call the model.
- Sends the results and prints the report.
Playbooks that pass are marked as validated on the new model, so recall ranks them fully as soon as you switch.
Write an entry point that takes a model
Use the same entry point as regression tests, with a third argument for the model:
def run_task(task: str, callbacks: list, model_id: str | None = None) -> str:
llm = ChatAnthropic(model=model_id or "claude-haiku-5-5")
graph = build_graph(llm)
result = graph.invoke({"messages": [("user", task)]}, config={"callbacks": callbacks})
return result["messages"][-1].contentA two-argument entry point still runs, with a warning, but then the replay cannot actually switch models, and the report says so.
Run a migration
retrack migrate --to claude-sonnet-5 --from claude-haiku-5-5 \
--entry my_agent:run_task --agent support-agent| Option | Meaning |
|---|---|
--to | The model to migrate to (required) |
--from | The current model, for the record |
--sample N, --seed S | Replay only N cases, chosen with seed S |
--entry, --agent | As for retrack test |
Reading the report
| Result | Meaning |
|---|---|
passes | The playbook's case passed on the new model |
needs_relearning | The agent finished, but an assertion failed: the procedure changed |
breaks | The agent raised an error |
fixed | A known failure no longer happens on the new model |
returns | A known failure still happens |
The command exits with 1 when anything breaks or a known failure returns, so you can use it as a gate before switching models in CI. Reports are also in the dashboard under Migrations, and through GET /v1/migrations/{id}.
Migration results never change the regression cases' last results, which always describe your current model.