Evaluating agents before they touch production

A demo that answers one task is not an agent in production. An agent takes a task, calls tools, and stops, or it does not stop. Before it sees real tools and real data, we need a way to tell a good run from a run that only looks finished. That way is an evaluation: a task suite, a grader, a regression set, a budget on turns and steps, and a result pinned to one revision.

Task suites, graders and a regression set

A task suite is the jobs the agent is supposed to do, written down before we watch it work. Each task has a starting state, the tools it may use, and what done means. The path you hope for is one task among several. The suite also holds the cases that tend to break an agent: missing information, a tool that returns nothing, a tool that returns the wrong thing, and a request the agent should refuse.

A grader scores a run. Part of the score is mechanical. The expected file is there. The expected tool was called. A forbidden tool was not. The final state matches a known fixture. Another part is a rubric, read by a person or by a second model that only sees the task and the trace. We keep the two apart. A rubric can miss a bad tool call that the mechanical check would catch. A mechanical check can pass a run that did the steps and still answered the wrong question.

The regression set is the slice we run again whenever the prompt, the model, or the tools change. It is smaller than every experiment we have tried, and stable enough that a failure points at the change, not at the test. Failures we do not want to see again get added. Tasks that no longer describe the product get removed on purpose, in the same change as the behavior, so the set does not quietly rot.

Turn budgets and step budgets

An agent that may call tools will sometimes keep calling them. A turn budget caps how many model turns a task may take. A step budget caps how many tool calls, or how many steps inside one turn, we are willing to pay for. The cap belongs to the task. It is not a hidden constant for every job. Looking something up and filing a ticket should not share a budget just because they share a model.

A run that uses up the budget is a fail, even when the last message sounds complete. The budget keeps a loop from counting as progress just because it kept going. It also makes two revisions comparable. If a revision only gets through by taking more turns than the cap allows, it did not get through.

When we raise a cap, we write down why. A task that never finishes inside the cap is a task we have not specified, or an agent that is lost. Adding turns until it happens to land on the answer teaches the suite to accept wandering.

Proof belongs to a revision

A result is proof only when it names what ran. We pin it to the prompt revision, the model revision, and the tool revision. A prompt edit, a model swap, a schema change, and a change to the text a tool returns are different revisions. A green run from last week does not travel with today's prompt.

The trace is part of the result: which tools were called, with what arguments, what they returned, and what the agent said at the end. A score without a trace cannot be debugged, and it cannot be compared. We keep the result next to the revision it measured, so a later reader can see which change the score belongs to.

If the suite, the grader, or the budget changes, that is a new revision of the evaluation too. A higher score after a softer grader is not an improvement in the agent.

Failure modes we look for on purpose

We do not wait for production to meet these. The suite should be able to draw them out.

Looping. The agent calls the same tool, or walks the same plan, without a new fact. The turn budget stops the spend. The grader should still mark the run as a loop, not as a timeout that happened to be unlucky.

Tool misuse. The agent calls a tool it was not given, calls the right tool with arguments that do not match the schema, or writes when the task was read-only. The trace shows it. A grader that only reads the final answer does not.

Hallucinated actions. The agent says the email went out, the record was updated, or the check ran. The trace has no such call. We grade the claim against the trace. A confident final message is not evidence.

Silent partial completion. The agent finishes some of the task, skips a required part, and reports success. This is the failure that looks healthiest in a demo. The task has to say what complete means. The grader has to fail a run that left a required part out.

Prompt injection via tool output. A tool returns text, and that text contains instructions. A page, a ticket, a document, or a log line can tell the agent to ignore the task, leak something, or call a tool nobody asked for. Fixtures like that belong in the suite. A pass means the agent follows the task and the permissions, not the text inside the tool result. The model's capability is not a control.

What good enough means

Good enough is a decision about access, not a feeling after a demo. We stage it.

Read-only tools first. The agent may look, search, and draft. It may not change customer data, send a message, or merge a change. The suite runs against those tools until the traces look like the task, not like improvisation.

Sandbox data next. The records, documents, and accounts are fake, or copied somewhere a wrong write does no harm. The tools still have the shape production will have, so a surprise in the schema shows up here.

Then a human approval gate. A write, a send, or anything hard to undo waits for a person. The evaluation includes tasks where the right behavior is to stop and ask, and tasks where a gate should not appear, because the action was never allowed.

Then wider access, one capability at a time. Real tools and real data come after the regression set passes on the revision that is supposed to ship, the budgets still hold, and a person can read the trace. Wider access is its own decision. A green suite on read-only tools is not a yes to a write tool.

The suite needs an owner after launch. The model will change, and the prompt will change. The regression set is what says whether that change ships. Keeping an agent that is already live is a different job from building it. The set has to keep running, and a failure has to reach a person.

Where this meets a delivery

A single agent that has to run in production is a Hephaestus Sprint: one system, a golden eval set, monitoring hooks, a runbook, and a decision on whether to scale. A team putting coding agents into a repo is an AI Coding Rollout: what the agent may touch, how a change is proved, and a gate before it lands. An agent that is already live, and needs the suite to keep running when the model or the prompt changes, sits on Managed AI Ops.

Evaluation does not replace the runtime around the model. Permissions, a loop that can stop, and a record of the turn still have to exist. The suite is how we decide that runtime may see a real task.

Frequently asked questions

What is a task suite? The jobs the agent must do, each with a starting state, the tools it may use, and a definition of done. It includes the awkward cases, not only the path you hope for.

What is the difference between a grader and a regression set? A grader scores one run. A regression set is the stable slice you run again on every prompt, model, and tool change, so a failure points at that change.

Why pin a result to a revision? A score without the prompt, the model, and the tools that produced it cannot be compared with the next change. Last week's green run does not apply to today's prompt.

What should fail an agent before production? A loop that spends the budget, a call the task did not allow, an action the agent claims but never took, a partial task reported as done, or an instruction hidden in tool output that the agent obeys. Any one of those is a reason to keep real data away.

When is an agent good enough for real tools? When the regression set passes on the revision that is supposed to ship, the turn and step budgets still hold, the trace matches what the agent claims, and a person still approves anything hard to undo. Read-only tools and sandbox data come first. Wider access is its own decision.