Agent harnesses: how we build the runtime around an LLM

TL;DR

• An agent harness is the runtime around a language model: which tools exist, what fits in the window, when a turn stops, and what gets written down.
• The model proposes the next step. The harness decides whether that step runs.
• We build that runtime ourselves, and we run it on our own engineering work.

Tools stay behind an allow-list

A tool is a name and a schema. The catalog stays off the model. We expand an allow-list against the tools that actually exist, and we keep the list the model sees short, so the window stays about the task. The longer instructions for a tool stay out until something asks for them.

Control-plane tools stay hidden even from a wildcard: the turn runner, the queue, the session store, and the raw call to the model. A call that matches nothing waits for a person.

Context is a budget. Memory is a store.

Context is assembled for one turn. If it does not fit, the turn never reaches the provider. Memory is a separate store: working state, observations, and retrieval.

What we recall is placed in the request. It is not written into the transcript as if the model had said it. If that memory step fails, the turn still runs.

A loop stops on a budget

A turn has a shape: understand the ask, choose how to work, plan, act, check, then answer and stop. If that choice is missing or errors, we fall back to a reactive loop. A path that must decide before it acts stops when the decision does not land.

A plan is a checklist a person can leave off, force, or gate. One step runs per turn. A missed check is replanned once, and then the unfinished tail is dropped. That limit lives in the harness, where the model cannot edit it.

function nextTurn(index, budget) {
  const next = index + 1;
  if (next >= budget) {
    return "stop";
  }
  return {
    index: next,
    budget,
  };
}

A child run cannot take more turns than the parent has left, and it does not inherit a forced plan.

A call is a grant, then a gate

Each run, and each child, gets its own sandbox. Access to files is granted, then checked again before it is used. A read-only run cannot pick up tools that change things. Checks before and after a tool fail closed.

Irreversible actions always ask a person, even when the loop would otherwise allow them: merge, rebase, land, and apply, patch, or delete on a cluster.

An eval can stop the run

An eval watches the session and may continue or abort, inside its own turn budget. The usual check looks for a substring.

A proof that a change is good is tied to one revision. It is not the eval, and it is not the goal. The goal says what done means. The eval says whether this run may continue.

Write the turn down

Every turn is appended: the call, the result, and token use by layer. If the provider omits a cost, we attach one from the model catalog, so the turn can be costed from its own record.

Traces can be shipped out. They are not required for the turn to run.

Retry only before output starts

A call to the provider is retried only before any real output has been forwarded. An idle timeout is not retried. A stream that runs too long is stopped, and that stop is not retried either.

Bad output, or a completion the harness denies, is retried a fixed number of times. Then the turn stops.

Where this meets a delivery

One agent that has to run in production is a Hephaestus Sprint: one system, a golden eval set, monitoring, a runbook, and a decision on whether to scale. A team putting coding agents in front of engineers is an AI Coding Rollout: repo context, skills, a security boundary, and a gate in CI before a change lands.

Frequently asked questions

What is an agent harness? It is the runtime around a language model. The model proposes the next step. The harness decides which tools exist, what may sit in the window, and whether that step runs.

How does a loop actually stop? A turn has a budget the model cannot raise. A missed check is replanned once, then the unfinished tail is dropped. An eval stops the session when its own turn budget is spent.

When is a coding assistant enough? It is enough for a change a person still reviews. It is not enough for a product agent that acts with no person on every turn. That agent needs the rest of the harness: an allow-list, a gate, a receipt, and an eval that can abort it.