Running agent evals at scale: thousands of environments in parallel

Evals are an infrastructure problem in disguise

Whether you're grading a coding agent, running an RL loop, or sweeping prompts across a benchmark, the shape is the same: run the same task in thousands of identical, isolated environments, collect the results, tear it all down. The model work is the interesting part. The infrastructure is where it falls apart.

Three things make eval infra hard: parallelism, reproducibility, and teardown.

Reproducibility starts with a snapshot

Every run needs the same starting state — same repo, same dependencies, same fixtures. Rebuilding that per run is slow and, worse, non-deterministic: a dependency updates, a network call flakes, and now your results aren't comparable.

Instead, build the environment once and snapshot it. Every eval run forks from the identical snapshot, so run #1 and run #9,999 start from a byte-for-byte identical world.

Parallelism is where the speed lives

A thousand sequential runs is a coffee break. A thousand parallel runs is a few seconds. The bottleneck is how fast you can spin up isolated environments — which comes back to cold starts and concurrency limits. Sub-second starts and high concurrency turn an overnight eval into a coffee-refill eval.

const base = await am.create({ model: "claude-opus-4.8" })
await base.exec("git clone https://github.com/acme/suite && make setup")
const snap = await base.snapshot()

// 5,000 identical, isolated eval environments — in parallel
const runs = await am.fork(snap, { count: 5000 })
const results = await Promise.all(runs.map((r) => r.exec("make eval")))

Don't forget teardown

Five thousand environments you forgot to kill is five thousand environments you're paying for. Per-second billing plus automatic teardown means an eval costs what the eval actually used — not what you forgot to clean up.

The pattern, in one line

Snapshot once, fork thousands, run in parallel, tear down automatically. That's how you turn evals and RL from an infra slog into a primitive.

agentmachines gives you snapshots, forking, and sub-second isolated environments out of the box. Start for free.