Open source · MIT · zero dependencies
bakeoff mines evaluation tasks from your own git history, runs any agent CLI in a sealed sandbox, and reports quality against cost with honest confidence intervals — so a cheaper configuration that is statistically tied shows up as tied.
config tasks passed pass rate (95% CI) mean $ $/pass ----------- ----- ------ ---------------------- -------- -------- flaky-cheap 3 3 100.0% [43.8-100.0] 0.0004 0.0004 budget 3 3 100.0% [43.8-100.0] 0.0015 0.0015 premium 3 3 100.0% [43.8-100.0] 0.0375 0.0375 Routing opportunities ===================== budget instead of premium: 96% cheaper, no measured quality loss (under-powered -- widen the suite)
The problem
You are choosing between agents and model-harness combinations that differ by up to 117× in cost. Here is why the public numbers no longer help.
Cursor measured it on SWE-bench Pro — 57% found merged pull requests on the public web, 9% mined git history bundled with the task. Seal git and restrict the network and the same agent drops from 87.1% to 73.0%.
Cursor — Reward hacking in coding benchmarks ↗Several models now clear 95%, and OpenAI publicly stopped evaluating against it. The official leaderboard has closed to product submissions to stay a research venue.
OpenAI — Why we no longer evaluate SWE-bench Verified ↗Against 53% on repositories outside the benchmark — models recall these specific repos rather than reasoning about them. Contamination is measurable, not hypothetical.
The SWE-Bench Illusion (arXiv 2506.12286) ↗Four maintainers reviewed 296 AI pull requests that passed the automated grader. Passing is not merging, and the gap is widening year on year.
METR — Many SWE-bench-passing PRs would not be merged ↗How it works
Every stage writes JSON Lines, so the pipeline splits across CI steps and every intermediate state stays inspectable.
Walks your history for commits that fix something and reconstructs each as a task: the repo at the parent commit, a problem statement synthesised from the message, and the tests the fix made pass. The human patch is held out.
Six detectors ask whether a task gives away its own answer. Leakage that sealing already removes is scored separately from leakage that it cannot.
Isolated checkout at the base commit. Remotes, tags, branches and reflog dropped, then gc --prune=now — the fix commit is absent, not hidden. Agents are shell commands, so anything with a CLI can be measured.
Wilson intervals, cost per passed task, and pass@k against pass^k — solved at least once versus solved every single time.
Re-runs your golden tasks against a stored baseline and gates CI on a significant regression — including a cost increase at unchanged quality.
Agents are configured as shell commands, so bakeoff is vendor-neutral by construction. No vendor has an incentive to publish a fair comparison against its competitors, and teams routinely run three at once.
{
"name": "claude-code",
"command": "claude -p {prompt} --output-format json",
"input_tokens_path": "usage.input_tokens",
"cost_per_1k_input": 0.015,
"timeout_sec": 900
}Install
Zero runtime dependencies, Python 3.9+ and git. It runs inside CI and inside air-gapped networks, where every dependency is a procurement conversation.
# clone and run — no install, no API key $ git clone https://github.com/phoenixking25/bakeoff && cd bakeoff $ python3 -m bakeoff demo # or put it on your PATH $ pip install . # then, on your own repository $ bakeoff mine ~/code/your-service -o tasks.jsonl --limit 30 $ bakeoff audit tasks.jsonl --filter-clean clean.jsonl $ bakeoff run clean.jsonl --agents agents.json --suite baseline --db bakeoff.db --report # after a model or harness upgrade $ bakeoff run clean.jsonl --agents agents.json --suite week-2 --db bakeoff.db $ bakeoff drift --db bakeoff.db --baseline baseline --current week-2 --fail-on-regression
Your code never leaves your machine. Everything runs locally — deliberately, because every comparable product ships your proprietary pull requests to a vendor cloud.
Honest limits
A measurement tool that oversells itself is worse than no measurement tool. So, plainly:
Small suites cannot detect small changes. Twelve tasks cannot resolve anything below a very large swing. bakeoff prints the minimum detectable effect so “no regression” is never read as “nothing changed.”
Mined tasks are bug fixes. That is a real slice of the work, not all of it. The bias largely cancels when comparing configurations against each other; it does not cancel when quoting an absolute score.
The sandbox seals git, not the network. Remotes, refs and reflog are removed and the fix commit is pruned. The agent still needs its own model API, so process-level egress is yours to control.
Mining heuristics favour Python and JavaScript with conventional test layouts. Other ecosystems work, but you may need to set the test command yourself.
Working on this
We work with a small number of teams to mine a suite from their repository, run the matrix across the agents they are actually choosing between, and hand back a routing recommendation with the cost delta quantified and the error bars attached.
tests 1,084 passing dependencies 0 license MIT python 3.9+ telemetry none