# bakeoff

> A command-line tool that benchmarks coding agents on evaluation tasks mined from a repository's own git history, rather than on public benchmarks. It runs any agent CLI in a sealed sandbox and reports pass rate against cost with Wilson confidence intervals, then detects regressions between runs.

bakeoff exists because public coding benchmarks no longer answer the question "which agent and model should we use on our codebase?". SWE-bench Verified is saturated above 95% and its leaderboard has closed to product submissions; measured contamination is high (Cursor found 63% of its successful SWE-bench Pro resolutions retrieved the fix rather than deriving it); and harness choice alone can swing the same model by 20 points.

Written in Python 3.9+ with zero runtime dependencies. MIT licensed. Runs entirely locally — repository contents are never transmitted anywhere.

## Commands

- `bakeoff mine <repo> -o tasks.jsonl` — reconstruct merged bug-fix commits as evaluation tasks. Holds out the human patch; synthesises a problem statement from the commit message with the remedy stripped.
- `bakeoff audit tasks.jsonl --filter-clean clean.jsonl` — score each task for solution leakage using six detectors; separates leakage that sandbox sealing removes from leakage that it cannot.
- `bakeoff run tasks.jsonl --agents agents.json --db bakeoff.db` — run each agent configuration against each task in an isolated checkout at the base commit, with remotes, tags, branches and reflog removed and unreachable objects pruned.
- `bakeoff report results.jsonl` — pass rate with Wilson 95% intervals, cost per passed task, and `pass@k` versus `pass^k`. Reports two configurations as tied when their intervals overlap.
- `bakeoff drift --db bakeoff.db --baseline v1 --current v2 --fail-on-regression` — two-proportion z-test on pass rate plus thresholds on cost and latency; exits non-zero on a significant regression, for use as a CI gate.
- `bakeoff demo` — runs the entire pipeline against a generated scratch repository. Requires no API key and no network.

## Agent configuration

Agents are plain shell commands, which makes the tool vendor-neutral. A configuration entry supplies a `command` template with `{prompt}`, `{prompt_file}`, `{repo}` and `{model}` placeholders, optional dot-paths into the agent's JSON output for token accounting, and per-1k token rates for cost.

## Docs

- [Repository and source](https://github.com/phoenixking25/bakeoff)
- [README with full usage](https://github.com/phoenixking25/bakeoff#readme)

## Known limits

- Suites below roughly 30 tasks are statistically underpowered; the tool reports its own minimum detectable effect so that "no regression detected" is not mistaken for "nothing changed".
- Mined tasks are bug fixes, which is a subset of engineering work. The bias largely cancels when comparing configurations to each other, but not when quoting an absolute score.
- The sandbox seals git, not process-level network access — the agent still needs to reach its own model API.
- Mining heuristics are tuned for Python and JavaScript repositories with conventional test layouts.