SWE-bench Verified
500 human-validated GitHub issues the model must resolve in the real repository so the hidden tests pass. Epoch AI runs every model in one standardized scaffold.
Writing, fixing and shipping real code, judged by tests or users. One of 7 comparable tests in coding, a category the Index averages.
Read with care. One shared scaffold for every model; labs' own numbers with bespoke scaffolds can run 10 points higher and are not used here.
Publisher: Epoch AI (internal runs)What a model is asked to do
Given a real GitHub issue and its repository, write a patch that makes the project's hidden tests pass.
For exampleIssue: the CSV exporter drops the header row when the first column is empty. Find the cause in the codebase and fix it without breaking existing exports.
Why it matters. This is the daily work of a software engineer: understanding an unfamiliar codebase and fixing the right thing.
The example is original and illustrative, not an item from the dataset.
- Models scored here
- 26
- Who produced the numbers
- Run by Epoch AI
- Items graded
- n = 500 (0.2% each)
- Best published result
- 83.5%
- License
- CC BY 4.0 (via Epoch AI)
- Direction
- Higher is better
- Scale
- Published as a share of items solved. For the Index, each score is a share of the best published result (83.5%), so the frontier reads 100. Guessing earns nothing on this test, so no chance correction applies.
- Provenance
- Epoch AI ran the evaluation itself, under settings it documents per model, and publishes the log of every answer.
Snapshot September 8, 2026
Ranking on this test
26 models on SWE-bench Verified
Best published run per model, and how close each one comes to the best published result on this test (the frontier reads 100). Every number links to the publisher that produced it. Internal runs are marked and carry their run details on the model page. One item on this test is 0.2 points, so read gaps smaller than that as noise.
The best models, ranked here, working inside Theo.
28 of the models on this page run inside Theo today. Theo picks the right one for each step and always shows which engine answered.