SWE-bench explained in one sentence: it is 2,294 GitHub issues whose fixes shipped with a test, and a model scores by making that test pass. The fastest way to understand what the number means is not to read a leaderboard — it is to run the benchmark's own construction filter over a repository you know. We did that here, on the repository this site is built from, and it produced zero task instances from 40 commits. This article covers how instances are built, which fields decide a score, and what the filter selects for.
Key takeaways
- SWE-bench instances survive a three-stage filter that keeps roughly 2.5% of the pull requests it sees: 2,294 instances from about 90,000 pull requests across 12 Python repositories.
- A pull request qualifies only if it both resolves an issue and modifies test files. No test, no instance — that criterion is doing most of the selection.
- Six of the twelve instance fields are unavailable in our repository, and all three that decide the score are among them. Everything describing a change is easy; everything establishing it was correct is not.
- SWE-bench Verified is a 500-instance human-checked subset, created because original instances could be underspecified or graded by tests that fail a valid fix.
- A score is evidence about a model plus a scaffold plus a task set, and only the first is something you can buy.
The short answer
SWE-bench measures whether a model can make a specific failing test pass, on a repository whose maintainers write tests alongside fixes. That is a real and hard capability. It is also narrower than "can this model do software engineering," and the gap between those two sentences is where most quoted scores get misused.
If you are choosing a tool rather than studying the benchmark, the best AI coding assistants comparison is the pillar this sits under, and best AI model for coding prices the current lineup on one measured turn.
What SWE-bench actually is
SWE-bench was introduced in SWE-bench: Can Language Models Resolve Real-World GitHub Issues? by Jimenez, Yang, Wettig, Yao, Pei, Press and Narasimhan, published at ICLR 2024. The abstract describes it as "an evaluation framework consisting of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories."
The task given to a model is deliberately unlike code generation. It receives a codebase at a specific commit and the text of an issue, and it must produce a patch. Nothing tells it which file to edit, and resolving an issue "frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously."
The scale of the challenge at publication is worth remembering when reading a modern score: the best-performing model in the paper, Claude 2, resolved 1.96% of instances. Published figures today are far higher. Some of that movement is model capability and some of it is scaffolding — the agent loop, the tools, the retrieval — and the benchmark does not separate the two for you.
The twelve repositories are all mature, well-tested Python projects, and their pull request counts are published in the paper:
| Repository | PRs scraped | Repository | PRs scraped |
|---|---|---|---|
| django | 16,914 | scikit-learn | 15,159 |
| matplotlib | 16,545 | sympy | 11,928 |
| astropy | 9,469 | pytest | 5,147 |
| sphinx | 4,931 | pylint | 3,848 |
| xarray | 3,416 | flask | 2,434 |
| requests | 2,344 | seaborn | 1,004 |
That selection is not incidental. The authors prioritised repositories with strong documentation and testing practices, because the construction pipeline cannot work without them.
How a task instance is built
Three stages, applied in order:
- Stage I — scrape. Collect pull requests from the 12 repositories, "producing about ∼90,000 PRs in total."
- Stage II — attribute filter. Keep pull requests that "(1) resolve a GitHub issue and (2) make changes to the test files of the repository, which indicates that the user likely contributed tests to check whether the issue has been resolved."
- Stage III — execution filter. Apply the pull request's test changes, run the suite before and after the code changes, and "filter out task instances without at least one test where its status changes from a fail to pass," along with anything producing installation or runtime errors.
Roughly 90,000 in, 2,294 out. A survival rate near 2.5%, and the second stage is where most of the loss happens.
That second criterion is the one to hold on to. A pull request that fixes a real bug, is reviewed, merged and shipped, but does not add a test, is not eligible. Not because it is bad work — because there is no way to grade it automatically.
Running the filter on a repository we know
Reading the criteria is one thing. Applying them is faster. The committed check implements the same three stages against this repository's git history, using commits in place of pull requests, because this repository has none.
npm run check:swebench
Stage I — scrape Changes in history 40 Stage II — attribute filter (both criteria required) Mentions a #number 1 Resolves a GitHub issue 0 Modifies the repository's test files 0 Both 0 Stage III — execution filter (at least one fail-to-pass test) Test command in package.json none Test files tracked by git 0 Can observe a test flip fail to pass no Task instances produced From this repository 0 of 40 (0.00%) From the paper's 12 repositories 2,294 of ~90,000 (2.55%)
Zero, and the run never reaches the interesting stage. This repository has no test files and no test script, so the execution filter has nothing to execute — the attribute filter has already emptied the set.
Required disclosure: the first version of this script got the answer wrong, and the way it was wrong is instructive. It matched issue references with a bare #\d+ regex, which found one commit — Add articles #45, #48, #49 — Road 1 batch three — where the numbers are roadmap entry IDs and no GitHub issue exists. On that reading the script reported a fetchable problem_statement, for a problem statement that cannot be fetched. The fix was to match GitHub's closing keywords (fixes #12), which are what actually associate a merged change with an issue:
The first two rows differ, and the gap is the reason the second one exists. "Add articles #45, #48, #49 — Road 1 batch three" mentions a #number that is a roadmap entry, not a GitHub issue.
A five-minute bug in a small script, on a repository whose entire history one person wrote. Scale that across 90,000 pull requests from twelve projects and you have an argument for the human annotation pass covered below.
The twelve fields, and which ones decide the score
A SWE-bench Verified instance carries twelve fields, documented on the dataset card. Splitting them by what they do is more useful than listing them, because they are not equally hard to produce:
FIELD SUPPLIED SCORING FROM instance_id yes — MuhammadKashif10__devventa-<n> repo yes — MuhammadKashif10/devventa base_commit yes — the change's parent patch yes — non-test half of the diff created_at yes — 2026-08-15T06:24:21+05:00 environment_setup_commit yes — any commit hash qualifies problem_statement no — issue title and body hints_text no — issue comments before the fix version no — install version for evaluation test_patch no yes test-file half of the diff FAIL_TO_PASS no yes tests the change makes pass PASS_TO_PASS no yes tests that must not regress 6 of 12 fields, and 0 of the 3 the harness reads to decide whether a submission resolved anything.
Every field describing the change is available. Every field establishing that the change was correct is missing. That split is the article in one table.
The three scoring fields work together. test_patch is the test-file half of the original pull request, applied to the repository before a submission is graded — so the model is evaluated against tests it never saw. FAIL_TO_PASS lists the tests the original fix made pass, and a submission must make them pass too. PASS_TO_PASS lists tests that passed before and after, so a submission cannot score by breaking the rest of the suite. An instance resolves only when both lists are satisfied.
This is a genuinely good grading design — it is objective, it is cheap to run, and it cannot be gamed by a plausible-looking diff. It also means the benchmark is structurally unable to contain a single instance from a project like this one.
SWE-bench Verified, and why it exists
SWE-bench Verified is described by the project as "500 instances from SWE-bench, created in collaboration with OpenAI," where "human annotators reviewed each instance to ensure the problem descriptions are clear, the test patches are correct, and the tasks are solvable given the available information."
Read that list of three checks against the pipeline above and the motivation is plain. Automated construction can produce an instance whose problem_statement does not contain enough information to identify the fix, or whose FAIL_TO_PASS tests assert implementation details specific to how the original author solved it — failing a correct alternative solution. Neither flaw is visible to the filter, because the filter only checks that a test flipped.
A caveat about the numbers usually quoted here. The annotation statistics that circulate — the share of samples flagged for underspecified problem statements, the share flagged for unfair tests — come from OpenAI's announcement post, which returned HTTP 403 to our fetch on 2026-08-17. We are not quoting figures we could not read at source. The three qualitative criteria above are from swebench.com and are quoted directly.
The practical consequence is one you can act on: a score on the original SWE-bench and a score on SWE-bench Verified are different measurements, and the same system scores higher on Verified because a chunk of unwinnable instances is gone. Comparing across the two is a category error, and a leaderboard screenshot that does not name the variant is not usable.
What a score is evidence of
Strip out everything the benchmark does not establish and this is what survives:
| Question | Answer | Basis |
|---|---|---|
| Can the system fix a described bug | Yes, at the measured rate | Fail-to-pass tests, objectively graded |
| Was it the model or the scaffold | Not separated | A submission is a whole system |
| Will it work on an untested repo | Not measured | Every instance ships a test patch |
| Does it hold outside Python | Not by this benchmark | 12 Python repositories |
The second row is the one that matters commercially. A published score belongs to a configuration — a model, an agent loop, a tool set, a retrieval strategy, a budget. You can license the model. The rest of that configuration is somebody's product, and swapping your own harness in changes the number. It is the same reason this site's Claude vs GPT for coding measurement found 2 of 32,005 input tokens were typed by a person: in an agent, the scaffold is most of the request.
The fourth row is narrowing rather than fixed — the project now publishes Multilingual and Multimodal variants alongside the original — but a bare "SWE-bench" number still means the Python set. We later counted that set row by row: of the 623 files the Verified gold patches change, 622 are .py, and 231 of the 500 instances come from Django alone. The full distribution is in what the evidence behind a Python model choice is made of.
What we are not claiming
This article does not report a leaderboard position for any model, and the omission is deliberate. Scores move between releases, they are reported against different variants, and this site has no way to verify a submission that it did not run.
Two limits on the measurement it does report. Commits are not pull requests — this repository has none, so the filter was applied to the closest available unit, and a project with a real pull request workflow would give the attribute filter more to chew on. And the test-file pattern is a heuristic: it matches the conventions of six ecosystems, generously, so that a zero cannot be blamed on looking in the wrong place. Both choices are visible in the script rather than described here.
Common mistakes reading a benchmark score
- Treating a score as a model property. It belongs to a model plus a scaffold plus a task set; two of those three are not for sale.
- Comparing across variants. SWE-bench and SWE-bench Verified are different measurements, and Verified reads higher for structural reasons.
- Comparing across years. A 2024 figure and a 2026 figure differ by scaffold as well as by model.
- Generalising to untested code. Every instance ships a test patch. That is the population the score describes.
- Assuming a hard benchmark is a representative one. SWE-bench is hard and narrow; those are independent properties.
Conclusion
Treat a SWE-bench number as a floor on a system's ability to fix a described bug in a well-tested Python project, name the variant whenever you quote it, and never read it as a forecast for your own repository until you have checked that your repository looks like the ones it is built from. The cheapest way to check is to run the filter: npm run check:swebench implements all three stages, and on the repository this article was written in it returned zero instances from forty commits — which told us more about our own testing practice than about any model.
Frequently asked questions
What is SWE-bench?
How are SWE-bench task instances created?
What is the difference between SWE-bench and SWE-bench Verified?
What do FAIL_TO_PASS and PASS_TO_PASS mean?
Does a high SWE-bench score mean a model will work on my codebase?
Muhammad Kashif
Founder and editor of Devventa, covering AI coding assistants, Next.js and the modern AI development stack.
