SWE-bench Verified Explained: What Is in the 500

SWE-bench Verified is a 500-instance subset of SWE-bench, re-checked by human annotators so every problem is solvable and every test is fair. That much is on the tin. What nobody prints is what the 500 actually contain, so we downloaded them: 46.2% come from one repository, 38.8% are annotated by those same humans as fixes taking under fifteen minutes, and the median instance is 22 lines in a single file with one test to make pass. Everything below is recomputed from the dataset every time we re-run our internal check script.

Key takeaways

• Verified is a strict subset: all 500 instance IDs appear in the full 2,294-instance test split, a selection rate of 21.8%.
• django/django supplies 231 of the 500. Three repositories cover more than two thirds of the benchmark.
• The human annotation shipped as a column. 194 instances are labelled <15 min fix; exactly three are labelled >4 hours.
• 345 of 500 instances have exactly one fail-to-pass test, and the median gold patch is 22 lines in one file.
• The newest issue in the set was filed on 2023-08-07, which is before every model that quotes a score on it.

The short answer

A SWE-bench Verified score is evidence that a system can make one failing test pass in a mature Python library, usually by changing about twenty lines of one file. That is a real capability and a hard one. It is not the same claim as "resolves real-world GitHub issues," and the distance between them is visible in the dataset rather than arguable.

This article is the companion to what a SWE-bench score measures in general, which ran the benchmark's construction filter over this repository. That one explains how an instance is built. This one opens the 500 that survived. If you are choosing a tool rather than reading leaderboards, the AI coding assistants comparison is the pillar this sits under.

What SWE-bench Verified actually is

Three public datasets share the name on the official SWE-bench benchmark, and only one of them is what a vendor means by "SWE-bench Verified":

The full dataset also ships a 19,008-instance train split and a 225-instance dev split, which is worth knowing before you read any claim about contamination.

Verified is a subset, not a rebuild. We checked rather than assumed by downloading the SWE-bench_Verified dataset on Hugging Face: every one of the 500 instance IDs is present in the full test split, so the annotation pass selected from what the automated filter had already produced. Lite is a third, separately drawn subset — not a stage of Verified, and not comparable to it.

The annotation pass ships as a column

Verified's schema has thirteen fields where the original has twelve, and the extra one is difficulty. That single column is the human annotation made readable, and it is the part of this benchmark almost nobody reads.

Nearly two in five instances in the benchmark every vendor quotes are annotated, by the humans who reviewed them, as quarter-hour fixes. Ninety-one percent are an hour or less. The entire long tail is forty-five instances, and the three hardest in the set are one each from xarray, sphinx and sympy.

This is the number the general SWE-bench explainer could not get — it noted that the annotation statistics circulating in commentary came from a page that returned HTTP 403 to our fetch on 2026-08-17, and declined to quote figures it could not read at source. The figures were in the dataset the whole time, as a column. Nobody had to be quoted.

Nearly half of it is one project

Twelve repositories, all Python. Here is the split, with the full dataset beside it so you can see what the annotators kept:

The Django concentration is not inherited, it is amplified. Django is 37.1% of the full test split and 46.2% of Verified, because it was selected at 27.2% against a split-wide rate of 21.8%. Seaborn and flask went the other way at 9.1% each and contribute three instances between them.

Nothing about that is misconduct — a repository with consistent tests and clear issues produces instances that survive human review, which is the filter working. But it means a Verified score is evidence about Django-shaped maintenance work, weighted roughly half, and about eleven other Python libraries the rest of the time. There is no JavaScript instance to weight at all.

What passing one instance actually requires

Two fields decide whether a submission scores: FAIL_TO_PASS is the tests your patch has to make pass, and PASS_TO_PASS is the tests it must not break.

345 of the 500 instances — 69% — have exactly one fail-to-pass test. The median is one. So for most of this benchmark, the target is a single assertion, and the other 60,142 tests are the evaluation confirming you did not break the library while hitting it.

Both distributions have long tails: one instance carries 438 fail-to-pass tests, several hundred times the median. And eleven instances carry no regression suite at all, which means for those eleven nothing is checking collateral damage.

The gold patches are small

Each instance ships the change its maintainers actually merged. Measured across all 500, and against the full dataset for contrast:

The annotation pass left a set that is measurably smaller as well as fairer. A median of 22 lines against 34, and five in six patches confined to a single file. The paper's framing of SWE-bench is that resolving an issue "frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously" — which is true of the full dataset and true of 71 instances here, and not true of the other 429.

The median Verified instance, stated plainly: read a 1,185-character issue, change about twenty lines of one Python file, make one test pass without breaking fifty others.

Everything in it predates every model quoting it

Every instance carries the date its issue was filed. The newest in Verified is 2023-08-07; the newest in the full test split is 2023-08-15, so even the subset's ceiling is eight days lower.

The oldest is from 2013-01-25, a decade before the newest. Every model that has ever published a Verified score was trained after every issue in it was filed, and all twelve repositories are public, as are the merged fixes. This does not establish contamination — we measured dates, not training sets, and nothing here can see inside a model. It does mean that "held out" is not a property this dataset has, and that a score improving over time has at least two available explanations.

335 of the 500 instances also ship non-empty hints_text: maintainer comments from the issue thread, before the fix landed. Whether an evaluation feeds those to the model is a harness choice, and it is not visible in a leaderboard number.

How to read a Verified score

• Ask which variant. Verified, Lite and the full 2,294 are three numbers. Only one of them is usually stated.
• Read it as Python library maintenance. Half Django, all Python, twelve projects. If your work is a React monorepo, the transfer is an assumption, not a finding.
• Halve your mental model of the difficulty. Ninety-one percent of the instances are labelled an hour or less by the people who checked them.
• Ask what the harness fed the model. Hints, retries, test visibility and scaffolding all move the number and none of them are in it.
• Treat a few points as noise. One point is five instances. Two systems separated by three points differ on fifteen problems, nearly half of which are quarter-hour Django fixes.

What we did not measure

Any model's score. Nothing here runs a model, grades a patch or ranks anything. This is a census of a dataset, and a guard in the fixture asserts that no output-quality claim is licensed by it.

Contamination. We measured when issues were filed, not what anything was trained on. The dates are a fact; the inference is not one we can make from here.

Whether the annotators were right. The difficulty column is what humans recorded. We report the distribution, not its accuracy, and we did not re-annotate a single instance.

The other SWE-bench family members. SWE-bench Multimodal, Multilingual and the live variants are not in scope; we read the three splits named above and nothing else.

How harnesses actually use the fields. hints_text is present in 335 instances. Whether a given evaluation supplies it is a property of that evaluation, and this site's own attempt at a bake-off could not get three vendors' agents to complete one shared task at all.

Common mistakes reading a Verified score

• Comparing a Verified number to a SWE-bench number. Different denominators and, as measured above, different difficulty. It is a category error, not a rounding issue.
• Reading 500 as a large sample. It is 500 instances from 12 projects, half of them one project, and one score point is five problems.
• Assuming "Verified" means "harder." It means human-checked. The surviving instances are smaller than the ones left behind.
• Treating it as a general software-engineering measure. No JavaScript, no Go, no Rust, no service code, no greenfield work, nothing after August 2023.
• Quoting the annotation statistics from commentary. They are a column in the dataset. Download it and count — that is what this article is.

Conclusion

Use a SWE-bench Verified score for what it supports: whether a system can land a small, well-specified fix in a mature Python library with a test to prove it. Ask which variant produced the number, then discount for the fact that nine in ten of the underlying problems were annotated as an hour's work or less. If you need the general version of this argument, read what a SWE-bench score measures first. Our check script downloads both datasets, recomputes every figure on this page, and fails loudly if the benchmark is ever revised underneath them.

Frequently asked questions

Muhammad Kashif

Founder and editor of Devventa, covering AI coding assistants, Next.js and the modern AI development stack.