What SWE-bench Verified Actually Measures

SWE-bench Verified, the flagship coding-agent benchmark, measures whether an AI agent can resolve real GitHub issues by generating code patches that pass unit tests. It uses a human-validated subset of 500 test cases. Top models scored about 78–79% on it in early 2026. The benchmark is narrow: 87% of its tasks are bug fixes from just five Python repositories. At least 59% of audited problems have test cases that reject functionally correct submissions. OpenAI officially retired SWE-bench Verified as a frontier evaluation.

Harder variants expose the gap. SWE-bench Pro, designed across 41 repositories in multiple languages to resist contamination through GPL-licensed and private code, sees top models score only about 23%, roughly 57 points below Verified. FeatureBench targets building features rather than fixing bugs; one frontier model scored 11.0% there versus 74.4% on SWE-bench. Feature development is about 80% of real engineering work. Older benchmarks such as HumanEval are saturated above 95%, and semantically transformed HumanEval problems produce drops of 19.6–47.7%, exposing memorization.

increasingly reflect how much the model was exposed to the benchmark at training time
— OpenAI, on retiring SWE-bench Verified

The Harness Matters More Than the Model

The system around the model matters as much as the model. The same model can score 69% or 81% depending on the agent harness wrapped around it, yet benchmarks report a single number. On SWE-bench Pro, swapping the agent scaffold changed scores by about 22%, while swapping between the top two frontier models changed them by about 1%. The harness controls how the model plans, edits files, runs tests, and iterates on failures. A benchmark result therefore describes the whole system, not just the model.

Field Evidence: Slower While Feeling Faster

Field evidence diverges sharply from benchmark results. In a controlled study by METR, AI tools made experienced developers 19% slower on real tasks, while the developers self-reported feeling about 20% faster. Faros AI found that high-AI teams merged 98% more pull requests. Review time increased 91%, pull requests grew 154% larger, and bugs per developer increased 9%. The result was no net delivery performance improvement. Veracode found AI-generated code carried 2.74 times more vulnerabilities than human-written code. DORA's 2024 report estimated that every 25% rise in AI adoption cut delivery stability by 7.2%. Its 2025 report calls AI "an amplifier" that magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones. The pattern holds across studies: gains in generation volume are offset by costs in review, testing, and rework.

Real-world coding is harder than benchmark tasks. It involves changes across codebases with thousands of interdependent files, ambiguous requirements, custom build toolchains, security concerns, and long multi-step evolution. A patch that passes unit tests on a benchmark issue is not the same as a feature shipped through production with review, testing, and deployment. The same tradeoff appears in unit tests with AI, where speed gains come with reliability costs.

What to Measure Instead

Teams should measure delivery, not model scores. Track cycle time from commit to production, change failure rate, and defect escape rate at the team level. Avoid lines of code and individual usage metrics; they create perverse incentives. A developer who generates more code but spends hours reviewing and fixing it is not faster. Invest in review pipelines, verification infrastructure, and observability. System-level improvements yield more sustainable throughput than buying more model capacity. The tools can help, but the infrastructure around them determines whether they help your team or just add review burden. When a tool ships with a benchmark score, ask what harness produced it, what tasks it contains, and whether those tasks resemble your codebase. If the answer is a vendor's own scaffold and a five-repo bug-fix set, treat the number as a starting point, not a promise. The same discipline applies to code generation in legacy codebases, where benchmark tasks rarely resemble production constraints.