SWE-Bench gives agents a GitHub issue — usually with reproduction steps, expected behavior, and a clear acceptance test — and scores whether their patch makes the tests pass. Agents have gotten remarkably good at it. Claude Opus 4.8 sits above 88% on SWE-Bench Verified. That number is real; the model genuinely can read a codebase, locate relevant code, and produce a patch that passes the test suite on a majority of issues.
What that number doesn’t tell you is how the same model handles the work that arrives before anyone has turned a bug report into a clean issue. The vague feature request. The “users are saying it feels slow.” The new requirement described in half a sentence by a product manager who assumes you know the context.
Senior SWE-Bench, released by Snorkel AI with Princeton and UW-Madison, is an attempt to measure performance on that harder problem. The premise, stated directly on the benchmark site: “we treat agents like senior engineers, so why evaluate them like junior engineers?”
The design change is intentional and specific. Requirements are shortened — 31% fewer characters than SWE-Bench Pro’s median — and written to reflect how engineers actually communicate work to each other, not how they write bug tickets. The three task types are features (vague, under-specified, edge cases left open), bugs (user reports requiring runtime investigation before the problem is even clearly located), and performance (requiring profiling and analysis before any optimization attempt). Evaluation scores solutions on runtime correctness tests, as usual, but adds quality metrics based on observed codebase practices — naming conventions, idioms already present in the repository, abstraction levels that match the surrounding code.
The leaderboard as of release: Claude Opus 4.8 at 24.0%, Claude Sonnet 5 at 19.4%, GPT-5.5 at 16.0%.
The gap from 88% to 24% for the same model is the number worth sitting with. On reflection, it’s not surprising — but having it precisely measured is useful. SWE-Bench’s tight specifications do the hard disambiguation work ahead of time. The agent’s job, once the issue is well-formed, is mostly translation: convert a clearly-stated problem into a correct code change. Senior SWE-Bench pushes the disambiguation work onto the agent. It has to figure out what “make the file upload faster” actually requires, which edge cases are in scope, what the codebase’s preferred way to abstract things is, and whether a particular approach is consistent with existing patterns. That’s a different task.
The quality dimension is also new and matters. Most benchmark evaluation stops at “does it compile and do the tests pass.” Senior SWE-Bench tries to capture something closer to code review: does the output fit the codebase it landed in? A technically correct solution that adds an unnecessary abstraction layer, violates naming conventions already established in the file, or duplicates a utility that exists three directories over fails code review in practice. Tests don’t catch this. The benchmark is attempting to measure it.
The academic pedigree — Snorkel AI, Princeton, and UW-Madison — follows SWE-Bench’s pattern of combining industry and academic resources to produce an evaluation that’s hard to game quickly. The benchmark is open-source, which means teams can run their own agents against it with their own scaffolding and compare to the leaderboard. That’s important because the field has learned that SWE-Bench scores depend heavily on the harness: the tool-calling loop, the context strategy, the edit format. A model with a strong harness can score significantly higher than the same model with a weak one, which makes cross-team comparisons on closed benchmarks difficult to interpret.
The 24% top score has a practical implication: you should not deploy a coding agent without human review on under-specified tasks if you need the first-pass result to be usable without significant rework. That’s not a surprising finding, but the benchmark gives you a calibrated number to reason about. If your team’s review process can catch and correct failures quickly, even 24% task completion on ambiguous work might be economically useful — you get rough first drafts on a quarter of the tasks and your team handles the rest. If you need reliable unsupervised execution, you’re not there yet.
What would move the number upward: better retrieval of codebase conventions so agents understand what “good code looks like here” without being told, improved planning that decomposes vague requirements into concrete steps before making edits, and more iterative refinement with self-review passes. Some of that is model capability. Some of it is harness design. A benchmark that measures them together — the way Senior SWE-Bench does — is a better guide to actual progress than one that holds half the problem constant.