When You Stop Holding the Agent's Hand
Snorkel AI, Princeton, and UW-Madison released Senior SWE-Bench, a coding agent benchmark that replaces precise issue specs with realistic, under-specified requirements and grades solutions on code quality as well as test correctness. Models that clear 88% on SWE-Bench Verified drop to around 24% here. The gap between those numbers is worth examining carefully.
Read more →