Agentic AI news, research notes, and industry analysis.
Category
2 stories in this category.
Specific Labs released Real-SWE, a benchmark that tests frontier AI coding agents on licensed private production codebases — tasks with real business stakes like billing systems and tax compliance. The best model (Fable 5.1) resolves 38.8% of tasks. The most expensive model doesn't win by much over models costing a third as much per rollout, and failure rates range from 61–84% depending on the model.
Read more →EEBench, a new benchmark for AI circuit design, uses atopile's declarative code interface to test whether models can produce functional circuits. Claude Opus 5 leads at 61.6%, with OpenAI's models trailing at 39–42%. The benchmark reveals a specific failure mode: designs that pass nominal simulation fail when real component tolerances are applied.
Read more →