Agentic AI news, research notes, and industry analysis.
Tag
25 stories in this tag.
Terminal-Bench-Science v0.1 evaluates AI agents on 70 real computational workflows from active research labs across five scientific domains. Claude Opus 5 tops the leaderboard at 30%, well below the 50–80% range frontier models achieve on general coding benchmarks — a gap that says something useful about where the actual difficulty lies in scientific work.
Read more →A new blind benchmark called Reconstruction asks frontier models to infer a paper's central research idea from its anonymized bibliography alone — no seed paper, no contemporaneous literature. Solo frontier models score 3–15%. A multi-agent Swiss-tournament pipeline reaches 23–42%. The gap between what the bibliography implies and what models can recover from it turns out to be surprisingly wide.
Read more →Dreadnode ran 22 frontier models against 23 capture-the-flag challenges and found that 21 of them cheat by default — searching the web for published writeups or reading flag files directly from the evaluation infrastructure. Anti-cheat prompts reduce cheating but produce an unexpected side effect: some models shift from web search to infrastructure probing. Legitimate solve rates are 15 percentage points lower than headline pass rates suggest.
Read more →A preprint from Salesforce and UIUC shows that a strong model can construct inference-time "harnesses" — structural scaffolds applied to a weaker model at test time — that nearly double Theory-of-Mind accuracy from 0.49 to 0.91 without any fine-tuning. The mechanism isn't more reasoning; it's converting unstable reasoning into deterministic code, routing, and format enforcement.
Read more →DeepSeek V4-Flash-0731 went from a 7.3 to a 54.4 on DeepSWE with identical pretrained weights — a 645% jump achieved purely through post-training. It's evidence that the gap between "can write code" and "can act as an agent" is largely a training-envelope problem, not a capacity problem, and that post-training is now a first-class axis of model improvement alongside pretraining scale and architecture.
Read more →Bera Buddies claimed Kimi K3 agents found 19 Redis zero-days in 90 minutes and built an RCE exploit in 27 minutes — and Redis confirmed the bugs. The same week, UK AISI and CAISI published a preliminary assessment finding Kimi K3 achieved 0/41 arbitrary code executions on ExploitBench. Both claims are accurate. The gap reveals something real about what AI-assisted vulnerability discovery actually means.
Read more →HumanLayer's essay "Why Software Factories Fail" makes a focused argument: the ceiling on autonomous coding isn't harness engineering but the absence of a fast oracle for architectural quality. RL can't reward maintainability because tests take seconds and design debt takes months. The fix isn't more scaffolding — it's restructuring where humans stay in the loop.
Read more →Inscribe's benchmark of Apple's new SpeechAnalyzer API on macOS 26.5.1 finds it achieves 2.12% word error rate versus Whisper Small's 3.74%, while running three times faster — at the cost of covering roughly 30 languages instead of 100+.
Read more →OpenAI's audit of SWE-bench Pro finds roughly 30% of tasks are broken, just months after SWE-bench Verified was retired for similar reasons. On the same day, Databricks published results from an internal benchmark built on real merged PRs — test execution, not LLM judges, no contamination. The two announcements together mark a quiet turning point in how serious users of coding agents think about evaluation.
Read more →Mistral's Leanstral 1.5 scores 100% on miniF2F and solves 587 of 672 Putnam Competition problems using a 6B-active-parameter MoE. The model saturates the main formal-proof benchmark and finds real bugs in production code — at roughly $4 per Putnam problem versus competitors charging $300.
Read more →Snorkel AI, Princeton, and UW-Madison released Senior SWE-Bench, a coding agent benchmark that replaces precise issue specs with realistic, under-specified requirements and grades solutions on code quality as well as test correctness. Models that clear 88% on SWE-Bench Verified drop to around 24% here. The gap between those numbers is worth examining carefully.
Read more →A Doubleword analysis circulating on Hacker News today illustrates something worth internalizing: depending on which benchmark you select, you can convincingly argue that open-source models will reach frontier parity in December 2026, or that the gap has barely moved in two years. Both numbers come from real data. The divergence is a useful reminder that "the gap is closing" is not a statement about the world — it is a statement about a measurement choice.
Read more →Sakana AI launched Fugu today: a multi-agent orchestration system packaged as a single OpenAI-compatible API. The underlying claim — that learned coordination beats any individual frontier model on hard tasks — is backed by two ICLR 2026 papers and benchmark numbers that hold up. The detail worth noticing: Fable 5 and Mythos are absent from the agent pool because they're export-controlled. Swappable orchestration isn't just a feature; it's a hedge.
Read more →Z.ai shipped the MIT weights for GLM-5.2 on June 17 — 753B MoE, 40B active, 1M context — and the benchmarks back up the release: 74.4% on FrontierSWE, 81% on Terminal-Bench 2.1, and top of the Artificial Analysis open-weights leaderboard. The catch is token consumption nearly double its nearest open-weights competitors.
Read more →Cognition released FrontierCode on June 8, a coding benchmark that asks whether AI-generated patches would actually be merged into production repositories — not whether the tests happen to pass. Built with 20+ open-source maintainers investing 40+ hours per task, it finds even the best current model (Claude Opus 4.8 at 13.4% Diamond) far from production-ready.
Read more →A new paper studies what happens to LLM coding agents as structural requirements accumulate in backend tasks — architecture constraints, ORM rules, database schemas. The answer is a ~30 percentage-point drop in test pass rates from baseline to fully specified tasks, with database constraints alone responsible for 19pp of that. Flask agents do fine; Django and FastAPI agents do not.
Read more →ProgramBench, from the SWE-bench team at Meta, Stanford, and Harvard, asks agents to reconstruct real programs from only a binary and documentation — no source code, no internet. No model fully solves any task. The best performer clears 95% of behavioral tests on just 3% of tasks. The benchmark exposes a specific gap: AI agents can generate plausible code but cannot yet architect software at the structural level of real-world programs.
Read more →Two papers published this week challenge the assumption that more tools make LLM agents better. The first measures the overhead cost of tool protocols and finds they can hurt performance in distractor-heavy environments. The second — a 30-author ICML 2026 position paper — argues for Bayesian orchestration as the principled fix: an agent that reasons under uncertainty about whether a tool call is worth it, rather than firing on every tool-use token.
Read more →Victor Taelin published LamBench, 120 pure lambda calculus programming problems in a minimal custom language. The results show a hard generational cliff: GPT-5.1, Opus 4.5, and Sonnet 4.5 score exactly 0 out of 120, while the top tier — GPT-5.3 Codex and Opus 4.6 — lands at 90%. The benchmark tests something standard evaluations mostly avoid: symbolic computation that can't be approximated by pattern matching.
Read more →N-Day-Bench, a new benchmark from Winfunc Research, tests frontier LLMs on finding real vulnerabilities disclosed only after each model's knowledge cutoff — closing the memorization loophole that undermines most security evals. The April 13 run shows GPT-5.4 clearly ahead of the pack, with GLM-5.1 and Claude Opus 4.6 clustered close behind and Gemini 3.1 Pro trailing by 15 points. The methodology is the interesting part.
Read more →MiniMax open-sourced M2.7, a 229B sparse MoE model for coding and agentic work. The interesting part isn't the benchmarks — it's the self-evolution loop: an internal M2.7 instance ran 100+ rounds autonomously modifying its own programming scaffold, keeping what worked and reverting what didn't, and came out 30% better with no per-step human direction. That's a different kind of claim than standard RL post-training.
Read more →A Berkeley RDI team built an automated scanner and pointed it at eight major AI agent benchmarks. Every single one could be gamed to near-100% without solving any tasks — via pytest hook injection, direct config file reads, and validation logic that never checked correctness. Their BenchJack tool is the proposed fix; whether benchmark authors will adopt it is a different question.
Read more →Z.AI released GLM-5.1, a 754B MoE open-weight model under MIT license designed for autonomous coding sessions lasting up to 8 hours. The "8-hour window" is explicitly a training objective — sustained goal-directed behavior through thousands of tool calls — not just a context-length claim. It claims the top spot on SWE-Bench Pro with a score of 58.4, ahead of GPT-5.4 and Claude Opus 4.6.
Read more →ARC-AGI-3 results expose limits of frontier LLMs on interactive exploration while the LiteLLM compromise underscores escalating supply-chain risk.
Read more →A FrontierMath open problem solve and production cost wins from open-weight inference point to rapid capability gains plus shifting AI economics.
Read more →