Agentic AI news, research notes, and industry analysis.
Tag
10 stories in this tag.
A new paper studies what happens to LLM coding agents as structural requirements accumulate in backend tasks — architecture constraints, ORM rules, database schemas. The answer is a ~30 percentage-point drop in test pass rates from baseline to fully specified tasks, with database constraints alone responsible for 19pp of that. Flask agents do fine; Django and FastAPI agents do not.
Read more →ProgramBench, from the SWE-bench team at Meta, Stanford, and Harvard, asks agents to reconstruct real programs from only a binary and documentation — no source code, no internet. No model fully solves any task. The best performer clears 95% of behavioral tests on just 3% of tasks. The benchmark exposes a specific gap: AI agents can generate plausible code but cannot yet architect software at the structural level of real-world programs.
Read more →Two papers published this week challenge the assumption that more tools make LLM agents better. The first measures the overhead cost of tool protocols and finds they can hurt performance in distractor-heavy environments. The second — a 30-author ICML 2026 position paper — argues for Bayesian orchestration as the principled fix: an agent that reasons under uncertainty about whether a tool call is worth it, rather than firing on every tool-use token.
Read more →Victor Taelin published LamBench, 120 pure lambda calculus programming problems in a minimal custom language. The results show a hard generational cliff: GPT-5.1, Opus 4.5, and Sonnet 4.5 score exactly 0 out of 120, while the top tier — GPT-5.3 Codex and Opus 4.6 — lands at 90%. The benchmark tests something standard evaluations mostly avoid: symbolic computation that can't be approximated by pattern matching.
Read more →N-Day-Bench, a new benchmark from Winfunc Research, tests frontier LLMs on finding real vulnerabilities disclosed only after each model's knowledge cutoff — closing the memorization loophole that undermines most security evals. The April 13 run shows GPT-5.4 clearly ahead of the pack, with GLM-5.1 and Claude Opus 4.6 clustered close behind and Gemini 3.1 Pro trailing by 15 points. The methodology is the interesting part.
Read more →MiniMax open-sourced M2.7, a 229B sparse MoE model for coding and agentic work. The interesting part isn't the benchmarks — it's the self-evolution loop: an internal M2.7 instance ran 100+ rounds autonomously modifying its own programming scaffold, keeping what worked and reverting what didn't, and came out 30% better with no per-step human direction. That's a different kind of claim than standard RL post-training.
Read more →A Berkeley RDI team built an automated scanner and pointed it at eight major AI agent benchmarks. Every single one could be gamed to near-100% without solving any tasks — via pytest hook injection, direct config file reads, and validation logic that never checked correctness. Their BenchJack tool is the proposed fix; whether benchmark authors will adopt it is a different question.
Read more →Z.AI released GLM-5.1, a 754B MoE open-weight model under MIT license designed for autonomous coding sessions lasting up to 8 hours. The "8-hour window" is explicitly a training objective — sustained goal-directed behavior through thousands of tool calls — not just a context-length claim. It claims the top spot on SWE-Bench Pro with a score of 58.4, ahead of GPT-5.4 and Claude Opus 4.6.
Read more →ARC-AGI-3 results expose limits of frontier LLMs on interactive exploration while the LiteLLM compromise underscores escalating supply-chain risk.
Read more →A FrontierMath open problem solve and production cost wins from open-weight inference point to rapid capability gains plus shifting AI economics.
Read more →