Forty-two physicists contributed 56 tasks from their own research to a benchmark called FrontierPhysics, published on arXiv last week. Each task covers a real open problem the contributor was working on, spanning nine domains from theoretical to experimental physics. The benchmark doesn’t just check whether an agent gets the right number — it asks agents to do the full research cycle: literature review, planning, running simulations or experiments, and writing a manuscript. Expert physicists grade the output using rubrics they wrote themselves.
The headline number is brutal: across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts. Claude Opus 5.5 tops the leaderboard at 27.4%. That’s not necessarily a condemnation of AI — it’s a rough picture of where frontier research agents actually sit on difficult expert tasks today.
The more interesting finding is why agents fail. The researchers did a manual analysis of failed trajectories and found that the most common failure mode wasn’t what you might expect. Physics knowledge gaps accounted for about 11% of failures. Missing physics intuition was behind 34%. The single largest category — 48% — was non-rigorous reasoning.
That deserves unpacking. When researchers talk about non-rigorous reasoning in this context, they mean things like: an agent proposing a calculation, working through it, arriving at an answer that’s dimensionally inconsistent, and moving on. Or proposing a measurement approach that couldn’t work for subtle physical reasons — not because the agent doesn’t know the relevant principle, but because it doesn’t stop to check whether its plan actually satisfies the constraints. These are errors of judgment, not gaps in a knowledge base.
This matters because a lot of work on improving AI for science has focused on knowledge: better retrieval, more scientific data in pretraining, domain-specific fine-tuning. The FrontierPhysics results suggest that for end-to-end research tasks, that’s not where the ceiling is. The ceiling is in whether an agent can maintain rigor across a multi-step process, where each step depends on correctly evaluating the last one.
The benchmark is legitimately hard to evaluate fairly. Expert rubrics are inherently subjective, physics domains vary enormously in how evaluable their outputs are, and a sample of 56 tasks means performance estimates carry wide error bars. The paper notes that experimental physics tasks are harder to verify than computational ones. And every benchmark of this kind has selection effects: the contributing physicists wrote tasks from their current work, which is both a strength (they’re real problems) and a limitation (the domains may skew toward what those 42 people happen to do).
Still, the structure is worth taking seriously. These aren’t constructed exam questions from a textbook. They’re the kind of problems that might appear in a Nature Physics paper — because they did, or they will. The tasks aren’t testing whether an agent knows things physicists know. They’re testing whether an agent can do what physicists do, which turns out to involve a different set of capabilities entirely.
Whether 27.4% feels high or low depends on your priors. For anyone who thought frontier agents were ready to run independent research programs: it’s low. For anyone who expected these models to score near zero on tasks where the researchers themselves might take weeks: there’s something real here. The gap isn’t closed by making agents smarter in the textbook sense. It’s closed by making them more careful — and that’s a considerably harder target.
