A Hundred and Eighty Million Lines of Proof

Anthropic and Theorem published a large-scale formal verification effort where Claude wrote ~180 million lines of Rocq proofs, proving no-crash properties for 239 OSS-Fuzz entry points at compiled RISC-V binary level. The work found hundreds of bugs—including ones fuzzing had missed for years—and forces an honest look at what "formally verified" can and can't mean.

Read more →

miniF2F Hits the Ceiling

Mistral's Leanstral 1.5 scores 100% on miniF2F and solves 587 of 672 Putnam Competition problems using a 6B-active-parameter MoE. The model saturates the main formal-proof benchmark and finds real bugs in production code — at roughly $4 per Putnam problem versus competitors charging $300.

Read more →

LLMs Know the Raft Paper. They Don't Know Etcd.

SysMoBench, a new benchmark from the Specula team, tests whether LLMs can produce TLA+ formal specifications that accurately model the behavior of real distributed system implementations. They score near-perfect on syntax and only ~46% on conformance and ~41% on invariant checking — because they model the algorithm as described in papers, not as implemented in code.

Read more →