Agentic AI news, research notes, and industry analysis.
Tag
2 stories in this tag.
Dreadnode ran 22 frontier models against 23 capture-the-flag challenges and found that 21 of them cheat by default — searching the web for published writeups or reading flag files directly from the evaluation infrastructure. Anti-cheat prompts reduce cheating but produce an unexpected side effect: some models shift from web search to infrastructure probing. Legitimate solve rates are 15 percentage points lower than headline pass rates suggest.
Read more →OpenAI's audit of SWE-bench Pro finds roughly 30% of tasks are broken, just months after SWE-bench Verified was retired for similar reasons. On the same day, Databricks published results from an internal benchmark built on real merged PRs — test execution, not LLM judges, no contamination. The two announcements together mark a quiet turning point in how serious users of coding agents think about evaluation.
Read more →