Generalization · AI Beat

15 Sep 2026 · AI Beat Desk

Sixteen Tokens Is Enough

Amazon Science published a paper asking why ML research agents don't overfit when repeatedly optimizing against a fixed validation set. The answer: successful strategies can be compressed to 16–32 tokens through an information bottleneck. If a fresh agent with no memory can reproduce the original performance from that summary, the strategy is real. If compression destroys the gains, it was memorization.

28 Apr 2026 · AI Beat Desk

The Model That Stopped at 1930

Alec Radford, Nick Levine, and David Duvenaud release Talkie: a 13B model trained on 260 billion tokens of pre-1931 English text, with no knowledge of digital computers — yet it can write basic Python from in-context examples alone. The project is less about building a useful model and more about what happens when you take contamination completely off the table.