Tokenization · AI Beat

11 Oct 2026 · AI Beat Desk

Byteification: Retrofitting Tokenized Models to Read Raw Bytes

A Nature paper from Ai2, LMU Munich, Cambridge, and UW describes how to convert a trained subword model into a byte-level one using less than 1% of a typical pretraining budget. The resulting Bolmo, Bwen, and Blama families retain most of the original capability while gaining the structural properties that tokenization normally destroys.

23 Jul 2026 · AI Beat Desk

Tokenizing at GB/s

Marcel Røed released GigaToken, a Rust tokenizer using SIMD and cache-optimized byte-pair encoding that runs 500–1,000× faster than HuggingFace's tokenizers and up to 681× faster than tiktoken — reaching 24 GB/s on a single CPU.