Tokenizing at GB/s
Marcel Røed released GigaToken, a Rust tokenizer using SIMD and cache-optimized byte-pair encoding that runs 500–1,000× faster than HuggingFace's tokenizers and up to 681× faster than tiktoken — reaching 24 GB/s on a single CPU.
Read more →