Byteification: Retrofitting Tokenized Models to Read Raw Bytes
A Nature paper from Ai2, LMU Munich, Cambridge, and UW describes how to convert a trained subword model into a byte-level one using less than 1% of a typical pretraining budget. The resulting Bolmo, Bwen, and Blama families retain most of the original capability while gaining the structural properties that tokenization normally destroys.
Read more →