Pretrained

Tokenizers now and in the future

This episode covers the history of tokenization, including early language modeling and modern BPE techniques. It also looks at the future of token-free bitestreams.

Listen