Tokenizers now and in the future
This episode covers the history of tokenization, including early language modeling and modern BPE techniques. It also looks at the future of token-free bitestreams.
This episode covers the history of tokenization, including early language modeling and modern BPE techniques. It also looks at the future of token-free bitestreams.