# SpaceByte

This is the implementation of SpaceByte used in [SpaceByte: Towards Deleting Tokenization from Large Language Modeling](https://arxiv.org/abs/2404.14408).
SpaceByte is a tokenization-free large language model (LLM) that uses multiscale modeling at the byte and word levels to model language with the same performance as standard LLM architectures that instead use tokenization.
See the `reproduce` directory for instructions for reproducing the results in our paper.

![SpaceByte architecture schematic](spacebyte_figure.svg)
