Hugging Face Tokenizers

A Rust-based tokenization library from Hugging Face, providing the subword algorithms used by modern LLMs. The core is written in Rust (the tokenizer crate) with Python bindings via PyO3. Version 0.21.x ships four algorithm implementations: BytePairEncoding (used to reproduce GPT-2 tiktoken encoding with 50,257 tokens), WordPiece (BERT/DistilBERT compatible with the bert-base-uncased 30,522-token vocab), Unigram (T5 SentencePiece-compatible with 32,000 tokens), and the TemplateProcessing post-processor. Concrete API: from tokenizers import Tokenizer; tokenizer = Tokenizer.from_file("tokenizer.json"); encoding = tokenizer.encode("Hello world"); returns Encoding object with .ids, .tokens, .offsets, .attention_mask. Training: from tokenizers import models, trainers; bpe = models.BPE(unl_token="<|endoftext|">"); trainer = trainers.BpeTrainer(vocab_size=50257, special_tokens=["<|endoftext|>"]). The library uses a pipeline architecture: Normalizer (Lowercase, NFD, StripAccents, BertNormalizer) feeds PreTokenizer (ByteLevel for GPT-2, WhitespaceSplit, BertPreTokenizer, Metaspace, Split, Punctuation) feeds Model (BPE, WordPiece, Unigram) feeds PostProcessor (TemplateProcessing for adding [CLS]/[SEP]). The Rust core achieves approximately 50,000 sentences/second on a single core. As of 2026, version 0.21.x has over 9,400 stars.