SentencePiece

SentencePiece is an unsupervised text tokenizer developed by Google (created by Taku Kudo and John Richardson), primarily for neural network-based text generation and machine translation. It solves a key limitation of traditional tokenizers: they rely on language-specific pre-tokenization rules (whitespace splitting works for English but fails for Chinese, Japanese, Thai). SentencePiece treats input as raw Unicode characters and applies subword segmentation directly, enabling a single tokenizer to work for any language. It implements two algorithms: BPE (Byte-Pair Encoding) and Unigram Language Model (providing probability scores for sampling-based decoding and n-best segmentation). Features include lossless detokenization, subword regularization (sampling multiple segmentations during training for robustness), BPE dropout for data augmentation, byte-fallback encoding (handling any character via byte decomposition), configurable vocabulary size and character coverage. SentencePiece is used by T5 (32,000 vocab), ALBERT (30,000 vocab), mT5 (250,000 vocab for 101 languages), XLNet, and NLLB. Available via C++, Python, and Rust bindings. As of 2026, it has over 11,000 stars.