PU-Tok — An Indic-First Multilingual Tokenizer

The problem

Standard BPE tokenizers are trained on Latin-heavy corpora and it shows: Hindi, Tamil, or Bengali text routinely needs 2-4x more tokens per character than English for the same content. That's slower inference and worse context utilization for exactly the languages we care most about.

Approach

• Indic Priority Fraction of 0.55 — vocabulary budget explicitly weighted toward Indic scripts rather than left to fall out of frequency statistics.
• Rust acceleration layer — Viterbi segmentation, trie-based matching, and parallel batch encoding via a sharded LRU cache.
• 200,000 vocabulary size, confirmed as the right target via fertility analysis across 32 languages.

Honest development notes

Not every idea worked. Morphology suffix protection caused a Hindi fertility regression before being fixed back to neutral. Iterative reallocation turned out to be a no-op on small corpora because the vocabulary was already saturated. Both are documented, not smoothed over.