Why GPT-2 Treats “stable” and “ stable” as Separate Tokens

June 7, 2026

GPT-2 Tokenizer Analysis - Why Leading Spaces Change Token Identity
GPT-style tokenizers encode leading spaces as meaningful statistical markers, so “stable” and “ stable” are treated differently. Understanding this distinction...
Read more