#LLMCost

The Non-English Token Tax — Korean Costs 1.4x, Measured

I tokenized 285 of my posts across ko/ja/en/zh with three tokenizers. Korean ran 1.38x English tokens, Japanese 1.34x. The non-English token tax, measured.