2. Tokenization and why it bites you

Text is split into tokens by a subword tokenizer (BPE, or byte-level BPE / SentencePiece). Common English words are one token; rare words, code identifiers, emoji, and non-Latin scripts split into many.

Practical consequences: - Cost and context limits are measured in tokens, not characters. A rough English heuristic is 1 token ≈ 4 characters ≈ 0.75 words. - Non-English text can cost 2-4x more tokens for the same meaning. - Character-level tasks ("reverse this string", "count the r's") are hard because the model never sees characters.

python
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
print(len(enc.encode("Hello world")))          # 2
print(len(enc.encode("नमस्ते दुनिया")))            # many more

Gotcha: never truncate a prompt by characters. Count tokens with the model's own tokenizer, or you will silently cut mid-word and corrupt structured payloads like JSON.

Read in another language

How to study this topic

Scoring

Check your understanding

Badges