← Curriculum map
L01 · Phase 2 · 20 min

From Words to Tokens

“Why does a longer conversation with AI Tool cost more, and why does it sometimes cut a word in a strange place?”

Token Explorer

There are shared evacuated-tube solar collectors on the roof. They're connected to a glycol circulation loop that runs down to a plant room.
34 tokens34 ÷ 1,000 × $0.002 = $0.000068 (illustrative rate)

Which costs more?

Predict which phrase uses more tokens, then reveal:

Prototype note: this is a heuristic stand-in tokenizer (a curated vocabulary plus a chunking fallback), not a production BPE tokenizer — close enough to teach the idea, not accurate enough to predict a real vendor’s exact count.

Depth ladder

AI Tool doesn't read your message the way you do, one whole word at a time. Before it can do anything else, it breaks your text into small pieces called tokens — common words usually stay whole ('the', 'tank', 'solar'), but longer or less common ones get split into chunks ('photovoltaic' might become three pieces: 'photo', 'volt', 'aic'). Every one of those pieces is a unit of work: more tokens means more for the model to process, and on most AI tools, a higher cost per message. That's part of why a short, plain-language question is often both cheaper and clearer than a long, jargon-heavy one.

Knowledge check

Given two similarly-long prompts with different vocabulary, predict which uses more tokens, then verify.