← Curriculum map
L09 · Phase 3 · 15 min

Why Multiple Attention Heads?

“The heater's power and the tank's volume are both relevant, but in different ways — can one attention pass capture that?”

Attention Heatmap — multiple heads

"Each apartment has a 300 L tank with two coils and a 2 kW electric immersion heater."
Each apartment has a 300 L tank with two coils and a 2 kW electric immersion heater .

Leans toward quantities and units — the numbers that feed the calculation.

Prototype note: these three heads are hand-authored to show plausibly different relevance patterns over the same sentence — a real model’s heads emerge from training and aren’t hand-labelled like this. Resist reading too much into any one head’s “job”; that’s a genuinely open research question, not something you can reliably read off a heatmap.

Depth ladder

One attention pass can only really answer one question about the sentence at a time — which words are relevant to 'quantity', say. Multi-head attention runs several of these spotlights in parallel, each free to catch a different kind of relationship, then combines what they all found. Toggle between three example heads above and see how their highlighting differs.

Knowledge check

Given heatmaps from two different heads on the same sentence, describe what relationship each seems to track — without claiming certainty about what the head 'means'.