Skip to content

Interpretability: What Attention Actually Learns

Overview

Attention weights give us a rare, direct window into a trained model's "reasoning": for any token, we can read off which other tokens it weighted most heavily. Decades of analysis (starting with the original 2017 paper's famous "visualizing attention" figures) show clear, reproducible patterns — syntax heads, coreference heads, induction heads. But attention is also not a perfect explanation of model behavior; weights tell part of the story, and value vectors + MLPs tell the rest.

  • Core finding: attention heads specialize in linguistic functions (syntax, coreference, position)
  • Emergence: specialization appears without any supervision — it is a by-product of the training loss
  • Caveat: "attention is not explanation" (Jain & Wallace, 2019) — weights ≠ full reasoning trace
  • Prerequisite: 04 Multi Head Attention (heads are the units we analyze)

The Analogy: Reading the Model's Highlighter

Every attention head is like a person with a highlighter:
  - given a sentence, they underline which words they "looked at"
  - head-by-head, the highlights reveal their strategy

"The cat sat on the mat"
  Head A highlights: sat ←→ cat      ("I track subject-verb links")
  Head B highlights: mat ← the       ("I track determiners")
  Head C highlights: every token ← its neighbor ("I track locality")

The attention matrix is literally this: weights per (query, key) pair.

Pattern 1: Syntactic Heads

Subject-verb agreement

"The cat sat on the mat"
Layer 3, Head 7 attention from "sat":
  the    cat    sat    on    the    mat
  0.01   0.62   0.12   0.03  0.01   0.21
              ↑ subject + object are the "sources of truth"

Interpretation: to predict what comes next, "sat" needs to know
WHO sat — the subject "cat". The head routes that information.

"the cat" / "a book" / "my dog"
Many heads in early layers link determiners to their noun:
  the → cat, a → book, my → dog

This mirrors linguistic structure (DP attachment) and emerges
naturally from predicting the next token.

Pattern 2: Coreference Chains

"John told Mary that he was sorry"

From "he":
  John → 0.71   Mary → 0.02   told → 0.03 ...

The model resolves "he" → "John" via attention alone —
this is the transformer's long-range memory in action.

Longer chains (5+ hops):
  "Alice gave Bob her book because he asked"
   her → Alice, he → Bob — the model tracks two chains
   simultaneously with different heads

💡 This is why transformers beat LSTMs on coreference: the connection is direct (query→key), not through a compressed hidden state.

Pattern 3: Induction Heads (Copy-Paste Pattern Matching)

Modern interpretability (Anthropic, 2021–2022) found a crucial pattern in GPT-style models:

Pattern: "A B ... A"  →  predict "B"

Example:
  "Rufus the dog ... Rufus"  →  model predicts "the dog"
  "The capital of France is ... Paris, and the capital of ..."

An induction head works in two phases:
  1. A "previous-token head" copies:  position of A  → position of B
  2. A "matching head" attends:       current A      → past A
     then moves to the copied B and predicts it

Why it matters:
  - explains how LLMs learn in-context patterns without any training step
  - emerges sharply at specific model scales ("phase change" in loss)
  - it is the mechanistic foundation of few-shot prompting

Layer-Wise Structure

Attention patterns change systematically as you go deeper:

Layer 1–2 (early):   local, positional
  - attention to the immediately previous token
  - word-piece structure, determiner–noun links

Layers 3–6 (middle): syntactic
  - subject–verb, object–verb, modifiers
  - coreference chains begin

Top layers (late):   semantic / global
  - attention to sentence boundaries, [SEP]/special tokens
  - long-range topic links, cross-sentence relations

Also typical: a few "universal" heads that attend to the
first/last token or special tokens — the model's "attention sinks".

Attention ≠ Explanation (The Debate)

The critique (Jain & Wallace, 2019: "Attention is not Explanation")

1. Alternative explanations fit the same weights
   - you can find attention distributions that "predict" outputs
     differently yet match behavior

2. Attention can be adversarial
   - perturbing attention weights often does NOT change predictions
   → the final output doesn't causally depend on attention alone

3. Weights are only part of the pipeline
   - V values multiply weights: a 0.9 weight on a tiny value
     can matter less than a 0.1 weight on a huge value
   - MLPs and residual streams carry most of the "meaning"

The rebuttal (Wiegreffe & Pinter, 2019: "Attention is not not Explanation")

- Attention is still correlated with behavior in most settings
- It remains a useful, cheap, faithful-enough summary
- Mechanistic methods (logit lens, activation patching) refine
  rather than invalidate attention analysis

Practical stance for engineers:
  - use attention as a DEBUGGING and VALIDATION tool
  - don't claim it is the model's full decision trace

How to Actually Inspect Attention

Code: dump attention weights

import torch, torch.nn.functional as F

def attention_weights(q, k, n_heads):
    """(batch, n_heads, seq, seq) weights for inspection."""
    b, s, _ = q.shape
    d = q.size(-1) // n_heads
    q = q.view(b, s, n_heads, d).transpose(1, 2)
    k = k.view(b, s, n_heads, d).transpose(1, 2)
    scores = q @ k.transpose(-2, -1) / (d ** 0.5)
    return F.softmax(scores, dim=-1)   # shape: (b, n_heads, s, s)

# To inspect: hooks on the model's attention module
# (see 07-PyTorch Implementation for the module to hook into)

Tools

BERTViz        — interactive head-by-head visualizer (web)
Lingvo/Lens    — attention rollout / input contribution analysis
Circuit tools  — Anthropic's "A Mathematical Framework" (induction heads),
                 transformer-lens for activation patching

Key Takeaways

🧠 Heads specialize: syntax, coreference, position — all emergent
🔗 Induction heads explain in-context learning mechanistically
🌐 Layers organize: local → syntactic → semantic as depth grows
⚖️ Weights ≠ full explanation: values/MLPs also carry meaning
🛠️ Still useful: attention is a top debugging + validation tool
🔬 Modern tooling: logit lens, patching, circuits — beyond raw weights