Skip to content

Attention Mechanisms

Overview

Attention is the mechanism that lets a model decide which parts of the input to focus on when producing each output. Instead of compressing the whole sequence into one fixed vector (like RNNs), attention keeps every token available and computes a weighted combination of them per position. It is the single most important idea in modern LLMs— the "A" in GPT, LLaMA, Claude, and every transformer-based model.

  • Paper: "Attention Is All You Need" (Vaswani et al., 2017)
  • Core idea: Every token looks at every other token and asks "how relevant are you to me?" → weighted sum of values
  • Why it matters: Solves RNNs' long-range dependency problem; enables parallel training
  • Cost: O(n²) compute and memory per layer (the main bottleneck of transformers)
  • Prerequisite: Tokenization— attention operates on token embeddings

Chapter Map

This chapter is split into focused sub-notes. Read them in order:

# File Covers
0 [00 Rnn & Lstm](/01-modeling/00-fundamentals/01-attention/(00-rnn-lstm/) Background: how recurrent models (RNN → LSTM → GRU) tried to fix long context— and why they fell short
1 [01 Query, Key & Value](/01-modeling/00-fundamentals/01-attention/(01-query-key-value/) The core intuition: attention as a fuzzy dictionary lookup; the three roles Q, K, V
2 02 Scaled Dot Product Attention The full formula, the four steps, why we scale by √d_k, worked examples
3 03 Types Of Attention Self-attention vs cross-attention vs causal/masked attention
4 04 Multi Head Attention Running many attention passes in parallel; how heads specialize
5 05 Position Information Why attention is position-blind; APE, RoPE, ALiBi
6 [06 Complexity & Cost](/01-modeling/00-fundamentals/01-attention/(06-complexity-cost/) The O(n²) bottleneck and the optimization techniques that attack it
7 07 Pytorch Implementation Minimal but complete code: single-head, multi-head, causal masking
8 08 Interpretability What attention weights actually reveal about model reasoning

-

The Problem: Why RNNs Fail at Long Context

Full story of the attempts to fix this: [00 Rnn & Lstm](/01-modeling/00-fundamentals/01-attention/(00-rnn-lstm/) (RNN → LSTM → GRU → Bahdanau attention → Transformer).

The Fixed-Vector Bottleneck

Before transformers, sequence models (RNNs, LSTMs, GRUs) read text one token at a time, updating a hidden state:

Input: "The chef added salt and pepper to the soup."

LSTM reads left → right, compressing everything into a single hidden state h:

 h₀ → h₁ → h₂ →... → h₁₂
 "The" "chef" "added"... "soup."

The final h₁₂ must contain ALL information about the sentence.
 - If the sentence is 100 tokens, h still has to hold everything
 - Information gets overwritten / diluted as more tokens arrive

Why Information Decays: Vanishing Gradients

Training RNNs on long sequences is mathematically hard. Backpropagation-through-time multiplies gradients across every time step:

Loss at position t:
 ∂L/∂W ∝ Π (from step t back to step 1) of the Jacobian of h

Chain rule: the gradient is a PRODUCT of many small matrices
 Example: if each step scales the gradient by ~0.9, then:
 10 steps back → 0.9¹⁰ ≈ 0.35 (35% remains)
 50 steps back → 0.9⁵⁰ ≈ 0.005 (0.5% remains!)
 100 steps back → 0.9¹⁰⁰ ≈ 0.00003 (nothing left)

Result: the model literally cannot "learn" to carry information
far back in time using gradient signals.
 - LSTMs/GRUs with gating help (they add additive paths)
 - But they remain bounded by a memory cell, not a direct link

The "Cat" Problem (Concrete Example)

Sentence: "The cat, which was adopted from the shelter last winter,
 finally settled into its new home."

Task: What does "its" refer to? → "cat"

But by the time the LSTM reaches "its":
 - It has processed 16 tokens since "cat"
 - The meaning of "cat" is buried under "adopted", "shelter",
 "winter", "settled", "new", "home"
 - Long-distance dependencies decay exponentially

LSTM result: frequently confuses "its" with "shelter" or "home"
Transformer: "its" attends directly to "cat" — the distance doesn't matter

The Second Failure: No Parallelism

RNNs are also sequential by construction:

RNN: t₀ → t₁ → t₂ →... → t₁₀₀₀
 each step DEPENDS on the previous hidden state
 → cannot be parallelized across time
 → 1000 steps = 1000 serial GPU operations
 → slow training

Attention: every token's output depends only on the FULL sequence,
 which exists all at once
 → all tokens processed in parallel on the GPU
 → one big matrix multiplication, huge speedup

Key Insight: The core failure is positional— an RNN can only access the past through a compressed memory. Attention removes that bottleneck by giving every token a direct connection to every other token.

-

Key Takeaways

Attention = soft lookup: Q asks, K labels, V delivers— weighted by similarity (softmax) Four steps: Score → Scale (÷√d_k) → Softmax → Weighted sum Multi-head: 8–32 parallel attention views, each specializing in a linguistic function Causal masking: autoregressive LLMs can only look backward O(n²) cost: the defining bottleneck → KV cache, FlashAttention, GQA, SWA Position-blind: must add position info (APE, RoPE, ALiBi) or word order disappears Learned, not programmed: Q/K/V matrices emerge from training data statistics

-