Skip to content

Types of Attention

Overview

The attention formula is always the same — what changes is where the queries, keys, and values come from, and which positions are allowed to attend to which. Three families dominate:

  1. Self-attention — tokens attend to tokens in the same sequence
  2. Cross-attention — queries come from one sequence, keys/values from another

  3. Causal (masked) self-attention — self-attention restricted to the past (autoregressive LLMs)

  4. Prerequisite: 02 Scaled Dot Product Attention — the shared formula

  5. Model archetypes: BERT = bidirectional self-attention; GPT = causal self-attention; T5 = causal self-attention + cross-attention
  6. Choice of type is a design decision about information flow, not a quality ranking

1. Self-Attention (Intra-Sentence)

Every token attends to every token in the same sequence. This is what lets a model resolve pronouns, check grammar agreement, and connect related phrases.

"The cat sat on the mat"

cat → attends to: itself, "The", "sat"  (and weakly to "mat")
mat → attends to: "sat", "on", itself, "the"

Where Q, K, V come from

X = token embeddings of ONE sequence: (seq_len, d_model)

Q = X·W_Q   K = X·W_K   V = X·W_V     ← all from the same X

Every token is simultaneously:
  - a QUERY  (seeking context from others)
  - a KEY    (offering itself as context)
  - a VALUE  (supplying content to others)

Used in

  • Encoder layers of any transformer (BERT, T5 encoder, GPT decoder)
  • Decoder layers of autoregressive models (with causal mask)
  • Every layer of GPT-family models

Why it's powerful (bidirectional version, like BERT)

"The bank approved the loan" → "bank" = financial institution
"The bank of the river"      → "bank" = edge of water

BERT's mask: token i attends to ALL j (left AND right)
  → "bank" can look right to see "loan"/"river" for disambiguation
  → bidirectional context is ideal for understanding tasks
    (classification, NER, QA)

Cost: the model "sees the answer" — not usable for generation directly

2. Cross-Attention (Between Two Sequences)

Queries come from one sequence, Keys and Values from another. The classic case is machine translation: the decoder's queries attend to the encoder's keys and values.

(Encoder output)  "Die Katze saß auf der Matte" → K, V
(Decoder)         "The cat" → Q

At each step, the decoder asks: "which parts of the German
sentence do I need to produce the next English word?"

Where Q, K, V come from (contrast with self-attention)

Decoder hidden states:   H_dec  (seq_len_dec, d_model)   → Q = H_dec·W_Q
Encoder final states:    H_enc  (seq_len_enc, d_model)   → K = H_enc·W_K
                                                          → V = H_enc·W_V

Lengths may differ!  seq_len_dec ≠ seq_len_enc
  → attention scores are (seq_len_dec × seq_len_enc), not square

The decoder "reads FROM" the encoder's memory but never writes to it.

Grammar of cross-attention layers

Standard encoder-decoder (T5, BART, mT5, original Transformer):

  Encoder:  [Self-Attention → FFN] × N layers     (bidirectional)
  Decoder:  [Masked Self-Attention → Cross-Attention → FFN] × N layers
                ↑                                   ↑
          looks at its own                        looks at the
          previous outputs                        encoder's memory

Flow per decoder step:
  1. Predict one target token
  2. Masked self-attention: see earlier target tokens only
  3. Cross-attention: pull relevant info from the source text
  4. FFN, softmax over vocab → next token

Also used in

  • Text-to-image (Stable Diffusion): the U-Net cross-attends to the text prompt
  • Audio/video models: decode from a context different from what's being decoded

3. Causal / Masked Self-Attention (Autoregressive Decoding)

GPT-style models predict the next token. They must never look ahead — otherwise the answer would leak. A causal mask forces token i to attend only to tokens j ≤ i.

Mask for 4 tokens (1 = allowed, 0 = masked):

        j=0  j=1  j=2  j=3
i=0:    1    0    0    0     ← "token 0 sees only itself"
i=1:    1    1    0    0
i=2:    1    1    1    0
i=3:    1    1    1    1     ← "last token sees everything before"

Implementation: add -∞ to masked positions BEFORE softmax
  -∞ → softmax gives 0 weight

Why -∞ instead of 0?

If we masked with 0 (not -∞):
  scores: [0.5, 0, 0.3]  → softmax → e^0.5 / (e^0.5 + e^1 + e^0.3)
  The "0" entries still receive attention!  (softmax never outputs 0
  unless the input is -∞)

With -∞:
  e^(-∞) = 0  →  those positions contribute exactly 0 weight ✅

Worked masked example

Example: "The cat sat" with causal mask (as logits before softmax):

Unmasked logits:      After adding -∞:        Softmax:
  [1, 0, 1]             [1, -∞, -∞]           [1.00, 0, 0]
  [0, 1, 0]             [0, 1, -∞]            [0.27, 0.73, 0]
  [1, 1, 1]             [1, 1, 1]             [0.33, 0.33, 0.33]

"The" can only see itself. "cat" sees "The" + itself.
"sat" sees everything. This is how next-token prediction works.

The training–inference symmetry

Training: all positions computed at once (parallel), because the mask
          guarantees Position 3 never saw later tokens.

Inference: generate ONE token at a time; feed it back with the mask
           growing by one row each step:
             step 1: mask 1×1      → predict token 2
             step 2: mask 2×2      → predict token 3
             step 3: mask 3×3      → predict token 4...

This is why inference re-computes previous tokens' K/V every step
unless cached — see [Kv Cache](/02-llm-modeling/01-architecture/03-memory-management/kv-cache/).

Variants of causal masking

1. Causal (GPT, LLaMA):          token i sees j ≤ i
2. Prefix attention (T5, PaLM):  a "prefix" region is bidirectional,
                                 the rest is causal
3. Block-causal (long-context):  causal over blocks, global attention
                                 on designated tokens (e.g., Longformer)
4. Sliding window (Mistral):     causal + only the last W tokens
                                 → [Sliding Window Attention](/02-llm-modeling/01-architecture/01-core-designs/sliding-window-attention/)

Comparison Table

Property Self-attention (bidirectional) Self-attention (causal) Cross-attention
Query source same sequence same sequence (past only) decoder hidden states
Key/Value source same sequence same sequence (past) encoder memory
Score shape n × n lower-triangular n × n n_dec × n_enc
Can look ahead? ✅ yes ❌ no ✅ yes (into source)
Typical layer encoder decoder decoder (after self-attn)
Signature models BERT, encoder of T5 GPT, LLaMA, Mistral T5, BART, mT5, diffusion
Best at understanding generation sequence-to-sequence

Which Combination Does Your Model Use?

BERT (encoder-only):     bidirectional self-attention only
                         → great at understanding, no generation

GPT (decoder-only):      causal self-attention only
                         → great at generation, still strong at NLU

T5 / BART:               bidirectional self-attn (enc)
                         + causal self-attn + cross-attn (dec)
                         → classic sequence-to-sequence

Why decoder-only won (GPT vs T5 era):
  - the causal objective scales to ANY text (no parallel pairs needed)
  - cross-attention adds complexity without helping the
    "predict the next token" task once models are big enough

Key Takeaways

🔀 Same formula, different sources: Q/K/V provenance defines the attention type
🔁 Self-attention: full-context understanding (BERT)
🧭 Cross-attention: a decoder reading from a source it doesn't edit
🚫 Causal mask: -∞ before softmax → strict "no peeking" into the future
⚙️ T5/BART stack: masked self → cross → FFN per decoder layer
🏆 Decoder-only won: causal self-attention scales best for generative LLMs