Skip to content

Types of Attention

Overview

The attention formula is always the same— what changes is where the queries, keys, and values come from, and which positions are allowed to attend to which. Three families dominate:

  1. Self-attention— tokens attend to tokens in the same sequence
  2. Cross-attention— queries come from one sequence, keys/values from another

  3. Causal (masked) self-attention— self-attention restricted to the past (autoregressive LLMs)

  4. Prerequisite: 02 Scaled Dot Product Attention— the shared formula

  5. Model archetypes: BERT = bidirectional self-attention; GPT = causal self-attention; T5 = causal self-attention + cross-attention
  6. Choice of type is a design decision about information flow, not a quality ranking

-

1. Self-Attention (Intra-Sentence)

Every token attends to every token in the same sequence. This is what lets a model resolve pronouns, check grammar agreement, and connect related phrases.

"The cat sat on the mat"

cat → attends to: itself, "The", "sat" (and weakly to "mat")
mat → attends to: "sat", "on", itself, "the"

Where Q, K, V come from

X = token embeddings of ONE sequence: (seq_len, d_model)

Q = X·W_Q K = X·W_K V = X·W_V ← all from the same X

Every token is simultaneously:
 - a QUERY (seeking context from others)
 - a KEY (offering itself as context)
 - a VALUE (supplying content to others)

Used in

  • Encoder layers of any transformer (BERT, T5 encoder, GPT decoder)
  • Decoder layers of autoregressive models (with causal mask)
  • Every layer of GPT-family models

Why it's powerful (bidirectional version, like BERT)

"The bank approved the loan" → "bank" = financial institution
"The bank of the river" → "bank" = edge of water

BERT's mask: token i attends to ALL j (left AND right)
 → "bank" can look right to see "loan"/"river" for disambiguation
 → bidirectional context is ideal for understanding tasks
 (classification, NER, QA)

Cost: the model "sees the answer" — not usable for generation directly

-

2. Cross-Attention (Between Two Sequences)

Queries come from one sequence, Keys and Values from another. The classic case is machine translation: the decoder's queries attend to the encoder's keys and values.

(Encoder output) "Die Katze saß auf der Matte" → K, V
(Decoder) "The cat" → Q

At each step, the decoder asks: "which parts of the German
sentence do I need to produce the next English word?"

Where Q, K, V come from (contrast with self-attention)

Decoder hidden states: H_dec (seq_len_dec, d_model) → Q = H_dec·W_Q
Encoder final states: H_enc (seq_len_enc, d_model) → K = H_enc·W_K
 → V = H_enc·W_V

Lengths may differ! seq_len_dec ≠ seq_len_enc
 → attention scores are (seq_len_dec × seq_len_enc), not square

The decoder "reads FROM" the encoder's memory but never writes to it.

Grammar of cross-attention layers

Standard encoder-decoder (T5, BART, mT5, original Transformer):

 Encoder: [Self-Attention → FFN] × N layers (bidirectional)
 Decoder: [Masked Self-Attention → Cross-Attention → FFN] × N layers
 ↑ ↑
 looks at its own looks at the
 previous outputs encoder's memory

Flow per decoder step:
 1. Predict one target token
 2. Masked self-attention: see earlier target tokens only
 3. Cross-attention: pull relevant info from the source text
 4. FFN, softmax over vocab → next token

Also used in

  • Text-to-image (Stable Diffusion): the U-Net cross-attends to the text prompt
  • Audio/video models: decode from a context different from what's being decoded

-

3. Causal / Masked Self-Attention (Autoregressive Decoding)

GPT-style models predict the next token. They must never look ahead— otherwise the answer would leak. A causal mask forces token i to attend only to tokens j ≤ i.

Mask for 4 tokens (1 = allowed, 0 = masked):

 j=0 j=1 j=2 j=3
i=0: 1 0 0 0 ← "token 0 sees only itself"
i=1: 1 1 0 0
i=2: 1 1 1 0
i=3: 1 1 1 1 ← "last token sees everything before"

Implementation: add -∞ to masked positions BEFORE softmax
 -∞ → softmax gives 0 weight

Why -∞ instead of 0?

If we masked with 0 (not -∞):
 scores: [0.5, 0, 0.3] → softmax → e^0.5 / (e^0.5 + e^1 + e^0.3)
 The "0" entries still receive attention! (softmax never outputs 0
 unless the input is -∞)

With -∞:
 e^(-∞) = 0 → those positions contribute exactly 0 weight

Worked masked example

Example: "The cat sat" with causal mask (as logits before softmax):

Unmasked logits: After adding -∞: Softmax:
 [1, 0, 1] [1, -∞, -∞] [1.00, 0, 0]
 [0, 1, 0] [0, 1, -∞] [0.27, 0.73, 0]
 [1, 1, 1] [1, 1, 1] [0.33, 0.33, 0.33]

"The" can only see itself. "cat" sees "The" + itself.
"sat" sees everything. This is how next-token prediction works.

The training–inference symmetry

Training: all positions computed at once (parallel), because the mask
 guarantees Position 3 never saw later tokens.

Inference: generate ONE token at a time; feed it back with the mask
 growing by one row each step:
 step 1: mask 1×1 → predict token 2
 step 2: mask 2×2 → predict token 3
 step 3: mask 3×3 → predict token 4...

This is why inference re-computes previous tokens' K/V every step
unless cached — see [Kv Cache](/01-modeling/01-architecture/03-memory-management/kv-cache/).

Variants of causal masking

1. Causal (GPT, LLaMA): token i sees j ≤ i
2. Prefix attention (T5, PaLM): a "prefix" region is bidirectional,
 the rest is causal
3. Block-causal (long-context): causal over blocks, global attention
 on designated tokens (e.g., Longformer)
4. Sliding window (Mistral): causal + only the last W tokens
 → [Sliding Window Attention](/01-modeling/01-architecture/01-core-designs/sliding-window-attention/)

Comparison Table

Property Self-attention (bidirectional) Self-attention (causal) Cross-attention
Query source same sequence same sequence (past only) decoder hidden states
Key/Value source same sequence same sequence (past) encoder memory
Score shape n × n lower-triangular n × n n_dec × n_enc
Can look ahead? yes no yes (into source)
Typical layer encoder decoder decoder (after self-attn)
Signature models BERT, encoder of T5 GPT, LLaMA, Mistral T5, BART, mT5, diffusion
Best at understanding generation sequence-to-sequence

-

Which Combination Does Your Model Use?

BERT (encoder-only): bidirectional self-attention only
 → great at understanding, no generation

GPT (decoder-only): causal self-attention only
 → great at generation, still strong at NLU

T5 / BART: bidirectional self-attn (enc)
 + causal self-attn + cross-attn (dec)
 → classic sequence-to-sequence

Why decoder-only won (GPT vs T5 era):
 - the causal objective scales to ANY text (no parallel pairs needed)
 - cross-attention adds complexity without helping the
 "predict the next token" task once models are big enough

-

Key Takeaways

Same formula, different sources: Q/K/V provenance defines the attention type Self-attention: full-context understanding (BERT) Cross-attention: a decoder reading from a source it doesn't edit Causal mask: -∞ before softmax → strict "no peeking" into the future T5/BART stack: masked self → cross → FFN per decoder layer Decoder-only won: causal self-attention scales best for generative LLMs

-