Skip to content

Position Information: The Missing Ingredient

Overview

Attention is position-agnostic β€” QΒ·Kα΅€ between "token 1" and "token 5" behaves identically to "token 1" and "token 3" if the embeddings are the same. Without position signals, "the cat chased the dog" and "the dog chased the cat" would produce identical representations. Every transformer must therefore inject position information explicitly β€” via absolute embeddings, relative biases, or rotary rotations.

  • Problem: permutation invariance β€” attention ignores word order by construction
  • Consequence: word order carries grammar; ignoring it breaks language modeling
  • Modern winner: Rope (LLaMA, Mistral, Qwen, GPT-4)
  • Side effect: position encoding is also the main lever for Length Extrapolation

The Problem: Permutation Invariance

The core issue

Attention(Q,K,V) = softmax(QΒ·Kα΅€ / √d_k) Β· V

The score Qα΅’Β·Kβ±Ό depends ONLY on the content of tokens i and j.
It does NOT depend on i and j themselves β€” where they sit.

"The cat chased the dog"     and     "The dog chased the cat"
        ↑ same tokens, same order swapped                    ↑

If embeddings were identical and position ignored:
  - identical pairwise scores (identical words β‡’ identical Q/K)
  - identical attention weights
  - identical output for "cat" in both sentences ❌

Why this is fatal for language

Word order encodes meaning:
  "dog bites man" β‰  "man bites dog"   (roles swapped)
  "only John eats pizza" β‰  "John eats only pizza"  (scope changes)
  "great danger" vs "danger, great"   (phrasal structure)

Also: verbs must agree with their subject in number/tense β€”
the model needs to know WHICH token is earlier to do syntax.

πŸ’‘ Key Insight: Attention says "what is similar to what"; position says "what is where". Transformers need both β€” that's why every production LLM adds position encoding before/inside attention.

The Solution Space: Four Families

Family How it encodes position Typical use
APE β€” Absolute Position Embeddings add a position vector to each token original Transformer, GPT-2/3
RPE β€” Relative Position Bias add a distance-based bias to scores T5, Shaw et al. 2018
RoPE β€” Rotary Position Embeddings rotate Q/K by position angle LLaMA, Mistral, Qwen, GPT-4
ALiBi β€” Attention with Linear Biases subtract distance Γ— slope from scores BLOOM, MPT

All four answer the same question: "how do I make position visible to attention?" β€” with different trade-offs for extrapolation, compute, and long-context behavior.

1. Absolute Position Embeddings (APE)

Sinusoidal (original Transformer)

pos = position index, i = embedding dimension

PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Why sine/cosine?
  - different frequencies β†’ the model can "see" coarse (low-freq)
    and fine (high-freq) position signals
  - linear in time: PE(pos+k) can be expressed via PE(pos) with
    rotation β€” lets the model learn relative offsets for free

Learned (GPT-2/3)

GPT-2: a learnable matrix of shape (max_positions, d_model)
       added to the token embeddings

X_final[pos, :] = X_token[pos, :] + PE_learned[pos, :]

Pros: model tunes positions to its data
Cons: max_positions is FIXED at training time
      β†’ cannot attend beyond it β†’ hard length extrapolation

Why APE struggles at long context

Training context: 4,096 β†’ learned PE exists for positions 0..4095
Prompt length:    8,000  β†’ positions 4096..7999 have NO learned vector

Options:
  - truncate (lose context) ❌
  - position interpolation (compress 8000 into 4096 slots) β€” a hack
  - random init for unseen positions (must be re-learned) ❌

This rigidity is why modern models moved to relative methods.

2. Relative Position Bias (RPE)

Instead of adding to embeddings, bias the attention scores by the distance between tokens:

score(i, j) = Qα΅’Β·Kβ±Ό / √d_k + b(j βˆ’ i)

b: a learned bias keyed by relative distance (j βˆ’ i)
  - nearby tokens get a distinct bias from far-away tokens
  - the model learns "immediate neighbors are usually relevant"

T5: uses a bucketed bias table (e.g., 32 buckets), shared across heads
   β†’ cheaper than full nΓ—n learned bias, decent extrapolation

Pros: relative distances generalize better than absolute indices. Cons: bucketing choices matter; still a fixed max distance.

3. Rotary Position Embeddings (RoPE) β€” the modern default

Key idea: rotate Q and K by an angle proportional to position.

Q' = Rotate(Q, pos·θ₁, ..., posΒ·ΞΈ_{d/2})
K' = Rotate(K, pos·θ₁, ...)

Properties:
  - dot product of rotated vectors depends ONLY on relative offset
    (Q'α΅’Β·K'β±Ό = f(i βˆ’ j)) β€” relative attention for free
  - works beyond training length (rotation is defined for ANY pos)
  - no extra parameters (rotation matrices are fixed/derived)
  - cheap to implement (complex-number arithmetic in fp32)

Adoption: LLaMA 1/2/3, Mistral, Qwen, Gemma, GPT-4 (speculated)

πŸ”— Full deep-dive: Rope.

4. ALiBi (Attention with Linear Biases)

score(i, j) = Qα΅’Β·Kβ±Ό / √d_k βˆ’ m Β· (i βˆ’ j)

m: a per-head slope (e.g., 1/2, 1/4, 1/8, ..., decreasing)
The farther back token j is, the more the score is penalized.

No position parameters to learn at all!
  - works out of the box on long contexts
  - used by BLOOM, MPT-7B

Trade-off: penalizes old tokens (can hurt long-range recall);
also a form of "learned recency bias".

Which Position Method Do Modern Models Use?

Model               Position method     Context
──────────────────────────────────────────────────────────
Transformer (2017)  Sinusoidal APE       512 (original paper)
GPT-2/3             Learned APE          1,024 / 2,048
BERT                Learned APE          512
T5                  Relative bias        512
BLOOM / MPT         ALiBi                2,048 / 8,192 (extrapolates)
LLaMA 1/2/3         RoPE                2,048 β†’ 8K β†’ 128K+
Mistral / Qwen      RoPE (+ SWA)         32K / 128K+
GPT-4               RoPE (reported)      ~128K

The trend: relative/rotary methods won because they extrapolate, and extrapolation is what lets models grow context windows without retraining from scratch.

Position vs. Attention: The Interaction

APE:   position added to embeddings BEFORE attention
       β†’ Q/K/V inherit it; scores become "content + position-ish"

RPE:   position added to scores AFTER QΒ·Kα΅€
       β†’ explicit "distance discount" on relevance

RoPE:  position applied to Q/K BEFORE the dot product
       β†’ relative offset emerges inside the score itself

All roads lead to: score(i,j) should depend on (i βˆ’ j),
which plain content dot products never capture.

Key Takeaways

🧭 Attention is order-blind β€” position encoding is mandatory, not optional
πŸ”€ Permutation test: swap words β†’ identical attention β†’ broken language model
πŸ“ Four families: APE (add), RPE (bias scores), RoPE (rotate), ALiBi (linear penalty)
πŸ“ˆ Modern default: RoPE β€” relative, extrapolatable, parameter-free
🚧 Extrapolation is the battle: learned APE caps context; RoPE/ALiBi stretch it
πŸ”— Position and context length are linked β€” see Length Extrapolation & Long Context