Skip to content

Query, Key & Value

Overview

Every word in a sentence plays one of three roles inside attention: Query (Q), Key (K), or Value (V). The Query asks "what am I looking for?", the Key answers "what am I?", and the Value supplies "what do I contribute?". Attention is the process of matching Queries to Keys and blending the corresponding Values. Nothing about Q/K/V is pre-programmed— all three are learned from data.

  • Analogy: attention works like a database / search engine lookup
  • Zero-setup: no hand-crafted rules; the model discovers the roles during training
  • Leads to: 02 Scaled Dot Product Attention— how Q, K, V combine mathematically
  • First seen in: "Neural Machine Translation by Jointly Learning to Align and Translate" (Bahdanau et al., 2015)— attention predates transformers!

-

The Search Engine Analogy

The simplest mental model for attention is a database query. Every word plays one of three roles— Query, Key, or Value:

You run a search engine (this is attention!):

 ┌──────────────────────────────┐
 QUERY = what I'm │ "best laptop under 1000$" │
 searching for └──────────────────────────────┘
 ┌──────────────────────────────┐
 KEY = the labels │ "laptop reviews" → page 3│
 of every stored │ "coffee machines" → page 8│
 document │ "budget laptops" → page 1│
 └──────────────────────────────┘
 ┌──────────────────────────────┐
 VALUE = the │ page 3 content, page 8 │
 actual content │ content, page 1 content │
 └──────────────────────────────┘

 1. QUERY is compared against every KEY → score (similarity)
 2. Scores are normalized → weights (how much to pay attention)
 3. We fetch the VALUES weighted by scores → output

Why is the search engine the right metaphor?

Query ≠ Key: "what I want" is different from "what each item is"
Query vs all: a query compares itself against EVERY key (all positions)
Soft retrieval: results are mixed proportionally to relevance —
 not a hard "yes/no" retrieval
Learned ranking: the "relevance function" is trained, not hand-written

The Three Roles, in Detail

In a transformer, every token projects to three vectors using learned weight matrices:

For token xᵢ:
 Qᵢ = W_Q · xᵢ ← "What am I looking for?" (query)
 Kᵢ = W_K · xᵢ ← "What am I?" (key)
 Vᵢ = W_V · xᵢ ← "What do I contribute?" (value)

Reading them in a concrete sentence

Example: in "The chef tasted the soup"
 "soup" as Query: "is there anything modifying me? (adjectives, verbs...)"

 "chef" sends a Key: "I am the subject, a person"
 "tasted" sends a Key: "I am a past-tense verb"
 "the" sends a Key: "I am a determiner"

 The softmax weights decide which Keys match the Query best.

What each role is "about"

Role Question it answers Governed by Emerges as
───── ────────────────────────── ──────────────── ─────────────────────
Q "What am I seeking?" the token's own needs object-of-verb links,
 modification needs
K "How should I be found?" the token's identity grammatical category,
 topic, name
V "What content do I give?" the token's meaning the "payload" that
 gets passed along

Key Insight: Q, K, V are learned during training. The model discovers which roles are useful for language understanding. Nobody tells "soup" to look for adjectives— it learns to, because that improves next-token prediction.

-

Why Three Vectors? (Why Not One?)

A reasonable question: why not just compute similarity between raw token embeddings? Three reasons:

1. Separation of "Matching" from "Content"

Similarity is about MATCHING, output is about CONTENT.

 "cat" and "kitten": high similarity (both noun-y, animal-y)
 But when generating the word after "the ___", you want the
 CONTENT of "cat"/"kitten" (the animal concept), not a blur.

 Q/K decide WHO talks to WHOM. V decides WHAT is said.
 Mixing them into one vector forces a compromise:
 - a vector that is both a good label AND good content
 - conflates "findability" with "meaning"

2. Different Projections → Different Semantics

Q, K, V come from three DIFFERENT learned matrices:

 Qᵢ = W_Q xᵢ (d_model → d_k)
 Kᵢ = W_K xᵢ (d_model → d_k)
 Vᵢ = W_V xᵢ (d_model → d_v)

Each matrix warps the embedding space for its own purpose:
 W_K warps space so that related items land close together
 (two words can be "far apart" in K-space but close in V-space)
 W_Q warps space relative to what the token needs at this position
 W_V warps space so the payload is well-separated for the MLP

One embedding vector cannot serve all three purposes at once.

3. Keys Are Compared in a Lower-Dimensional Space

Typically d_k < d_model (e.g., d_model = 4096, d_k = 128).

Projecting to a smaller key space:
 - cheap dot products (fewer FLOPs per head)
 - forces the key to compress to "essentially relevant" features
 - the compression is learned, so it keeps what matters for matching

-

The Projection Matrices

Token embeddings X: (seq_len, d_model) e.g., (6, 4)

Learned parameters (all trainable, shared across positions):
 W_Q: (d_model, d_k) W_K: (d_model, d_k) W_V: (d_model, d_v)

Computing Q, K, V — one matrix multiply each:
 Q = X @ W_Q → (seq_len, d_k)
 K = X @ W_K → (seq_len, d_k)
 V = X @ W_V → (seq_len, d_v)

Concrete (toy):
 X = 6 tokens × 4-dim embedding
 W_Q: 4×3, W_K: 4×3, W_V: 4×4
 → Q: 6×3, K: 6×3, V: 6×4

Typical dimensions in real models

Model d_model d_k (head) heads
GPT-2 small 768 64 12
BERT base 768 64 12
LLaMA-2 7B 4096 128 32
GPT-3 12288 128 96

Common ratio: d_k = d_model / heads (128, 64, 80,...)

-

Attention as a Biological/Perceptual Idea

Attention isn't an AI invention— it mirrors natural attention:

🟢 Visual attention: your eyes scan a scene and "focus" on salient
 regions; peripheral regions contribute less
🟢 Cognitive attention: when reading, you focus on words that resolve
 ambiguity ("bank" → depends on "river" or "money")
🟢 Machine translation origin: Bahdanau (2015) let the decoder "look back"
 at relevant encoder words for each target word —
 instead of squeezing the whole source into
 one context vector

Transformer = the same idea, applied at every layer, for every token,
in parallel, with learned Q/K/V.

-

Where Q, K, V Come From: The Full Pipeline

Raw text
 │ tokenization
 ▼
Token IDs ──► Embedding lookup ──► X (token embeddings, may already
 │ include position info — see
 │ [05 Position Information](/01-modeling/00-fundamentals/01-attention/05-position-information/))
 ▼
Linear layer: X·W_Q → Q X·W_K → K X·W_V → V
 │
 ▼
scaled_dot_product_attention(Q, K, V) ← [02 Scaled Dot Product Attention](/01-modeling/00-fundamentals/01-attention/02-scaled-dot-product-attention/)
 │
 ▼
Context-aware token representations (one per position)

-

Key Takeaways

Q asks, K labels, V delivers— three learned roles per token Fuzzy retrieval: relevance is soft (softmax weights), not binary Three separate matrices: matching space (Q/K) ≠ content space (V) Fully learned: no hand-crafted linguistic rules anywhere Universal: self-attention, cross-attention and multi-head all reuse the same Q/K/V machinery Parallel by design: X·W_Q, X·W_K, X·W_V are one batched matmul each

-