How does an attention mechanism work?
Attention compares a query against keys, then mixes the matching values. The form used in transformers is scaled dot-product attention, softmax(QKᵀ/√d_k)V, defined in “Attention Is All You Need” (Vaswani et al., arXiv 1706.03762, read 16 November 2026). The √d_k scaling exists because large dot products push the softmax “into regions where it has extremely small gradients.” The output is always a weighted average of values, so attention routes information rather than creating it.
A worked example with three input tokens: one query scores 2.0, 1.0 and 0.1 against their keys. Softmax turns those scores into weights of about 0.66, 0.24 and 0.10, so the output is 0.66 of the first value vector plus 0.24 of the second plus 0.10 of the third. In an LLM, queries, keys and values are learned projections of each token, recomputed in every layer.
Why did attention replace recurrent networks?
Attention first fixed a bottleneck in recurrent translation models. Bahdanau, Cho and Bengio argued that “the use of a fixed-length vector is a bottleneck” in encoder-decoder translation and let the decoder “(soft-)search” the source for each target word (arXiv 1409.0473, first submitted 1 September 2014, read 16 November 2026). The 2017 transformer architecture then dropped recurrence entirely and built on attention alone.
Removing recurrence made every position computable in parallel, which is what made training at scale practical. The cost moved: the 2017 paper puts a self-attention layer at O(n²·d) against O(n·d²) for a recurrent one, where n is sequence length, the trade-off behind every long context window.
What types of attention mechanism are there?
Attention variants differ in where queries, keys and values come from and how much state they keep.
| Variant | Queries, keys and values | What it changes | Source |
|---|---|---|---|
| Global and local attention | Decoder queries over all source words, or a window of them | Local attention looks at a subset of source words at a time | Luong et al., 2015 |
| Self-attention | All from the same sequence | Every token can read every other token | Vaswani et al., 2017 |
| Cross (encoder-decoder) attention | Queries from the decoder, keys and values from the encoder | The output side reads the whole input | Vaswani et al., 2017 |
| Causal (masked) self-attention | Same sequence, earlier positions only | Future scores set to minus infinity, so generation stays left to right | Vaswani et al., 2017 |
| Multi-head attention | Several projections in parallel | 8 heads of 64 dimensions each in the original model | Vaswani et al., 2017 |
| Multi-query and grouped-query attention | One key-value head, or a few shared by groups of query heads | Faster decoding, near multi-head quality for GQA | Ainslie et al., 2023 |
All sources read 16 November 2026. These rows stack rather than compete: one decoder layer can be causal, multi-head and grouped-query at once. Fewer key-value heads also mean a smaller KV cache per request. FlashAttention keeps the math exact but computes it in tiles to cut GPU memory reads and writes, reporting about 3 times faster GPT-2 training at sequence length 1K (Dao et al., arXiv 2205.14135, read 16 November 2026).
What does attention help a model do, and what does it not do?
Attention lets each output draw on whichever inputs matter to it, at any distance. That covers cases such as the name a pronoun points to three paragraphs back. It does not check facts: a confident blend of the wrong passage is still wrong.
Nor are attention weights a reliable explanation. Jain and Wallace tested whether attention weights explain predictions: “We find that they largely do not” (Attention is not Explanation, arXiv 1902.10186, NAACL 2019, read 16 November 2026). Treat an attention heatmap as a debugging hint, not as evidence of why a model decided. When a decision needs justifying, use AI explainability methods tested against the model’s behavior, and keep the evidence in what the model was given, not in where it looked.