Attention Mechanism Explained: Formula, Types and Worked Example

LLM foundationsModels and inferencePublished By Simon Budziak

An attention mechanism is a neural network layer that weighs which parts of an input matter for each output. It scores every input position against a query, turns the scores into weights that sum to one, and returns a weighted blend of values. Self-attention, cross-attention and multi-head attention are variants of this one operation.

How does an attention mechanism work?

Attention compares a query against keys, then mixes the matching values. The form used in transformers is scaled dot-product attention, softmax(QKᵀ/√d_k)V, defined in “Attention Is All You Need” (Vaswani et al., arXiv 1706.03762, read 16 November 2026). The √d_k scaling exists because large dot products push the softmax “into regions where it has extremely small gradients.” The output is always a weighted average of values, so attention routes information rather than creating it.

A worked example with three input tokens: one query scores 2.0, 1.0 and 0.1 against their keys. Softmax turns those scores into weights of about 0.66, 0.24 and 0.10, so the output is 0.66 of the first value vector plus 0.24 of the second plus 0.10 of the third. In an LLM, queries, keys and values are learned projections of each token, recomputed in every layer.

Why did attention replace recurrent networks?

Attention first fixed a bottleneck in recurrent translation models. Bahdanau, Cho and Bengio argued that “the use of a fixed-length vector is a bottleneck” in encoder-decoder translation and let the decoder “(soft-)search” the source for each target word (arXiv 1409.0473, first submitted 1 September 2014, read 16 November 2026). The 2017 transformer architecture then dropped recurrence entirely and built on attention alone.

Removing recurrence made every position computable in parallel, which is what made training at scale practical. The cost moved: the 2017 paper puts a self-attention layer at O(n²·d) against O(n·d²) for a recurrent one, where n is sequence length, the trade-off behind every long context window.

What types of attention mechanism are there?

Attention variants differ in where queries, keys and values come from and how much state they keep.

Variant Queries, keys and values What it changes Source
Global and local attention Decoder queries over all source words, or a window of them Local attention looks at a subset of source words at a time Luong et al., 2015
Self-attention All from the same sequence Every token can read every other token Vaswani et al., 2017
Cross (encoder-decoder) attention Queries from the decoder, keys and values from the encoder The output side reads the whole input Vaswani et al., 2017
Causal (masked) self-attention Same sequence, earlier positions only Future scores set to minus infinity, so generation stays left to right Vaswani et al., 2017
Multi-head attention Several projections in parallel 8 heads of 64 dimensions each in the original model Vaswani et al., 2017
Multi-query and grouped-query attention One key-value head, or a few shared by groups of query heads Faster decoding, near multi-head quality for GQA Ainslie et al., 2023

All sources read 16 November 2026. These rows stack rather than compete: one decoder layer can be causal, multi-head and grouped-query at once. Fewer key-value heads also mean a smaller KV cache per request. FlashAttention keeps the math exact but computes it in tiles to cut GPU memory reads and writes, reporting about 3 times faster GPT-2 training at sequence length 1K (Dao et al., arXiv 2205.14135, read 16 November 2026).

What does attention help a model do, and what does it not do?

Attention lets each output draw on whichever inputs matter to it, at any distance. That covers cases such as the name a pronoun points to three paragraphs back. It does not check facts: a confident blend of the wrong passage is still wrong.

Nor are attention weights a reliable explanation. Jain and Wallace tested whether attention weights explain predictions: “We find that they largely do not” (Attention is not Explanation, arXiv 1902.10186, NAACL 2019, read 16 November 2026). Treat an attention heatmap as a debugging hint, not as evidence of why a model decided. When a decision needs justifying, use AI explainability methods tested against the model’s behavior, and keep the evidence in what the model was given, not in where it looked.

This entry was drafted with AI assistance.

Frequently asked questions

Is an attention mechanism the same as self-attention?

No. Attention is the general operation of weighting values by how well their keys match a query. Self-attention is the case where queries, keys and values all come from the same sequence; cross-attention takes them from two different sequences.

Why is attention divided by the square root of d_k?

Dot products of long vectors grow large, which pushes the softmax into a region where gradients almost vanish. Dividing by the square root of the key dimension keeps the scores in a range where the model can still learn.

Summarize this page with

Train your team to build this