How does self-attention actually work?
Each token’s embedding is projected three ways: into a query (what am I looking for), a key (what do I offer) and a value (what I carry). Every token’s query is scored against every other token’s key, the scores become weights, and each token’s new representation is the weighted blend of the values. Nothing in the mechanism is sequential, every comparison happens at once, which is what let transformers train on orders of magnitude more data than the recurrent models they replaced. “Attention Is All You Need”, the 2017 paper that introduced the transformer architecture, is named for exactly this removal.
What is multi-head attention?
The same trick run several times in parallel with different learned projections. One head might track syntax, another coreference, another position; the model learns the division itself. Concatenated, the heads give each token a composite view of the sequence. The heads are why the mechanism scales in quality, not just in speed: a single attention pattern could not carry grammar, meaning and position at once.
Why does self-attention set the limits of LLMs?
Because its cost grows with the square of the sequence. Doubling the context window roughly quadruples the attention compute, which is why long contexts are priced the way they are and why models can degrade on material buried mid-context even when it technically fits. Most practical context engineering is working around the economics of this one mechanism. Every token you feed the model joins that all-pairs comparison, which is the literal reason trimming context cuts cost.
Where else does self-attention show up?
Anywhere data can be written as a sequence of embeddings. Vision transformers attend over image patches, audio models over sound frames, protein models over amino acids; the mechanism is indifferent to what the tokens mean. That generality is why one architecture took over AI: the field converged on a single trainable way to ask which parts of the input matter to which.