Explainer
How transformer models work: attention, heads and layers
The Transformer is a neural network architecture introduced in 2017 that relies on attention mechanisms instead of reading text one word at a time.[1][2] For each token, self-attention scores how much every other token should contribute to its meaning, and several attention heads stacked in many layers build up richer representations.[3][4] Because the whole sequence is processed in parallel, Transformers train efficiently on modern chips, and they are now the most widely used architecture for building large language models.[5][6]
This page goes inside the Transformer architecture. For how Transformers become chatbots through scaling and human-feedback training, see how large language models work.
The problem the Transformer solved
Before 2017, recurrent neural networks were the typical architecture for machine translation.[2] They read language one word at a time, left to right or right to left.[2] That made decisions depending on distant words harder to learn, and it suited parallel chips such as GPUs and TPUs poorly, because those excel at parallel rather than sequential work.[2]
The 2017 paper “Attention Is All You Need” proposed the Transformer, an architecture based solely on attention mechanisms that dropped recurrence and convolutions entirely.[1] It scored 28.4 BLEU on a standard English-to-German translation test and set a single-model record of 41.8 BLEU on English-to-French after training for 3.5 days on eight GPUs.[7] Google said it needed less computation to train and sped up training by up to an order of magnitude.[5]
Step 1: tokens
A Transformer reads text as tokens, which can be whole words, pieces of words or single characters.[8] Each token is represented as an embedding, a list of numbers, and self-attention layers transform one sequence of embeddings into another.[9]
Step 2: self-attention
Attention is how the model decides which other words matter for understanding each word. Take “I arrived at the bank after crossing the river.” To work out that “bank” means a riverbank, the model compares “bank” with every other word and gives each a score; “river” can score highly.[3] The scores then decide how much each word contributes to the model’s new picture of “bank”.[3]
The paper defines an attention function as mapping a query and a set of key-value pairs to an output, all of them vectors.[10] The output is a weighted sum of the values, with each weight computed by a compatibility function of the query with the corresponding key.[10] In a self-attention layer, the queries, keys and values all come from the same place,[10] so each step directly models relationships between all words in a sentence regardless of position.[3]
Step 3: many heads, many layers
One attention calculation captures one kind of relationship. The original Transformer ran eight attention layers in parallel, called heads, so the model could attend to information from different representation subspaces at once.[11] Because each head starts with random parameters, different heads can learn different relationships between words.[4]
A complete model stacks many such layers, each feeding the next.[4] In the original design, the encoder and decoder were each six identical layers, and every layer paired multi-head self-attention with a simple fully connected network applied to each position.[12]
Step 4: keeping track of word order
Because the Transformer has no recurrence or convolution, nothing in it tracks word order by default, so it adds information about each token’s position to its input.[13] Without that, “dog bites man” and “man bites dog” would contain the same tokens with no record of which came first.[13] The original paper used sine and cosine waves of different frequencies as these positional encodings.[13]
Encoders, decoders and text generation
A full Transformer has two halves. An encoder turns input text into an intermediate representation, and a decoder turns that representation into output text, such as a translation.[14] Encoders use bidirectional attention, looking at words on both sides, while decoders that generate text token by token use unidirectional attention, gathering context from one side only.[15]
The decoder generates one symbol at a time and feeds each output back in as input for the next, which the paper calls auto-regressive.[16] Encoder-only and decoder-only variants also exist, and decoder-only models typically excel at generating sequences, such as continuing a conversation.[14] BERT, a 2018 encoder model, pre-trained on unlabeled text using context from both directions and set new records on eleven language tasks.[17] The 2020 GPT-3 paper reported that scaling up such generative language models lets them do new tasks from text prompts alone; see how large language models work.[18]
Strengths and costs
Transformers proved general. The 2020 Vision Transformer showed that a pure Transformer applied to sequences of image patches can perform very well at image classification after pre-training on large amounts of data, with excellent results compared with state-of-the-art convolutional networks while needing substantially less compute to train.[19]
The price is scale. Self-attention makes every token learn its relevance to every other token, so the work grows with the square of the context length.[20] Google’s course notes that Transformers contain hundreds of billions or even trillions of parameters.[21] One analysis argues that memory bandwidth, rather than compute, has become the main bottleneck for AI workloads; see the memory wall.[22] Because generative models produce output that approximates the statistical patterns of their training data, for example by predicting the next token, they can also produce confident but false answers; see AI hallucinations.[23]
Read next
AI hallucinations: why chatbots make things up
Why AI chatbots sometimes state false things confidently, what NIST calls confabulation, what research says about the causes, and how to reduce the risk.
More in Frontier AI
Questions readers ask
What is a Transformer in AI?
A neural network architecture introduced in the 2017 paper "Attention Is All You Need". It is based solely on attention mechanisms and dispenses with the recurrence and convolutions used by earlier language models.[1]
What does attention do?
To compute a new representation for a word, the Transformer compares it with every other word and produces attention scores that set how much each one contributes. In "I arrived at the bank after crossing the river", the word "river" can get a high score when interpreting "bank".[3]
What are queries, keys and values?
The original paper describes attention as mapping a query and a set of key-value pairs to an output. The output is a weighted sum of the values, with each weight based on how well the query matches the corresponding key.[10]
What are attention heads?
Parallel attention layers. The original Transformer used eight, letting the model attend to different kinds of information at different positions at once; different heads can learn different relationships between words.[11][4]
Why do long prompts cost more?
Self-attention makes every token learn the relevance of every other token, so the work grows roughly with the square of the number of tokens in the context.[20]
Are Transformers only used for text?
No. The 2020 Vision Transformer paper showed that a pure Transformer applied to sequences of image patches can perform very well at image classification.[19]
Sources
· 23 claims from 6 publishers
Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.
- [1]
The 2017 paper "Attention Is All You Need" proposed the Transformer, an architecture based solely on attention mechanisms that dispenses with recurrence and convolutions. confirmedas of 2017-06-12
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Abstract (retrieved 2026-11-07)
- [2]
Before the Transformer, recurrent neural networks were the typical architecture for machine translation; they read text one word at a time, which makes decisions that depend on distant words harder and does not suit parallel hardware such as GPUs and TPUs. confirmedas of 2017-08-31
- Transformer: A Novel Neural Network Architecture for Language Understanding · Google Research · Primary source · 2017-08-31 · Accuracy and Efficiency in Language Understanding (retrieved 2026-11-07)
- Transformer: A Novel Neural Network Architecture for Language Understanding · Google Research · Primary source · 2017-08-31 · Accuracy and Efficiency in Language Understanding (retrieved 2026-11-07)
- [3]
To compute a new representation for a word, the Transformer compares it with every other word in the sentence, producing attention scores that set how much each other word contributes; in "I arrived at the bank after crossing the river", the word "river" can receive a high score for "bank". confirmedas of 2017-08-31
- Transformer: A Novel Neural Network Architecture for Language Understanding · Google Research · Primary source · 2017-08-31 · The Transformer (retrieved 2026-11-07)
- Transformer: A Novel Neural Network Architecture for Language Understanding · Google Research · Primary source · 2017-08-31 · The Transformer (retrieved 2026-11-07)
- [4]
A complete Transformer stacks multiple self-attention layers, each with several heads whose parameters start random so that different heads can learn different relationships between words. confirmedas of 2026-11-07
- LLMs: What's a large language model? (Machine Learning Crash Course) · Google for Developers · Primary source · LLMs: What's a large language model?, What is multi-head multi-layer self-attention? (retrieved 2026-11-07)
- LLMs: What's a large language model? (Machine Learning Crash Course) · Google for Developers · Primary source · LLMs: What's a large language model?, What is multi-head multi-layer self-attention? (retrieved 2026-11-07)
- [5]
Google said the Transformer needed less computation to train than earlier translation models and fitted modern machine learning hardware better, speeding up training by up to an order of magnitude. confirmedas of 2017-08-31
- Transformer: A Novel Neural Network Architecture for Language Understanding · Google Research · Primary source · 2017-08-31 · Introduction (retrieved 2026-11-07)
- [6]
Google's machine learning course calls the Transformer the most successful and widely used architecture for building large language models. confirmedas of 2026-11-07
- LLMs: What's a large language model? (Machine Learning Crash Course) · Google for Developers · Primary source · LLMs: What's a large language model?, What's a Transformer? (retrieved 2026-11-07)
- [7]
The original Transformer scored 28.4 BLEU on the WMT 2014 English-to-German translation task and set a single-model record of 41.8 BLEU on English-to-French after training for 3.5 days on eight GPUs. confirmedas of 2017-06-12
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Abstract (retrieved 2026-11-07)
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Abstract (retrieved 2026-11-07)
- [8]
Language models work on tokens, which can be whole words, subwords (parts of a word) or single characters. confirmedas of 2026-11-07
- LLMs: What's a large language model? (Machine Learning Crash Course) · Google for Developers · Primary source · LLMs: What's a large language model? (retrieved 2026-11-07)
- [9]
A self-attention layer transforms a sequence of embeddings, such as token embeddings, into another sequence of embeddings, and self-attention is one of the main building blocks of Transformers. confirmedas of 2026-11-07
- Machine Learning Glossary · Google for Developers · Primary source · Glossary, self-attention (retrieved 2026-11-07)
- Machine Learning Glossary · Google for Developers · Primary source · Glossary, self-attention (retrieved 2026-11-07)
- [10]
The paper describes an attention function as mapping a query and a set of key-value pairs to an output; the output is a weighted sum of the values, with each weight computed from how well the query matches the corresponding key; in a self-attention layer, the queries, keys and values all come from the same place. confirmedas of 2017-06-12
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Section 3.2, Attention (retrieved 2026-11-07)
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Section 3.2, Attention (retrieved 2026-11-07)
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Section 3.2.3, Applications of Attention in our Model (retrieved 2026-11-07)
- [11]
The original Transformer used eight parallel attention layers, or heads; multi-head attention lets the model attend to information from different representation subspaces at different positions at once. confirmedas of 2017-06-12
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Section 3.2.2, Multi-Head Attention (retrieved 2026-11-07)
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Section 3.2.2, Multi-Head Attention (h = 8) (retrieved 2026-11-07)
- [12]
The original Transformer's encoder and decoder were each a stack of six identical layers, each containing a multi-head self-attention mechanism and a position-wise fully connected feed-forward network. confirmedas of 2017-06-12
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Section 3.1, Encoder and Decoder Stacks (N = 6 for both stacks) (retrieved 2026-11-07)
- [13]
Because the Transformer has no recurrence or convolution, it adds positional encodings to its input embeddings so the model can use the order of the tokens; the original paper used sine and cosine functions of different frequencies. confirmedas of 2017-06-12
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Section 3.5, Positional Encoding (retrieved 2026-11-07)
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Section 3.5, Positional Encoding (retrieved 2026-11-07)
- [14]
A full Transformer has an encoder, which turns input text into an intermediate representation, and a decoder, which turns that representation into useful text; encoder-only and decoder-only designs also exist, and decoder-only models typically excel at generating sequences. confirmedas of 2026-11-07
- LLMs: What's a large language model? (Machine Learning Crash Course) · Google for Developers · Primary source · LLMs: What's a large language model?, What's a Transformer? (retrieved 2026-11-07)
- LLMs: What's a large language model? (Machine Learning Crash Course) · Google for Developers · Primary source · LLMs: What's a large language model?, What's a Transformer? (retrieved 2026-11-07)
- [15]
Encoders use bidirectional self-attention, which gathers context from words on both sides, while decoders that generate text token by token use unidirectional self-attention. confirmedas of 2026-11-07
- LLMs: What's a large language model? (Machine Learning Crash Course) · Google for Developers · Primary source · LLMs: What's a large language model?, What is self-attention? (retrieved 2026-11-07)
- [16]
The original Transformer's decoder generates output one symbol at a time, feeding each generated symbol back in as input for the next, which the paper calls auto-regressive. confirmedas of 2017-06-12
- Attention Is All You Need (arXiv 1706.03762v7, full text) · arXiv · Primary source · 2017-06-12 · Section 3, Model Architecture (retrieved 2026-11-07)
- [17]
BERT (Bidirectional Encoder Representations from Transformers), introduced in 2018, pre-trains deep bidirectional representations from unlabeled text and can be fine-tuned with one extra output layer; it set new state-of-the-art results on eleven language tasks. confirmedas of 2018-10-11
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · arXiv · Primary source · 2018-10-11 · Abstract (retrieved 2026-11-07)
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · arXiv · Primary source · 2018-10-11 · Abstract (retrieved 2026-11-07)
- [18]
The 2020 GPT-3 paper reported that scaling up language models greatly improves few-shot performance, with the 175-billion-parameter model doing new tasks from text prompts alone, without gradient updates. confirmedas of 2020-05-28
- Language Models are Few-Shot Learners · arXiv (Brown et al.) · Primary source · 2020-05-28 · Abstract (retrieved 2026-10-10)
- [19]
The 2020 Vision Transformer (ViT) paper showed that a pure Transformer applied to sequences of image patches can perform very well at image classification, matching state-of-the-art convolutional networks with substantially less training compute when pre-trained on large datasets. confirmedas of 2020-10-22
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale · arXiv · Primary source · 2020-10-22 · Abstract (retrieved 2026-11-07)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale · arXiv · Primary source · 2020-10-22 · Abstract (retrieved 2026-11-07)
- [20]
Because self-attention makes every token in the context learn the relevance of every other token, its cost grows with the square of the number of tokens. confirmedas of 2026-11-07
- LLMs: What's a large language model? (Machine Learning Crash Course) · Google for Developers · Primary source · LLMs: What's a large language model?, Big O for LLMs (retrieved 2026-11-07)
- [21]
Google's course says Transformers contain hundreds of billions or even trillions of parameters. confirmedas of 2026-11-07
- LLMs: What's a large language model? (Machine Learning Crash Course) · Google for Developers · Primary source · LLMs: What's a large language model?, Why are Transformers so large? (retrieved 2026-11-07)
- [22]
The same analysis argues that memory bandwidth, rather than compute, has become the primary bottleneck for AI workloads, especially when serving models. confirmedas of 2024-03-21
- AI and Memory Wall · arXiv (published in IEEE Micro) · Primary source · 2024-03-21 · Abstract (retrieved 2026-10-10)
- [23]
NIST describes confabulations as a natural result of how generative models are designed, since they generate outputs that approximate the statistical distribution of their training data, for example by predicting the next word. confirmedas of 2024-07-26
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 · National Institute of Standards and Technology · Primary source · 2024-07-26 · NIST AI 600-1, Section 2.2, Confabulation, PDF p. 10 (retrieved 2026-11-07)
Revision history (1)
- Page created.
Created Nov 7, 2026. Last reviewed by an editor on Nov 8, 2026. Next scheduled review: Feb 7, 2027.
Cite this page
"How transformer models work: attention, heads and layers." ContentLora, updated Nov 7, 2026. https://contentlora.com/explain/how-transformers-work
Spotted an error? Suggest a correction or emailcorrections@contentlora.com.