Attention Is All You Need: A Complete Breakdown of the Transformer Architecture

Attention Is All You Need (Vaswani et al., 2017) is the architectural starting point of modern large language models: the paper itself addresses the seq2seq problem of machine translation, proposing the Transformer, which completely discards recurrence (RNNs) and convolution in favor of stacking only attention mechanisms. Today, the vast majority of large language models are variants of this architecture, but most of them keep only the decoder and make quite a few adjustments to the details — they aren’t fully identical to the original paper. This article first breaks the original paper’s architecture down to the finest detail, then explains exactly what modern decoder-only LLMs have changed.

1. Why Use Attention Instead of an RNN

When an RNN processes a sequence, each time step’s computation depends on the previous step’s hidden state, making it inherently sequential: computing the $t$th token requires first finishing the $(t-1)$th. This means an RNN can’t parallelize along the sequence-length dimension, so a GPU’s parallel compute capability goes unused when training long sequences, and long-range dependencies must be carried step by step through the hidden state, prone to decaying as distance increases.

The Transformer’s core idea is: letting information flow between any two positions in a sequence require only a single step of attention computation, without needing to propagate step by step through intermediate positions. During training (when the full target sequence is available), attention at every position can be computed fully in parallel; a longer sequence doesn’t increase the number of steps needed to pass information between positions. The cost is that self-attention’s compute grows quadratically with sequence length — a problem that a large body of subsequent long-sequence optimization work has had to address, though it isn’t something the paper itself tackles.

2. Scaled Dot-Product Attention

2.1 Query, Key, Value, and tensor shapes

Attention takes three groups of vectors as input: Query ($Q$), Key ($K$), and Value ($V$). Within a single attention head, if the sequence length is $n$, then $Q\in\mathbb R^{n\times d_k}$, and $K,V\in\mathbb R^{m\times d_k}$ and $\mathbb R^{m\times d_v}$ respectively ($m$ is the length of the sequence being queried; in self-attention $m=n$, while in encoder-decoder cross-attention $m$ is the length of the encoder’s output sequence). The formula is:

$$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

$QK^\top$ computes a similarity score between every query and every key, with shape $n\times m$; applying softmax to each row gives every query position’s attention weights over all key positions; multiplying the weights by $V$ is equivalent to a weighted average of the value vectors using those weights, giving an output of shape $n\times d_v$.

2.2 Why divide by $\sqrt{d_k}$

If each component of $Q,K$ is approximately independent with mean 0 and variance 1, then each dot product in $QK^\top$ is a sum of $d_k$ terms, and its variance grows linearly with $d_k$. When $d_k$ is large, the dot-product values themselves become large, pushing softmax into a region where gradients are vanishingly small, making training difficult. Dividing by $\sqrt{d_k}$ pulls the dot product’s variance back to a constant order of magnitude — this is explicitly noted in the paper, and is also where the word “scaled” comes from.

3. Multi-Head Attention

A single attention head can only learn one kind of similarity metric. Multi-head attention projects $Q,K,V$ each with $h$ different linear projections $W_i^Q,W_i^K,W_i^V$ down to a lower dimension, computes the attention output for all $h$ heads in parallel, then concatenates the $h$ outputs and projects them back to the original dimension with another linear layer $W^O$:

$$\operatorname{MultiHead}(Q,K,V)=\operatorname{Concat}(\text{head}_1,\dots,\text{head}_h)W^O,\quad \text{head}_i=\operatorname{Attention}(QW_i^Q,KW_i^K,VW_i^V)$$

The original paper’s base model uses $d_{\text{model}}=512$ and $h=8$, giving each head $d_k=d_v=d_{\text{model}}/h=64$. Splitting into several narrower heads instead of using one wide head lets the model attend to different kinds of relationships in different subspaces simultaneously (for example, some heads tend to capture syntactic structure, others tend to capture coreference), while the total compute is roughly the same as a single full-width head.

4. The Block Structure of Encoder and Decoder

4.1 The encoder block: two sublayers

Each encoder block consists of two sublayers: self-attention, followed by a position-wise feed-forward network (see Section 5). Each sublayer is wrapped in a residual connection plus Layer Normalization. The original paper uses post-LN: compute the sublayer itself, add the residual back, then normalize — that is, $\operatorname{LayerNorm}(x+\operatorname{Sublayer}(x))$, in the order “sublayer → add residual → LayerNorm,” with normalization happening after the residual addition. The base model stacks 6 such encoder blocks.

4.2 The decoder block: three sublayers, including cross-attention

A decoder block has one more sublayer than the encoder, three in total: the first is self-attention with a causal mask (see 4.3); the second is encoder-decoder cross-attention — this sublayer’s $Q$ comes from the previous decoder sublayer’s output, while $K$ and $V$ come from the encoder’s final layer output, letting every decoder position query the entire input sequence; the third is the same FFN as in the encoder. All three sublayers are likewise each wrapped in a residual connection plus post-LN.

4.3 The causal mask: keeping causality consistent between training and inference

When the decoder generates the $t$th token, it can only depend on the already-generated tokens from position 1 to $t-1$ — it cannot see future tokens. Training uses teacher forcing, feeding in the entire target sequence at once; without any special handling, self-attention would let every position see the tokens after it, effectively cheating. The fix is to set the scores in the upper triangle (corresponding to “the query position comes before the position being looked at”) to $-\infty$ right after computing $QK^\top$ and before applying softmax; after softmax, the weights at these positions become 0, effectively forcing every position to attend only to itself and earlier positions. This mechanism keeps the parallel computation used during training consistent with the “can only see what’s already been generated, one step at a time” constraint at inference time.

5. The Position-wise Feed-Forward Network

The FFN applies the same two-layer fully connected transformation, with the same parameters, independently and identically to every position in the sequence:

$$\operatorname{FFN}(x)=\max(0,xW_1+b_1)W_2+b_2$$

Both the input and output dimensions are $d_{\text{model}}$, while the hidden dimension $d_{ff}$ in between is usually much larger than $d_{\text{model}}$ (the base model uses $d_{ff}=2048$, four times $d_{\text{model}}=512$), with ReLU as the activation function. The FFN isn’t what converts attention’s $Q,K,V$ into the output dimension — the attention sublayer’s own output projection $W^O$ already does that — the FFN’s role is to apply a nonlinear, position-wise transformation to the result computed by the attention sublayer at each position, giving the model, after it has mixed information across positions, the capacity to further process a single position’s representation nonlinearly. Modern models often swap ReLU for variants like GELU or SwiGLU, but the role of “a two-layer nonlinear transformation applied per position” remains unchanged.

6. Embeddings and Positional Encoding

6.1 Input embeddings, weight scaling, and weight tying with the output layer

An input token is first looked up and converted into a $d_{\text{model}}$-dimensional embedding vector; in the paper, this embedding is then multiplied by $\sqrt{d_{\text{model}}}$, so its numerical scale matches the positional encoding added afterward. On the output side, the decoder’s final representation is projected back to the vocabulary size and passed through softmax to predict the next token; this projection matrix shares the same weights as the input embedding table (weight tying), reducing the parameter count and keeping the input and output vector spaces in a consistent geometric relationship.

6.2 The original paper’s sinusoidal positional encoding

Self-attention itself is insensitive to input order — shuffling the input positions doesn’t change the weighted average that attention computes — so positional information needs to be injected separately. The original paper uses a fixed (non-learned) sinusoidal function:

$$PE_{(pos,2i)}=\sin\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right),\quad PE_{(pos,2i+1)}=\cos\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$

$pos$ is the position in the sequence, and $i$ is the dimension index. This encoding is added directly to the input embedding. One reason for choosing a sinusoidal function is that it lets the positional encoding at any fixed offset $k$, $PE_{pos+k}$, be written as a linear function of $PE_{pos}$, which in theory makes it easier for the model to learn patterns that depend on relative position, and also gives the model a chance to handle inputs longer than any sequence it saw during training.

Most modern decoder-only LLMs don’t use this kind of absolute positional encoding, instead using relative-position methods such as RoPE (Rotary Positional Encoding) or ALiBi, which fold positional information directly into the attention score itself rather than adding it to the input embedding; these methods generally extrapolate to long sequences better than the original paper’s sinusoidal encoding, and were developed after the original paper — they are not part of the original Transformer architecture.

7. Differences Between Training and Inference

The original paper’s training uses teacher forcing (feeding in the true preceding tokens at every step, rather than the model’s own prediction from the previous step), the Adam optimizer paired with a learning-rate schedule that linearly warms up and then decays based on step count, dropout applied after summing each sublayer’s output with the embeddings, and label smoothing ($\epsilon_{ls}=0.1$, making the target distribution not a hard one-hot, to avoid the model becoming overconfident in its predictions). Translation quality is evaluated using beam search.

At inference time, there’s no true target sequence to reference, so the decoder must be autoregressive: generating one token at a time, appending it to the already-generated sequence, then feeding that back into the model to generate the next token. The original paper’s decoding methods are only greedy search and beam search; sampling strategies like top-k, top-p (nucleus sampling), and temperature are methods proposed by later research (roughly after 2018, developed alongside the needs of open-ended text generation) for open-ended generation tasks — they aren’t part of the original 2017 paper, but they are standard practice for modern decoder-only LLM inference: temperature scales how sharp the logits distribution is, top-k samples only from the $k$ highest-probability tokens, top-p instead takes the smallest candidate set whose cumulative probability reaches $p$ before sampling, and greedy decoding (always picking the highest-probability token) can be seen as the special case of top-k=1 or temperature $\to0$.

8. From the Original Paper to Modern Decoder-Only LLMs

Modern large language models (the GPT series and its successors) mostly keep only the decoder, dropping the encoder and cross-attention, relying on causal self-attention to handle both input and generation at once — this is where the name “decoder-only” comes from. Beyond this architectural trade-off, other common concrete differences include: replacing post-LN with pre-LN (placing LayerNorm before the sublayer, so the residual path itself never passes through normalization, $x+\operatorname{Sublayer}(\operatorname{LayerNorm}(x))$), which noticeably improves training stability for deep models and is one of the key reasons very large models can be stacked to dozens or even hundreds of layers; swapping the positional encoding for RoPE or ALiBi; and swapping the FFN’s activation function for GELU or SwiGLU. Most of these changes are engineering improvements proposed gradually after 2017, aimed at training deeper and larger models — understanding what each component of the original paper actually does is a prerequisite for judging exactly what problem each of these improvements solves.