The point of this document

Everything in Attention Is All You Need (Vaswani et al., 2017) is a stack of matrix multiplications. The architecture diagram looks intimidating because \(d_{\text{model}} = 512\), there are 8 heads, and 6 identical layers. None of that changes the shapes of the operations.

So we shrink it until it fits on a napkin:

Shrinking the paper down to a hand calculation
Quantity Paper Here
\(d_{\text{model}}\) 512 2
heads \(h\) 8 1 (then 2, briefly)
\(d_k = d_v = d_{\text{model}}/h\) 64 2
\(d_{\text{ff}}\) 2048 4
encoder layers \(N\) 6 1
sequence length up to 512 3

Everything below is arithmetic you could do on paper. We only follow the encoder and the self-attention mechanism inside it.

The corpus

Two sentences, three tokens each:

S1: cats chase mice

S2: dogs chase cats

These are chosen so the same word (chase) appears in both, and the same word (cats) plays two different grammatical roles: subject in S1, object in S2. That contrast is the whole punchline of the document, so keep an eye on it.

Vocabulary: \(\mathcal{V} = \{\texttt{cats}, \texttt{dogs}, \texttt{mice}, \texttt{chase}\}\), so \(|\mathcal{V}| = 4\).

Step 0: The embedding table

The embedding layer is a lookup table \(E \in \mathbb{R}^{|\mathcal{V}| \times d_{\text{model}}} = \mathbb{R}^{4 \times 2}\). Each row is one word. Because \(d_{\text{model}} = 2\), we can name both axes:

  • axis 1 = “predator-ness”: how much of a hunter is this thing?
  • axis 2 = “action-ness”: how much is this word a verb?

\[ E = \begin{array}{r|rr} & \text{pred} & \text{act} \\ \hline \texttt{dogs} & 2 & 0 \\ \texttt{cats} & 1 & 0 \\ \texttt{mice} & -1 & 0 \\ \texttt{chase} & 0 & 1 \end{array} \]

Dogs hunt, cats hunt-but-get-hunted, mice only get hunted, and chase is pure verb. In a real model these numbers are learned; here we hand-pick them so that the attention pattern later is legible.

What we’re skipping

The paper multiplies the embeddings by \(\sqrt{d_{\text{model}}}\) before adding positional encodings. We omit that scale factor (imagine it absorbed into the table above) purely to keep the arithmetic clean.

Looking up the tokens gives one matrix per sentence, of shape \((\text{seq}, d_{\text{model}}) = (3, 2)\):

\[ E_{S1} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ -1 & 0 \end{pmatrix} \qquad E_{S2} = \begin{pmatrix} 2 & 0 \\ 0 & 1 \\ 1 & 0 \end{pmatrix} \]

Step 1: Positional encoding

Self-attention is a set operation: permute the rows of the input and the output rows just permute with them. Nothing in \(QK^\top\) knows about word order. Position has to be injected into the vectors themselves.

The paper uses fixed sinusoids,

\[ PE_{(pos,\,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right), \qquad PE_{(pos,\,2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) \]

which with \(d_{\text{model}}=2\) degenerates to a single frequency, \(PE(pos) = (\sin pos,\ \cos pos)\): and that would swamp our two semantic axes with values of size 1. So we use a toy positional encoding: a small ramp

\[ PE(pos) = (0.1 \cdot pos,\ \ 0.1 \cdot pos) \]

Small enough to perturb rather than destroy the meaning, distinct at every position. It plays the same structural role.

The encoder input is the sum \(X = E + PE\), still \((3, 2)\):

\[ X_{S1} = \begin{array}{l|rr} & \text{pred} & \text{act} \\ \hline \texttt{cats}_{\,0} & 1.0 & 0.0 \\ \texttt{chase}_{\,1} & 0.1 & 1.1 \\ \texttt{mice}_{\,2} & -0.8 & 0.2 \end{array} \qquad X_{S2} = \begin{array}{l|rr} & \text{pred} & \text{act} \\ \hline \texttt{dogs}_{\,0} & 2.0 & 0.0 \\ \texttt{chase}_{\,1} & 0.1 & 1.1 \\ \texttt{cats}_{\,2} & 1.2 & 0.2 \end{array} \]

Note that chase has the same input vector \((0.1, 1.1)\) in both sentences. It is at position 1 in both. Whatever makes it mean something different in S2 has to come from attention.

Batching and shapes

Stack the two sentences and you get the tensor every implementation actually passes around: \((\text{batch}, \text{seq}, d_{\text{model}}) = (2, 3, 2)\). Our sentences are the same length, so no padding is needed. If they weren’t, we’d pad the short one and add a padding mask (\(-\infty\) in the score matrix) so attention never spends weight on filler tokens.

Step 2: Q, K, V: three views of the same token

Self-attention is a soft dictionary lookup. Each token emits:

  • a query \(q\): what am I looking for?
  • a key \(k\): what do I advertise about myself?
  • a value \(v\): what do I hand over if you attend to me?

These are three different linear projections of the same row of \(X\):

\[ Q = XW^Q, \qquad K = XW^K, \qquad V = XW^V \]

with \(W^Q, W^K \in \mathbb{R}^{d_{\text{model}} \times d_k} = \mathbb{R}^{2\times 2}\) and \(W^V \in \mathbb{R}^{2 \times 2}\). So \(Q, K, V\) are all \((3,2)\).

We pick, by hand:

\[ W^Q = \begin{pmatrix} 0 & 1 \\ 1 & 0\end{pmatrix} \qquad W^K = \begin{pmatrix} 1 & 0 \\ 0 & 1\end{pmatrix} \qquad W^V = \begin{pmatrix} 1 & 0 \\ 0 & 2\end{pmatrix} \]

\(W^Q\) swaps the two axes. That single choice creates the entire story:

  • A token’s key is its identity: \(k = (\text{pred}, \text{act})\).
  • A token’s query is its identity flipped: \(q = (\text{act}, \text{pred})\).

So a verb (high action-ness) emits a query pointing along the predator axis: it goes looking for a hunter. A noun (high predator-ness) emits a query pointing along the action axis: it goes looking for a verb. Nouns seek verbs, verbs seek their agent. That is a crude cartoon of syntax, and it is the kind of structure real attention heads do learn.

For S1:

\[ Q_{S1} = \begin{pmatrix} 0.0 & 1.0 \\ 1.1 & 0.1 \\ 0.2 & -0.8 \end{pmatrix} \quad K_{S1} = \begin{pmatrix} 1.0 & 0.0 \\ 0.1 & 1.1 \\ -0.8 & 0.2 \end{pmatrix} \quad V_{S1} = \begin{pmatrix} 1.0 & 0.0 \\ 0.1 & 2.2 \\ -0.8 & 0.4 \end{pmatrix} \]

Step 3: Scores: every token queries every token

\[ S = QK^\top \in \mathbb{R}^{3 \times 3} \]

Row \(i\), column \(j\) is \(q_i \cdot k_j\): how much should token \(i\) pay attention to token \(j\)? This is the only place in the entire encoder where positions talk to each other. Note the shape change: we went from \((3,2)\) to \((3,3)\): the matrix is sequence-length squared, which is exactly why transformers are \(O(n^2)\) in memory.

Take one entry by hand. Query of chase is \((1.1, 0.1)\); key of cats is \((1.0, 0.0)\):

\[ S_{\texttt{chase},\,\texttt{cats}} = 1.1(1.0) + 0.1(0.0) = 1.1 \]

Doing all nine:

\[ S_{S1} = \begin{array}{l|rrr} q \backslash k & \texttt{cats} & \texttt{chase} & \texttt{mice} \\ \hline \texttt{cats} & 0.00 & 1.10 & 0.20 \\ \texttt{chase} & 1.10 & 0.22 & -0.86 \\ \texttt{mice} & 0.20 & -0.86 & -0.32 \end{array} \]

chase scores highest against cats (1.10): the verb found its subject.

Why divide by \(\sqrt{d_k}\)

\[ S^{\text{scaled}} = \frac{QK^\top}{\sqrt{d_k}} \]

If the components of \(q\) and \(k\) are roughly independent with unit variance, their dot product has variance \(d_k\): so scores grow with dimension. Large scores push softmax into a corner where one weight is \(\approx 1\) and the rest are \(\approx 0\), and the gradient of softmax there is \(\approx 0\). Dividing by \(\sqrt{d_k}\) keeps the variance at \(\approx 1\) and the gradients alive. Here \(\sqrt{d_k} = \sqrt{2} \approx 1.414\):

\[ S^{\text{scaled}}_{S1} = \begin{array}{l|rrr} & \texttt{cats} & \texttt{chase} & \texttt{mice} \\ \hline \texttt{cats} & 0.000 & 0.778 & 0.141 \\ \texttt{chase} & 0.778 & 0.156 & -0.608 \\ \texttt{mice} & 0.141 & -0.608 & -0.226 \end{array} \]

Step 4: Softmax: scores become a weighted average

Softmax is applied row-wise, so each row becomes a probability distribution over the tokens the row is looking at:

\[ A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right), \qquad \sum_j A_{ij} = 1 \]

For the chase row: \(e^{0.778} = 2.18\), \(e^{0.156} = 1.17\), \(e^{-0.608} = 0.54\), summing to \(3.89\). Dividing:

\[ A_{S1} = \begin{array}{l|rrr} & \texttt{cats} & \texttt{chase} & \texttt{mice} \\ \hline \texttt{cats} & 0.231 & 0.503 & 0.266 \\ \textbf{\texttt{chase}} & \textbf{0.560} & 0.300 & 0.140 \\ \texttt{mice} & 0.462 & 0.218 & 0.320 \end{array} \]

\[ A_{S2} = \begin{array}{l|rrr} & \texttt{dogs} & \texttt{chase} & \texttt{cats} \\ \hline \texttt{dogs} & 0.142 & 0.671 & 0.188 \\ \textbf{\texttt{chase}} & \textbf{0.558} & 0.138 & 0.304 \\ \texttt{cats} & 0.250 & 0.486 & 0.264 \end{array} \]

Read the bold rows side by side. chase: with an identical input vector in both sentences: spends 56% of its attention on cats in S1 and 56% on dogs in S2. And cats, which received 56% of the verb’s attention as a subject in S1, receives only 30% as an object in S2. The same token is being read differently because of its context. Nothing else in the encoder does this.

Step 5: The output: mix the values

\[ Z = AV \in \mathbb{R}^{3 \times 2} \]

Shape is back to \((\text{seq}, d_{\text{model}})\): attention is shape preserving, which is what lets you stack it. Each output row is a convex combination of value vectors, weighted by the attention row.

For chase in S1, with \(A_{\texttt{chase}} = (0.560, 0.300, 0.140)\) and values \((1.0, 0.0)\), \((0.1, 2.2)\), \((-0.8, 0.4)\):

\[ z_{\texttt{chase}} = 0.560\begin{pmatrix}1.0\\0.0\end{pmatrix} + 0.300\begin{pmatrix}0.1\\2.2\end{pmatrix} + 0.140\begin{pmatrix}-0.8\\0.4\end{pmatrix} = \begin{pmatrix}0.48\\0.72\end{pmatrix} \]

Doing the same for S2 gives \(z_{\texttt{chase}} = (1.50, 0.42)\).

The punchline

chase entered both sentences as \((0.1,\ 1.1)\): barely any predator-ness. It leaves self-attention as:

  • S1 (cats chase mice): \((0.48,\ 0.72)\)
  • S2 (dogs chase cats): \((1.50,\ 0.42)\)

Recall cats has predator-ness 1 and dogs has predator-ness 2. The contextualized chase vector has absorbed the identity of its subject. The token no longer means “the verb chase”; it means “a chasing done by this particular animal.” That is what “contextual embedding” means, and it is produced by nothing more than a weighted average.

Full outputs:

\[ Z_{S1} = \begin{pmatrix} 0.07 & 1.21 \\ 0.48 & 0.72 \\ 0.23 & 0.61 \end{pmatrix} \qquad Z_{S2} = \begin{pmatrix} 0.58 & 1.55 \\ 1.50 & 0.42 \\ 0.87 & 1.17 \end{pmatrix} \]

The whole thing in one line

\[ \boxed{\ \operatorname{Attention}(Q,K,V) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V\ } \]

The shape journey, which is the thing to memorize:

\[ \underbrace{(3,2)}_{X} \xrightarrow{\ \times W^{Q,K,V}\ } \underbrace{(3,2)}_{Q,K,V} \xrightarrow{\ QK^\top\ } \underbrace{(3,3)}_{\text{scores}} \xrightarrow{\ \text{softmax}\ } \underbrace{(3,3)}_{A} \xrightarrow{\ \times V\ } \underbrace{(3,2)}_{Z} \]

In, out, same shape. The \((3,3)\) in the middle is where all the interaction happens, and it is the only part that scales badly.

Step 6: Multi-head attention

One head can learn one relation. The paper runs \(h\) of them in parallel on lower-dimensional projections, so the cost is roughly the same as one full-width head:

\[ \text{head}_i = \operatorname{Attention}(XW_i^Q, XW_i^K, XW_i^V), \qquad \operatorname{MultiHead}(X) = \operatorname{Concat}(\text{head}_1, \dots, \text{head}_h)\,W^O \]

With \(d_{\text{model}}=2\) and \(h=2\) we get \(d_k = d_v = d_{\text{model}}/h = 1\). Each head projects our 2-D tokens down to one number:

Head | \(W_i^Q\) | \(W_i^K\) | what it computes |

||||| | 1 | \(\binom{0}{1}\) | \(\binom{1}{0}\) | score \(=\) action-ness\((i)\ \times\) predator-ness\((j)\) → verbs find their agent | | 2 | \(\binom{1}{0}\) | \(\binom{0}{1}\) | score \(=\) predator-ness\((i)\ \times\) action-ness\((j)\) → nouns find the verb |

: Two heads, one syntactic relation each {.striped}

Shapes:

\[ \underbrace{(3,2)}_{X} \times \underbrace{(2,1)}_{W_i^Q} = \underbrace{(3,1)}_{Q_i} \ \Longrightarrow\ \text{head}_i \in \mathbb{R}^{3\times 1} \ \Longrightarrow\ \operatorname{Concat} \in \mathbb{R}^{3\times 2} \ \xrightarrow{\ W^O \in \mathbb{R}^{2\times 2}\ } \mathbb{R}^{3\times 2} \]

Each head still produces a \((3,3)\) attention matrix: you get \(h\) of them, which is exactly what attention-visualization tools plot. The concatenation restores \(d_{\text{model}}\), and \(W^O\) lets the heads’ findings mix. Shape in = shape out, again.

Step 7: Residual connection and layer norm

Every sublayer in the encoder is wrapped:

\[ \text{output} = \operatorname{LayerNorm}\big(x + \operatorname{Sublayer}(x)\big) \]

The residual \(x + \operatorname{Sublayer}(x)\) is what makes 6 layers (or 96) trainable: gradients get a highway back to the input, and each sublayer only has to learn a correction to the running representation (“the residual stream”) rather than rebuild it. Note this addition is only well-defined because attention is shape-preserving.

\[ X_{S1} + Z_{S1} = \begin{pmatrix} 1.0 & 0.0 \\ 0.1 & 1.1 \\ -0.8 & 0.2 \end{pmatrix} + \begin{pmatrix} 0.07 & 1.21 \\ 0.48 & 0.72 \\ 0.23 & 0.61 \end{pmatrix} = \begin{pmatrix} 1.07 & 1.21 \\ 0.58 & 1.82 \\ -0.57 & 0.81 \end{pmatrix} \]

LayerNorm then standardizes each token vector across its features (not across the batch):

\[ \operatorname{LN}(x) = \gamma \odot \frac{x - \mu}{\sigma + \epsilon} + \beta, \qquad \mu, \sigma \text{ computed over the } d_{\text{model}} \text{ features of one token} \]

Where our toy breaks: and why that’s instructive

Apply LayerNorm to a 2-dimensional vector \((a,b)\). The mean is \((a+b)/2\) and the standard deviation is \(|a-b|/2\), so

\[ \operatorname{LN}(a,b) = \big(\operatorname{sign}(a-b),\ -\operatorname{sign}(a-b)\big) \]

Every token collapses to \((-1, 1)\) or \((1, -1)\). All three of our S1 rows have \(a < b\), so LayerNorm maps them all to the same vector and destroys everything attention just computed.

This is not a bug in LayerNorm; it’s a demonstration that \(d_{\text{model}}=2\) is absurdly small. Normalizing away 1 degree of freedom (the mean) and 1 more (the scale) leaves \(d_{\text{model}} - 2 = 0\) dimensions of information. At \(d_{\text{model}} = 512\) you lose 2 of 512 and nobody notices.

For the rest of the walkthrough we carry the un-normalized residual stream forward, so the numbers stay meaningful. Just remember that a real encoder normalizes here.

Step 8: The position-wise feed-forward network

\[ \operatorname{FFN}(x) = \max(0,\ xW_1 + b_1)\,W_2 + b_2 \]

Two crucial words: position-wise. The same small MLP is applied to each token vector independently: it is a \(1\times1\) convolution over the sequence. It cannot move information between positions. That is attention’s job, and attention’s only job. The FFN’s job is to be the nonlinearity: it takes the mixed-up vector attention produced and transforms it.

Shapes, with \(d_{\text{ff}} = 4\) (the paper uses \(4 \times d_{\text{model}}\); we use 4 so the matrices print):

\[ \underbrace{(3,2)}_{\text{input}} \xrightarrow{\ W_1 \in \mathbb{R}^{2\times4}\ } \underbrace{(3,4)}_{\text{expand}} \xrightarrow{\ \text{ReLU}\ } \underbrace{(3,4)}_{\text{gate}} \xrightarrow{\ W_2 \in \mathbb{R}^{4\times2}\ } \underbrace{(3,2)}_{\text{project back}} \]

Wide in the middle, back to \(d_{\text{model}}\) at the ends. Take

\[ W_1 = \begin{pmatrix} 1 & -1 & 0 & 0 \\ 0 & 0 & 1 & -1 \end{pmatrix}, \quad b_1 = 0 \]

so that the hidden layer is \(\operatorname{ReLU}(x_1, -x_1, x_2, -x_2)\): it splits each channel into its positive and negative parts. That’s the entire trick behind ReLU MLPs: rectification lets \(W_2\) treat a feature differently depending on its sign. Choosing

\[ W_2 = \begin{pmatrix} 2 & 0 \\ -1 & 0 \\ 0 & 1 \\ 0 & -1\end{pmatrix}, \quad b_2 = 0 \qquad \Longrightarrow \qquad \operatorname{FFN}(x) = \big(\,2\max(x_1,0) + \min(x_1,0),\ \ x_2\,\big) \]

gives a “leaky amplifier”: double predator-ness if positive, leave it alone if negative. For chase in S1, the residual stream value is \((0.58, 1.82)\):

\[ h = \operatorname{ReLU}(0.58,\ -0.58,\ 1.82,\ -1.82) = (0.58,\ 0,\ 1.82,\ 0) \ \xrightarrow{W_2}\ (1.16,\ 1.82) \]

Then the second residual connection:

\[ \text{layer output}_{\texttt{chase}} = (0.58, 1.82) + (1.16, 1.82) = (1.73,\ 3.63) \]

Doing all rows:

\[ \text{Encoder}_{S1} \approx \begin{array}{l|rr} \texttt{cats} & 3.21 & 2.43 \\ \texttt{chase} & 1.73 & 3.63 \\ \texttt{mice} & -1.14 & 1.62 \end{array} \qquad \text{Encoder}_{S2} \approx \begin{array}{l|rr} \texttt{dogs} & 7.73 & 3.10 \\ \texttt{chase} & 4.79 & 3.05 \\ \texttt{cats} & 6.20 & 2.75 \\ \end{array} \]

\((3,2)\) in, \((3,2)\) out.

Step 9: Stack it

One encoder layer is exactly:

x → multi-head self-attention → add x → LayerNorm
  → position-wise FFN         → add    → LayerNorm

Because the output shape equals the input shape, you can feed it straight into another identical layer with its own weights. The paper uses \(N = 6\). The attention pattern in layer 1 tends to be shallow (nearby words, syntax); by layer 6 the “tokens” are really composite objects that have already absorbed their neighborhoods, and heads attend to much more abstract relations.

The encoder’s final \((\text{seq}, d_{\text{model}})\) output is the sequence of contextual embeddings. In the full model these are handed to the decoder’s cross-attention as its \(K\) and \(V\): but that’s a different document.

What to take away

  1. Attention is a soft lookup. Queries dot keys, softmax the result, average the values.
  2. It is the only place tokens interact. Embedding, FFN, LayerNorm, residuals are all per-token.
  3. It is shape-preserving. \((\text{seq}, d_{\text{model}})\) in, out. That is why it stacks and why residuals work.
  4. The \((\text{seq}, \text{seq})\) score matrix is the bottleneck: quadratic in sequence length.
  5. The same token gets different vectors in different sentences. That’s the whole game.