Evaluating a Recurrent Neural Network by Hand

From raw text to a prediction, one number at a time

This notebook walks through the forward pass of a vanilla RNN in complete detail. We are not training anything here. We assume someone has already handed us a set of weight matrices, and our only job is to answer one question:

Given a sentence, how does an RNN turn it into a number?

Everything is small enough to do with a pencil. Every hand calculation is immediately mirrored with Keras code, so you can see that the library is doing exactly the arithmetic you just did yourself.

Roadmap

Step What we do Result
1 Define a two-document corpus raw strings
2 Tokenize and stem lists of normalized words
3 Build the vocabulary word \(\to\) index map
4 Integer-encode shape (samples, timesteps)
5 One-hot encode shape (2, 4, 5)
6 Dense embed shape (2, 4, 2)
7 Unroll the RNN by hand \(h_1, h_2, h_3, h_4\)
8 Apply the output layer a single probability
9 Reproduce all of it in Keras identical numbers
import numpy as np

np.set_printoptions(precision=4, suppress=True)

1. The corpus

Our entire dataset is two sentences. That is deliberate: a corpus this small means every tensor we build can be printed in full and inspected with the naked eye.

corpus = [
    "I love deep learning",
    "Deep Learning Loves Me",
]

for i, doc in enumerate(corpus):
    print(f"Document {i}: {doc!r}")
Document 0: 'I love deep learning'
Document 1: 'Deep Learning Loves Me'

2. Tokenization and stemming

A neural network cannot consume a Python string. The first move is always to break the string into tokens.

Our tokenizer does two normalization jobs.

Lowercase. Without it, "Deep" in document 1 and "deep" in document 0 would become two unrelated vocabulary entries, and the model would have no way to know they are the same word.

Stem. Document 0 has love and document 1 has loves. These are the same word wearing different grammatical clothing, and we would like the model to treat them as one. A stemmer strips inflectional endings to reduce a word to its root form. Real stemmers (Porter, Snowball) apply dozens of ordered rewrite rules; here our corpus is so small that a single rule does the job, so we just chop a trailing s:

\[\texttt{loves} \;\longrightarrow\; \texttt{love}\]

This is a crude hand-rolled stand-in, but it is behaving exactly the way a real stemmer would on this input, and it buys us the same thing a real one does: love and loves collapse into a single vocabulary entry, so whatever the model learns about one automatically applies to the other.

(Being crude, it would also mangle words like "this" into "thi". We guard slightly with a length check. In production you would reach for nltk.stem.PorterStemmer and not think about it again.)

def stem(token):
    '''A one-rule stemmer: drop a trailing 's'. Stands in for Porter/Snowball.'''
    if token.endswith("s") and len(token) > 3:
        return token[:-1]
    return token


def tokenize(text):
    return [stem(tok) for tok in text.lower().split()]


tokenized = [tokenize(doc) for doc in corpus]

for i, doc in enumerate(corpus):
    raw = doc.lower().split()
    print(f"Document {i}")
    print(f"    raw      : {raw}")
    print(f"    stemmed  : {tokenized[i]}")

print("\nEffect of stemming:", stem("loves"), "<- 'loves' and 'love' are now the SAME token")
Document 0
    raw      : ['i', 'love', 'deep', 'learning']
    stemmed  : ['i', 'love', 'deep', 'learning']
Document 1
    raw      : ['deep', 'learning', 'loves', 'me']
    stemmed  : ['deep', 'learning', 'love', 'me']

Effect of stemming: love <- 'loves' and 'love' are now the SAME token

Notice that both documents happen to have exactly 4 tokens. This is a lucky accident that lets us skip padding entirely, so we can keep our attention on the RNN itself.

In the language of the standard tensor shape, that 4 is our number of time steps, \(T = 4\). The RNN will read each document one word at a time, so “time” here simply means “position in the sentence.”

3. Building the vocabulary

The vocabulary is the sorted-out set of every distinct token in the corpus, with an integer index assigned to each. We build it by walking the corpus in order and assigning the next available index the first time we see a new word.

vocab = {}
for toks in tokenized:
    for tok in toks:
        if tok not in vocab:
            vocab[tok] = len(vocab)

inv_vocab = {i: w for w, i in vocab.items()}

V = len(vocab)
print(f"Vocabulary size V = {V}\n")
for word, idx in vocab.items():
    print(f"  {idx} -> {word!r}")
Vocabulary size V = 5

  0 -> 'i'
  1 -> 'love'
  2 -> 'deep'
  3 -> 'learning'
  4 -> 'me'

So our vocabulary is:

Index Word
0 i
1 love
2 deep
3 learning
4 me

\(V = 5\) words, not 6. Stemming bought us that. Had we skipped it, love and loves would sit at two different indices, the model would see two unrelated symbols, and it would have to learn from data that they mean nearly the same thing. Instead we handed it that knowledge for free during preprocessing.

This is the general trade of normalization: every collapse you make shrinks \(V\), shrinks the embedding matrix, and gives each remaining word more training examples. The cost is that you throw away a distinction, and occasionally that distinction mattered.

4. Integer encoding: the (samples, timesteps) matrix

Now replace every word with its vocabulary index. This gives a 2-D integer array.

\[\text{shape} = (\underbrace{2}_{\text{samples}},\ \underbrace{4}_{\text{time steps}})\]

This array is not yet the input to the RNN. It is a compact intermediate representation, an index into a lookup table. There is no arithmetic meaning to these integers. The number 3 (learning) is not “three times” the number 1 (love).

X_int = np.array([[vocab[tok] for tok in toks] for toks in tokenized])

print("X_int =")
print(X_int)
print("\nshape:", X_int.shape, " -> (samples, timesteps)")

print("\nDecoded back:")
for row in X_int:
    print("  ", [inv_vocab[i] for i in row])
X_int =
[[0 1 2 3]
 [2 3 1 4]]

shape: (2, 4)  -> (samples, timesteps)

Decoded back:
   ['i', 'love', 'deep', 'learning']
   ['deep', 'learning', 'love', 'me']

5. Representation 1: one-hot encoded vectors

To give the network something numerically meaningful, we replace each integer index with a one-hot vector: a vector of length \(V\) that is 0 everywhere except a single 1 at the position of that word.

\[\text{one-hot}(\texttt{deep}) = \text{one-hot}(2) = \begin{bmatrix} 0 & 0 & 1 & 0 & 0\end{bmatrix}\]

Stacking these up for every word of every document produces the standard 3-D tensor that every sequence model expects:

\[\text{shape} = (\underbrace{2}_{\text{samples}},\ \underbrace{4}_{\text{time steps}},\ \underbrace{5}_{\text{features}})\]

The rule to memorize: (samples, time steps, features). The first axis indexes which document, the second indexes which word within that document, and the third holds the numeric vector for that word.

X_onehot = np.eye(V)[X_int]

print("shape:", X_onehot.shape, " -> (samples, timesteps, features)")

header = " " * 18 + " ".join(f"{inv_vocab[j][:4]:>5}" for j in range(V))

for i in range(X_onehot.shape[0]):
    print(f"\nDocument {i}: {corpus[i]!r}")
    print(f"  (time steps) x (features), shape {X_onehot[i].shape}\n")
    print(header)
    for t in range(X_onehot.shape[1]):
        word = inv_vocab[X_int[i, t]]
        row = " ".join(f"{v:>5.0f}" for v in X_onehot[i, t])
        print(f"  t={t+1} {word:>9} : {row}")
shape: (2, 4, 5)  -> (samples, timesteps, features)

Document 0: 'I love deep learning'
  (time steps) x (features), shape (4, 5)

                      i  love  deep  lear    me
  t=1         i :     1     0     0     0     0
  t=2      love :     0     1     0     0     0
  t=3      deep :     0     0     1     0     0
  t=4  learning :     0     0     0     1     0

Document 1: 'Deep Learning Loves Me'
  (time steps) x (features), shape (4, 5)

                      i  love  deep  lear    me
  t=1      deep :     0     0     1     0     0
  t=2  learning :     0     0     0     1     0
  t=3      love :     0     1     0     0     0
  t=4        me :     0     0     0     0     1

Read that block as a picture. Each row is one time step, each column is one vocabulary word, and the single 1 in each row marks which word appeared at that position.

The problem with one-hot

One-hot works, but it has two properties we dislike:

  1. It is enormous and mostly zeros. Feature dimension = vocabulary size. A realistic vocabulary of 50,000 words means 50,000 numbers per word, 49,999 of which are 0.
  2. It carries no notion of similarity. Every pair of distinct one-hot vectors is orthogonal, so their dot product is exactly 0. The vector for i is precisely as “far” from me as it is from deep, even though the first pair are both first-person pronouns. All the structure of language is thrown away.

Stemming rescued love/loves for us, but it can only ever merge words that happen to be spelled alike. It cannot tell one-hot that i and me belong together, because that is a semantic relationship, not a spelling one. To capture it, we need a representation where words can be near each other rather than merely equal or unequal.

oh_i, oh_me, oh_deep = np.eye(V)[vocab["i"]], np.eye(V)[vocab["me"]], np.eye(V)[vocab["deep"]]

print("one-hot(i) . one-hot(me)   =", oh_i @ oh_me)
print("one-hot(i) . one-hot(deep) =", oh_i @ oh_deep)
print("\nEvery word is equidistant from every other word. No similarity structure at all.")
one-hot(i) . one-hot(me)   = 0.0
one-hot(i) . one-hot(deep) = 0.0

Every word is equidistant from every other word. No similarity structure at all.

6. Representation 2: dense embeddings

The fix is a dense embedding: represent each word as a short vector of real numbers, living in a \(d\)-dimensional space where \(d \ll V\). Similar words can now sit near each other.

We choose \(d = 2\) and simply declare the embedding values by hand. In practice these numbers are learned parameters, but since our goal is a hand-computable forward pass, we pick small integers:

Index Word Embedding \(\mathbf{e}\)
0 i \([1,\ 0]\)
1 love \([0,\ 1]\)
2 deep \([1,\ 1]\)
3 learning \([2,\ 1]\)
4 me \([2,\ 0]\)

These were not chosen at random. Look at the geometry: i \(=[1,0]\) and me \(=[2,0]\) point in the same direction. This is the relationship stemming could not give us, since the two words share no spelling, and it is exactly the kind of structure a real embedding learns from data. Here we are just installing it by hand.

Notice the division of labor. Stemming merged love and loves into one symbol before the model ever saw them. Embedding places two genuinely different symbols, i and me, close together in a continuous space. Preprocessing handles surface form; the embedding handles meaning.

Stacked together, these rows form the embedding matrix \(E\), of shape \((V, d) = (5, 2)\).

d = 2

E = np.array([
    [1.0, 0.0],   # 0 i
    [0.0, 1.0],   # 1 love
    [1.0, 1.0],   # 2 deep
    [2.0, 1.0],   # 3 learning
    [2.0, 0.0],   # 4 me
])

print("Embedding matrix E, shape", E.shape, "= (V, d)\n")
for i, row in enumerate(E):
    print(f"  {inv_vocab[i]:>9} : {row}")
Embedding matrix E, shape (5, 2) = (V, d)

          i : [1. 0.]
       love : [0. 1.]
       deep : [1. 1.]
   learning : [2. 1.]
         me : [2. 0.]

The embedding lookup is just a matrix multiplication

Here is a small insight that ties the two representations together. Multiplying a one-hot vector by the embedding matrix selects a row of \(E\):

\[\underbrace{\begin{bmatrix} 0 & 0 & 1 & 0 & 0\end{bmatrix}}_{\text{one-hot}(\texttt{deep})} \cdot \underbrace{\begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \\ 2 & 1 \\ 2 & 0 \end{bmatrix}}_{E} = \underbrace{\begin{bmatrix} 1 & 1 \end{bmatrix}}_{\mathbf{e}_{\texttt{deep}}}\]

So an embedding layer is not a fundamentally new operation. It is a dense layer applied to a one-hot vector, implemented efficiently as a row lookup instead of a wasteful multiply-by-mostly-zeros.

onehot_deep = np.eye(V)[vocab["deep"]]

print("one-hot(deep) @ E =", onehot_deep @ E)
print("E[vocab['deep']]  =", E[vocab["deep"]])
print("\nIdentical. The lookup IS the matrix product.")
one-hot(deep) @ E = [1. 1.]
E[vocab['deep']]  = [1. 1.]

Identical. The lookup IS the matrix product.

The dense-embedded tensor

Applying the lookup to every token gives us the same standard 3-D shape, but now with a feature dimension of 2 instead of 6:

\[\text{shape} = (\underbrace{2}_{\text{samples}},\ \underbrace{4}_{\text{time steps}},\ \underbrace{2}_{\text{features}})\]

X_dense = E[X_int]

print("shape:", X_dense.shape, " -> (samples, timesteps, features)\n")
for i in range(len(corpus)):
    print(f"Document {i}: {corpus[i]!r}")
    for t in range(X_dense.shape[1]):
        print(f"    t={t+1}  {inv_vocab[X_int[i, t]]:>9} -> {X_dense[i, t]}")
    print()
shape: (2, 4, 2)  -> (samples, timesteps, features)

Document 0: 'I love deep learning'
    t=1          i -> [1. 0.]
    t=2       love -> [0. 1.]
    t=3       deep -> [1. 1.]
    t=4   learning -> [2. 1.]

Document 1: 'Deep Learning Loves Me'
    t=1       deep -> [1. 1.]
    t=2   learning -> [2. 1.]
    t=3       love -> [0. 1.]
    t=4         me -> [2. 0.]
# The similarity structure that one-hot destroyed is now present
def cos(a, b):
    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))

print("cosine(i, me)   =", round(cos(E[vocab['i']], E[vocab['me']]),   4))
print("cosine(i, deep)  =", round(cos(E[vocab['i']], E[vocab['deep']]), 4))

print("\n'i' and 'me' are now perfectly aligned; 'i' and 'deep' are not.")
print("One-hot could never express this. Stemming could never find it.")
cosine(i, me)   = 1.0
cosine(i, deep)  = 0.7071

'i' and 'me' are now perfectly aligned; 'i' and 'deep' are not.
One-hot could never express this. Stemming could never find it.

7. Decision point

We have built the same corpus in two valid input representations:

Representation Tensor shape Feature dim
One-hot (2, 4, 5) \(V = 5\)
Dense embedding (2, 4, 2) \(d = 2\)

From this point forward we use the dense embeddings. Everything below operates on the (2, 4, 2) tensor. The RNN math would be structurally identical with one-hot inputs, but the input weight matrix would have to be \(2 \times 5\) instead of \(2 \times 2\), and the arithmetic would be far more tedious to do by hand.

X = X_dense          # <- our input tensor from here on
print("Input tensor X:", X.shape, "= (samples, timesteps, features)")
Input tensor X: (2, 4, 2) = (samples, timesteps, features)

8. The vanilla RNN cell

An RNN reads a sequence one time step at a time, carrying a hidden state \(\mathbf{h}_t\) that summarizes everything it has seen so far. At each step it mixes two sources of information: the new word, and its own memory.

\[\boxed{\ \mathbf{h}_t = \tanh\!\left(W_x\, \mathbf{x}_t \; + \; W_h\, \mathbf{h}_{t-1} \; + \; \mathbf{b}\right)\ }\]

where

  • \(\mathbf{x}_t \in \mathbb{R}^{d}\) is the embedding of the word at time step \(t\) (here \(d = 2\)),
  • \(\mathbf{h}_t \in \mathbb{R}^{u}\) is the hidden state, with \(u = 2\) units,
  • \(W_x \in \mathbb{R}^{u \times d}\) is the input kernel, which maps the new word into hidden space,
  • \(W_h \in \mathbb{R}^{u \times u}\) is the recurrent kernel, which maps the previous memory into hidden space,
  • \(\mathbf{b} \in \mathbb{R}^{u}\) is the bias,
  • \(\tanh\) is applied elementwise, squashing every component into \((-1, 1)\).

The single most important thing to see in that formula: \(W_x\) and \(W_h\) do not change with \(t\). The same two matrices are reused at every time step. That weight sharing is what makes it recurrent, and it is why an RNN can handle sequences of any length with a fixed number of parameters.

The recursion has to start somewhere, so by convention the initial hidden state is zero:

\[\mathbf{h}_0 = \begin{bmatrix} 0 \\ 0 \end{bmatrix}\]

The weights we were handed

Remember, we are not training. Somebody hands us these matrices and we evaluate the network. They are chosen to be easy to multiply by hand.

\[W_x = \begin{bmatrix} 0.5 & 0.0 \\ 0.0 & 0.5 \end{bmatrix} \qquad W_h = \begin{bmatrix} 0.0 & 0.5 \\ 0.2 & 0.0 \end{bmatrix} \qquad \mathbf{b} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}\]

These have a clean interpretation:

  • \(W_x = \tfrac{1}{2}I\), so the new word simply contributes half of its embedding, component by component.
  • \(W_h\) is off-diagonal, so the memory gets crossed over: the old \(h^{(2)}\) feeds the new \(h^{(1)}\) (scaled by 0.5), and the old \(h^{(1)}\) feeds the new \(h^{(2)}\) (scaled by 0.2). Concretely, \[W_h \mathbf{h} = \begin{bmatrix} 0.5\, h^{(2)} \\ 0.2\, h^{(1)} \end{bmatrix}\]
  • The bias is zero, purely to keep the arithmetic short. It is still a real parameter of the layer.
u = 2   # hidden units

W_x = np.array([[0.5, 0.0],
                [0.0, 0.5]])

W_h = np.array([[0.0, 0.5],
                [0.2, 0.0]])

b = np.array([0.0, 0.0])

print("W_x", W_x.shape, "= (units, features)\n", W_x)
print("\nW_h", W_h.shape, "= (units, units)\n", W_h)
print("\nb  ", b.shape, "\n", b)
W_x (2, 2) = (units, features)
 [[0.5 0. ]
 [0.  0.5]]

W_h (2, 2) = (units, units)
 [[0.  0.5]
 [0.2 0. ]]

b   (2,) 
 [0. 0.]
def rnn_step(x_t, h_prev, W_x=W_x, W_h=W_h, b=b, verbose=True):
    '''One time step of a vanilla RNN, with the arithmetic spelled out.'''
    input_term = W_x @ x_t
    memory_term = W_h @ h_prev
    z = input_term + memory_term + b
    h = np.tanh(z)
    if verbose:
        print(f"  x_t            = {x_t}")
        print(f"  h_prev         = {h_prev}")
        print(f"  W_x @ x_t      = {input_term}      (contribution of the new word)")
        print(f"  W_h @ h_prev   = {memory_term}      (contribution of memory)")
        print(f"  z = sum + b    = {z}")
        print(f"  h_t = tanh(z)  = {h}")
    return h

9. Unrolling the RNN on document 0

We now evaluate the network on "I love deep learning", whose embedded time steps are

\[\mathbf{x}_1 = \begin{bmatrix}1\\0\end{bmatrix}_{\texttt{i}} \quad \mathbf{x}_2 = \begin{bmatrix}0\\1\end{bmatrix}_{\texttt{love}} \quad \mathbf{x}_3 = \begin{bmatrix}1\\1\end{bmatrix}_{\texttt{deep}} \quad \mathbf{x}_4 = \begin{bmatrix}2\\1\end{bmatrix}_{\texttt{learning}}\]

We will apply the boxed formula four times, carrying \(\mathbf{h}\) forward each time.

doc = 0
x_seq = X[doc]
print(f"Evaluating: {corpus[doc]!r}\n")
for t in range(x_seq.shape[0]):
    print(f"  x_{t+1} = {x_seq[t]}   ({inv_vocab[X_int[doc, t]]})")
Evaluating: 'I love deep learning'

  x_1 = [1. 0.]   (i)
  x_2 = [0. 1.]   (love)
  x_3 = [1. 1.]   (deep)
  x_4 = [2. 1.]   (learning)

Time step 1: reading “i”

The memory is empty, \(\mathbf{h}_0 = \mathbf{0}\), so the recurrent term vanishes and only the word matters.

\[W_x \mathbf{x}_1 = \begin{bmatrix} 0.5 & 0 \\ 0 & 0.5\end{bmatrix}\begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}0.5\\0\end{bmatrix} \qquad W_h \mathbf{h}_0 = \begin{bmatrix}0\\0\end{bmatrix}\]

\[\mathbf{z}_1 = \begin{bmatrix}0.5\\0\end{bmatrix} + \begin{bmatrix}0\\0\end{bmatrix} + \begin{bmatrix}0\\0\end{bmatrix} = \begin{bmatrix}0.5\\0\end{bmatrix}\]

\[\mathbf{h}_1 = \tanh\begin{bmatrix}0.5\\0\end{bmatrix} = \begin{bmatrix}\tanh(0.5)\\ \tanh(0)\end{bmatrix} = \begin{bmatrix}0.4621\\ 0.0000\end{bmatrix}\]

h_0 = np.zeros(u)

print("TIME STEP 1  ('i')")
h_1 = rnn_step(x_seq[0], h_0)
TIME STEP 1  ('i')
  x_t            = [1. 0.]
  h_prev         = [0. 0.]
  W_x @ x_t      = [0.5 0. ]      (contribution of the new word)
  W_h @ h_prev   = [0. 0.]      (contribution of memory)
  z = sum + b    = [0.5 0. ]
  h_t = tanh(z)  = [0.4621 0.    ]

Time step 2: reading “love”

Now memory is live. Apply the cross-over rule \(W_h \mathbf{h} = [\,0.5 h^{(2)},\ 0.2 h^{(1)}\,]^\top\) to \(\mathbf{h}_1 = [0.4621,\ 0]^\top\):

\[W_h \mathbf{h}_1 = \begin{bmatrix} 0.5 \times 0.0000 \\ 0.2 \times 0.4621 \end{bmatrix} = \begin{bmatrix} 0.0000 \\ 0.0924 \end{bmatrix} \qquad W_x \mathbf{x}_2 = \begin{bmatrix}0\\0.5\end{bmatrix}\]

\[\mathbf{z}_2 = \begin{bmatrix}0\\0.5\end{bmatrix} + \begin{bmatrix}0.0000\\0.0924\end{bmatrix} = \begin{bmatrix}0.0000\\0.5924\end{bmatrix}\]

\[\mathbf{h}_2 = \tanh\begin{bmatrix}0.0000\\0.5924\end{bmatrix} = \begin{bmatrix}0.0000\\0.5316\end{bmatrix}\]

The second component, \(0.5316\), is larger than the \(0.5\) that “love” alone would have produced. That extra \(0.0316\) is the trace of the word “i”, carried forward through memory. This is the whole point of an RNN in one number.

print("TIME STEP 2  ('love')")
h_2 = rnn_step(x_seq[1], h_1)
TIME STEP 2  ('love')
  x_t            = [0. 1.]
  h_prev         = [0.4621 0.    ]
  W_x @ x_t      = [0.  0.5]      (contribution of the new word)
  W_h @ h_prev   = [0.     0.0924]      (contribution of memory)
  z = sum + b    = [0.     0.5924]
  h_t = tanh(z)  = [0.     0.5316]

Time step 3: reading “deep”

\[W_x \mathbf{x}_3 = \begin{bmatrix}0.5\\0.5\end{bmatrix} \qquad W_h \mathbf{h}_2 = \begin{bmatrix} 0.5 \times 0.5316 \\ 0.2 \times 0.0000 \end{bmatrix} = \begin{bmatrix}0.2658\\0.0000\end{bmatrix}\]

\[\mathbf{z}_3 = \begin{bmatrix}0.7658\\0.5000\end{bmatrix} \qquad \mathbf{h}_3 = \tanh\begin{bmatrix}0.7658\\0.5000\end{bmatrix} = \begin{bmatrix}0.6445\\0.4621\end{bmatrix}\]

print("TIME STEP 3  ('deep')")
h_3 = rnn_step(x_seq[2], h_2)
TIME STEP 3  ('deep')
  x_t            = [1. 1.]
  h_prev         = [0.     0.5316]
  W_x @ x_t      = [0.5 0.5]      (contribution of the new word)
  W_h @ h_prev   = [0.2658 0.    ]      (contribution of memory)
  z = sum + b    = [0.7658 0.5   ]
  h_t = tanh(z)  = [0.6445 0.4621]

Time step 4: reading “learning”

\[W_x \mathbf{x}_4 = \begin{bmatrix} 0.5 & 0 \\ 0 & 0.5\end{bmatrix}\begin{bmatrix}2\\1\end{bmatrix} = \begin{bmatrix}1.0\\0.5\end{bmatrix} \qquad W_h \mathbf{h}_3 = \begin{bmatrix} 0.5 \times 0.4621 \\ 0.2 \times 0.6445 \end{bmatrix} = \begin{bmatrix}0.2311\\0.1289\end{bmatrix}\]

\[\mathbf{z}_4 = \begin{bmatrix}1.2311\\0.6289\end{bmatrix} \qquad \mathbf{h}_4 = \tanh\begin{bmatrix}1.2311\\0.6289\end{bmatrix} = \begin{bmatrix}0.8429\\0.5573\end{bmatrix}\]

\(\mathbf{h}_4\) is the final hidden state. Two numbers that encode the whole sentence “I love deep learning”. Every word contributed to it, and the earlier words reached it only by being passed along, hand to hand, through \(W_h\).

print("TIME STEP 4  ('learning')")
h_4 = rnn_step(x_seq[3], h_3)

print("\nFinal hidden state h_4 =", h_4)
TIME STEP 4  ('learning')
  x_t            = [2. 1.]
  h_prev         = [0.6445 0.4621]
  W_x @ x_t      = [1.  0.5]      (contribution of the new word)
  W_h @ h_prev   = [0.2311 0.1289]      (contribution of memory)
  z = sum + b    = [1.2311 0.6289]
  h_t = tanh(z)  = [0.8429 0.5573]

Final hidden state h_4 = [0.8429 0.5573]

The same thing as a loop

Four hand-written steps collapse into three lines of code. This loop is the RNN.

def rnn_forward(x_seq, W_x=W_x, W_h=W_h, b=b):
    '''Run a vanilla RNN over one sequence; return every hidden state.'''
    h = np.zeros(W_h.shape[0])
    H = []
    for t in range(x_seq.shape[0]):
        h = np.tanh(W_x @ x_seq[t] + W_h @ h + b)   # the entire model
        H.append(h)
    return np.array(H)

H_doc0 = rnn_forward(X[0])

print("Hidden state at every time step, shape", H_doc0.shape, "= (timesteps, units)\n")
print("   step   word          h^(1)     h^(2)")
for t, h in enumerate(H_doc0):
    print(f"    {t+1}    {inv_vocab[X_int[0, t]]:<10} {h[0]:>8.4f}  {h[1]:>8.4f}")
Hidden state at every time step, shape (4, 2) = (timesteps, units)

   step   word          h^(1)     h^(2)
    1    i            0.4621    0.0000
    2    love         0.0000    0.5316
    3    deep         0.6445    0.4621
    4    learning     0.8429    0.5573

10. The output layer

The hidden state is an internal summary, not a prediction. To get an answer we push \(\mathbf{h}_4\) through one ordinary dense layer. Suppose the task is binary sentiment, so we use a single output unit with a sigmoid:

\[\hat{y} = \sigma\!\left(W_y\, \mathbf{h}_4 + b_y\right), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}\]

Again the weights are given to us:

\[W_y = \begin{bmatrix} 1.0 & -1.0 \end{bmatrix}, \qquad b_y = 0\]

Substituting \(\mathbf{h}_4 = [0.8429,\ 0.5573]^\top\):

\[z_y = (1.0)(0.8429) + (-1.0)(0.5573) + 0 = 0.2856\]

\[\hat{y} = \sigma(0.2856) = \frac{1}{1 + e^{-0.2856}} = 0.5709\]

Note that only the last hidden state is used. Steps 1 through 3 influence the prediction, but only indirectly, by shaping \(\mathbf{h}_4\). This “many-to-one” pattern is exactly what return_sequences=False means in Keras.

W_y = np.array([[1.0, -1.0]])   # (1, units)
b_y = np.array([0.0])

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

z_y = W_y @ h_4 + b_y
y_hat = sigmoid(z_y)

print("h_4     =", h_4)
print("z_y     =", z_y)
print("y_hat   =", y_hat, "  <- the model's prediction for document 0")
h_4     = [0.8429 0.5573]
z_y     = [0.2856]
y_hat   = [0.5709]   <- the model's prediction for document 0

11. The same computation in Keras

Everything above was NumPy we wrote ourselves. Now we build the identical model in Keras, force it to use our weights instead of random initialization, and check that it reproduces our numbers to the last decimal.

The architecture maps one-to-one onto what we just did:

Our math Keras layer
\(E\) lookup Embedding(input_dim=6, output_dim=2)
\(\mathbf{h}_t = \tanh(W_x\mathbf{x}_t + W_h\mathbf{h}_{t-1} + \mathbf{b})\) SimpleRNN(units=2, activation='tanh')
\(\sigma(W_y \mathbf{h}_4 + b_y)\) Dense(1, activation='sigmoid')
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

print("TensorFlow", tf.__version__, "| Keras", keras.__version__)
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1784024565.253114     590 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1784024567.365407     590 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
TensorFlow 2.21.0 | Keras 3.15.0

The one thing that trips everyone up: Keras transposes the weights

We wrote the math with \(\mathbf{x}_t\) as a column vector, multiplying from the left:

\[\mathbf{h}_t = \tanh\left(W_x \mathbf{x}_t + W_h \mathbf{h}_{t-1} + \mathbf{b}\right)\]

Keras stores data with samples in rows, so internally it treats \(\mathbf{x}_t\) as a row vector and multiplies from the right:

\[\mathbf{h}_t = \tanh\left(\mathbf{x}_t K + \mathbf{h}_{t-1} U + \mathbf{b}\right)\]

Transposing our version shows these are the same equation, provided

\[K = W_x^\top \ \ \text{(the \texttt{kernel}, shape } (d, u) = (2,2)) \qquad U = W_h^\top \ \ \text{(the \texttt{recurrent\_kernel}, shape } (u,u) = (2,2))\]

So when we call set_weights, we must hand Keras the transposes. The same applies to the Dense layer, whose kernel has shape (units_in, units_out) = (2, 1).

T = X_int.shape[1]   # 4 time steps

inputs = layers.Input(shape=(T,), dtype="int32", name="tokens")

embed_layer = layers.Embedding(input_dim=V, output_dim=d, name="embed")
rnn_layer   = layers.SimpleRNN(units=u, activation="tanh",
                               return_sequences=True, name="rnn")

emb_out = embed_layer(inputs)
rnn_out = rnn_layer(emb_out)

# Two "inspector" models that let us peek at intermediate tensors
embed_model  = keras.Model(inputs, emb_out, name="embed_inspector")
hidden_model = keras.Model(inputs, rnn_out, name="hidden_state_inspector")

# Install OUR weights, transposed to Keras's row-vector convention
embed_layer.set_weights([E])
rnn_layer.set_weights([W_x.T, W_h.T, b])

k, r, bb = rnn_layer.get_weights()
print("Keras kernel K = W_x.T          ", k.shape, "\n", k)
print("\nKeras recurrent_kernel U = W_h.T", r.shape, "\n", r)
print("\nKeras bias                      ", bb.shape, "\n", bb)
Keras kernel K = W_x.T           (2, 2) 
 [[0.5 0. ]
 [0.  0.5]]

Keras recurrent_kernel U = W_h.T (2, 2) 
 [[0.  0.2]
 [0.5 0. ]]

Keras bias                       (2,) 
 [0. 0.]

Check 1: does the Embedding layer reproduce our lookup?

keras_emb = embed_model.predict(X_int, verbose=0)

print("Keras Embedding output, doc 0:\n", keras_emb[0])
print("\nOur NumPy X[0]:\n", X[0])
print("\nMatch:", np.allclose(keras_emb, X, atol=1e-6))
Keras Embedding output, doc 0:
 [[1. 0.]
 [0. 1.]
 [1. 1.]
 [2. 1.]]

Our NumPy X[0]:
 [[1. 0.]
 [0. 1.]
 [1. 1.]
 [2. 1.]]

Match: True

Check 2: does SimpleRNN reproduce our four hand-computed hidden states?

With return_sequences=True, Keras returns \(\mathbf{h}_t\) at every time step, shape (samples, timesteps, units). That is exactly the list we built by hand.

keras_H = hidden_model.predict(X_int, verbose=0)

print("Keras hidden states, shape", keras_H.shape, "= (samples, timesteps, units)\n")
print("            KERAS h_t              OUR HAND-COMPUTED h_t")
for t in range(T):
    word = inv_vocab[X_int[0, t]]
    print(f"  t={t+1} {word:>9}  {keras_H[0, t]}        {H_doc0[t]}")

print("\nMatch:", np.allclose(keras_H[0], H_doc0, atol=1e-6))
Keras hidden states, shape (2, 4, 2) = (samples, timesteps, units)

            KERAS h_t              OUR HAND-COMPUTED h_t
  t=1         i  [0.4621 0.    ]        [0.4621 0.    ]
  t=2      love  [0.     0.5316]        [0.     0.5316]
  t=3      deep  [0.6445 0.4621]        [0.6445 0.4621]
  t=4  learning  [0.8429 0.5573]        [0.8429 0.5573]

Match: True

Check 3: the full classifier, end to end

Now the real thing, with return_sequences=False so that only \(\mathbf{h}_4\) survives, followed by the sigmoid output layer.

rnn_last  = layers.SimpleRNN(units=u, activation="tanh",
                             return_sequences=False, name="rnn_last")
out_layer = layers.Dense(1, activation="sigmoid", name="out")

h_last = rnn_last(embed_layer(inputs))     # reuse the same Embedding layer
y_out  = out_layer(h_last)

classifier = keras.Model(inputs, y_out, name="rnn_classifier")

rnn_last.set_weights([W_x.T, W_h.T, b])
out_layer.set_weights([W_y.T, b_y])        # Dense kernel is (units_in, units_out)

classifier.summary(line_length=72)
Model: "rnn_classifier"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Layer (type)                  ┃ Output Shape          ┃      Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ tokens (InputLayer)           │ (None, 4)             │            0 │
├───────────────────────────────┼───────────────────────┼──────────────┤
│ embed (Embedding)             │ (None, 4, 2)          │           10 │
├───────────────────────────────┼───────────────────────┼──────────────┤
│ rnn_last (SimpleRNN)          │ (None, 2)             │           10 │
├───────────────────────────────┼───────────────────────┼──────────────┤
│ out (Dense)                   │ (None, 1)             │            3 │
└───────────────────────────────┴───────────────────────┴──────────────┘
 Total params: 23 (92.00 B)
 Trainable params: 23 (92.00 B)
 Non-trainable params: 0 (0.00 B)
preds = classifier.predict(X_int, verbose=0)

print("Keras prediction, doc 0 :", preds[0])
print("Our  prediction, doc 0 :", y_hat)
print("\nMatch:", np.allclose(preds[0], y_hat, atol=1e-6))
Keras prediction, doc 0 : [0.5709]
Our  prediction, doc 0 : [0.5709]

Match: True

Where did those 23 parameters come from?

The summary reports 23 trainable parameters. Every one is accounted for by the matrices we wrote down:

Layer Parameters Count
Embedding \(E\), shape \((V, d) = (5, 2)\) \(10\)
SimpleRNN \(W_x\ (2\times 2)\), \(W_h\ (2 \times 2)\), \(\mathbf{b}\ (2)\) \(4 + 4 + 2 = 10\)
Dense \(W_y\ (1 \times 2)\), \(b_y\ (1)\) \(2 + 1 = 3\)
Total 23

Stemming shaved two parameters off the embedding matrix by removing one vocabulary row. On a toy corpus that is a rounding error. On a real one, where collapsing inflections can cut \(V\) by 20 percent or more, it is \(0.2 \times V \times d\) fewer parameters to learn.

In general, a SimpleRNN has \(u \cdot d + u^2 + u\) parameters. Crucially, that count does not depend on \(T\). A sequence of 4 words and a sequence of 4,000 words use the identical weights, because the same cell is applied over and over.

12. Both documents at once

Nothing about the model changes when we process the second document. It travels down the very same weights. Batching simply means the leading samples axis is 2 instead of 1, and the whole computation runs in parallel across it.

H_all = np.array([rnn_forward(X[i]) for i in range(X.shape[0])])
y_all = sigmoid(H_all[:, -1, :] @ W_y.T + b_y)

for i in range(len(corpus)):
    print(f"Document {i}: {corpus[i]!r}")
    for t in range(T):
        print(f"    t={t+1}  {inv_vocab[X_int[i, t]]:>9}  h = {H_all[i, t]}")
    print(f"    final h_4 = {H_all[i, -1]}   ->   y_hat = {y_all[i][0]:.4f}\n")

print("NumPy y_hat:", y_all.ravel())
print("Keras y_hat:", preds.ravel())
print("Match:", np.allclose(y_all, preds, atol=1e-6))
Document 0: 'I love deep learning'
    t=1          i  h = [0.4621 0.    ]
    t=2       love  h = [0.     0.5316]
    t=3       deep  h = [0.6445 0.4621]
    t=4   learning  h = [0.8429 0.5573]
    final h_4 = [0.8429 0.5573]   ->   y_hat = 0.5709

Document 1: 'Deep Learning Loves Me'
    t=1       deep  h = [0.4621 0.4621]
    t=2   learning  h = [0.8429 0.5316]
    t=3       love  h = [0.2597 0.584 ]
    t=4         me  h = [0.8597 0.0519]
    final h_4 = [0.8597 0.0519]   ->   y_hat = 0.6916

NumPy y_hat: [0.5709 0.6916]
Keras y_hat: [0.5709 0.6916]
Match: True

Look at the two predictions. After stemming, the documents share three of their four words: love, deep, and learning. Only i versus me differs. Yet they land on clearly different outputs. Two reasons:

  1. The one word that differs (i versus me) has a different embedding.
  2. The shared words appear in a different order, and an RNN is order-sensitive by construction. deep learning arriving at steps 3 and 4 produces a different memory trajectory than the same pair arriving at steps 1 and 2.

That second point is worth dwelling on. A bag-of-words model would give these two sentences nearly identical treatment. The RNN cannot, because \(\mathbf{h}_t\) depends on the entire ordered prefix that came before it.

13. Recap

Here is the whole journey in one place, with the shape at every stage.

Stage Object Shape
Raw text 2 strings
Tokenized + stemmed lists of words (2, 4)
Integer encoded X_int (2, 4)
One-hot X_onehot (2, 4, 5)
Dense embedded X (2, 4, 2)
Hidden states H_all (2, 4, 2)
Final hidden state H_all[:, -1, :] (2, 2)
Prediction y_hat (2, 1)

The three ideas worth carrying away:

  1. The standard shape is (samples, time steps, features). Every sequence model expects it. The only question is what fills the features axis: a sparse one-hot vector of length \(V\), or a dense embedding of length \(d \ll V\).

  2. An RNN is one small equation applied repeatedly. \(\mathbf{h}_t = \tanh(W_x \mathbf{x}_t + W_h \mathbf{h}_{t-1} + \mathbf{b})\), with the same \(W_x\) and \(W_h\) at every step. The loop over \(t\) is the entire architecture.

  3. Memory is the recurrent term. Delete \(W_h \mathbf{h}_{t-1}\) and you are left with a plain dense layer applied independently to each word, blind to order. That one term is what makes the network a sequence model.

Everything we did here was a forward pass with fixed weights. Training would add one more piece: compare \(\hat{y}\) to a true label, then backpropagate the error through this same unrolled graph to update \(W_x\), \(W_h\), \(\mathbf{b}\), and \(E\). The forward computation you just did by hand is exactly the graph that gradient would flow backwards through.