Vanilla Recurrent Neural Networks with Keras

A hands-on, visual walkthrough of the canonical RNN forecasting workflow.

We use a SimpleRNN — the “vanilla” recurrent cell,

\[h_t = \tanh\!\big(W_{x} x_t + W_{h} h_{t-1} + b\big), \qquad \hat{y} = W_o h_T + b_o\]

— because everything you learn here (windowing, chronological splitting, scaling, regularization, diagnostics) transfers unchanged to LSTMs, GRUs, and Transformers. Only the cell changes.

Roadmap

Section Data Task
1 One synthetic series (exponential trend + 2 sinusoids + noise) Predict the next value from the last 40
2 Three series with different parameterizations Predict the next value of all three jointly

The workflow in both sections is identical:

generate → visualize → split chronologically → scale → window → build → train → diagnose → evaluate

0. Setup

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import keras
from keras import layers
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

SEED = 42
keras.utils.set_random_seed(SEED)   # seeds python, numpy, and tensorflow at once

plt.rcParams.update({
    "figure.dpi": 110, "font.size": 10, "axes.grid": True,
    "grid.alpha": 0.3, "axes.spines.top": False, "axes.spines.right": False,
})
C = {"train": "#2c7fb8", "val": "#f0a500", "test": "#d62728", "pred": "#000000"}

print("Keras", keras.__version__)
Keras 2.13.1

Shared helpers

Four small functions carry the entire notebook. Read them once — they are reused verbatim in Section 2.

  1. make_series — the data-generating process (DGP).
  2. make_windows — turns a 1-D series into supervised (X, y) pairs.
  3. regression_metrics — MAE / RMSE / R² / MAPE in one dict.
  4. plot_history — training curves.
# Data-generating process:  exponential trend + two sinusoids + Gaussian noise.
#   y(t) = level*exp(growth*t) + a1*sin(2*pi*t/p1) + a2*sin(2*pi*t/p2 + phase) + eps
def make_series(n, level, growth, a1, p1, a2, p2, phase, noise, rng):
    t = np.arange(n)
    trend    = level * np.exp(growth * t)          # slow multiplicative growth
    seasonal = (a1 * np.sin(2 * np.pi * t / p1)    # long oscillation
                + a2 * np.sin(2 * np.pi * t / p2 + phase))  # short oscillation
    eps      = noise * rng.standard_normal(n)      # irreducible noise -> sets the error floor
    return trend + seasonal + eps, trend, seasonal, eps


# Slide a window of length `window` over `series`; the target is the value `horizon` steps after
# the window ends.  X -> (samples, window, features);  y -> (samples, features)
def make_windows(series, window, horizon=1):
    series = np.asarray(series)
    if series.ndim == 1:                       # (T,) -> (T, 1) so univariate and multivariate share code
        series = series[:, None]
    X, y = [], []
    for i in range(len(series) - window - horizon + 1):
        X.append(series[i : i + window])                 # inputs:  t-window ... t-1
        y.append(series[i + window + horizon - 1])       # target:  t  (one step ahead)
    return np.array(X), np.array(y)


def regression_metrics(y_true, y_pred):
    return {
        "MAE":  mean_absolute_error(y_true, y_pred),
        "RMSE": np.sqrt(mean_squared_error(y_true, y_pred)),
        "R2":   r2_score(y_true, y_pred),
        "MAPE%": 100 * np.mean(np.abs((y_true - y_pred) / y_true)),
    }


# Loss (MSE) and MAE vs epoch, train vs validation.
def plot_history(history, title="Learning curves"):
    h = history.history
    fig, ax = plt.subplots(1, 2, figsize=(10, 3.4))
    for a, key, name in zip(ax, ["loss", "mae"], ["Loss (MSE, scaled units)", "MAE (scaled units)"]):
        a.plot(h[key],           color=C["train"], label="train")
        a.plot(h["val_" + key],  color=C["val"],   label="validation")
        best = int(np.argmin(h["val_" + key]))
        a.axvline(best, ls=":", c="grey")
        a.scatter([best], [h["val_" + key][best]], c=C["val"], zorder=5)
        a.set(xlabel="epoch", ylabel=name, yscale="log")
        a.legend(frameon=False)
    fig.suptitle(title)
    fig.tight_layout()
    plt.show()

Section 1 — Univariate time series

1.1 Generate and inspect the data

We build the series from three transparent ingredients so we always know the ground truth: a gentle exponential trend, two sinusoids of different periods, and small Gaussian noise.

The noise is the irreducible part: no model can do better than its standard deviation. That gives us a target to compare against.

N       = 3000
WINDOW  = 40      # how many past steps the RNN sees
HORIZON = 1       # predict one step ahead

rng = np.random.default_rng(SEED)
y, trend, seasonal, eps = make_series(
    n=N, level=10.0, growth=8e-5,     # trend: 10 -> ~12.7 over the run
    a1=2.0, p1=200,                   # slow oscillation, period 200
    a2=1.0, p2=25, phase=0.8,         # fast oscillation, period 25
    noise=0.15, rng=rng,
)
t = np.arange(N)

fig, ax = plt.subplots(2, 1, figsize=(11, 5), sharex=True)
ax[0].plot(t, y, lw=0.8, color="#333")
ax[0].set(ylabel="y(t)", title="Observed series  y = trend + seasonal + noise")
ax[1].plot(t, trend, label="exponential trend", lw=2)
ax[1].plot(t, seasonal, label="two sinusoids", lw=0.8, alpha=0.9)
ax[1].plot(t, eps, label=f"noise (sd={eps.std():.2f})", lw=0.5, alpha=0.6)
ax[1].set(xlabel="t", ylabel="component", title="Ground-truth components")
ax[1].legend(frameon=False, ncol=3)
fig.tight_layout(); plt.show()

print(f"Irreducible noise sd = {eps.std():.3f}  -> the best achievable RMSE")

Irreducible noise sd = 0.151  -> the best achievable RMSE

1.2 Chronological split

The single most important rule in time-series ML: never shuffle before splitting. A random split lets the model see the future, and your validation score becomes a lie.

We cut the series in time: the first 70 % trains, the next 15 % validates, the last 15 % tests.

Note the small overlap we allow: to build a window whose target is the first test point, the model must look at the last WINDOW points of validation. That is legitimate — those are past observations at prediction time. What matters is that no target ever appears in two splits.

n_train = int(0.70 * N)
n_val   = int(0.15 * N)

# raw (unscaled) slices; val/test carry WINDOW points of history so their first target is predictable
train_raw = y[:n_train]
val_raw   = y[n_train - WINDOW : n_train + n_val]
test_raw  = y[n_train + n_val - WINDOW :]

fig, ax = plt.subplots(figsize=(11, 3))
ax.plot(t[:n_train], y[:n_train], color=C["train"], lw=0.8, label=f"train ({n_train})")
ax.plot(t[n_train:n_train+n_val], y[n_train:n_train+n_val], color=C["val"], lw=0.8, label=f"val ({n_val})")
ax.plot(t[n_train+n_val:], y[n_train+n_val:], color=C["test"], lw=0.8, label=f"test ({N-n_train-n_val})")
ax.set(xlabel="t", ylabel="y(t)", title="Chronological hold-out split")
ax.legend(frameon=False, ncol=3); fig.tight_layout(); plt.show()

1.3 Scale — using training statistics only

Neural nets train best on inputs of order 1. We standardize with

\[z = \frac{y - \mu_{\text{train}}}{\sigma_{\text{train}}}\]

The mean and standard deviation come only from the training slice. Using the full series would leak information about the future into the model — a subtle but very common bug.

mu, sd = train_raw.mean(), train_raw.std()      # <- fitted on TRAIN ONLY
scale   = lambda a: (a - mu) / sd
unscale = lambda a: a * sd + mu

X_tr, y_tr = make_windows(scale(train_raw), WINDOW, HORIZON)
X_va, y_va = make_windows(scale(val_raw),   WINDOW, HORIZON)
X_te, y_te = make_windows(scale(test_raw),  WINDOW, HORIZON)

print(f"train mu={mu:.3f}  sd={sd:.3f}")
for name, X, yy in [("train", X_tr, y_tr), ("val", X_va, y_va), ("test", X_te, y_te)]:
    print(f"{name:>5}: X {X.shape}  y {yy.shape}")
train mu=10.941  sd=1.675
train: X (2060, 40, 1)  y (2060, 1)
  val: X (450, 40, 1)  y (450, 1)
 test: X (450, 40, 1)  y (450, 1)

X has shape (samples, timesteps, features) — the shape every Keras recurrent layer expects. Here features = 1 because the series is univariate. In Section 2 only that last number changes.

1.4 Build the model

Standard practice for a first recurrent baseline:

  • Two stacked SimpleRNN layers. The first returns the full sequence (return_sequences=True) so the second can consume it; the second returns only the final hidden state \(h_T\) — a fixed-length summary of the window.
  • Dropout as the regularizer — the standard, default choice. Randomly zeroing activations during training stops the network from over-relying on any single unit. Rates of 0.1–0.3 are typical; with a small recurrent net, too much dropout simply underfits, so we start at 0.1.
  • Dense(1) head, linear activation — this is regression, so no squashing at the output.
  • MSE loss, the canonical regression loss; we also monitor MAE because it is in interpretable units.
def build_rnn(window, n_features, n_outputs, units=32, dropout=0.1):
    model = keras.Sequential([
        keras.Input(shape=(window, n_features)),
        layers.SimpleRNN(units, return_sequences=True),   # sequence -> sequence
        layers.Dropout(dropout),                          # regularization
        layers.SimpleRNN(units),                          # sequence -> final hidden state
        layers.Dropout(dropout),
        layers.Dense(n_outputs),                          # linear head for regression
    ])
    model.compile(optimizer=keras.optimizers.Adam(1e-3), loss="mse", metrics=["mae"])
    return model

model = build_rnn(WINDOW, n_features=1, n_outputs=1)
model.summary()
WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.Adam` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.Adam`.
WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., `tf.keras.optimizers.legacy.Adam`.
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 simple_rnn (SimpleRNN)      (None, 40, 32)            1088      
                                                                 
 dropout (Dropout)           (None, 40, 32)            0         
                                                                 
 simple_rnn_1 (SimpleRNN)    (None, 32)                2080      
                                                                 
 dropout_1 (Dropout)         (None, 32)                0         
                                                                 
 dense (Dense)               (None, 1)                 33        
                                                                 
=================================================================
Total params: 3201 (12.50 KB)
Trainable params: 3201 (12.50 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

1.5 Train

Two callbacks do the heavy lifting, and both are regularizers in their own right:

  • EarlyStopping halts training once validation loss stops improving and — crucially — restore_best_weights=True rolls the model back to its best epoch. This is early stopping, the cheapest form of regularization there is.
  • ReduceLROnPlateau anneals the learning rate when progress stalls, letting the optimizer settle into a sharper minimum.
callbacks = [
    keras.callbacks.EarlyStopping(monitor="val_loss", patience=20, restore_best_weights=True),
    keras.callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.5, patience=8, min_lr=1e-5),
]

history = model.fit(
    X_tr, y_tr,
    validation_data=(X_va, y_va),
    epochs=150,
    batch_size=64,
    shuffle=True,       # OK: the *windows* may be shuffled, the series was never shuffled
    callbacks=callbacks,
    verbose=0,
)
print(f"stopped after {len(history.history['loss'])} epochs; "
      f"best val_loss = {min(history.history['val_loss']):.5f}")
stopped after 99 epochs; best val_loss = 0.01243

1.6 Diagnose the training run

The two curves should descend together. A validation curve that turns upward while training keeps falling is the signature of overfitting — the dotted line marks the epoch EarlyStopping restored.

plot_history(history, "Section 1 — univariate RNN")

1.7 Results on the test set

Predictions are made in scaled space, then mapped back to the original units with \(y = z\,\sigma_{\text{train}} + \mu_{\text{train}}\) so every metric below is in the units of the data.

We also compute a persistence baseline (\(\hat{y}_t = y_{t-1}\), “tomorrow looks like today”). Any forecasting model that cannot beat persistence has learned nothing useful.

def evaluate(model, X, y_scaled, unscale_fn):
    y_true = unscale_fn(y_scaled).ravel()
    y_pred = unscale_fn(model.predict(X, verbose=0)).ravel()
    return y_true, y_pred

rows = {}
for name, X, ys in [("train", X_tr, y_tr), ("val", X_va, y_va), ("test", X_te, y_te)]:
    yt, yp = evaluate(model, X, ys, unscale)
    rows[f"RNN ({name})"] = regression_metrics(yt, yp)

# persistence baseline on the test set: predict the last observed value of each window
y_true_te, y_pred_te = evaluate(model, X_te, y_te, unscale)
y_naive_te = unscale(X_te[:, -1, 0])
rows["Persistence (test)"] = regression_metrics(y_true_te, y_naive_te)

pd.DataFrame(rows).T.round(4)
MAE RMSE R2 MAPE%
RNN (train) 0.1381 0.1743 0.9893 1.2948
RNN (val) 0.1481 0.1867 0.9863 1.2536
RNN (test) 0.1625 0.2085 0.9820 1.3194
Persistence (test) 0.2291 0.2837 0.9667 1.8821

The RNN should beat persistence comfortably and land close to the noise floor (RMSE \(\approx\) 0.15) — that is the sign of a model that has learned the signal and is no longer able to do better only because the remaining error is pure randomness.

fig, ax = plt.subplots(2, 1, figsize=(11, 5.5))

ax[0].plot(y_true_te, color=C["test"], lw=1.1, label="actual")
ax[0].plot(y_pred_te, color=C["pred"], lw=0.9, ls="--", label="RNN prediction")
ax[0].set(ylabel="y", title="Test set — full horizon"); ax[0].legend(frameon=False, ncol=2)

z = slice(0, 200)   # zoom in: at full width the two curves lie on top of each other
ax[1].plot(y_true_te[z], color=C["test"], lw=1.4, marker="o", ms=2.5, label="actual")
ax[1].plot(y_pred_te[z], color=C["pred"], lw=1.2, ls="--", label="RNN prediction")
ax[1].set(xlabel="step within test set", ylabel="y", title="Test set — first 200 steps (zoom)")
ax[1].legend(frameon=False, ncol=2)
fig.tight_layout(); plt.show()

Parity plot and residual diagnostics

  • Parity plot — predicted vs actual. Perfect predictions sit on the 45° line; systematic curvature or a slope \(\neq 1\) reveals bias.
  • Residuals vs time — should look like structureless noise. Any leftover pattern (drift, oscillation) is signal the model failed to capture.
  • Residual histogram — should be centred at zero, roughly Gaussian, with a spread close to the irreducible noise level.
resid = y_true_te - y_pred_te
m = regression_metrics(y_true_te, y_pred_te)

fig, ax = plt.subplots(1, 3, figsize=(12.5, 3.6))

lo, hi = y_true_te.min(), y_true_te.max()
ax[0].scatter(y_true_te, y_pred_te, s=6, alpha=0.35, color=C["test"], edgecolors="none")
ax[0].plot([lo, hi], [lo, hi], "k--", lw=1)
ax[0].set(xlabel="actual", ylabel="predicted", title="Parity plot (test)", aspect="equal")
ax[0].text(0.05, 0.92, f"$R^2$ = {m['R2']:.4f}\nMAE = {m['MAE']:.3f}\nRMSE = {m['RMSE']:.3f}",
           transform=ax[0].transAxes, va="top", fontsize=9,
           bbox=dict(fc="white", ec="0.8"))

ax[1].scatter(np.arange(len(resid)), resid, s=5, alpha=0.35, color=C["test"], edgecolors="none")
ax[1].axhline(0, color="k", lw=1)
ax[1].set(xlabel="step within test set", ylabel="residual", title="Residuals vs time")

ax[2].hist(resid, bins=40, color=C["test"], alpha=0.75, edgecolor="white")
ax[2].axvline(0, color="k", lw=1)
ax[2].set(xlabel="residual", ylabel="count",
          title=f"Residuals (sd={resid.std():.3f}, noise={eps.std():.3f})")

fig.tight_layout(); plt.show()

Section 2 — Three time series

Same DGP, three different parameterizations: different growth rates, amplitudes, periods, phases and noise levels. The three series are stacked into a matrix of shape (T, 3).

What changes: the input window becomes (WINDOW, 3) and the output head becomes Dense(3). The RNN now consumes all three channels at every timestep and predicts the next value of all three at once, so it can exploit any shared structure across them.

What does not change: every other line of the workflow.

PARAMS = [
    dict(name="A", level=10.0, growth=8.0e-5, a1=2.0, p1=200, a2=1.0, p2=25, phase=0.8, noise=0.15),
    dict(name="B", level= 5.0, growth=1.5e-4, a1=1.2, p1=120, a2=0.6, p2=18, phase=2.1, noise=0.10),
    dict(name="C", level=20.0, growth=4.0e-5, a1=3.0, p1=300, a2=1.5, p2=40, phase=4.0, noise=0.25),
]
NAMES = [p["name"] for p in PARAMS]
K = len(PARAMS)

rng = np.random.default_rng(SEED + 1)
series, noises = [], []
for p in PARAMS:
    q = {k: v for k, v in p.items() if k != "name"}
    s, _, _, e = make_series(n=N, rng=rng, **q)
    series.append(s); noises.append(e.std())
Y = np.column_stack(series)            # (T, 3)

fig, ax = plt.subplots(K, 1, figsize=(11, 6), sharex=True)
for k in range(K):
    ax[k].plot(t, Y[:, k], lw=0.7, color="#333")
    ax[k].set(ylabel=f"series {NAMES[k]}")
    ax[k].set_title(f"periods {PARAMS[k]['p1']}/{PARAMS[k]['p2']}, "
                    f"growth {PARAMS[k]['growth']:.0e}, noise sd {noises[k]:.2f}", fontsize=9)
ax[-1].set_xlabel("t")
fig.suptitle("Three series from the same DGP, different parameterizations", y=1.0)
fig.tight_layout(); plt.show()

2.1 Split, scale, window

Identical to Section 1, with two details generalized:

  • the split is applied to rows of the matrix, so all three channels are cut at the same instants;
  • scaling is per column — each series has its own \(\mu, \sigma\) from the training rows, because their levels and spreads differ by design.
Tr_raw = Y[:n_train]
Va_raw = Y[n_train - WINDOW : n_train + n_val]
Te_raw = Y[n_train + n_val - WINDOW :]

mu_k, sd_k = Tr_raw.mean(axis=0), Tr_raw.std(axis=0)     # per-series, TRAIN ONLY
scale_k   = lambda A: (A - mu_k) / sd_k
unscale_k = lambda A: A * sd_k + mu_k

MX_tr, My_tr = make_windows(scale_k(Tr_raw), WINDOW, HORIZON)
MX_va, My_va = make_windows(scale_k(Va_raw), WINDOW, HORIZON)
MX_te, My_te = make_windows(scale_k(Te_raw), WINDOW, HORIZON)

print("per-series train mu:", np.round(mu_k, 2), " sd:", np.round(sd_k, 2))
print("X shape (samples, timesteps, features):", MX_tr.shape, " y:", My_tr.shape)
per-series train mu: [10.95  5.9  20.86]  sd: [1.67 1.09 2.38]
X shape (samples, timesteps, features): (2060, 40, 3)  y: (2060, 3)

2.2 Build and train

Same architecture, wider (64 units) because there are now three interacting channels to model, and a three-unit output head. Same regularization: dropout + early stopping + LR annealing. Note that the extra width is doing real work here — with only 32 units the shared hidden state cannot represent three different oscillation periods at once and the model fails to beat persistence.

mmodel = build_rnn(WINDOW, n_features=K, n_outputs=K, units=64, dropout=0.1)
mmodel.summary()

mcallbacks = [
    keras.callbacks.EarlyStopping(monitor="val_loss", patience=20, restore_best_weights=True),
    keras.callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.5, patience=8, min_lr=1e-5),
]

mhistory = mmodel.fit(
    MX_tr, My_tr,
    validation_data=(MX_va, My_va),
    epochs=150, batch_size=64, shuffle=True,
    callbacks=mcallbacks, verbose=0,
)
print(f"stopped after {len(mhistory.history['loss'])} epochs; "
      f"best val_loss = {min(mhistory.history['val_loss']):.5f}")

plot_history(mhistory, "Section 2 — multivariate RNN")
WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.Adam` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.Adam`.
WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., `tf.keras.optimizers.legacy.Adam`.
Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 simple_rnn_2 (SimpleRNN)    (None, 40, 64)            4352      
                                                                 
 dropout_2 (Dropout)         (None, 40, 64)            0         
                                                                 
 simple_rnn_3 (SimpleRNN)    (None, 64)                8256      
                                                                 
 dropout_3 (Dropout)         (None, 64)                0         
                                                                 
 dense_1 (Dense)             (None, 3)                 195       
                                                                 
=================================================================
Total params: 12803 (50.01 KB)
Trainable params: 12803 (50.01 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
stopped after 119 epochs; best val_loss = 0.02063

2.3 Results

The loss is the average MSE across the three channels, so we now report metrics per series — an aggregate number could hide one badly-fit channel behind two good ones. Check that the RNN beats persistence on every series, not just on average.

My_true = unscale_k(My_te)                                   # (samples, 3)
My_pred = unscale_k(mmodel.predict(MX_te, verbose=0))
My_naive = unscale_k(MX_te[:, -1, :])                        # persistence baseline

rows = {}
for k, nm in enumerate(NAMES):
    rows[f"RNN — series {nm}"]         = regression_metrics(My_true[:, k], My_pred[:, k])
    rows[f"Persistence — series {nm}"] = regression_metrics(My_true[:, k], My_naive[:, k])
pd.DataFrame(rows).T.round(4)
MAE RMSE R2 MAPE%
RNN — series A 0.1865 0.2308 0.9781 1.5086
Persistence — series A 0.2230 0.2756 0.9688 1.8353
RNN — series B 0.1564 0.1993 0.9563 2.0265
Persistence — series B 0.1721 0.2139 0.9496 2.3184
RNN — series C 0.2719 0.3434 0.9771 1.2510
Persistence — series C 0.3007 0.3779 0.9722 1.3990
fig, ax = plt.subplots(K, 1, figsize=(11, 6.5), sharex=True)
z = slice(0, 250)
for k, nm in enumerate(NAMES):
    ax[k].plot(My_true[z, k], color=C["test"], lw=1.3, label="actual")
    ax[k].plot(My_pred[z, k], color=C["pred"], lw=1.1, ls="--", label="RNN prediction")
    ax[k].set(ylabel=f"series {nm}")
    if k == 0: ax[k].legend(frameon=False, ncol=2)
ax[-1].set_xlabel("step within test set")
fig.suptitle("Test-set predictions, first 250 steps", y=1.0)
fig.tight_layout(); plt.show()

fig, ax = plt.subplots(2, K, figsize=(12.5, 6.5))
for k, nm in enumerate(NAMES):
    yt, yp = My_true[:, k], My_pred[:, k]
    r = yt - yp
    mk = regression_metrics(yt, yp)

    lo, hi = yt.min(), yt.max()
    ax[0, k].scatter(yt, yp, s=6, alpha=0.3, color=C["test"], edgecolors="none")
    ax[0, k].plot([lo, hi], [lo, hi], "k--", lw=1)
    ax[0, k].set(xlabel="actual", ylabel="predicted" if k == 0 else "",
                 title=f"Parity — series {nm}", aspect="equal")
    ax[0, k].text(0.05, 0.92, f"$R^2$={mk['R2']:.4f}\nRMSE={mk['RMSE']:.3f}",
                  transform=ax[0, k].transAxes, va="top", fontsize=9,
                  bbox=dict(fc="white", ec="0.8"))

    ax[1, k].hist(r, bins=35, color=C["test"], alpha=0.75, edgecolor="white")
    ax[1, k].axvline(0, color="k", lw=1)
    ax[1, k].set(xlabel="residual", ylabel="count" if k == 0 else "",
                 title=f"sd={r.std():.3f}  (noise {noises[k]:.2f})")

fig.suptitle("Test-set diagnostics per series", y=1.0)
fig.tight_layout(); plt.show()

Takeaways

  1. Windowing turns forecasting into supervised regression. (samples, timesteps, features) is the only tensor shape you need to remember; univariate and multivariate differ only in features.
  2. Split by time, never at random, and fit the scaler on the training slice alone. These two rules prevent the overwhelming majority of “my model was great in validation and useless in production”.
  3. Regularize with dropout + early stopping. The learning curves tell you immediately whether you have the capacity/regularization balance right.
  4. Always compare against a naive baseline. Persistence is free and surprisingly hard to beat.
  5. Judge the model against the noise floor. Residuals whose standard deviation approaches the irreducible noise mean the model has extracted essentially all the available signal — pushing further only fits noise.

Natural extensions

  • Swap SimpleRNN for LSTM or GRU — a one-line change — and compare. Vanilla RNNs struggle to propagate gradients across long windows; that is exactly what gated cells were invented to fix.
  • Forecast further ahead (HORIZON > 1), or emit a whole trajectory (Dense(H)), or feed predictions back in autoregressively and watch the errors compound.
  • Our exponential trend is mild, so the test values stay inside the training range. Push the growth rate up and the model will systematically under-predict: an RNN interpolates, it does not extrapolate a trend it has never seen. The standard remedies are differencing the series or normalizing each window by its own last value.
"""
Multivariate correlogram ("cross-correlation matrix plot").

Diagonal    : autocorrelation function (ACF) of each series
Off-diagonal: cross-correlation function (CCF) over +/- lags

Sign convention (Box-Jenkins):
    r_xy(k) = corr(x[t+k], y[t])
A peak at POSITIVE k means y LEADS x by k steps.
"""

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator

# ----------------------------------------------------------------------
# Correlation machinery
# ----------------------------------------------------------------------

def ccf(x, y, maxlag):
    """Cross-correlation r_xy(k) = corr(x[t+k], y[t]) for k in [-maxlag, maxlag]."""
    x = np.asarray(x, float)
    y = np.asarray(y, float)
    n = len(x)
    xc = x - x.mean()
    yc = y - y.mean()
    denom = n * x.std() * y.std()

    lags = np.arange(-maxlag, maxlag + 1)
    r = np.empty_like(lags, dtype=float)
    for i, k in enumerate(lags):
        if k >= 0:
            r[i] = np.dot(xc[k:], yc[: n - k]) / denom
        else:
            r[i] = np.dot(xc[: n + k], yc[-k:]) / denom
    return lags, r


def acf(x, maxlag):
    """One-sided autocorrelation, lags 0..maxlag."""
    lags, r = ccf(x, x, maxlag)
    return lags[maxlag:], r[maxlag:]


# ----------------------------------------------------------------------
# Sample data: cosine leads sine by a quarter period
# ----------------------------------------------------------------------

rng = np.random.default_rng(7)

N        = 600      # samples
PERIOD   = 50       # samples per cycle
MAXLAG   = 60       # lags to display
NOISE    = 0.25

t = np.arange(N)
s1 = np.sin(2 * np.pi * t / PERIOD) + NOISE * rng.standard_normal(N)
s2 = np.cos(2 * np.pi * t / PERIOD) + NOISE * rng.standard_normal(N)

series = {"sine": s1, "cosine": s2}
names  = list(series)
k      = len(names)

# ----------------------------------------------------------------------
# Style
# ----------------------------------------------------------------------

INK    = "#2B2B2B"
MUTED  = "#9A9A94"
GRID   = "#E6E4DF"
BAND   = "#B9B5AC"
PAPER  = "#FDFCF9"
COLORS = {"sine": "#C1666B", "cosine": "#3C8D8A"}   # rose / teal
CROSS  = "#6B6560"                                   # neutral for off-diagonal

plt.rcParams.update({
    "font.family": "DejaVu Sans",
    "font.size": 9,
    "axes.edgecolor": MUTED,
    "axes.linewidth": 0.8,
    "xtick.color": MUTED,
    "ytick.color": MUTED,
    "xtick.direction": "out",
    "ytick.direction": "out",
})

conf = 1.96 / np.sqrt(N)   # white-noise band -- see caveat in the write-up


def stems(ax, lags, r, color):
    ax.vlines(lags, 0, r, color=color, lw=1.5, alpha=0.85, zorder=3)
    ax.plot(lags, r, "o", ms=3.0, color=color,
            mec=PAPER, mew=0.6, zorder=4)


# ----------------------------------------------------------------------
# Figure
# ----------------------------------------------------------------------

fig, axes = plt.subplots(
    k, k, figsize=(9.2, 7.4), sharey=True,
    gridspec_kw=dict(hspace=0.30, wspace=0.12,
                     left=0.10, right=0.97, top=0.83, bottom=0.11),
)
fig.patch.set_facecolor(PAPER)

for i, row in enumerate(names):        # row  -> x  (the "t+k" series)
    for j, col in enumerate(names):    # col  -> y  (the reference series)
        ax = axes[i, j]
        ax.set_facecolor(PAPER)

        # confidence band + zero line
        ax.axhspan(-conf, conf, color=BAND, alpha=0.22, lw=0, zorder=0)
        ax.axhline(0, color=MUTED, lw=0.8, zorder=1)

        if i == j:
            lags, r = acf(series[row], MAXLAG)
            stems(ax, lags, r, COLORS[row])
            ax.set_title(f"ACF · {row}", fontsize=9.5, color=INK,
                         pad=6, loc="left", fontweight="bold")
            ax.set_xlim(-2, MAXLAG + 2)
        else:
            lags, r = ccf(series[row], series[col], MAXLAG)
            stems(ax, lags, r, CROSS)
            ax.axvline(0, color=MUTED, lw=0.8, ls=(0, (3, 3)), zorder=1)
            ax.set_title(f"CCF · {row} vs {col}", fontsize=9.5, color=INK,
                         pad=6, loc="left", fontweight="bold")
            ax.set_xlim(-MAXLAG - 2, MAXLAG + 2)

            # Annotate the dominant peak -> this is the lead/lag you care about.
            # NB: a periodic CCF has equally-good peaks every PERIOD steps, so
            # take the strongest POSITIVE correlation within one half-period.
            win = np.abs(lags) <= PERIOD // 2
            idx = np.flatnonzero(win)[np.argmax(r[win])]
            pk = lags[idx]
            ax.annotate(
                f"peak k = {pk:+d}",
                xy=(pk, r[idx]),
                xytext=(0.5, 0.06), textcoords="axes fraction",
                fontsize=8, color=INK, ha="center",
                arrowprops=dict(arrowstyle="-", lw=0.8, color=INK,
                                shrinkA=0, shrinkB=3,
                                connectionstyle="arc3,rad=0.2"),
                bbox=dict(boxstyle="round,pad=0.3", fc="white",
                          ec=GRID, lw=0.7),
            )

        ax.set_ylim(-1.12, 1.12)
        ax.grid(axis="y", color=GRID, lw=0.7, zorder=0)
        ax.xaxis.set_major_locator(MultipleLocator(25))
        ax.yaxis.set_major_locator(MultipleLocator(0.5))
        for side in ("top", "right"):
            ax.spines[side].set_visible(False)
        ax.spines["left"].set_color(MUTED)
        ax.spines["bottom"].set_color(MUTED)

        if i == k - 1:
            ax.set_xlabel("lag  $k$", color=INK, fontsize=9)
        if j == 0:
            ax.set_ylabel("correlation", color=INK, fontsize=9)

fig.suptitle("Multivariate correlogram",
             x=0.10, y=0.955, ha="left",
             fontsize=15, color=INK, fontweight="bold")
fig.text(0.10, 0.905,
         r"$r_{xy}(k)=\mathrm{corr}\,(x_{t+k},\,y_t)$   ·   "
         "peak at $k>0$ ⟹ the column series leads the row series",
         ha="left", fontsize=9.5, color=MUTED)
fig.text(0.10, 0.030,
         f"n = {N}   ·   shaded band = ±1.96/√n (white-noise null)",
         ha="left", fontsize=8, color=MUTED)

fig.savefig("correlogram.png", dpi=200, facecolor=PAPER)
plt.show()

# Sanity check: the recovered lead/lag should be the quarter period.
lags, r = ccf(s1, s2, MAXLAG)
win = np.abs(lags) <= PERIOD // 2
print(f"recovered peak lag : {lags[np.flatnonzero(win)[np.argmax(r[win])]]:+d}")
print(f"quarter period     : {PERIOD / 4:+.1f}")

recovered peak lag : +12
quarter period     : +12.5
"""
Multivariate correlogram: a pair-plot for time series.
  diagonal     -> autocorrelation (ACF) of each series
  off-diagonal -> cross-correlation (CCF) over +/- lags

Sign convention (Box-Jenkins):   r_xy(k) = corr( x[t+k], y[t] )
so in panel (row = x, col = y), a peak at POSITIVE k means the COLUMN
series LEADS the ROW series by k steps.

Since r_xy(k) = r_yx(-k), the upper and lower triangles are mirror images.
That redundancy is deliberate: each panel is independently readable.
(R's acf() halves it, showing +lags above the diagonal and -lags below.)

Self-contained: numpy + pandas + matplotlib only.
"""

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.ticker import MaxNLocator

MAX_SERIES = 6

# ==========================================================================
# Correlation machinery
# ==========================================================================

def ccf(x, y, maxlag):
    """Cross-correlation r_xy(k) = corr(x[t+k], y[t]) for k in [-maxlag, maxlag]."""
    x = np.asarray(x, dtype=float)
    y = np.asarray(y, dtype=float)
    if x.shape != y.shape:
        raise ValueError("x and y must have the same length")

    ok = np.isfinite(x) & np.isfinite(y)
    x, y = x[ok], y[ok]
    n = x.size
    if n <= maxlag:
        raise ValueError(f"need more than maxlag={maxlag} valid points, got {n}")

    xc, yc = x - x.mean(), y - y.mean()
    denom = n * x.std() * y.std()
    if denom == 0:
        raise ValueError("a series is constant; correlation undefined")

    lags = np.arange(-maxlag, maxlag + 1)
    r = np.empty(lags.size, dtype=float)
    for i, k in enumerate(lags):
        if k >= 0:
            r[i] = np.dot(xc[k:], yc[: n - k]) / denom
        else:
            r[i] = np.dot(xc[: n + k], yc[-k:]) / denom
    return lags, r


def acf(x, maxlag):
    """One-sided autocorrelation, lags 0..maxlag."""
    lags, r = ccf(x, x, maxlag)
    return lags[maxlag:], r[maxlag:]


# ==========================================================================
# The plot
# ==========================================================================

INK, MUTED, GRID, BAND, PAPER = "#2B2B2B", "#9A9A94", "#E6E4DF", "#B9B5AC", "#FDFCF9"
CROSS = "#6B6560"
PALETTE = ["#C1666B", "#3C8D8A", "#5A6C97", "#C98A3C", "#7C6A93", "#6E8B4E"]


def correlogram_matrix(
    series,
    names=None,
    maxlag=40,
    conf=True,
    annotate_peak=None,   # None -> auto (on for <= 3 series)
    peak_window=None,     # restrict peak search to |k| <= this
    ylim=None,
    title="Multivariate correlogram",
    subtitle=None,
    figsize=None,
):
    """ACF/CCF matrix for up to six time series.

    series : list of 1-D arrays, or a DataFrame (columns = series)
    names  : list of str; taken from DataFrame columns if omitted
    conf   : draw the +/- 1.96/sqrt(n) white-noise band (valid only under
             a white-noise null -- see the prices-vs-returns demo below)
    peak_window : for periodic data, whose CCF has equally-good peaks every
             period, search the peak only within |k| <= peak_window

    Returns (fig, axes).
    """
    # ---- unpack -----------------------------------------------------------
    if hasattr(series, "columns"):                       # DataFrame
        if names is None:
            names = [str(c) for c in series.columns]
        series = [series[c].to_numpy(dtype=float) for c in series.columns]
    else:
        series = [np.asarray(s, dtype=float).ravel() for s in series]
        if names is None:
            names = [f"series {i+1}" for i in range(len(series))]

    k = len(series)
    if not 1 <= k <= MAX_SERIES:
        raise ValueError(f"supply between 1 and {MAX_SERIES} series, got {k}")
    if len(names) != k:
        raise ValueError("len(names) must match len(series)")
    if len({s.size for s in series}) != 1:
        raise ValueError("all series must be the same length")

    n = min(np.isfinite(s).sum() for s in series)
    if annotate_peak is None:
        annotate_peak = k <= 3

    # ---- adaptive density -------------------------------------------------
    dense  = k >= 4
    lw     = 0.8 if dense else 1.5
    ms     = 0.0 if dense else 3.0          # drop markers once panels shrink
    lab_fs = 8 if dense else 9.5
    if figsize is None:
        figsize = (2.05 * k + 1.4, 1.75 * k + 1.9)

    band = 1.96 / np.sqrt(n)

    plt.rcParams.update({
        "font.family": "DejaVu Sans", "font.size": 8.5,
        "axes.edgecolor": MUTED, "axes.linewidth": 0.8,
        "xtick.color": MUTED, "ytick.color": MUTED,
    })

    fig, axes = plt.subplots(
        k, k, figsize=figsize, squeeze=False, sharey=True,
        gridspec_kw=dict(hspace=0.22, wspace=0.10,
                         left=0.115, right=0.975, top=0.845, bottom=0.105),
    )
    fig.patch.set_facecolor(PAPER)

    lo, hi = 0.0, 0.0
    for i in range(k):            # row -> x  (the "t+k" series)
        for j in range(k):        # col -> y  (the reference series)
            ax = axes[i, j]
            ax.set_facecolor(PAPER)

            if conf:
                ax.axhspan(-band, band, color=BAND, alpha=0.22, lw=0, zorder=0)
            ax.axhline(0, color=MUTED, lw=0.8, zorder=1)

            if i == j:
                lags, r = acf(series[i], maxlag)
                color = PALETTE[i % len(PALETTE)]
                ax.set_xlim(-1, maxlag + 1)
            else:
                lags, r = ccf(series[i], series[j], maxlag)
                color = CROSS
                ax.axvline(0, color=MUTED, lw=0.8, ls=(0, (3, 3)), zorder=1)
                ax.set_xlim(-maxlag - 1, maxlag + 1)

            ax.vlines(lags, 0, r, color=color, lw=lw, alpha=0.85, zorder=3)
            if ms:
                ax.plot(lags, r, "o", ms=ms, color=color,
                        mec=PAPER, mew=0.6, zorder=4)

            lo, hi = min(lo, r.min()), max(hi, r.max())

            if i != j and annotate_peak:
                sel = (np.abs(lags) <= peak_window) if peak_window \
                      else np.ones_like(lags, dtype=bool)
                idx = np.flatnonzero(sel)[np.argmax(r[sel])]
                ax.annotate(
                    f"$k={lags[idx]:+d}$",
                    xy=(lags[idx], r[idx]),
                    xytext=(0.5, 0.06), textcoords="axes fraction",
                    fontsize=7.5, color=INK, ha="center",
                    arrowprops=dict(arrowstyle="-", lw=0.7, color=INK,
                                    shrinkA=0, shrinkB=3,
                                    connectionstyle="arc3,rad=0.2"),
                    bbox=dict(boxstyle="round,pad=0.25", fc="white",
                              ec=GRID, lw=0.6),
                )

            ax.grid(axis="y", color=GRID, lw=0.7, zorder=0)
            ax.xaxis.set_major_locator(MaxNLocator(3, integer=True))
            ax.yaxis.set_major_locator(MaxNLocator(4))
            for side in ("top", "right"):
                ax.spines[side].set_visible(False)

            # pair-plot margin labels: names on the outside, not on each panel
            if j == 0:
                ax.set_ylabel(names[i], color=PALETTE[i % len(PALETTE)],
                              fontsize=lab_fs, fontweight="bold", labelpad=8)
            if i == 0:
                ax.set_title(names[j], color=PALETTE[j % len(PALETTE)],
                             fontsize=lab_fs, fontweight="bold", pad=8)
            if i == k - 1:
                ax.set_xlabel("lag $k$", color=INK, fontsize=8.5)
            else:
                ax.tick_params(labelbottom=False)

    pad = 0.10 * max(hi - lo, 0.2)
    for ax in axes.ravel():
        ax.set_ylim((lo - pad, hi + pad) if ylim is None else ylim)

    if subtitle is None:
        subtitle = (r"$r_{xy}(k)=\mathrm{corr}(x_{t+k},\,y_t)$   ·   "
                    r"peak at $k>0$  ⟹  column leads row")
    fig.suptitle(title, x=0.115, y=0.965, ha="left",
                 fontsize=14, color=INK, fontweight="bold")
    fig.text(0.115, 0.920, subtitle, ha="left", fontsize=9, color=MUTED)
    foot = f"n = {n}   ·   rows = $x$,  columns = $y$"
    if conf:
        foot += "   ·   shaded band = ±1.96/√n (white-noise null)"
    fig.text(0.115, 0.028, foot, ha="left", fontsize=7.5, color=MUTED)

    return fig, axes


# ==========================================================================
# Data
# ==========================================================================

TICKERS = ["AAPL", "MSFT", "NVDA", "JPM", "XOM", "JNJ"]
START, END = "2013-02-08", "2018-02-07"


def load_prices(tickers=TICKERS, start=START, end=END):
    """Daily closes as a DataFrame. Tries yfinance, falls back to a CSV."""
    try:
        import yfinance as yf
        df = yf.download(tickers, start=start, end=end,
                         auto_adjust=True, progress=False)["Close"]
        if not df.empty:
            return df[tickers].dropna()
    except Exception as e:
        print(f"[yfinance unavailable: {e}]")

    print("[falling back to S&P-500 daily CSV on GitHub]")
    url = ("https://raw.githubusercontent.com/plotly/datasets/"
           "master/all_stocks_5yr.csv")
    raw = pd.read_csv(url, parse_dates=["date"])
    return (raw[raw["Name"].isin(tickers)]
            .pivot(index="date", columns="Name", values="close")
            .loc[start:end, tickers]
            .dropna())


# ==========================================================================
# Demo
# ==========================================================================

if __name__ == "__main__":

    # --- 1. sanity check: cosine leads sine by a quarter period ------------
    rng = np.random.default_rng(7)
    N, PERIOD = 600, 50
    t = np.arange(N)
    demo = pd.DataFrame({
        "sine":   np.sin(2 * np.pi * t / PERIOD) + 0.25 * rng.standard_normal(N),
        "cosine": np.cos(2 * np.pi * t / PERIOD) + 0.25 * rng.standard_normal(N),
    })
    fig, _ = correlogram_matrix(
        demo, maxlag=60, peak_window=PERIOD // 2,
        title="Correlogram · synthetic sine & cosine",
        subtitle=f"cosine leads sine by a quarter period = {PERIOD/4:.1f} samples",
    )
    fig.savefig("corr_sine.png", dpi=190, facecolor=fig.get_facecolor())
    print(f"sine/cosine: expect a peak near k = +{PERIOD/4:.1f}")

    # --- 2. raw prices: the WRONG way (non-stationary) ---------------------
    px = load_prices()
    print(f"\nloaded {px.shape[0]} days x {px.shape[1]} tickers "
          f"({px.index.min().date()} to {px.index.max().date()})")

    fig, _ = correlogram_matrix(
        px, maxlag=40, conf=False, ylim=(-1.1, 1.1),
        title="Correlogram · raw prices  (DON'T DO THIS)",
        subtitle="Prices have a unit root. Everything correlates with "
                 "everything at every lag; the plot says nothing.",
    )
    fig.savefig("corr_prices.png", dpi=190, facecolor=fig.get_facecolor())

    # --- 3. log returns: the right way ------------------------------------
    ret = np.log(px).diff().dropna()

    fig, _ = correlogram_matrix(
        ret, maxlag=20, ylim=(-0.25, 1.05),
        title="Correlogram · daily log returns",
        subtitle="Differencing induces stationarity. Structure now sits almost "
                 "entirely at $k=0$: the market factor.",
    )
    fig.savefig("corr_returns.png", dpi=190, facecolor=fig.get_facecolor())

    print("\nlag-0 cross-correlation of returns:")
    print(ret.corr().round(2).to_string())

    worst = max(
        np.abs(ccf(ret[a], ret[b], 20)[1][ccf(ret[a], ret[b], 20)[0] != 0]).max()
        for a in ret.columns for b in ret.columns if a != b
    )
    print(f"\nmax |CCF| at any nonzero lag, over all pairs: {worst:.3f}")
    print(f"white-noise band (±1.96/√n):                  "
          f"{1.96/np.sqrt(len(ret)):.3f}")

    plt.show()
sine/cosine: expect a peak near k = +12.5
[yfinance unavailable: No module named 'yfinance']
[falling back to S&P-500 daily CSV on GitHub]

loaded 1259 days x 6 tickers (2013-02-08 to 2018-02-07)

lag-0 cross-correlation of returns:
Name  AAPL  MSFT  NVDA   JPM   XOM   JNJ
Name                                    
AAPL  1.00  0.37  0.30  0.33  0.26  0.24
MSFT  0.37  1.00  0.36  0.41  0.33  0.34
NVDA  0.30  0.36  1.00  0.31  0.20  0.20
JPM   0.33  0.41  0.31  1.00  0.50  0.42
XOM   0.26  0.33  0.20  0.50  1.00  0.44
JNJ   0.24  0.34  0.20  0.42  0.44  1.00

max |CCF| at any nonzero lag, over all pairs: 0.115
white-noise band (±1.96/√n):                  0.055