The Canonical Keras Workflow

Training a Multi-Layer Perceptron on MNIST

Author

DSAN

Published

July 15, 2026

Almost every deep learning project — no matter how fancy — follows the same steps:

# Step Question it answers
1 Load the data What am I working with?
2 Explore & visualize What does it actually look like?
3 Preprocess What shape/scale does the network need?
4 Split train / validation How will I know if I’m overfitting?
5 Build & compile the model What function am I fitting, and how do I score it?
6 Train Turn the crank.
7 Evaluate Did it work? Where does it fail?
8 Tune Could a different setting do better?

Our task: look at a 28×28 grayscale image of a handwritten digit and say which digit (0–9) it is.

Keep one idea in the back of your mind the whole time: a neural network is just a function. It takes 784 numbers in and produces 10 numbers out. Training is the process of nudging its internal knobs (weights) until that function is usually right.

0.1 Setup

import numpy as np
import matplotlib.pyplot as plt

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, regularizers

# Fix the random seeds so the notebook is reproducible run-to-run.
np.random.seed(42)
tf.random.set_seed(42)

print("TensorFlow:", tf.__version__)
TensorFlow: 2.13.0

0.2 Load the data

MNIST ships with Keras, so loading is a one-liner. It arrives already divided into a training set (60,000 images) and a test set (10,000 images).

The test set is a locked box. We will not look at it again until the very end.

(x_train_full, y_train_full), (x_test, y_test) = keras.datasets.mnist.load_data()

print("Training images:", x_train_full.shape, "| labels:", y_train_full.shape)
print("Test images:    ", x_test.shape, "| labels:", y_test.shape)
Training images: (60000, 28, 28) | labels: (60000,)
Test images:     (10000, 28, 28) | labels: (10000,)

0.3 Explore and visualize

Never train on data you haven’t looked at. Three quick questions:

  1. What type and range are the pixel values?
  2. Are the classes balanced?
  3. What do the images actually look like?
print("dtype:", x_train_full.dtype)
print("pixel min:", x_train_full.min(), " pixel max:", x_train_full.max())
print("labels present:", np.unique(y_train_full))
dtype: uint8
pixel min: 0  pixel max: 255
labels present: [0 1 2 3 4 5 6 7 8 9]

So each image is a 28 × 28 grid of integers from 0 to 255, where 0 = black (background) and 255 = white (ink).

That’s the key insight: an image is just a matrix of numbers. Let’s prove it by printing a corner of one.

img = x_train_full[0]
label = y_train_full[0]

fig, ax = plt.subplots(1, 2, figsize=(11, 5))

# Left: the image as we see it
ax[0].imshow(img, cmap="gray")
ax[0].set_title(f"Image #0 — label = {label}")
ax[0].axis("off")

# Right: the same image as the computer sees it (a 14x14 crop, values printed)
crop = img[7:21, 7:21]
ax[1].imshow(crop, cmap="gray")
for i in range(crop.shape[0]):
    for j in range(crop.shape[1]):
        v = crop[i, j]
        ax[1].text(j, i, str(v), ha="center", va="center", fontsize=6,
                   color="red" if v < 128 else "black")
ax[1].set_title("...the same thing, as numbers (center crop)")
ax[1].axis("off")

plt.tight_layout()
plt.show()

0.3.1 One example of each digit

fig, axes = plt.subplots(2, 5, figsize=(10, 4.5))

for digit, ax in enumerate(axes.flat):
    # find the first image whose label is this digit
    idx = np.where(y_train_full == digit)[0][0]
    ax.imshow(x_train_full[idx], cmap="gray")
    ax.set_title(f"{digit}")
    ax.axis("off")

plt.suptitle("One sample of each class")
plt.tight_layout()
plt.show()

0.3.2 Are the classes balanced?

counts = np.bincount(y_train_full)

plt.figure(figsize=(7, 3.5))
plt.bar(range(10), counts)
plt.xticks(range(10))
plt.xlabel("digit")
plt.ylabel("# of training images")
plt.title("Class balance")
plt.show()

print(counts)

[5923 6742 5958 6131 5842 5421 5918 6265 5851 5949]

Roughly 6,000 images per digit — close enough to balanced that we don’t need to do anything special. (If one class had 50× fewer examples, accuracy would become a misleading metric and we’d have to be more careful.)

0.4 Preprocess

Two changes are needed to feed a plain MLP:

(a) Flatten. A dense layer takes a vector, not a grid. So we unroll each 28 × 28 image into a 784-long vector by reading it row by row.

Note what this throws away: the network no longer knows that pixel 5 and pixel 33 were vertical neighbors. An MLP has to rediscover spatial structure from scratch. This is exactly the weakness that convolutional networks fix — but it’s remarkable how far an MLP still gets.

(b) Rescale to [0, 1]. Divide by 255. Neural networks train much more stably when inputs are small and comparable in size; feeding in raw values up to 255 makes gradients large and erratic.

# (a) flatten: (N, 28, 28) -> (N, 784)
x_train_full_flat = x_train_full.reshape(-1, 28 * 28)
x_test_flat       = x_test.reshape(-1, 28 * 28)

# (b) rescale: 0..255 -> 0.0..1.0
x_train_full_flat = x_train_full_flat.astype("float32") / 255.0
x_test_flat       = x_test_flat.astype("float32") / 255.0

print("shape:", x_train_full_flat.shape)
print("range:", x_train_full_flat.min(), "to", x_train_full_flat.max())
shape: (60000, 784)
range: 0.0 to 1.0

0.5 Split into training and validation

We already have a test set — so why split again?

  • Training set → the model learns from this (weights are updated).
  • Validation set → the model never learns from this. We watch its score during training to detect overfitting, and later we use it to choose hyperparameters.
  • Test set → touched exactly once, at the end, for an honest final number.

If we tuned our choices by peeking at the test set, the test score would stop being an honest estimate of performance on new data. So we carve a validation set out of the training data.

from sklearn.model_selection import train_test_split

x_train, x_val, y_train, y_val = train_test_split(
    x_train_full_flat, y_train_full,
    test_size=0.2,          # 20% held out for validation
    stratify=y_train_full,  # keep the class balance in both halves
    random_state=42,
)

print("train:", x_train.shape[0], "images")
print("val:  ", x_val.shape[0], "images")
print("test: ", x_test_flat.shape[0], "images")
train: 48000 images
val:   12000 images
test:  10000 images

0.6 Build and compile the model

Here is the whole architecture:

784 inputs  →  Dense(128, relu)  →  Dense(64, relu)  →  Dense(10, softmax)

Reading it left to right:

  • Dense = “fully connected”: every neuron sees every value from the previous layer. Each neuron computes w · x + b — a weighted vote over its inputs.
  • relu = max(0, z). Without this nonlinearity, stacking layers would be pointless: a chain of linear maps collapses into a single linear map. ReLU is what lets the network build curved decision boundaries.
  • The layers get narrower (784 → 128 → 64 → 10). Think of it as a funnel: raw pixels → coarse strokes/edges → digit-ish features → a decision.
  • softmax on the last layer turns 10 arbitrary scores into 10 probabilities that sum to 1.

0.6.1 One more ingredient: L2 regularization

Our model has ~109,000 knobs but only 48,000 training images. With that much freedom it can simply memorize the training set — including its noise — instead of learning what a “3” looks like.

L2 regularization fights this by changing what we ask the optimizer to minimize:

\[ \text{total loss} \;=\; \underbrace{\text{cross-entropy}}_{\text{fit the data}} \;+\; \lambda \underbrace{\sum_i w_i^2}_{\text{keep weights small}} \]

The mechanical picture: think of a spring attached to every weight, pulling it toward zero. The data pulls each weight outward to whatever value fits the training images; the spring pulls it back. Where they balance depends on \(\lambda\), the spring stiffness.

Why does shrinking weights help? Large weights mean the network’s output swings violently in response to small input changes — a sharp, wiggly function that can contort itself around individual training points. Small weights mean a smoother function, and smooth functions generalize better.

  • \(\lambda = 0\) → no spring. Free to memorize.
  • \(\lambda\) too large → springs dominate, all weights crushed toward zero, the model can’t even fit the data (underfitting).
  • Somewhere in between is the sweet spot. We’ll go looking for it in the last section.

In Keras this is one argument: kernel_regularizer=regularizers.l2(lam). Note it penalizes the weights (kernel), not the biases — a bias just shifts the function, it doesn’t make it wiggly.

L2_LAMBDA = 1e-4   # our starting guess; we'll tune this at the end

model = keras.Sequential([
    layers.Input(shape=(784,)),
    layers.Dense(128, activation="relu",
                 kernel_regularizer=regularizers.l2(L2_LAMBDA)),
    layers.Dense(64,  activation="relu",
                 kernel_regularizer=regularizers.l2(L2_LAMBDA)),
    layers.Dense(10,  activation="softmax"),
])

model.summary()
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 128)               100480    
                                                                 
 dense_1 (Dense)             (None, 64)                8256      
                                                                 
 dense_2 (Dense)             (None, 10)                650       
                                                                 
=================================================================
Total params: 109386 (427.29 KB)
Trainable params: 109386 (427.29 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

Sanity-check the parameter count. Layer 1 has 784 × 128 weights plus 128 biases = 100,480 knobs. Layer 2: 128 × 64 + 64 = 8,256. Layer 3: 64 × 10 + 10 = 650. Total ≈ 109,386 numbers that training will adjust. (Regularization doesn’t add parameters — it only changes how they’re penalized.)

0.6.2 Compile = choose the scoring rule and the search strategy

compile() doesn’t train anything. It just attaches three things to the model:

  • loss — the number we are trying to make small. sparse_categorical_crossentropy is essentially “how surprised was the model by the correct answer?” It’s near 0 when the model puts high probability on the true digit, and blows up when it is confidently wrong. “Sparse” just means our labels are plain integers (7) rather than one-hot vectors ([0,...,1,...,0]).
  • optimizer — the rule for updating weights from the gradient. adam is the reliable default.
  • metrics — numbers we want reported but that aren’t used for the update. Accuracy is for humans; the loss is for the optimizer.
model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"],
)

0.7 Train

fit() is the crank. In each epoch the model sees every training image once, in shuffled mini-batches of 128. For every batch it:

  1. predicts,
  2. measures the loss (cross-entropy + the L2 penalty),
  3. computes the gradient of that loss w.r.t. all 109,386 weights (backpropagation),
  4. takes a small step downhill.

At the end of each epoch, Keras also scores the untouched validation set so we can watch for overfitting.

history = model.fit(
    x_train, y_train,
    validation_data=(x_val, y_val),
    epochs=20,
    batch_size=128,
    verbose=2,
)
Epoch 1/20
375/375 - 1s - loss: 0.3983 - accuracy: 0.8969 - val_loss: 0.2158 - val_accuracy: 0.9465 - 843ms/epoch - 2ms/step
Epoch 2/20
375/375 - 1s - loss: 0.1810 - accuracy: 0.9556 - val_loss: 0.1763 - val_accuracy: 0.9576 - 567ms/epoch - 2ms/step
Epoch 3/20
375/375 - 1s - loss: 0.1395 - accuracy: 0.9686 - val_loss: 0.1529 - val_accuracy: 0.9647 - 665ms/epoch - 2ms/step
Epoch 4/20
375/375 - 1s - loss: 0.1174 - accuracy: 0.9745 - val_loss: 0.1358 - val_accuracy: 0.9707 - 630ms/epoch - 2ms/step
Epoch 5/20
375/375 - 1s - loss: 0.1016 - accuracy: 0.9799 - val_loss: 0.1299 - val_accuracy: 0.9714 - 592ms/epoch - 2ms/step
Epoch 6/20
375/375 - 1s - loss: 0.0913 - accuracy: 0.9831 - val_loss: 0.1229 - val_accuracy: 0.9740 - 569ms/epoch - 2ms/step
Epoch 7/20
375/375 - 1s - loss: 0.0811 - accuracy: 0.9859 - val_loss: 0.1218 - val_accuracy: 0.9737 - 568ms/epoch - 2ms/step
Epoch 8/20
375/375 - 1s - loss: 0.0761 - accuracy: 0.9880 - val_loss: 0.1259 - val_accuracy: 0.9746 - 546ms/epoch - 1ms/step
Epoch 9/20
375/375 - 1s - loss: 0.0724 - accuracy: 0.9891 - val_loss: 0.1358 - val_accuracy: 0.9706 - 570ms/epoch - 2ms/step
Epoch 10/20
375/375 - 1s - loss: 0.0696 - accuracy: 0.9897 - val_loss: 0.1212 - val_accuracy: 0.9757 - 702ms/epoch - 2ms/step
Epoch 11/20
375/375 - 1s - loss: 0.0634 - accuracy: 0.9926 - val_loss: 0.1398 - val_accuracy: 0.9680 - 612ms/epoch - 2ms/step
Epoch 12/20
375/375 - 1s - loss: 0.0606 - accuracy: 0.9927 - val_loss: 0.1185 - val_accuracy: 0.9760 - 557ms/epoch - 1ms/step
Epoch 13/20
375/375 - 1s - loss: 0.0600 - accuracy: 0.9931 - val_loss: 0.1303 - val_accuracy: 0.9736 - 550ms/epoch - 1ms/step
Epoch 14/20
375/375 - 1s - loss: 0.0609 - accuracy: 0.9924 - val_loss: 0.1222 - val_accuracy: 0.9762 - 615ms/epoch - 2ms/step
Epoch 15/20
375/375 - 1s - loss: 0.0569 - accuracy: 0.9939 - val_loss: 0.1232 - val_accuracy: 0.9747 - 612ms/epoch - 2ms/step
Epoch 16/20
375/375 - 1s - loss: 0.0551 - accuracy: 0.9945 - val_loss: 0.1287 - val_accuracy: 0.9758 - 562ms/epoch - 1ms/step
Epoch 17/20
375/375 - 1s - loss: 0.0528 - accuracy: 0.9953 - val_loss: 0.1144 - val_accuracy: 0.9791 - 537ms/epoch - 1ms/step
Epoch 18/20
375/375 - 1s - loss: 0.0500 - accuracy: 0.9958 - val_loss: 0.1203 - val_accuracy: 0.9766 - 509ms/epoch - 1ms/step
Epoch 19/20
375/375 - 1s - loss: 0.0523 - accuracy: 0.9948 - val_loss: 0.1237 - val_accuracy: 0.9759 - 537ms/epoch - 1ms/step
Epoch 20/20
375/375 - 1s - loss: 0.0522 - accuracy: 0.9943 - val_loss: 0.1177 - val_accuracy: 0.9762 - 509ms/epoch - 1ms/step

0.8 Training curves

history.history is just a dictionary of lists — one value per epoch. Plotting it is the single most informative diagnostic in deep learning.

How to read these plots: - Both curves still falling → keep training. - Training loss falls, validation loss turns back up → overfitting: the model is memorizing training images instead of learning digits. - Both curves flat and high → underfitting (too small a model, too much regularization, or too few epochs).

⚠️ One caveat now that we have L2: the reported loss includes the penalty term \(\lambda \sum w^2\), so it is no longer a pure measure of data fit. When comparing models with different \(\lambda\), compare accuracy, not loss — otherwise you’re comparing apples to oranges.

h = history.history
epochs = range(1, len(h["loss"]) + 1)

fig, ax = plt.subplots(1, 2, figsize=(12, 4))

ax[0].plot(epochs, h["loss"], "o-", label="train")
ax[0].plot(epochs, h["val_loss"], "o-", label="validation")
ax[0].set_xlabel("epoch"); ax[0].set_ylabel("loss (incl. L2 penalty)")
ax[0].set_title("Loss")
ax[0].legend(); ax[0].grid(alpha=0.3)

ax[1].plot(epochs, h["accuracy"], "o-", label="train")
ax[1].plot(epochs, h["val_accuracy"], "o-", label="validation")
ax[1].set_xlabel("epoch"); ax[1].set_ylabel("accuracy")
ax[1].set_title("Accuracy")
ax[1].legend(); ax[1].grid(alpha=0.3)

plt.tight_layout()
plt.show()

gap = h["accuracy"][-1] - h["val_accuracy"][-1]
print(f"Final generalization gap (train acc - val acc): {gap:.4f}")

Final generalization gap (train acc - val acc): 0.0180

The gap between the two accuracy curves is overfitting, made visible: the model does better on images it has seen than on images it hasn’t. L2 keeps that gap smaller than it would otherwise be, but it doesn’t eliminate it. Remember this number — we’ll watch it shrink as we crank up \(\lambda\).

0.9 Final evaluation on the test set

Now — and only now — we open the locked box.

test_loss, test_acc = model.evaluate(x_test_flat, y_test, verbose=0)

print(f"Test accuracy: {test_acc:.4f}   ({test_acc*100:.2f}%)")
print(f"That's about {int(round((1-test_acc)*len(y_test)))} mistakes out of {len(y_test)} images.")
Test accuracy: 0.9778   (97.78%)
That's about 222 mistakes out of 10000 images.

0.9.1 What the model actually outputs

predict() returns 10 probabilities per image. To get a hard prediction we take the argmax — the index of the largest probability.

probs  = model.predict(x_test_flat, verbose=0)   # shape (10000, 10)
y_pred = probs.argmax(axis=1)                    # shape (10000,)

# Inspect a single test image and its full probability vector
i = 0
fig, ax = plt.subplots(1, 2, figsize=(10, 3.5))

ax[0].imshow(x_test[i], cmap="gray")
ax[0].set_title(f"true = {y_test[i]}   predicted = {y_pred[i]}")
ax[0].axis("off")

ax[1].bar(range(10), probs[i])
ax[1].set_xticks(range(10))
ax[1].set_xlabel("digit")
ax[1].set_ylabel("probability")
ax[1].set_title("Softmax output (sums to 1)")

plt.tight_layout()
plt.show()

print("probabilities sum to:", probs[i].sum())

probabilities sum to: 0.9999999

0.9.2 Confusion matrix

Accuracy is one number, and one number hides everything interesting. The confusion matrix tells us which digits get mistaken for which.

Row = the true digit. Column = what the model guessed. The diagonal is correct; everything off-diagonal is a mistake.

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay, classification_report

cm = confusion_matrix(y_test, y_pred)

fig, ax = plt.subplots(figsize=(7, 6))
ConfusionMatrixDisplay(cm, display_labels=range(10)).plot(
    ax=ax, cmap="Blues", colorbar=False, values_format="d"
)
ax.set_title("Confusion matrix (test set)")
plt.tight_layout()
plt.show()

The diagonal dominates, so let’s zero it out and look only at the errors — that’s where the story is.

cm_errors = cm.copy()
np.fill_diagonal(cm_errors, 0)

fig, ax = plt.subplots(figsize=(7, 6))
im = ax.imshow(cm_errors, cmap="Reds")
ax.set_xticks(range(10)); ax.set_yticks(range(10))
ax.set_xlabel("predicted"); ax.set_ylabel("true")
ax.set_title("Errors only (diagonal removed)")

for i in range(10):
    for j in range(10):
        if cm_errors[i, j] > 0:
            ax.text(j, i, cm_errors[i, j], ha="center", va="center", fontsize=8)

plt.colorbar(im)
plt.tight_layout()
plt.show()

i, j = np.unravel_index(cm_errors.argmax(), cm_errors.shape)
print(f"Most common mistake: true {i} predicted as {j}  ({cm_errors[i, j]} times)")

Most common mistake: true 7 predicted as 9  (14 times)

The mistakes are not random — they cluster where humans would also hesitate (4↔︎9, 3↔︎5, 7↔︎2). That’s reassuring: the model has learned something about shape, not noise.

0.9.3 Per-class scores

print(classification_report(y_test, y_pred, digits=3))
              precision    recall  f1-score   support

           0      0.985     0.988     0.986       980
           1      0.989     0.993     0.991      1135
           2      0.977     0.979     0.978      1032
           3      0.963     0.983     0.973      1010
           4      0.977     0.975     0.976       982
           5      0.982     0.967     0.975       892
           6      0.983     0.980     0.982       958
           7      0.984     0.957     0.970      1028
           8      0.977     0.972     0.975       974
           9      0.961     0.981     0.971      1009

    accuracy                          0.978     10000
   macro avg      0.978     0.978     0.978     10000
weighted avg      0.978     0.978     0.978     10000

Quick glossary for the columns: - precision — of everything the model called a 9, what fraction really was a 9? - recall — of all the actual 9s, what fraction did it catch? - f1-score — the harmonic mean of the two, a single balanced summary.

0.9.4 Look at the failures

Always look at the images the model got wrong. It is the fastest way to build intuition about what the model is (and isn’t) doing.

wrong = np.where(y_pred != y_test)[0]
print(f"{len(wrong)} misclassified test images")

fig, axes = plt.subplots(2, 6, figsize=(12, 4.5))
for ax, idx in zip(axes.flat, wrong[:12]):
    ax.imshow(x_test[idx], cmap="gray")
    confidence = probs[idx].max()
    ax.set_title(f"true {y_test[idx]} → pred {y_pred[idx]}\n({confidence:.0%} sure)", fontsize=9)
    ax.axis("off")

plt.suptitle("Where the model went wrong")
plt.tight_layout()
plt.show()
222 misclassified test images

Many of these are genuinely ambiguous — sloppy handwriting that you might get wrong too. Notice also that the model is sometimes confidently wrong. A softmax probability is not a measure of how much the model actually knows.

1 Hyperparameter tuning

We picked \(\lambda = 10^{-4}\) out of thin air. Why that number?

Parameters vs. hyperparameters — the key distinction:

Who sets it? Example How many?
Parameter Learned by gradient descent the 109,386 weights huge
Hyperparameter Chosen by you, before training \(\lambda\), learning rate, layer widths, batch size a handful

Gradient descent cannot tune \(\lambda\) for us — it would just set it to 0, since that always lowers the training loss. So we have to search by brute force: try several values, train a model for each, and keep the one with the best score on the validation set.

The rule that matters: we choose using the validation set, never the test set. The moment we start picking based on test performance, the test score becomes optimistic and stops being an honest estimate of real-world accuracy.

Our search here is the simplest possible: a 1-D grid search over \(\lambda\), spaced logarithmically (orders of magnitude matter far more than the exact digits).

1.0.1 Step 1: wrap model construction in a function

Anytime you’re going to build the same model many times with a knob turned, put it in a function. Notice clear_session() and the seed reset — without them, models built in a loop share state and the comparison isn’t fair.

def build_model(l2_lambda, seed=42):
    """Build + compile our MLP with a given L2 strength."""
    keras.backend.clear_session()
    tf.random.set_seed(seed)          # same starting weights every time -> fair comparison

    m = keras.Sequential([
        layers.Input(shape=(784,)),
        layers.Dense(128, activation="relu",
                     kernel_regularizer=regularizers.l2(l2_lambda)),
        layers.Dense(64,  activation="relu",
                     kernel_regularizer=regularizers.l2(l2_lambda)),
        layers.Dense(10,  activation="softmax"),
    ])
    m.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
    return m

1.0.2 Step 2: sweep

Five values, spanning five orders of magnitude, including \(\lambda = 0\) (no regularization at all) as our baseline.

This cell trains 5 models — expect a few minutes on CPU.

lambdas = [0.0, 1e-5, 1e-4, 1e-3, 1e-2]
EPOCHS  = 15

results = {}   # lambda -> history dict

for lam in lambdas:
    m = build_model(lam)
    hist = m.fit(
        x_train, y_train,
        validation_data=(x_val, y_val),
        epochs=EPOCHS, batch_size=128,
        verbose=0,                       # quiet — we print a summary instead
    )
    results[lam] = hist.history

    tr = hist.history["accuracy"][-1]
    va = hist.history["val_accuracy"][-1]
    print(f"lambda = {lam:<8.0e}  train acc = {tr:.4f}   val acc = {va:.4f}   gap = {tr - va:+.4f}")
lambda = 0e+00     train acc = 0.9974   val acc = 0.9762   gap = +0.0213
lambda = 1e-05     train acc = 0.9961   val acc = 0.9736   gap = +0.0225
lambda = 1e-04     train acc = 0.9939   val acc = 0.9772   gap = +0.0167
lambda = 1e-03     train acc = 0.9819   val acc = 0.9728   gap = +0.0092
lambda = 1e-02     train acc = 0.9545   val acc = 0.9551   gap = -0.0005

1.0.3 Step 3: read the results

Two things to watch as \(\lambda\) increases:

  1. The gap (train − validation) should shrink. That’s regularization doing its job.
  2. Validation accuracy should rise, peak, then fall. Too little regularization → overfits. Too much → underfits. The peak is the sweet spot.

This U-shape (upside-down here, since higher accuracy is better) is the bias–variance tradeoff, and you are about to see it with your own eyes.

train_acc = [results[lam]["accuracy"][-1]     for lam in lambdas]
val_acc   = [results[lam]["val_accuracy"][-1] for lam in lambdas]
gaps      = [t - v for t, v in zip(train_acc, val_acc)]

xs     = range(len(lambdas))                       # evenly spaced positions...
labels = ["0"] + [f"$10^{{{int(np.log10(l))}}}$" for l in lambdas[1:]]   # ...with log labels

fig, ax = plt.subplots(1, 2, figsize=(12, 4))

ax[0].plot(xs, train_acc, "o-", label="train")
ax[0].plot(xs, val_acc,   "o-", label="validation")
ax[0].set_xticks(xs); ax[0].set_xticklabels(labels)
ax[0].set_xlabel(r"L2 strength  $\lambda$")
ax[0].set_ylabel("accuracy after 15 epochs")
ax[0].set_title("Accuracy vs. regularization strength")
ax[0].legend(); ax[0].grid(alpha=0.3)

ax[1].bar(xs, gaps, color="firebrick")
ax[1].set_xticks(xs); ax[1].set_xticklabels(labels)
ax[1].set_xlabel(r"L2 strength  $\lambda$")
ax[1].set_ylabel("train acc  −  val acc")
ax[1].set_title("Generalization gap (overfitting)")
ax[1].grid(alpha=0.3, axis="y")

plt.tight_layout()
plt.show()

And the same story told through the training curves — one line per \(\lambda\):

plt.figure(figsize=(8, 4.5))

for lam in lambdas:
    ep = range(1, EPOCHS + 1)
    plt.plot(ep, results[lam]["val_accuracy"], "o-", label=f"$\\lambda$ = {lam:.0e}")

plt.xlabel("epoch")
plt.ylabel("validation accuracy")
plt.title("Validation accuracy for each L2 strength")
plt.legend()
plt.grid(alpha=0.3)
plt.show()

1.0.4 Step 4: pick the winner and retrain

best_lambda = lambdas[int(np.argmax(val_acc))]
print(f"Best lambda by validation accuracy: {best_lambda:.0e}  (val acc = {max(val_acc):.4f})")

# Retrain that configuration and give it the honest test-set number.
best_model = build_model(best_lambda)
best_model.fit(x_train, y_train,
               validation_data=(x_val, y_val),
               epochs=EPOCHS, batch_size=128, verbose=0)

_, tuned_test_acc = best_model.evaluate(x_test_flat, y_test, verbose=0)

print(f"\nOriginal model (lambda = 1e-04): test accuracy = {test_acc:.4f}")
print(f"Tuned model    (lambda = {best_lambda:.0e}): test accuracy = {tuned_test_acc:.4f}")
Best lambda by validation accuracy: 1e-04  (val acc = 0.9772)

Original model (lambda = 1e-04): test accuracy = 0.9778
Tuned model    (lambda = 1e-04): test accuracy = 0.9747

1.0.5 Reality check

Be honest about what this sweep does and doesn’t tell you:

  • The differences are small. MNIST with an MLP is an easy problem; every \(\lambda\) we tried lands in the high 97s. A 0.2% difference between two runs may just be noise — we trained one model per setting, with one random seed. A rigorous comparison would average several seeds and report the spread.
  • We searched one dimension. Real tuning involves learning rate, width, depth, dropout, batch size, epochs… Trying 5 values of 5 hyperparameters means \(5^5 = 3{,}125\) models. This is why people use random search, Bayesian optimization, or tools like keras-tuner instead of nested for loops.
  • Every peek at the validation set costs you a little. Search hard enough and you’ll start overfitting the validation set too. That is precisely why the test set stayed locked until the last line.

2 Recap

You just executed the entire canonical workflow:

  1. Load — keras.datasets.mnist.load_data()
  2. Explore — shapes, ranges, class balance, sample images
  3. Preprocess — flatten to 784, rescale to [0, 1]
  4. Split — train / validation / (locked) test
  5. Build & compile — Sequential + Dense layers, L2 regularization; loss, optimizer, metrics
  6. Train — fit(), watching validation each epoch
  7. Evaluate — training curves, test accuracy, confusion matrix, error inspection
  8. Tune — grid search over \(\lambda\), selected on validation, confirmed on test

This exact skeleton scales up. Swap MNIST for satellite images and the Dense stack for convolutions, and the eight steps do not change — only the contents of step 5.

2.0.1 Things to try

  • Remove the two hidden layers (leave only Dense(10, softmax)). You now have plain logistic regression — how much accuracy do you lose? That difference is what “deep” bought you.
  • Delete the activation="relu" arguments. Accuracy should collapse toward the linear model. Why?
  • Push the sweep further: add \(\lambda = 10^{-1}\) and \(\lambda = 1\). Watch the model underfit into uselessness.
  • Swap regularizers.l2 for regularizers.l1 and compare. L1 drives weights to exactly zero rather than merely shrinking them — inspect model.layers[0].get_weights()[0] and count how many are zero.
  • Replace L2 with layers.Dropout(0.2) and see whether it closes the generalization gap more cheaply.
  • Skip the /255.0 rescaling and watch training struggle.