Almost every deep learning project — no matter how fancy — follows the same steps:
#
Step
Question it answers
1
Load the data
What am I working with?
2
Explore & visualize
What does it actually look like?
3
Preprocess
What shape/scale does the network need?
4
Split train / validation
How will I know if I’m overfitting?
5
Build & compile the model
What function am I fitting, and how do I score it?
6
Train
Turn the crank.
7
Evaluate
Did it work? Where does it fail?
8
Tune
Could a different setting do better?
Our task: look at a 28×28 grayscale image of a handwritten digit and say which digit (0–9) it is.
Keep one idea in the back of your mind the whole time: a neural network is just a function. It takes 784 numbers in and produces 10 numbers out. Training is the process of nudging its internal knobs (weights) until that function is usually right.
0.1 Setup
import numpy as npimport matplotlib.pyplot as pltimport tensorflow as tffrom tensorflow import kerasfrom tensorflow.keras import layers, regularizers# Fix the random seeds so the notebook is reproducible run-to-run.np.random.seed(42)tf.random.set_seed(42)print("TensorFlow:", tf.__version__)
TensorFlow: 2.13.0
0.2 Load the data
MNIST ships with Keras, so loading is a one-liner. It arrives already divided into a training set (60,000 images) and a test set (10,000 images).
The test set is a locked box. We will not look at it again until the very end.
So each image is a 28 × 28 grid of integers from 0 to 255, where 0 = black (background) and 255 = white (ink).
That’s the key insight: an image is just a matrix of numbers. Let’s prove it by printing a corner of one.
img = x_train_full[0]label = y_train_full[0]fig, ax = plt.subplots(1, 2, figsize=(11, 5))# Left: the image as we see itax[0].imshow(img, cmap="gray")ax[0].set_title(f"Image #0 — label = {label}")ax[0].axis("off")# Right: the same image as the computer sees it (a 14x14 crop, values printed)crop = img[7:21, 7:21]ax[1].imshow(crop, cmap="gray")for i inrange(crop.shape[0]):for j inrange(crop.shape[1]): v = crop[i, j] ax[1].text(j, i, str(v), ha="center", va="center", fontsize=6, color="red"if v <128else"black")ax[1].set_title("...the same thing, as numbers (center crop)")ax[1].axis("off")plt.tight_layout()plt.show()
0.3.1 One example of each digit
fig, axes = plt.subplots(2, 5, figsize=(10, 4.5))for digit, ax inenumerate(axes.flat):# find the first image whose label is this digit idx = np.where(y_train_full == digit)[0][0] ax.imshow(x_train_full[idx], cmap="gray") ax.set_title(f"{digit}") ax.axis("off")plt.suptitle("One sample of each class")plt.tight_layout()plt.show()
0.3.2 Are the classes balanced?
counts = np.bincount(y_train_full)plt.figure(figsize=(7, 3.5))plt.bar(range(10), counts)plt.xticks(range(10))plt.xlabel("digit")plt.ylabel("# of training images")plt.title("Class balance")plt.show()print(counts)
Roughly 6,000 images per digit — close enough to balanced that we don’t need to do anything special. (If one class had 50× fewer examples, accuracy would become a misleading metric and we’d have to be more careful.)
0.4 Preprocess
Two changes are needed to feed a plain MLP:
(a) Flatten. A dense layer takes a vector, not a grid. So we unroll each 28 × 28 image into a 784-long vector by reading it row by row.
Note what this throws away: the network no longer knows that pixel 5 and pixel 33 were vertical neighbors. An MLP has to rediscover spatial structure from scratch. This is exactly the weakness that convolutional networks fix — but it’s remarkable how far an MLP still gets.
(b) Rescale to [0, 1]. Divide by 255. Neural networks train much more stably when inputs are small and comparable in size; feeding in raw values up to 255 makes gradients large and erratic.
Training set → the model learns from this (weights are updated).
Validation set → the model never learns from this. We watch its score during training to detect overfitting, and later we use it to choose hyperparameters.
Test set → touched exactly once, at the end, for an honest final number.
If we tuned our choices by peeking at the test set, the test score would stop being an honest estimate of performance on new data. So we carve a validation set out of the training data.
from sklearn.model_selection import train_test_splitx_train, x_val, y_train, y_val = train_test_split( x_train_full_flat, y_train_full, test_size=0.2, # 20% held out for validation stratify=y_train_full, # keep the class balance in both halves random_state=42,)print("train:", x_train.shape[0], "images")print("val: ", x_val.shape[0], "images")print("test: ", x_test_flat.shape[0], "images")
Dense = “fully connected”: every neuron sees every value from the previous layer. Each neuron computes w · x + b — a weighted vote over its inputs.
relu = max(0, z). Without this nonlinearity, stacking layers would be pointless: a chain of linear maps collapses into a single linear map. ReLU is what lets the network build curved decision boundaries.
The layers get narrower (784 → 128 → 64 → 10). Think of it as a funnel: raw pixels → coarse strokes/edges → digit-ish features → a decision.
softmax on the last layer turns 10 arbitrary scores into 10 probabilities that sum to 1.
0.6.1 One more ingredient: L2 regularization
Our model has ~109,000 knobs but only 48,000 training images. With that much freedom it can simply memorize the training set — including its noise — instead of learning what a “3” looks like.
L2 regularization fights this by changing what we ask the optimizer to minimize:
The mechanical picture: think of a spring attached to every weight, pulling it toward zero. The data pulls each weight outward to whatever value fits the training images; the spring pulls it back. Where they balance depends on \(\lambda\), the spring stiffness.
Why does shrinking weights help? Large weights mean the network’s output swings violently in response to small input changes — a sharp, wiggly function that can contort itself around individual training points. Small weights mean a smoother function, and smooth functions generalize better.
\(\lambda = 0\) → no spring. Free to memorize.
\(\lambda\) too large → springs dominate, all weights crushed toward zero, the model can’t even fit the data (underfitting).
Somewhere in between is the sweet spot. We’ll go looking for it in the last section.
In Keras this is one argument: kernel_regularizer=regularizers.l2(lam). Note it penalizes the weights (kernel), not the biases — a bias just shifts the function, it doesn’t make it wiggly.
L2_LAMBDA =1e-4# our starting guess; we'll tune this at the endmodel = keras.Sequential([ layers.Input(shape=(784,)), layers.Dense(128, activation="relu", kernel_regularizer=regularizers.l2(L2_LAMBDA)), layers.Dense(64, activation="relu", kernel_regularizer=regularizers.l2(L2_LAMBDA)), layers.Dense(10, activation="softmax"),])model.summary()
Sanity-check the parameter count. Layer 1 has 784 × 128 weights plus 128 biases = 100,480 knobs. Layer 2: 128 × 64 + 64 = 8,256. Layer 3: 64 × 10 + 10 = 650. Total ≈ 109,386 numbers that training will adjust. (Regularization doesn’t add parameters — it only changes how they’re penalized.)
0.6.2 Compile = choose the scoring rule and the search strategy
compile() doesn’t train anything. It just attaches three things to the model:
loss — the number we are trying to make small. sparse_categorical_crossentropy is essentially “how surprised was the model by the correct answer?” It’s near 0 when the model puts high probability on the true digit, and blows up when it is confidently wrong. “Sparse” just means our labels are plain integers (7) rather than one-hot vectors ([0,...,1,...,0]).
optimizer — the rule for updating weights from the gradient. adam is the reliable default.
metrics — numbers we want reported but that aren’t used for the update. Accuracy is for humans; the loss is for the optimizer.
history.history is just a dictionary of lists — one value per epoch. Plotting it is the single most informative diagnostic in deep learning.
How to read these plots: - Both curves still falling → keep training. - Training loss falls, validation loss turns back up → overfitting: the model is memorizing training images instead of learning digits. - Both curves flat and high → underfitting (too small a model, too much regularization, or too few epochs).
⚠️ One caveat now that we have L2: the reported loss includes the penalty term \(\lambda \sum w^2\), so it is no longer a pure measure of data fit. When comparing models with different\(\lambda\), compare accuracy, not loss — otherwise you’re comparing apples to oranges.
Final generalization gap (train acc - val acc): 0.0180
The gap between the two accuracy curves is overfitting, made visible: the model does better on images it has seen than on images it hasn’t. L2 keeps that gap smaller than it would otherwise be, but it doesn’t eliminate it. Remember this number — we’ll watch it shrink as we crank up \(\lambda\).
0.9 Final evaluation on the test set
Now — and only now — we open the locked box.
test_loss, test_acc = model.evaluate(x_test_flat, y_test, verbose=0)print(f"Test accuracy: {test_acc:.4f} ({test_acc*100:.2f}%)")print(f"That's about {int(round((1-test_acc)*len(y_test)))} mistakes out of {len(y_test)} images.")
Test accuracy: 0.9778 (97.78%)
That's about 222 mistakes out of 10000 images.
0.9.1 What the model actually outputs
predict() returns 10 probabilities per image. To get a hard prediction we take the argmax — the index of the largest probability.
probs = model.predict(x_test_flat, verbose=0) # shape (10000, 10)y_pred = probs.argmax(axis=1) # shape (10000,)# Inspect a single test image and its full probability vectori =0fig, ax = plt.subplots(1, 2, figsize=(10, 3.5))ax[0].imshow(x_test[i], cmap="gray")ax[0].set_title(f"true = {y_test[i]} predicted = {y_pred[i]}")ax[0].axis("off")ax[1].bar(range(10), probs[i])ax[1].set_xticks(range(10))ax[1].set_xlabel("digit")ax[1].set_ylabel("probability")ax[1].set_title("Softmax output (sums to 1)")plt.tight_layout()plt.show()print("probabilities sum to:", probs[i].sum())
probabilities sum to: 0.9999999
0.9.2 Confusion matrix
Accuracy is one number, and one number hides everything interesting. The confusion matrix tells us which digits get mistaken for which.
Row = the true digit. Column = what the model guessed. The diagonal is correct; everything off-diagonal is a mistake.
The diagonal dominates, so let’s zero it out and look only at the errors — that’s where the story is.
cm_errors = cm.copy()np.fill_diagonal(cm_errors, 0)fig, ax = plt.subplots(figsize=(7, 6))im = ax.imshow(cm_errors, cmap="Reds")ax.set_xticks(range(10)); ax.set_yticks(range(10))ax.set_xlabel("predicted"); ax.set_ylabel("true")ax.set_title("Errors only (diagonal removed)")for i inrange(10):for j inrange(10):if cm_errors[i, j] >0: ax.text(j, i, cm_errors[i, j], ha="center", va="center", fontsize=8)plt.colorbar(im)plt.tight_layout()plt.show()i, j = np.unravel_index(cm_errors.argmax(), cm_errors.shape)print(f"Most common mistake: true {i} predicted as {j} ({cm_errors[i, j]} times)")
Most common mistake: true 7 predicted as 9 (14 times)
The mistakes are not random — they cluster where humans would also hesitate (4↔︎9, 3↔︎5, 7↔︎2). That’s reassuring: the model has learned something about shape, not noise.
Quick glossary for the columns: - precision — of everything the model called a 9, what fraction really was a 9? - recall — of all the actual 9s, what fraction did it catch? - f1-score — the harmonic mean of the two, a single balanced summary.
0.9.4 Look at the failures
Always look at the images the model got wrong. It is the fastest way to build intuition about what the model is (and isn’t) doing.
wrong = np.where(y_pred != y_test)[0]print(f"{len(wrong)} misclassified test images")fig, axes = plt.subplots(2, 6, figsize=(12, 4.5))for ax, idx inzip(axes.flat, wrong[:12]): ax.imshow(x_test[idx], cmap="gray") confidence = probs[idx].max() ax.set_title(f"true {y_test[idx]} → pred {y_pred[idx]}\n({confidence:.0%} sure)", fontsize=9) ax.axis("off")plt.suptitle("Where the model went wrong")plt.tight_layout()plt.show()
222 misclassified test images
Many of these are genuinely ambiguous — sloppy handwriting that you might get wrong too. Notice also that the model is sometimes confidently wrong. A softmax probability is not a measure of how much the model actually knows.
1 Hyperparameter tuning
We picked \(\lambda = 10^{-4}\) out of thin air. Why that number?
Parameters vs. hyperparameters — the key distinction:
Gradient descent cannot tune \(\lambda\) for us — it would just set it to 0, since that always lowers the training loss. So we have to search by brute force: try several values, train a model for each, and keep the one with the best score on the validation set.
The rule that matters: we choose using the validation set, never the test set. The moment we start picking based on test performance, the test score becomes optimistic and stops being an honest estimate of real-world accuracy.
Our search here is the simplest possible: a 1-D grid search over \(\lambda\), spaced logarithmically (orders of magnitude matter far more than the exact digits).
1.0.1 Step 1: wrap model construction in a function
Anytime you’re going to build the same model many times with a knob turned, put it in a function. Notice clear_session() and the seed reset — without them, models built in a loop share state and the comparison isn’t fair.
def build_model(l2_lambda, seed=42):"""Build + compile our MLP with a given L2 strength.""" keras.backend.clear_session() tf.random.set_seed(seed) # same starting weights every time -> fair comparison m = keras.Sequential([ layers.Input(shape=(784,)), layers.Dense(128, activation="relu", kernel_regularizer=regularizers.l2(l2_lambda)), layers.Dense(64, activation="relu", kernel_regularizer=regularizers.l2(l2_lambda)), layers.Dense(10, activation="softmax"), ]) m.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])return m
1.0.2 Step 2: sweep
Five values, spanning five orders of magnitude, including \(\lambda = 0\) (no regularization at all) as our baseline.
This cell trains 5 models — expect a few minutes on CPU.
lambdas = [0.0, 1e-5, 1e-4, 1e-3, 1e-2]EPOCHS =15results = {} # lambda -> history dictfor lam in lambdas: m = build_model(lam) hist = m.fit( x_train, y_train, validation_data=(x_val, y_val), epochs=EPOCHS, batch_size=128, verbose=0, # quiet — we print a summary instead ) results[lam] = hist.history tr = hist.history["accuracy"][-1] va = hist.history["val_accuracy"][-1]print(f"lambda = {lam:<8.0e} train acc = {tr:.4f} val acc = {va:.4f} gap = {tr - va:+.4f}")
lambda = 0e+00 train acc = 0.9974 val acc = 0.9762 gap = +0.0213
lambda = 1e-05 train acc = 0.9961 val acc = 0.9736 gap = +0.0225
lambda = 1e-04 train acc = 0.9939 val acc = 0.9772 gap = +0.0167
lambda = 1e-03 train acc = 0.9819 val acc = 0.9728 gap = +0.0092
lambda = 1e-02 train acc = 0.9545 val acc = 0.9551 gap = -0.0005
1.0.3 Step 3: read the results
Two things to watch as \(\lambda\) increases:
The gap (train − validation) should shrink. That’s regularization doing its job.
Validation accuracy should rise, peak, then fall. Too little regularization → overfits. Too much → underfits. The peak is the sweet spot.
This U-shape (upside-down here, since higher accuracy is better) is the bias–variance tradeoff, and you are about to see it with your own eyes.
train_acc = [results[lam]["accuracy"][-1] for lam in lambdas]val_acc = [results[lam]["val_accuracy"][-1] for lam in lambdas]gaps = [t - v for t, v inzip(train_acc, val_acc)]xs =range(len(lambdas)) # evenly spaced positions...labels = ["0"] + [f"$10^{{{int(np.log10(l))}}}$"for l in lambdas[1:]] # ...with log labelsfig, ax = plt.subplots(1, 2, figsize=(12, 4))ax[0].plot(xs, train_acc, "o-", label="train")ax[0].plot(xs, val_acc, "o-", label="validation")ax[0].set_xticks(xs); ax[0].set_xticklabels(labels)ax[0].set_xlabel(r"L2 strength $\lambda$")ax[0].set_ylabel("accuracy after 15 epochs")ax[0].set_title("Accuracy vs. regularization strength")ax[0].legend(); ax[0].grid(alpha=0.3)ax[1].bar(xs, gaps, color="firebrick")ax[1].set_xticks(xs); ax[1].set_xticklabels(labels)ax[1].set_xlabel(r"L2 strength $\lambda$")ax[1].set_ylabel("train acc − val acc")ax[1].set_title("Generalization gap (overfitting)")ax[1].grid(alpha=0.3, axis="y")plt.tight_layout()plt.show()
And the same story told through the training curves — one line per \(\lambda\):
plt.figure(figsize=(8, 4.5))for lam in lambdas: ep =range(1, EPOCHS +1) plt.plot(ep, results[lam]["val_accuracy"], "o-", label=f"$\\lambda$ = {lam:.0e}")plt.xlabel("epoch")plt.ylabel("validation accuracy")plt.title("Validation accuracy for each L2 strength")plt.legend()plt.grid(alpha=0.3)plt.show()
1.0.4 Step 4: pick the winner and retrain
best_lambda = lambdas[int(np.argmax(val_acc))]print(f"Best lambda by validation accuracy: {best_lambda:.0e} (val acc = {max(val_acc):.4f})")# Retrain that configuration and give it the honest test-set number.best_model = build_model(best_lambda)best_model.fit(x_train, y_train, validation_data=(x_val, y_val), epochs=EPOCHS, batch_size=128, verbose=0)_, tuned_test_acc = best_model.evaluate(x_test_flat, y_test, verbose=0)print(f"\nOriginal model (lambda = 1e-04): test accuracy = {test_acc:.4f}")print(f"Tuned model (lambda = {best_lambda:.0e}): test accuracy = {tuned_test_acc:.4f}")
Best lambda by validation accuracy: 1e-04 (val acc = 0.9772)
Original model (lambda = 1e-04): test accuracy = 0.9778
Tuned model (lambda = 1e-04): test accuracy = 0.9747
1.0.5 Reality check
Be honest about what this sweep does and doesn’t tell you:
The differences are small. MNIST with an MLP is an easy problem; every \(\lambda\) we tried lands in the high 97s. A 0.2% difference between two runs may just be noise — we trained one model per setting, with one random seed. A rigorous comparison would average several seeds and report the spread.
We searched one dimension. Real tuning involves learning rate, width, depth, dropout, batch size, epochs… Trying 5 values of 5 hyperparameters means \(5^5 = 3{,}125\) models. This is why people use random search, Bayesian optimization, or tools like keras-tuner instead of nested for loops.
Every peek at the validation set costs you a little. Search hard enough and you’ll start overfitting the validation set too. That is precisely why the test set stayed locked until the last line.
2 Recap
You just executed the entire canonical workflow:
Load — keras.datasets.mnist.load_data()
Explore — shapes, ranges, class balance, sample images
Evaluate — training curves, test accuracy, confusion matrix, error inspection
Tune — grid search over \(\lambda\), selected on validation, confirmed on test
This exact skeleton scales up. Swap MNIST for satellite images and the Dense stack for convolutions, and the eight steps do not change — only the contents of step 5.
2.0.1 Things to try
Remove the two hidden layers (leave only Dense(10, softmax)). You now have plain logistic regression — how much accuracy do you lose? That difference is what “deep” bought you.
Delete the activation="relu" arguments. Accuracy should collapse toward the linear model. Why?
Push the sweep further: add \(\lambda = 10^{-1}\) and \(\lambda = 1\). Watch the model underfit into uselessness.
Swap regularizers.l2 for regularizers.l1 and compare. L1 drives weights to exactly zero rather than merely shrinking them — inspect model.layers[0].get_weights()[0] and count how many are zero.
Replace L2 with layers.Dropout(0.2) and see whether it closes the generalization gap more cheaply.
Skip the /255.0 rescaling and watch training struggle.