NumPy Arrays and PyTorch Tensors

Both libraries are built on a single idea: the n-dimensional array — a grid of numbers, all of the same type, stored in one contiguous block of memory.

Working with a grid as a whole object (instead of looping over it) is what makes scientific Python fast and readable. Learn the idea once in NumPy, and PyTorch is mostly the same idea with two superpowers added: it can run on a GPU, and it can compute derivatives automatically.

Section 1: NumPy Arrays

1.1 Why not just use Python lists?

A Python list is a flexible bag of pointers to arbitrary objects. A NumPy array is a rigid block of raw numbers. Rigidity buys speed and gives us vectorized operations: one line acts on every element.

import numpy as np

nums = [1, 2, 3, 4]

# A list "multiplied" repeats itself, and elementwise math needs a loop.
print("list * 2      :", nums * 2)
print("loop doubling :", [x * 2 for x in nums])

# An array behaves like a mathematical vector.
arr = np.array(nums)
print("array * 2     :", arr * 2)
print("array + array :", arr + arr)
list * 2      : [1, 2, 3, 4, 1, 2, 3, 4]
loop doubling : [2, 4, 6, 8]
array * 2     : [2 4 6 8]
array + array : [2 4 6 8]
# The speed difference is the reason we care.
big_list = list(range(1_000_000))
big_arr = np.arange(1_000_000)

%timeit sum(x * x for x in big_list)
%timeit (big_arr ** 2).sum()
34.3 ms ± 1.18 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
1.08 ms ± 28.8 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

1.2 Creating arrays

Beyond np.array(...), most arrays come from a handful of factory functions.

rng = np.random.default_rng(seed=0)   # reproducible randomness

print(np.zeros(3))               # [0. 0. 0.]
print(np.ones((2, 3)))           # 2 rows, 3 columns of 1.0
print(np.arange(0, 10, 2))       # like range(): start, stop, step
print(np.linspace(0, 1, 5))      # 5 evenly spaced points, endpoints included
print(np.eye(3))                 # identity matrix
print(rng.normal(size=(2, 2)))   # samples from a standard normal
[0. 0. 0.]
[[1. 1. 1.]
 [1. 1. 1.]]
[0 2 4 6 8]
[0.   0.25 0.5  0.75 1.  ]
[[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]
[[ 0.12573022 -0.13210486]
 [ 0.64042265  0.10490012]]

1.3 The four attributes worth memorizing

Every array carries its own description. shape is the one you will check a hundred times a day.

X = np.array([[1.0, 2.0, 3.0],
              [4.0, 5.0, 6.0]])

print("shape:", X.shape)   # (rows, columns)
print("ndim :", X.ndim)    # number of axes/dimensions
print("size :", X.size)    # total number of elements
print("dtype:", X.dtype)   # the single type shared by all elements
shape: (2, 3)
ndim : 2
size : 6
dtype: float64

The dtype is fixed at creation. If you build an array from integers, it stores integers — and assignments get silently truncated to fit.

ints = np.array([1, 2, 3])
print(ints.dtype)

ints[0] = 3.99          # truncated, not rounded, to fit an int array
print(ints)

floats = ints.astype(np.float64)   # .astype() makes a converted copy
floats[0] = 3.99
print(floats, floats.dtype)
int64
[3 2 3]
[3.99 2.   3.  ] float64

1.4 Indexing and slicing

Index with one number per axis: X[row, col]. A slice start:stop takes everything up to but not including stop. A bare : means “the whole axis.”

X = np.arange(12).reshape(3, 4)
print(X)

print("\nX[1, 2]   ->", X[1, 2])      # single element: row 1, column 2
print("X[0]      ->", X[0])          # entire first row
print("X[:, 1]   ->", X[:, 1])       # entire second column
print("X[0:2, 1:3] ->\n", X[0:2, 1:3])  # a 2x2 sub-block
[[ 0  1  2  3]
 [ 4  5  6  7]
 [ 8  9 10 11]]

X[1, 2]   -> 6
X[0]      -> [0 1 2 3]
X[:, 1]   -> [1 5 9]
X[0:2, 1:3] ->
 [[1 2]
 [5 6]]

A slice is a view, not a copy. It looks into the original array’s memory. Writing to it writes to the original. This is a feature (no wasted memory) and a classic source of bugs.

a = np.arange(6)
view = a[1:4]
view[0] = 99
print("a after writing to a view:", a)

b = np.arange(6)
copy = b[1:4].copy()     # ask explicitly for independent memory
copy[0] = 99
print("b after writing to a copy:", b)
a after writing to a view: [ 0 99  2  3  4  5]
b after writing to a copy: [0 1 2 3 4 5]

1.5 Broadcasting

Operations happen elementwise. When shapes differ, NumPy stretches the smaller one — without ever copying the data — as long as each axis is either equal in size or of length 1.

A (3, 1) column plus a (1, 4) row both stretch to (3, 4):

col = np.array([[0], [10], [20]])    # shape (3, 1)
row = np.array([[1, 2, 3, 4]])       # shape (1, 4)

print(col.shape, "+", row.shape, "->", (col + row).shape)
print(col + row)
(3, 1) + (1, 4) -> (3, 4)
[[ 1  2  3  4]
 [11 12 13 14]
 [21 22 23 24]]
# The everyday case: a scalar or a single row applied to a whole matrix.
X = np.array([[1.0, 2.0, 3.0],
              [4.0, 5.0, 6.0]])

print(X * 10)                        # scalar broadcasts to every element
print(X - np.array([1.0, 1.0, 0.0])) # one row broadcasts down all rows
[[10. 20. 30.]
 [40. 50. 60.]]
[[0. 1. 3.]
 [3. 4. 6.]]

1.6 Reductions and the axis argument

A reduction collapses an axis. The key mental rule: axis=k is the axis that disappears.

X = np.array([[1, 2, 3],
              [4, 5, 6]])      # shape (2, 3)

print("X.sum()        ->", X.sum())          # everything -> a scalar
print("X.sum(axis=0)  ->", X.sum(axis=0))    # collapse rows -> one value per column
print("X.sum(axis=1)  ->", X.sum(axis=1))    # collapse columns -> one value per row

print("\nmean, max, argmax along axis 0:",
      X.mean(axis=0), X.max(axis=0), X.argmax(axis=0))
X.sum()        -> 21
X.sum(axis=0)  -> [5 7 9]
X.sum(axis=1)  -> [ 6 15]

mean, max, argmax along axis 0: [2.5 3.5 4.5] [4 5 6] [1 1 1]

1.7 Reshaping and transposing

reshape reinterprets the same numbers under a new shape; the element count must match. Use -1 for “you figure this dimension out.”

v = np.arange(6)
print(v.reshape(2, 3))
print(v.reshape(3, -1))    # -1 is inferred as 2

M = np.arange(6).reshape(2, 3)
print("\nM.T has shape", M.T.shape)
print(M.T)
[[0 1 2]
 [3 4 5]]
[[0 1]
 [2 3]
 [4 5]]

M.T has shape (3, 2)
[[0 3]
 [1 4]
 [2 5]]

1.8 Matrix multiplication

* is elementwise. @ is the linear-algebra matrix product. Confusing them is the single most common NumPy error.

A = np.array([[1, 2],
              [3, 4]])
B = np.array([[10, 0],
              [0, 10]])

print("A * B (elementwise):\n", A * B)
print("\nA @ B (matrix product):\n", A @ B)

# Shapes must line up: (2,3) @ (3,) -> (2,)
M = np.arange(6).reshape(2, 3)
x = np.array([1, 0, -1])
print("\nM @ x ->", M @ x)
A * B (elementwise):
 [[10  0]
 [ 0 40]]

A @ B (matrix product):
 [[10 20]
 [30 40]]

M @ x -> [-2 -2]

1.9 Boolean masks

Comparisons produce arrays of True/False, which can then be used to select or overwrite elements. This replaces most if statements inside loops.

data = np.array([3.0, -1.0, 4.0, -1.5, 0.0, 2.5])

mask = data > 0
print("mask       :", mask)
print("data[mask] :", data[mask])          # keep only the elements where mask is True
print("count > 0  :", mask.sum())          # True counts as 1
print("any / all  :", mask.any(), mask.all())

clipped = data.copy()
clipped[clipped < 0] = 0                   # ReLU, in one line
print("clipped    :", clipped)
mask       : [ True False  True False False  True]
data[mask] : [3.  4.  2.5]
count > 0  : 3
any / all  : True False
clipped    : [3.  0.  4.  0.  0.  2.5]

1.10 Putting it together: standardizing a dataset

A typical preprocessing step. Rows are samples, columns are features; we rescale each feature to have mean 0 and standard deviation 1. Broadcasting and axis do all the work.

rng = np.random.default_rng(0)
data = rng.normal(loc=[100.0, 0.5], scale=[15.0, 0.1], size=(1000, 2))

mu = data.mean(axis=0)       # shape (2,) — one mean per feature
sigma = data.std(axis=0)     # shape (2,) — one std per feature

standardized = (data - mu) / sigma    # (1000, 2) - (2,) broadcasts down every row

print("original mean/std :", mu.round(3), sigma.round(3))
print("new mean/std      :", standardized.mean(axis=0).round(3),
                             standardized.std(axis=0).round(3))
original mean/std : [99.551  0.497] [15.289  0.098]
new mean/std      : [-0. -0.] [1. 1.]

Section 2: PyTorch Tensors

A torch.Tensor is the same object as a NumPy array — a typed n-dimensional grid — with the same creation functions, the same indexing, the same broadcasting, the same axis reductions (spelled dim), and the same @.

What it adds:

  1. It can live on a GPU.
  2. It can track the operations performed on it and compute derivatives with respect to it, automatically. This is what makes neural networks trainable.

2.1 The familiar half

If you know Section 1, you already know this section. Almost everything transfers.

import torch

t = torch.tensor([[1.0, 2.0, 3.0],
                  [4.0, 5.0, 6.0]])

print(t)
print("shape:", t.shape, "| dtype:", t.dtype, "| device:", t.device)
tensor([[1., 2., 3.],
        [4., 5., 6.]])
shape: torch.Size([2, 3]) | dtype: torch.float32 | device: cpu
torch.manual_seed(0)

print(torch.zeros(3))
print(torch.ones(2, 3))
print(torch.arange(0, 10, 2))
print(torch.linspace(0, 1, 5))
print(torch.randn(2, 2))        # standard normal, like rng.normal
tensor([0., 0., 0.])
tensor([[1., 1., 1.],
        [1., 1., 1.]])
tensor([0, 2, 4, 6, 8])
tensor([0.0000, 0.2500, 0.5000, 0.7500, 1.0000])
tensor([[ 1.5410, -0.2934],
        [-2.1788,  0.5684]])
X = torch.arange(12).reshape(3, 4)

print(X[1, 2], X[:, 1])              # indexing and slicing: identical to NumPy
print(X * 10)                        # broadcasting: identical
print(X.sum(dim=0))                  # 'dim' is torch's word for numpy's 'axis'
print(X.reshape(2, 6).shape, X.T.shape)

A = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
print(A @ A)                         # matrix product: identical
print(A[A > 2])                      # boolean masks: identical
tensor(6) tensor([1, 5, 9])
tensor([[  0,  10,  20,  30],
        [ 40,  50,  60,  70],
        [ 80,  90, 100, 110]])
tensor([12, 15, 18, 21])
torch.Size([2, 6]) torch.Size([4, 3])
tensor([[ 7., 10.],
        [15., 22.]])
tensor([3., 4.])

Two small vocabulary differences worth noting:

NumPy PyTorch
axis=0 dim=0
arr.astype(np.float32) t.float() or t.to(torch.float32)
np.concatenate torch.cat
arr.copy() t.clone()

Also, methods ending in an underscore modify the tensor in place: t.add_(1) changes t, while t.add(1) returns a new tensor.

2.2 The NumPy bridge

Conversion is free in both directions — and on the CPU the two objects share the same memory. Changing one changes the other.

a = np.ones(3)
t = torch.from_numpy(a)     # numpy -> torch

t.add_(1)                   # in-place change to the tensor...
print("numpy array:", a)    # ...is visible in the numpy array

back = t.numpy()            # torch -> numpy
print("round trip:", back, type(back))
numpy array: [2. 2. 2.]
round trip: [2. 2. 2.] <class 'numpy.ndarray'>

2.3 Devices

A tensor lives on a device. Moving it to a GPU makes the same operations run in parallel over thousands of cores. The standard, portable idiom:

device = "cuda" if torch.cuda.is_available() else "cpu"
print("using device:", device)

x = torch.randn(3, 3).to(device)
y = torch.randn(3, 3).to(device)
z = x @ y                   # runs on the GPU if one is available

print(z.device)
# Note: tensors must be on the SAME device to interact.
using device: cpu
cpu

2.4 Autograd: the real reason PyTorch exists

Set requires_grad=True and the tensor starts recording every operation it takes part in, building a graph. Calling .backward() on a scalar result walks that graph backwards and deposits \(\partial(\text{result}) / \partial(\text{each input})\) into each input’s .grad.

Take \(y = x^2 + 3x\). Calculus says \(dy/dx = 2x + 3\), so at \(x = 2\) the derivative is \(7\). PyTorch should agree.

x = torch.tensor(2.0, requires_grad=True)

y = x**2 + 3*x     # forward pass: torch quietly records what happened
y.backward()       # backward pass: differentiate y with respect to x

print("y      =", y.item())
print("dy/dx  =", x.grad.item(), " (analytic answer: 2*2 + 3 = 7)")
y      = 10.0
dy/dx  = 7.0  (analytic answer: 2*2 + 3 = 7)

This works for any number of inputs at once, which is the whole trick: a neural network is just a function of millions of parameters, and .backward() gets the derivative with respect to every one of them in a single pass.

w = torch.tensor([1.0, 2.0], requires_grad=True)
b = torch.tensor(0.5, requires_grad=True)
x = torch.tensor([3.0, 4.0])

loss = ((w @ x + b) - 10.0)**2    # a scalar
loss.backward()

print("dloss/dw =", w.grad)       # one gradient per parameter
print("dloss/db =", b.grad)
dloss/dw = tensor([ 9., 12.])
dloss/db = tensor(3.)

2.5 A complete (tiny) training loop

Everything above, combined: fit a line \(y = wx + b\) to noisy data generated from \(y = 2x + 1\). The loop is gradient descent — nudge each parameter in the direction that decreases the loss.

Three things to notice:

  • loss.backward() fills in .grad for w and b.
  • Updates happen under torch.no_grad() because adjusting the parameters is bookkeeping, not part of the function we are differentiating.
  • Gradients accumulate by default, so we zero them each step.
torch.manual_seed(0)

x = torch.linspace(-1, 1, 100)
y = 2*x + 1 + 0.1*torch.randn(100)     # true parameters: w = 2, b = 1

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
lr = 0.1

for step in range(101):
    y_pred = w * x + b                 # forward
    loss = ((y_pred - y)**2).mean()    # mean squared error

    loss.backward()                    # compute dloss/dw and dloss/db

    with torch.no_grad():              # update, untracked
        w -= lr * w.grad
        b -= lr * b.grad

    w.grad.zero_()                     # reset for the next step
    b.grad.zero_()

    if step % 25 == 0:
        print(f"step {step:3d} | loss {loss.item():.4f} | w {w.item():.3f} | b {b.item():.3f}")

print("\nrecovered w, b =", round(w.item(), 3), round(b.item(), 3), " (truth: 2.0, 1.0)")
step   0 | loss 2.3780 | w 0.136 | b 0.201
step  25 | loss 0.0507 | w 1.679 | b 1.001
step  50 | loss 0.0117 | w 1.945 | b 1.004
step  75 | loss 0.0105 | w 1.990 | b 1.004
step 100 | loss 0.0105 | w 1.998 | b 1.004

recovered w, b = 1.998 1.004  (truth: 2.0, 1.0)
# plot results 
import matplotlib.pyplot as plt
plt.scatter(x.detach().numpy(), y.detach().numpy(), label='data')
plt.plot(x.detach().numpy(), (w*x + b).detach().numpy(), "-r", label='fitted line')
plt.legend()
plt.show()

Takeaways

  • An array/tensor is a typed grid. Its shape is the first thing to check when anything goes wrong.
  • Vectorize: express operations on whole arrays, not element by element in a loop.
  • Broadcasting aligns mismatched shapes; axis/dim says which axis to collapse; @ is matrix multiplication and * is not.
  • Slices are views into shared memory. Use .copy() / .clone() when you need independence.
  • A PyTorch tensor is a NumPy array that can move to a GPU and remember its own history — and that history is what .backward() turns into gradients.