A working refresher on the Python language, followed by a first look at the data stack
This notebook mirrors the R fundamentals notebook section by section, so each idea appears next to the R idiom it replaces.
Run every cell as we go. The fastest way to learn Python is to break things and read the traceback.
Aside: Data driven narrative
https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-APOCALYPSE/egpbykdxjvq/
R prints every top-level expression in a cell. Python prints only the last one. The following cell restores the R behavior so the two notebooks produce comparable output; don’t worry about the details.
= is the only assignment operator; there is no <-# starts a commentprint() does it explicitlycat()+ - * / behave as expected, but exponentiation is **, not ^^ is legal Python and means bitwise exclusive or, which makes it a silent R-to-Python bug% is modulo (remainder), // is floor division/ always returns a float, even between two integers== != < > <= >=and, or, not&, |, ~, with each side in parenthesesTrue or False, which are themselves a data typePython has a small set of built-in types. Everything else is built from them.
float: real numbers, the default for anything with a decimal point, R’s doubleint: whole numbers, unlimited precision, no L suffix neededstr: text, in single or double quotes, R’s characterbool: True and False, capitalized, R’s logicalcomplex and bytes: rarely needed in an intro courseThree questions worth asking about any object you did not create yourself.
type(): what is this, covering both R’s typeof() and class()isinstance(): the analogue of is.numeric() and friendsdir(): every attribute and method the object carries, the usual way to discover what you can do nextA number in Python is a scalar. Unlike R, it is not a length-1 vector, and len() on it is an error.
int(), float(), str(), bool() convert explicitlyValueError and stop execution; R returns NA with a warningint() truncates toward zero; it does not roundpd.to_numeric(..., errors="coerce"), which yields missing values instead of raisingNone: the empty object, R’s NULLfloat("nan") or np.nan: missing or undefined, covering both R’s NA and NaNnp.inf and -np.inf: overflow or division by zeropd.isna() or math.isnan(), never with == np.nan, because nan is not equal to itselfNumPy propagates missingness the way R does. pandas goes the other way and skips missing values by default, which is the single most common source of quiet disagreement between an R script and its Python translation.
import numpy as np
import pandas as pd
v = np.array([1, 2, np.nan, 4])
v.mean() # nan: missingness propagates
np.nanmean(v) # drop the nan first, R's na.rm = TRUE
pd.Series(v).mean() # pandas drops it without being asked
pd.isna(v) # element-wise test, R's is.na()
np.nan == np.nan # False: never test for missingness with ==
with np.errstate(divide="ignore", invalid="ignore"):
print(np.float64(0) / np.float64(0)) # nan
print(np.float64(1) / np.float64(0)) # infnp.float64(nan)
np.float64(2.3333333333333335)
np.float64(2.3333333333333335)
array([False, False, True, False])
False
nan
inf
The base Python workhorse. A list is an ordered, mutable sequence that can mix types freely.
[ ] builds a list literal, the closest analogue of c()range() builds an integer sequence, but it is lazy; wrap it in list() to see the valuesx * 2 repeats the list, it does not double each element[10, 15, 20, 25, 30, 35]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
['a', 'b', 'a', 'b', 'a', 'b']
6
[10, 15, 20, 25, 30, 35, 40]
[10, 15, 20, 25, 30, 35, 10, 15, 20, 25, 30, 35]
The true analogue of an R vector: one type, one fixed length, arithmetic applied element by element.
np.array() converts a list; np.arange() is R’s seq(by = )np.linspace() is R’s seq(length.out = ).dtype reports the single stored type, .shape and .size report the extentarray([10, 15, 20, 25, 30, 35])
dtype('int64')
array([0. , 0.25, 0.5 , 0.75, 1. ])
array([ 0. , 2.5, 5. , 7.5, 10. ])
array(['a', 'b', 'a', 'b', 'a', 'b'], dtype='<U1')
array(['a', 'a', 'a', 'b', 'b', 'b'], dtype='<U1')
6
a:b includes position a and excludes position bTrue, exactly as in Rx = np.array([10, 15, 20, 25, 30, 35])
x[0] # first element, R's x[1]
x[-1] # last element, R's x[length(x)]
x[1:4] # a contiguous slice, R's x[2:4]
x[[0, 4]] # arbitrary positions, R's x[c(1, 5)]
np.delete(x, 0) # drop the first element, R's x[-1]
x[x > 20] # boolean subsetting: the key idiom, identical in both languagesnp.int64(10)
np.int64(35)
array([15, 20, 25])
array([10, 30])
array([15, 20, 25, 30, 35])
array([25, 30, 35])
Operations apply element by element with no loop required. This is the central habit to build in NumPy.
np.std() defaults to the population formula; R’s sd() is the sample formula, so pass ddof=1 to matcharray([2, 4, 6, 8])
array([2, 4, 6, 8])
array([ 1, 4, 9, 16])
array([1. , 1.41421356, 1.73205081, 2. ])
(np.int64(10), np.float64(2.5), np.int64(4), np.float64(1.2909944487358056))
ValueError: operands could not be broadcast together with shapes (4,) (3,)
Series is an array plus an index of labels: R’s named vectordict is the other analogue, though it supports no arithmeticA matrix is a two-dimensional NumPy array.
reshape() fills row-first by default; pass order="F" to match R, which fills column-first[row, column]; a bare : keeps that whole dimensionarray([[ 1, 6, 11, 16],
[ 2, 7, 12, 17],
[ 3, 8, 13, 18],
[ 4, 9, 14, 19],
[ 5, 10, 15, 20]])
(5, 4)
np.int64(8)
array([ 3, 8, 13, 18])
array([ 6, 7, 8, 9, 10])
array([[ 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10],
[11, 12, 13, 14, 15],
[16, 17, 18, 19, 20]])
A dict holds elements of different types and different lengths: the analogue of an R list().
$ and [[ ]] both become [ ], with no partial matching of names.get() supplies a default instead of raising a KeyError{'name': 'Ada', 'age': 36, 'scores': [90, 95, 99]}
dict_keys(['name', 'age', 'scores'])
'Ada'
[90, 95, 99]
95
'not recorded'
A pandas Categorical represents categorical data with a fixed set of possible values, called categories.
['small', 'large', 'small', 'medium']
Categories (3, object): ['small' < 'medium' < 'large']
Index(['small', 'medium', 'large'], dtype='object')
small 2
medium 1
large 1
Name: count, dtype: int64
array([0, 2, 0, 1], dtype=int8)
A pandas DataFrame is a dict of equal-length columns displayed as a table: rows are observations, columns are variables.
| id | color | passed | |
|---|---|---|---|
| 0 | 10 | red | True |
| 1 | 20 | white | True |
| 2 | 30 | red | True |
| 3 | 40 | None | False |
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 3 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 id 4 non-null int64
1 color 3 non-null object
2 passed 4 non-null bool
dtypes: bool(1), int64(1), object(1)
memory usage: 200.0+ bytes
(4, 3)
['id', 'color', 'passed']
df["id"] pulls a single column out as a Seriesdf.loc[rows, columns] subsets by label, df.iloc[rows, columns] by positionhead(), describe(), and info() are the first three things to run on any new datasetPython ships no datasets with the language, so the cell below embeds R’s mtcars directly. For real work, seaborn.load_dataset() and statsmodels.api.datasets.get_rdataset() download the familiar R datasets on demand.
from io import StringIO
MTCARS_CSV = """\
model,mpg,cyl,disp,hp,drat,wt,qsec,vs,am,gear,carb
Mazda RX4,21,6,160,110,3.9,2.62,16.46,0,1,4,4
Mazda RX4 Wag,21,6,160,110,3.9,2.875,17.02,0,1,4,4
Datsun 710,22.8,4,108,93,3.85,2.32,18.61,1,1,4,1
Hornet 4 Drive,21.4,6,258,110,3.08,3.215,19.44,1,0,3,1
Hornet Sportabout,18.7,8,360,175,3.15,3.44,17.02,0,0,3,2
Valiant,18.1,6,225,105,2.76,3.46,20.22,1,0,3,1
Duster 360,14.3,8,360,245,3.21,3.57,15.84,0,0,3,4
Merc 240D,24.4,4,146.7,62,3.69,3.19,20,1,0,4,2
Merc 230,22.8,4,140.8,95,3.92,3.15,22.9,1,0,4,2
Merc 280,19.2,6,167.6,123,3.92,3.44,18.3,1,0,4,4
Merc 280C,17.8,6,167.6,123,3.92,3.44,18.9,1,0,4,4
Merc 450SE,16.4,8,275.8,180,3.07,4.07,17.4,0,0,3,3
Merc 450SL,17.3,8,275.8,180,3.07,3.73,17.6,0,0,3,3
Merc 450SLC,15.2,8,275.8,180,3.07,3.78,18,0,0,3,3
Cadillac Fleetwood,10.4,8,472,205,2.93,5.25,17.98,0,0,3,4
Lincoln Continental,10.4,8,460,215,3,5.424,17.82,0,0,3,4
Chrysler Imperial,14.7,8,440,230,3.23,5.345,17.42,0,0,3,4
Fiat 128,32.4,4,78.7,66,4.08,2.2,19.47,1,1,4,1
Honda Civic,30.4,4,75.7,52,4.93,1.615,18.52,1,1,4,2
Toyota Corolla,33.9,4,71.1,65,4.22,1.835,19.9,1,1,4,1
Toyota Corona,21.5,4,120.1,97,3.7,2.465,20.01,1,0,3,1
Dodge Challenger,15.5,8,318,150,2.76,3.52,16.87,0,0,3,2
AMC Javelin,15.2,8,304,150,3.15,3.435,17.3,0,0,3,2
Camaro Z28,13.3,8,350,245,3.73,3.84,15.41,0,0,3,4
Pontiac Firebird,19.2,8,400,175,3.08,3.845,17.05,0,0,3,2
Fiat X1-9,27.3,4,79,66,4.08,1.935,18.9,1,1,4,1
Porsche 914-2,26,4,120.3,91,4.43,2.14,16.7,0,1,5,2
Lotus Europa,30.4,4,95.1,113,3.77,1.513,16.9,1,1,5,2
Ford Pantera L,15.8,8,351,264,4.22,3.17,14.5,0,1,5,4
Ferrari Dino,19.7,6,145,175,3.62,2.77,15.5,0,1,5,6
Maserati Bora,15,8,301,335,3.54,3.57,14.6,0,1,5,8
Volvo 142E,21.4,4,121,109,4.11,2.78,18.6,1,1,4,2
"""
mtcars = pd.read_csv(StringIO(MTCARS_CSV), index_col="model")
mtcars.head()| mpg | cyl | disp | hp | drat | wt | qsec | vs | am | gear | carb | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| model | |||||||||||
| Mazda RX4 | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.620 | 16.46 | 0 | 1 | 4 | 4 |
| Mazda RX4 Wag | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.875 | 17.02 | 0 | 1 | 4 | 4 |
| Datsun 710 | 22.8 | 4 | 108.0 | 93 | 3.85 | 2.320 | 18.61 | 1 | 1 | 4 | 1 |
| Hornet 4 Drive | 21.4 | 6 | 258.0 | 110 | 3.08 | 3.215 | 19.44 | 1 | 0 | 3 | 1 |
| Hornet Sportabout | 18.7 | 8 | 360.0 | 175 | 3.15 | 3.440 | 17.02 | 0 | 0 | 3 | 2 |
0 10
1 20
2 30
3 40
Name: id, dtype: int64
| id | color | passed | |
|---|---|---|---|
| 0 | 10 | red | True |
| 1 | 20 | white | True |
| 2 | 30 | red | True |
| id | color | |
|---|---|---|
| 0 | 10 | red |
| 1 | 20 | white |
| 2 | 30 | red |
| id | color | |
|---|---|---|
| 0 | 10 | red |
| 1 | 20 | white |
| 2 | 30 | red |
| 3 | 40 | None |
| mpg | cyl | disp | hp | drat | wt | qsec | vs | am | gear | carb | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| model | |||||||||||
| Mazda RX4 | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.620 | 16.46 | 0 | 1 | 4 | 4 |
| Mazda RX4 Wag | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.875 | 17.02 | 0 | 1 | 4 | 4 |
| Datsun 710 | 22.8 | 4 | 108.0 | 93 | 3.85 | 2.320 | 18.61 | 1 | 1 | 4 | 1 |
| Hornet 4 Drive | 21.4 | 6 | 258.0 | 110 | 3.08 | 3.215 | 19.44 | 1 | 0 | 3 | 1 |
| Hornet Sportabout | 18.7 | 8 | 360.0 | 175 | 3.15 | 3.440 | 17.02 | 0 | 0 | 3 | 2 |
count 32.000000
mean 20.090625
std 6.026948
min 10.400000
25% 15.425000
50% 19.200000
75% 22.800000
max 33.900000
Name: mpg, dtype: float64
if / elif / else takes a single True or Falsenp.where() is the vectorized version, R’s ifelse()for item in sequence: iterates over any list, array, string, or dictenumerate() yields position and value together, the safer analogue of seq_along()i = 1 i^2 = 1
i = 2 i^2 = 4
i = 3 i^2 = 9
i = 4 i^2 = 16
i = 5 i^2 = 25
[1, 4, 9, 16, 25]
[1, 4, 9, 16, 25]
0 a
1 b
2 c
Truei += 1 is shorthand for i = i + 1; make sure something inside can flip the conditiondef name(arguments): followed by an indented bodyreturn is required. Unlike R, the last expression is not returned automatically, and a function without return hands back Nonehelp() displaysmean 18.00
sd 13.49
n 6.00
dtype: float64
mean 18.0000
sd 13.4907
n 6.0000
dtype: float64
Idiomatic Python replaces most loops with a comprehension or with a vectorized method call.
map(): R’s sapply() and lapply()axis=: R’s apply()axis=0 collapses down the rows and returns one value per column, which is R’s apply(m, 2, ...), so the numbers are swapped relative to Rdf.apply() runs a function column by column across a data frame[i ** 2 for i in range(1, 6)] # sapply(1:5, function(i) i^2)
list(map(lambda i: i ** 2, range(1, 6))) # the map() form
m.sum(axis=0) # column sums, R's apply(m, 2, sum)
m.max(axis=1) # row maxima, R's apply(m, 1, max)
mtcars[["mpg", "hp", "wt"]].mean()
mtcars[["mpg", "hp", "wt"]].apply(lambda col: col.max() - col.min())[1, 4, 9, 16, 25]
[1, 4, 9, 16, 25]
array([15, 40, 65, 90])
array([16, 17, 18, 19, 20])
mpg 20.090625
hp 146.687500
wt 3.217250
dtype: float64
mpg 23.500
hp 283.000
wt 3.911
dtype: float64
help(np.mean), or np.mean? in Jupyter, opens the documentationtype(x) when a result surprises you: the problem is almost always the typedir(x) lists what the object can do%whos is the IPython magic closest to R’s ls()No variables match your requested type.
<class 'pandas.core.frame.DataFrame'>
Index: 32 entries, Mazda RX4 to Volvo 142E
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 mpg 32 non-null float64
1 cyl 32 non-null int64
2 disp 32 non-null float64
3 hp 32 non-null int64
4 drat 32 non-null float64
5 wt 32 non-null float64
6 qsec 32 non-null float64
7 vs 32 non-null int64
8 am 32 non-null int64
9 gear 32 non-null int64
10 carb 32 non-null int64
dtypes: float64(5), int64(6)
memory usage: 3.0+ KB
Python has no single tidyverse. A handful of independent packages divide the same work, each imported on its own.
pandas: data manipulation and reshaping, covering dplyr and tidyrnumpy: arrays and numerical computation, the layer pandas is built onmatplotlib, seaborn, plotnine: visualization, where plotnine is a direct port of ggplot2pandas.read_csv and pyarrow: file import, covering readrscipy, statsmodels, scikit-learn: statistics, models, and machine learningInstall once with conda or pip, import every session.
The same rule governs everything downstream.
pandas calls this long form. Data in this shape works directly with groupby, melt, and every plotting library.
| mpg | cyl | disp | hp | drat | wt | qsec | vs | am | gear | carb | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| model | |||||||||||
| Mazda RX4 | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.620 | 16.46 | 0 | 1 | 4 | 4 |
| Mazda RX4 Wag | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.875 | 17.02 | 0 | 1 | 4 | 4 |
| Datsun 710 | 22.8 | 4 | 108.0 | 93 | 3.85 | 2.320 | 18.61 | 1 | 1 | 4 | 1 |
| Hornet 4 Drive | 21.4 | 6 | 258.0 | 110 | 3.08 | 3.215 | 19.44 | 1 | 0 | 3 | 1 |
| Hornet Sportabout | 18.7 | 8 | 360.0 | 175 | 3.15 | 3.440 | 17.02 | 0 | 0 | 3 | 2 |
<class 'pandas.core.frame.DataFrame'>
Index: 32 entries, Mazda RX4 to Volvo 142E
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 mpg 32 non-null float64
1 cyl 32 non-null int64
2 disp 32 non-null float64
3 hp 32 non-null int64
4 drat 32 non-null float64
5 wt 32 non-null float64
6 qsec 32 non-null float64
7 vs 32 non-null int64
8 am 32 non-null int64
9 gear 32 non-null int64
10 carb 32 non-null int64
dtypes: float64(5), int64(6)
memory usage: 3.0+ KB
Python has no pipe operator. Methods return a new object, so calls chain left to right instead.
.pipe(f) inserts a function that is not a method, which is the nearest thing to |>| mpg | cyl | disp | hp | drat | wt | qsec | vs | am | gear | carb | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| model | |||||||||||
| Datsun 710 | 22.8 | 4 | 108.0 | 93 | 3.85 | 2.32 | 18.61 | 1 | 1 | 4 | 1 |
| Merc 240D | 24.4 | 4 | 146.7 | 62 | 3.69 | 3.19 | 20.00 | 1 | 0 | 4 | 2 |
| Merc 230 | 22.8 | 4 | 140.8 | 95 | 3.92 | 3.15 | 22.90 | 1 | 0 | 4 | 2 |
| mpg | cyl | disp | hp | drat | wt | qsec | vs | am | gear | carb | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| model | |||||||||||
| Datsun 710 | 22.8 | 4 | 108.0 | 93 | 3.85 | 2.32 | 18.61 | 1 | 1 | 4 | 1 |
| Merc 240D | 24.4 | 4 | 146.7 | 62 | 3.69 | 3.19 | 20.00 | 1 | 0 | 4 | 2 |
| Merc 230 | 22.8 | 4 | 140.8 | 95 | 3.92 | 3.15 | 22.90 | 1 | 0 | 4 | 2 |
Five operations cover most day-to-day data manipulation.
query(), or a boolean mask: filter()[[ ]], or .filter(items = ): select()assign(): mutate()sort_values(ascending = False): arrange() with desc()agg(): summarize()One false friend to watch: the pandas method named filter() selects columns, where the dplyr verb of the same name selects rows.
| mpg | hp | wt | km_per_liter | |
|---|---|---|---|---|
| model | ||||
| Toyota Corolla | 33.9 | 65 | 1.835 | 14.4075 |
| Fiat 128 | 32.4 | 66 | 2.200 | 13.7700 |
| Honda Civic | 30.4 | 52 | 1.615 | 12.9200 |
| Lotus Europa | 30.4 | 113 | 1.513 | 12.9200 |
| Fiat X1-9 | 27.3 | 66 | 1.935 | 11.6025 |
groupby() splits the data, agg() collapses each group, and the result is one row per group.
"size" counts rows within the current group, R’s n()reset_index() turns them back into an ordinary columnmelt() turns wide data into long data, which is what plotting functions expect: R’s pivot_longer()pivot() and pivot_table() reverse it: R’s pivot_wider()| model | measure | value | |
|---|---|---|---|
| 0 | Mazda RX4 | mpg | 21.0 |
| 1 | Mazda RX4 Wag | mpg | 21.0 |
| 2 | Datsun 710 | mpg | 22.8 |
| 3 | Hornet 4 Drive | mpg | 21.4 |
| 4 | Hornet Sportabout | mpg | 18.7 |
| measure | hp | mpg |
|---|---|---|
| model | ||
| AMC Javelin | 150.0 | 15.2 |
| Cadillac Fleetwood | 205.0 | 10.4 |
| Camaro Z28 | 245.0 | 13.3 |
| Chrysler Imperial | 230.0 | 14.7 |
| Datsun 710 | 93.0 | 22.8 |
| Task | R | Python |
|---|---|---|
| Assign | x <- 5 |
x = 5 |
| Exponent | 2^10 |
2 ** 10 |
| Integer division, remainder | %/%, %% |
//, % |
| Storage type | typeof(x), class(x) |
type(x) |
| Structure summary | str(x) |
df.info(), type(x), dir(x) |
| Combine values | c(1, 2, 3) |
[1, 2, 3], np.array([1, 2, 3]) |
| Integer sequence | 1:10 |
range(1, 11), np.arange(1, 11) |
| Sequence by step, by length | seq(by =), seq(length.out =) |
np.arange(), np.linspace() |
| Repeat | rep(x, times =), rep(x, each =) |
np.tile(), np.repeat() |
| First element | x[1] |
x[0] |
| Last element | x[length(x)] |
x[-1] |
| Drop first element | x[-1] |
np.delete(x, 0), x[1:] |
| Boolean subset | x[x > 20] |
x[x > 20] |
| Missing value | NA, is.na() |
np.nan, pd.isna() |
| Empty object | NULL |
None |
| Ignore missing | mean(x, na.rm = TRUE) |
np.nanmean(x), Series.mean() |
| Sample standard deviation | sd(x) |
x.std(ddof=1) |
| Matrix | matrix(1:20, nrow = 5) |
np.arange(1, 21).reshape(5, 4, order="F") |
| Dimensions, transpose | dim(m), t(m) |
m.shape, m.T |
| Heterogeneous container | list() |
dict, list |
| Categorical | factor(), levels() |
pd.Categorical(), .categories |
| Frequency table | table(x) |
Series.value_counts() |
| Data frame | data.frame() |
pd.DataFrame() |
| Column | df$id |
df["id"] |
| Row subset | df[df$passed, ] |
df[df["passed"]], df.query() |
| Vectorized if | ifelse() |
np.where() |
| Apply over elements | sapply(), lapply() |
list comprehension, map() |
| Apply over columns, rows | apply(m, 2, f), apply(m, 1, f) |
axis=0, axis=1 |
| Function | f <- function(x) { } |
def f(x): |
| Pipe | x \|> f() \|> g() |
x.f().g(), .pipe(f) |
| dplyr verbs | filter, select, mutate, arrange |
query, [[ ]], assign, sort_values |
| Group and summarize | group_by() \|> summarize() |
groupby().agg() |
| Reshape | pivot_longer(), pivot_wider() |
melt(), pivot() |
| List the environment | ls() |
%whos |
| Help | ?mean |
help(np.mean), np.mean? |