# Show the value of every top-level expression, not just the last one
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"Python Fundamentals
Review Fundamentals
A working refresher on the Python language, followed by a first look at the data stack
- Part 2: data structures
- Part 3: control flow and functions
- Part 4: NumPy and pandas
This notebook mirrors the R fundamentals notebook section by section, so each idea appears next to the R idiom it replaces.
Run every cell as we go. The fastest way to learn Python is to break things and read the traceback.
Aside: Data driven narrative
https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-APOCALYPSE/egpbykdxjvq/
Formatting
R prints every top-level expression in a cell. Python prints only the last one. The following cell restores the R behavior so the two notebooks produce comparable output; don’t worry about the details.
Part 1: The Basics
Why Python
- General purpose language: data work lives in the same language as scripting, web services, and production code
- Single values are the native unit of computation; vectorization is provided by NumPy, not by the language
- Huge package ecosystem on PyPI and conda-forge, with strong tooling for notebooks and reproducible environments
- Two dialects coexist: base Python (this notebook, Part 1 to 3) and the array/table stack (Part 4)
Assignment and printing
=is the only assignment operator; there is no<-#starts a comment- A bare object name prints itself as the last line of a cell;
print()does it explicitly - An f-string is the closest analogue of
cat()
x = 5
y = 3
z = x + y
z # auto-print
print(z) # explicit print
print("z is", z) # print: space separated, newline added for you
print(f"z is {z}") # f-string: any expression inside the braces8
8
z is 8
z is 8
Arithmetic operators
+ - * /behave as expected, but exponentiation is**, not^^is legal Python and means bitwise exclusive or, which makes it a silent R-to-Python bug%is modulo (remainder),//is floor division/always returns a float, even between two integers
import math
7 + 2
7 / 2 # 3.5, always a float
7 % 2 # remainder, R's %%
7 // 2 # floor division, R's %/%
2 ** 10 # R's 2^10
math.sqrt(16)
round(3.14159, 2)9
3.5
1
3
1024
4.0
3.14
Comparison and logical operators
- Comparison:
== != < > <= >= - Logical on single values:
and,or,not - Logical element by element on arrays and Series:
&,|,~, with each side in parentheses - Comparisons return
TrueorFalse, which are themselves a data type - Comparisons can be chained, which R cannot do
5 > 3
5 == 5
5 != 5
(5 > 3) and (2 > 1) # both must be True
(5 > 3) or (2 > 10) # at least one True
not (5 > 3) # negation
3 < 5 < 10 # chained comparisonTrue
True
False
True
True
False
True
The fundamental data types
Python has a small set of built-in types. Everything else is built from them.
float: real numbers, the default for anything with a decimal point, R’sdoubleint: whole numbers, unlimited precision, noLsuffix neededstr: text, in single or double quotes, R’scharacterbool:TrueandFalse, capitalized, R’slogicalcomplexandbytes: rarely needed in an intro course
a = 3.14 # float
b = 3 # int
d = "hello" # str
e = True # bool
type(a)
type(b)
type(d)
type(e)float
int
str
bool
Inspecting an object
Three questions worth asking about any object you did not create yourself.
type(): what is this, covering both R’stypeof()andclass()isinstance(): the analogue ofis.numeric()and friendsdir(): every attribute and method the object carries, the usual way to discover what you can do next
A number in Python is a scalar. Unlike R, it is not a length-1 vector, and len() on it is an error.
x = 3.14
type(x)
isinstance(x, float)
isinstance(x, str)
len("hello") # len() works on containers, not on numbers
dir(x)[-8:] # the last few methods available on a floatfloat
True
False
5
['__trunc__',
'as_integer_ratio',
'conjugate',
'fromhex',
'hex',
'imag',
'is_integer',
'real']
Type conversion
int(),float(),str(),bool()convert explicitly- Failed conversions raise a
ValueErrorand stop execution; R returnsNAwith a warning int()truncates toward zero; it does not round- For the R-style behavior on a whole column, use
pd.to_numeric(..., errors="coerce"), which yields missing values instead of raising
int(3.99) # 3, truncated
str(3.14) # '3.14'
float("42") # 42.0
bool(0) # False
float(True) # 1.0
try:
float("banana")
except ValueError as err:
print("ValueError:", err)3
'3.14'
42.0
False
1.0
ValueError: could not convert string to float: 'banana'
Special values
None: the empty object, R’sNULLfloat("nan")ornp.nan: missing or undefined, covering both R’sNAandNaNnp.infand-np.inf: overflow or division by zero- Test with
pd.isna()ormath.isnan(), never with== np.nan, because nan is not equal to itself
NumPy propagates missingness the way R does. pandas goes the other way and skips missing values by default, which is the single most common source of quiet disagreement between an R script and its Python translation.
import numpy as np
import pandas as pd
v = np.array([1, 2, np.nan, 4])
v.mean() # nan: missingness propagates
np.nanmean(v) # drop the nan first, R's na.rm = TRUE
pd.Series(v).mean() # pandas drops it without being asked
pd.isna(v) # element-wise test, R's is.na()
np.nan == np.nan # False: never test for missingness with ==
with np.errstate(divide="ignore", invalid="ignore"):
print(np.float64(0) / np.float64(0)) # nan
print(np.float64(1) / np.float64(0)) # infnp.float64(nan)
np.float64(2.3333333333333335)
np.float64(2.3333333333333335)
array([False, False, True, False])
False
nan
inf
Part 2: Data Structures
Lists
The base Python workhorse. A list is an ordered, mutable sequence that can mix types freely.
[ ]builds a list literal, the closest analogue ofc()range()builds an integer sequence, but it is lazy; wrap it inlist()to see the values- A list does no arithmetic:
x * 2repeats the list, it does not double each element - For numeric work, reach for a NumPy array instead
x = [10, 15, 20, 25, 30, 35]
x
list(range(1, 11)) # R's 1:10, the upper bound is excluded
["a", "b"] * 3 # R's rep(c("a", "b"), times = 3)
len(x)
x + [40] # concatenation, not addition
x * 2 # repetition, not element-wise doubling[10, 15, 20, 25, 30, 35]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
['a', 'b', 'a', 'b', 'a', 'b']
6
[10, 15, 20, 25, 30, 35, 40]
[10, 15, 20, 25, 30, 35, 10, 15, 20, 25, 30, 35]
NumPy arrays
The true analogue of an R vector: one type, one fixed length, arithmetic applied element by element.
np.array()converts a list;np.arange()is R’sseq(by = )np.linspace()is R’sseq(length.out = ).dtypereports the single stored type,.shapeand.sizereport the extent
x = np.array([10, 15, 20, 25, 30, 35])
x
x.dtype
np.arange(0, 1.25, 0.25) # seq(0, 1, by = 0.25), upper bound excluded
np.linspace(0, 10, 5) # seq(0, 10, length.out = 5)
np.tile(["a", "b"], 3) # rep(..., times = 3)
np.repeat(["a", "b"], 3) # rep(..., each = 3)
x.sizearray([10, 15, 20, 25, 30, 35])
dtype('int64')
array([0. , 0.25, 0.5 , 0.75, 1. ])
array([ 0. , 2.5, 5. , 7.5, 10. ])
array(['a', 'b', 'a', 'b', 'a', 'b'], dtype='<U1')
array(['a', 'a', 'a', 'b', 'b', 'b'], dtype='<U1')
6
Indexing
- Python indexes from 0, not from 1
- A slice
a:bincludes positionaand excludes positionb - A negative index counts backward from the end; it does not mean “everything except”, which is the R meaning
- A boolean array selects the positions that are
True, exactly as in R
x = np.array([10, 15, 20, 25, 30, 35])
x[0] # first element, R's x[1]
x[-1] # last element, R's x[length(x)]
x[1:4] # a contiguous slice, R's x[2:4]
x[[0, 4]] # arbitrary positions, R's x[c(1, 5)]
np.delete(x, 0) # drop the first element, R's x[-1]
x[x > 20] # boolean subsetting: the key idiom, identical in both languagesnp.int64(10)
np.int64(35)
array([15, 20, 25])
array([10, 30])
array([15, 20, 25, 30, 35])
array([25, 30, 35])
Vectorized operations
Operations apply element by element with no loop required. This is the central habit to build in NumPy.
- Arrays of different shapes are combined by broadcasting, which stretches length-1 dimensions only
- Unlike R recycling, a genuine length mismatch raises an error instead of failing quietly
np.std()defaults to the population formula; R’ssd()is the sample formula, so passddof=1to match
x = np.array([1, 2, 3, 4])
x * 2 # every element doubled
x + x
x ** 2
np.sqrt(x)
x.sum(), x.mean(), x.max(), x.std(ddof=1)
try:
x + np.array([1, 2, 3]) # R would recycle and warn; NumPy refuses
except ValueError as err:
print("ValueError:", err)array([2, 4, 6, 8])
array([2, 4, 6, 8])
array([ 1, 4, 9, 16])
array([1. , 1.41421356, 1.73205081, 2. ])
(np.int64(10), np.float64(2.5), np.int64(4), np.float64(1.2909944487358056))
ValueError: operands could not be broadcast together with shapes (4,) (3,)
Labelled data and coercion
- A pandas
Seriesis an array plus an index of labels: R’s named vector - A plain
dictis the other analogue, though it supports no arithmetic - An array holds exactly one type; mixing types silently coerces everything upward
- The coercion ladder: bool, then int, then float, then string
scores = pd.Series({"math": 90, "art": 75, "bio": 88})
scores
scores["art"]
np.array([1, 2, True]) # bool becomes int
np.array([1, 2, "three"]) # everything becomes a stringmath 90
art 75
bio 88
dtype: int64
np.int64(75)
array([1, 2, 1])
array(['1', '2', 'three'], dtype='<U21')
Matrices
A matrix is a two-dimensional NumPy array.
reshape()fills row-first by default; passorder="F"to match R, which fills column-first- Indexed as
[row, column]; a bare:keeps that whole dimension
m = np.arange(1, 21).reshape(5, 4, order="F") # matrix(1:20, nrow = 5, ncol = 4)
m
m.shape # R's dim()
m[2, 1] # single element, R's m[3, 2]
m[2, :] # third row, R's m[3, ]
m[:, 1] # second column, R's m[, 2]
m.T # transpose, R's t()array([[ 1, 6, 11, 16],
[ 2, 7, 12, 17],
[ 3, 8, 13, 18],
[ 4, 9, 14, 19],
[ 5, 10, 15, 20]])
(5, 4)
np.int64(8)
array([ 3, 8, 13, 18])
array([ 6, 7, 8, 9, 10])
array([[ 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10],
[11, 12, 13, 14, 15],
[16, 17, 18, 19, 20]])
Dictionaries
A dict holds elements of different types and different lengths: the analogue of an R list().
- Keys are usually strings; values can be anything, including other dicts
- R’s
$and[[ ]]both become[ ], with no partial matching of names .get()supplies a default instead of raising aKeyError
person = {"name": "Ada", "age": 36, "scores": [90, 95, 99]}
person # the str() analogue for a small object
person.keys()
person["name"]
person["scores"]
person["scores"][1] # second score, R's person$scores[2]
person.get("height", "not recorded") # safe lookup with a fallback{'name': 'Ada', 'age': 36, 'scores': [90, 95, 99]}
dict_keys(['name', 'age', 'scores'])
'Ada'
[90, 95, 99]
95
'not recorded'
Categoricals
A pandas Categorical represents categorical data with a fixed set of possible values, called categories.
- Stored as integer codes plus a label lookup table, exactly like an R factor
- Category order controls the order categories appear in tables and plots
- The codes start at 0, where R’s factor codes start at 1
sizes = pd.Categorical(
["small", "large", "small", "medium"],
categories=["small", "medium", "large"],
ordered=True,
)
sizes
sizes.categories # R's levels()
pd.Series(sizes).value_counts(sort=False) # R's table()
sizes.codes # the underlying codes['small', 'large', 'small', 'medium']
Categories (3, object): ['small' < 'medium' < 'large']
Index(['small', 'medium', 'large'], dtype='object')
small 2
medium 1
large 1
Name: count, dtype: int64
array([0, 2, 0, 1], dtype=int8)
Data frames
A pandas DataFrame is a dict of equal-length columns displayed as a table: rows are observations, columns are variables.
- Each column has its own dtype
- Every row also carries an index label, which a base R data frame does not have
- This is the structure nearly all real analysis uses
df = pd.DataFrame({
"id": [10, 20, 30, 40],
"color": ["red", "white", "red", None],
"passed": [True, True, True, False],
})
df
df.info() # R's str()
df.shape # R's dim()
list(df.columns) # R's names()| id | color | passed | |
|---|---|---|---|
| 0 | 10 | red | True |
| 1 | 20 | white | True |
| 2 | 30 | red | True |
| 3 | 40 | None | False |
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 3 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 id 4 non-null int64
1 color 3 non-null object
2 passed 4 non-null bool
dtypes: bool(1), int64(1), object(1)
memory usage: 200.0+ bytes
(4, 3)
['id', 'color', 'passed']
Working with data frames
df["id"]pulls a single column out as a Seriesdf.loc[rows, columns]subsets by label,df.iloc[rows, columns]by positionhead(),describe(), andinfo()are the first three things to run on any new dataset
Python ships no datasets with the language, so the cell below embeds R’s mtcars directly. For real work, seaborn.load_dataset() and statsmodels.api.datasets.get_rdataset() download the familiar R datasets on demand.
from io import StringIO
MTCARS_CSV = """\
model,mpg,cyl,disp,hp,drat,wt,qsec,vs,am,gear,carb
Mazda RX4,21,6,160,110,3.9,2.62,16.46,0,1,4,4
Mazda RX4 Wag,21,6,160,110,3.9,2.875,17.02,0,1,4,4
Datsun 710,22.8,4,108,93,3.85,2.32,18.61,1,1,4,1
Hornet 4 Drive,21.4,6,258,110,3.08,3.215,19.44,1,0,3,1
Hornet Sportabout,18.7,8,360,175,3.15,3.44,17.02,0,0,3,2
Valiant,18.1,6,225,105,2.76,3.46,20.22,1,0,3,1
Duster 360,14.3,8,360,245,3.21,3.57,15.84,0,0,3,4
Merc 240D,24.4,4,146.7,62,3.69,3.19,20,1,0,4,2
Merc 230,22.8,4,140.8,95,3.92,3.15,22.9,1,0,4,2
Merc 280,19.2,6,167.6,123,3.92,3.44,18.3,1,0,4,4
Merc 280C,17.8,6,167.6,123,3.92,3.44,18.9,1,0,4,4
Merc 450SE,16.4,8,275.8,180,3.07,4.07,17.4,0,0,3,3
Merc 450SL,17.3,8,275.8,180,3.07,3.73,17.6,0,0,3,3
Merc 450SLC,15.2,8,275.8,180,3.07,3.78,18,0,0,3,3
Cadillac Fleetwood,10.4,8,472,205,2.93,5.25,17.98,0,0,3,4
Lincoln Continental,10.4,8,460,215,3,5.424,17.82,0,0,3,4
Chrysler Imperial,14.7,8,440,230,3.23,5.345,17.42,0,0,3,4
Fiat 128,32.4,4,78.7,66,4.08,2.2,19.47,1,1,4,1
Honda Civic,30.4,4,75.7,52,4.93,1.615,18.52,1,1,4,2
Toyota Corolla,33.9,4,71.1,65,4.22,1.835,19.9,1,1,4,1
Toyota Corona,21.5,4,120.1,97,3.7,2.465,20.01,1,0,3,1
Dodge Challenger,15.5,8,318,150,2.76,3.52,16.87,0,0,3,2
AMC Javelin,15.2,8,304,150,3.15,3.435,17.3,0,0,3,2
Camaro Z28,13.3,8,350,245,3.73,3.84,15.41,0,0,3,4
Pontiac Firebird,19.2,8,400,175,3.08,3.845,17.05,0,0,3,2
Fiat X1-9,27.3,4,79,66,4.08,1.935,18.9,1,1,4,1
Porsche 914-2,26,4,120.3,91,4.43,2.14,16.7,0,1,5,2
Lotus Europa,30.4,4,95.1,113,3.77,1.513,16.9,1,1,5,2
Ford Pantera L,15.8,8,351,264,4.22,3.17,14.5,0,1,5,4
Ferrari Dino,19.7,6,145,175,3.62,2.77,15.5,0,1,5,6
Maserati Bora,15,8,301,335,3.54,3.57,14.6,0,1,5,8
Volvo 142E,21.4,4,121,109,4.11,2.78,18.6,1,1,4,2
"""
mtcars = pd.read_csv(StringIO(MTCARS_CSV), index_col="model")
mtcars.head()| mpg | cyl | disp | hp | drat | wt | qsec | vs | am | gear | carb | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| model | |||||||||||
| Mazda RX4 | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.620 | 16.46 | 0 | 1 | 4 | 4 |
| Mazda RX4 Wag | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.875 | 17.02 | 0 | 1 | 4 | 4 |
| Datsun 710 | 22.8 | 4 | 108.0 | 93 | 3.85 | 2.320 | 18.61 | 1 | 1 | 4 | 1 |
| Hornet 4 Drive | 21.4 | 6 | 258.0 | 110 | 3.08 | 3.215 | 19.44 | 1 | 0 | 3 | 1 |
| Hornet Sportabout | 18.7 | 8 | 360.0 | 175 | 3.15 | 3.440 | 17.02 | 0 | 0 | 3 | 2 |
df["id"]
df[df["passed"]] # rows where passed is True
df.loc[df["passed"], ["id", "color"]] # rows and columns at once
df[["id", "color"]] # selected columns
mtcars.head()
mtcars["mpg"].describe() # R's summary(mtcars$mpg)0 10
1 20
2 30
3 40
Name: id, dtype: int64
| id | color | passed | |
|---|---|---|---|
| 0 | 10 | red | True |
| 1 | 20 | white | True |
| 2 | 30 | red | True |
| id | color | |
|---|---|---|
| 0 | 10 | red |
| 1 | 20 | white |
| 2 | 30 | red |
| id | color | |
|---|---|---|
| 0 | 10 | red |
| 1 | 20 | white |
| 2 | 30 | red |
| 3 | 40 | None |
| mpg | cyl | disp | hp | drat | wt | qsec | vs | am | gear | carb | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| model | |||||||||||
| Mazda RX4 | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.620 | 16.46 | 0 | 1 | 4 | 4 |
| Mazda RX4 Wag | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.875 | 17.02 | 0 | 1 | 4 | 4 |
| Datsun 710 | 22.8 | 4 | 108.0 | 93 | 3.85 | 2.320 | 18.61 | 1 | 1 | 4 | 1 |
| Hornet 4 Drive | 21.4 | 6 | 258.0 | 110 | 3.08 | 3.215 | 19.44 | 1 | 0 | 3 | 1 |
| Hornet Sportabout | 18.7 | 8 | 360.0 | 175 | 3.15 | 3.440 | 17.02 | 0 | 0 | 3 | 2 |
count 32.000000
mean 20.090625
std 6.026948
min 10.400000
25% 15.425000
50% 19.200000
75% 22.800000
max 33.900000
Name: mpg, dtype: float64
Part 3: Control Flow and Functions
Conditionals
if/elif/elsetakes a singleTrueorFalse- The header line ends in a colon and the body is indented; there are no braces
np.where()is the vectorized version, R’sifelse()
score = 85
if score >= 90:
print("A")
elif score >= 80:
print("B")
else:
print("C")
scores = np.array([95, 85, 60])
np.where(scores >= 80, "pass", "fail") # vectorizedB
array(['pass', 'pass', 'fail'], dtype='<U4')
For loops
for item in sequence:iterates over any list, array, string, or dictenumerate()yields position and value together, the safer analogue ofseq_along()- A list comprehension replaces most short loops and is the idiomatic form
- Appending to a list is cheap, so preallocation is not the concern it is in R
for i in range(1, 6):
print(f"i = {i} i^2 = {i ** 2}")
# grow a list
squares = []
for i in range(1, 6):
squares.append(i ** 2)
squares
[i ** 2 for i in range(1, 6)] # list comprehension, same result
for pos, val in enumerate(["a", "b", "c"]):
print(pos, val)i = 1 i^2 = 1
i = 2 i^2 = 4
i = 3 i^2 = 9
i = 4 i^2 = 16
i = 5 i^2 = 25
[1, 4, 9, 16, 25]
[1, 4, 9, 16, 25]
0 a
1 b
2 c
While loops
- Repeats while a condition stays
True - Use when the number of iterations is not known in advance
i += 1is shorthand fori = i + 1; make sure something inside can flip the condition
i = 1
while i <= 5:
print(i)
i += 11
2
3
4
5
Writing functions
def name(arguments):followed by an indented bodyreturnis required. Unlike R, the last expression is not returned automatically, and a function withoutreturnhands backNone- Arguments can carry default values
- The triple-quoted docstring on the first line is what
help()displays
def summarize_vec(x, digits=2):
"""Return the mean, standard deviation, and length of x."""
x = np.asarray(x)
return pd.Series({
"mean": x.mean(),
"sd": x.std(ddof=1),
"n": len(x),
}).round(digits)
v = [4, 8, 15, 16, 23, 42]
summarize_vec(v)
summarize_vec(v, digits=4)mean 18.00
sd 13.49
n 6.00
dtype: float64
mean 18.0000
sd 13.4907
n 6.0000
dtype: float64
The apply family
Idiomatic Python replaces most loops with a comprehension or with a vectorized method call.
- List comprehension, or
map(): R’ssapply()andlapply() - An array method with
axis=: R’sapply() - Watch the axis:
axis=0collapses down the rows and returns one value per column, which is R’sapply(m, 2, ...), so the numbers are swapped relative to R df.apply()runs a function column by column across a data frame
[i ** 2 for i in range(1, 6)] # sapply(1:5, function(i) i^2)
list(map(lambda i: i ** 2, range(1, 6))) # the map() form
m.sum(axis=0) # column sums, R's apply(m, 2, sum)
m.max(axis=1) # row maxima, R's apply(m, 1, max)
mtcars[["mpg", "hp", "wt"]].mean()
mtcars[["mpg", "hp", "wt"]].apply(lambda col: col.max() - col.min())[1, 4, 9, 16, 25]
[1, 4, 9, 16, 25]
array([15, 40, 65, 90])
array([16, 17, 18, 19, 20])
mpg 20.090625
hp 146.687500
wt 3.217250
dtype: float64
mpg 23.500
hp 283.000
wt 3.911
dtype: float64
Getting unstuck
help(np.mean), ornp.mean?in Jupyter, opens the documentationtype(x)when a result surprises you: the problem is almost always the typedir(x)lists what the object can do%whosis the IPython magic closest to R’sls()- Read the traceback from the bottom up: the last line is the actual error, the lines above are the call path
a = 1
b = "two"
%whos # everything defined right now
# help(np.mean) # uncomment to open the docs
mtcars.info() # R's str() on a data frameNo variables match your requested type.
<class 'pandas.core.frame.DataFrame'>
Index: 32 entries, Mazda RX4 to Volvo 142E
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 mpg 32 non-null float64
1 cyl 32 non-null int64
2 disp 32 non-null float64
3 hp 32 non-null int64
4 drat 32 non-null float64
5 wt 32 non-null float64
6 qsec 32 non-null float64
7 vs 32 non-null int64
8 am 32 non-null int64
9 gear 32 non-null int64
10 carb 32 non-null int64
dtypes: float64(5), int64(6)
memory usage: 3.0+ KB
Part 4: NumPy and pandas
What the stack is
Python has no single tidyverse. A handful of independent packages divide the same work, each imported on its own.
pandas: data manipulation and reshaping, coveringdplyrandtidyrnumpy: arrays and numerical computation, the layer pandas is built onmatplotlib,seaborn,plotnine: visualization, whereplotnineis a direct port of ggplot2pandas.read_csvandpyarrow: file import, coveringreadrscipy,statsmodels,scikit-learn: statistics, models, and machine learning
Install once with conda or pip, import every session.
import numpy as np
import pandas as pd
np.__version__
pd.__version__'2.4.2'
'2.3.3'
Tidy data
The same rule governs everything downstream.
- Each variable is a column
- Each observation is a row
- Each cell is a single value
pandas calls this long form. Data in this shape works directly with groupby, melt, and every plotting library.
mtcars.head()
mtcars.info() # the pandas counterpart to glimpse()| mpg | cyl | disp | hp | drat | wt | qsec | vs | am | gear | carb | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| model | |||||||||||
| Mazda RX4 | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.620 | 16.46 | 0 | 1 | 4 | 4 |
| Mazda RX4 Wag | 21.0 | 6 | 160.0 | 110 | 3.90 | 2.875 | 17.02 | 0 | 1 | 4 | 4 |
| Datsun 710 | 22.8 | 4 | 108.0 | 93 | 3.85 | 2.320 | 18.61 | 1 | 1 | 4 | 1 |
| Hornet 4 Drive | 21.4 | 6 | 258.0 | 110 | 3.08 | 3.215 | 19.44 | 1 | 0 | 3 | 1 |
| Hornet Sportabout | 18.7 | 8 | 360.0 | 175 | 3.15 | 3.440 | 17.02 | 0 | 0 | 3 | 2 |
<class 'pandas.core.frame.DataFrame'>
Index: 32 entries, Mazda RX4 to Volvo 142E
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 mpg 32 non-null float64
1 cyl 32 non-null int64
2 disp 32 non-null float64
3 hp 32 non-null int64
4 drat 32 non-null float64
5 wt 32 non-null float64
6 qsec 32 non-null float64
7 vs 32 non-null int64
8 am 32 non-null int64
9 gear 32 non-null int64
10 carb 32 non-null int64
dtypes: float64(5), int64(6)
memory usage: 3.0+ KB
Method chaining
Python has no pipe operator. Methods return a new object, so calls chain left to right instead.
- Wrap the whole chain in parentheses to spread it over several lines
.pipe(f)inserts a function that is not a method, which is the nearest thing to|>- Each step returns a copy, so the original frame is untouched
# without chaining, read from the inside out
mtcars[mtcars["cyl"] == 4].head(3)
# with chaining: read it as "take mtcars, then query, then take the head"
(
mtcars
.query("cyl == 4")
.head(3)
)| mpg | cyl | disp | hp | drat | wt | qsec | vs | am | gear | carb | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| model | |||||||||||
| Datsun 710 | 22.8 | 4 | 108.0 | 93 | 3.85 | 2.32 | 18.61 | 1 | 1 | 4 | 1 |
| Merc 240D | 24.4 | 4 | 146.7 | 62 | 3.69 | 3.19 | 20.00 | 1 | 0 | 4 | 2 |
| Merc 230 | 22.8 | 4 | 140.8 | 95 | 3.92 | 3.15 | 22.90 | 1 | 0 | 4 | 2 |
| mpg | cyl | disp | hp | drat | wt | qsec | vs | am | gear | carb | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| model | |||||||||||
| Datsun 710 | 22.8 | 4 | 108.0 | 93 | 3.85 | 2.32 | 18.61 | 1 | 1 | 4 | 1 |
| Merc 240D | 24.4 | 4 | 146.7 | 62 | 3.69 | 3.19 | 20.00 | 1 | 0 | 4 | 2 |
| Merc 230 | 22.8 | 4 | 140.8 | 95 | 3.92 | 3.15 | 22.90 | 1 | 0 | 4 | 2 |
The core pandas verbs
Five operations cover most day-to-day data manipulation.
query(), or a boolean mask:filter()[[ ]], or.filter(items = ):select()assign():mutate()sort_values(ascending = False):arrange()withdesc()agg():summarize()
One false friend to watch: the pandas method named filter() selects columns, where the dplyr verb of the same name selects rows.
(
mtcars
.query("cyl == 4")
.filter(["mpg", "hp", "wt"])
.assign(km_per_liter=lambda d: d["mpg"] * 0.425)
.sort_values("mpg", ascending=False)
.head(5)
)| mpg | hp | wt | km_per_liter | |
|---|---|---|---|---|
| model | ||||
| Toyota Corolla | 33.9 | 65 | 1.835 | 14.4075 |
| Fiat 128 | 32.4 | 66 | 2.200 | 13.7700 |
| Honda Civic | 30.4 | 52 | 1.615 | 12.9200 |
| Lotus Europa | 30.4 | 113 | 1.513 | 12.9200 |
| Fiat X1-9 | 27.3 | 66 | 1.935 | 11.6025 |
Group and summarize
groupby() splits the data, agg() collapses each group, and the result is one row per group.
- This single pattern replaces most of what pivot tables do
"size"counts rows within the current group, R’sn()- The group labels become the index, so
reset_index()turns them back into an ordinary column
(
mtcars
.groupby("cyl")
.agg(
n=("mpg", "size"),
mean_mpg=("mpg", "mean"),
mean_hp=("hp", "mean"),
)
.sort_values("mean_mpg", ascending=False)
.reset_index()
)| cyl | n | mean_mpg | mean_hp | |
|---|---|---|---|---|
| 0 | 4 | 11 | 26.663636 | 82.636364 |
| 1 | 6 | 7 | 19.742857 | 122.285714 |
| 2 | 8 | 14 | 15.100000 | 209.214286 |
Reshaping
melt()turns wide data into long data, which is what plotting functions expect: R’spivot_longer()pivot()andpivot_table()reverse it: R’spivot_wider()
long = (
mtcars
.reset_index()
.filter(["model", "mpg", "hp"])
.melt(id_vars="model", var_name="measure", value_name="value")
)
long.head()
long.pivot(index="model", columns="measure", values="value").head()| model | measure | value | |
|---|---|---|---|
| 0 | Mazda RX4 | mpg | 21.0 |
| 1 | Mazda RX4 Wag | mpg | 21.0 |
| 2 | Datsun 710 | mpg | 22.8 |
| 3 | Hornet 4 Drive | mpg | 21.4 |
| 4 | Hornet Sportabout | mpg | 18.7 |
| measure | hp | mpg |
|---|---|---|
| model | ||
| AMC Javelin | 150.0 | 15.2 |
| Cadillac Fleetwood | 205.0 | 10.4 |
| Camaro Z28 | 245.0 | 13.3 |
| Chrysler Imperial | 230.0 | 14.7 |
| Datsun 710 | 93.0 | 22.8 |
Quick reference
| Task | R | Python |
|---|---|---|
| Assign | x <- 5 |
x = 5 |
| Exponent | 2^10 |
2 ** 10 |
| Integer division, remainder | %/%, %% |
//, % |
| Storage type | typeof(x), class(x) |
type(x) |
| Structure summary | str(x) |
df.info(), type(x), dir(x) |
| Combine values | c(1, 2, 3) |
[1, 2, 3], np.array([1, 2, 3]) |
| Integer sequence | 1:10 |
range(1, 11), np.arange(1, 11) |
| Sequence by step, by length | seq(by =), seq(length.out =) |
np.arange(), np.linspace() |
| Repeat | rep(x, times =), rep(x, each =) |
np.tile(), np.repeat() |
| First element | x[1] |
x[0] |
| Last element | x[length(x)] |
x[-1] |
| Drop first element | x[-1] |
np.delete(x, 0), x[1:] |
| Boolean subset | x[x > 20] |
x[x > 20] |
| Missing value | NA, is.na() |
np.nan, pd.isna() |
| Empty object | NULL |
None |
| Ignore missing | mean(x, na.rm = TRUE) |
np.nanmean(x), Series.mean() |
| Sample standard deviation | sd(x) |
x.std(ddof=1) |
| Matrix | matrix(1:20, nrow = 5) |
np.arange(1, 21).reshape(5, 4, order="F") |
| Dimensions, transpose | dim(m), t(m) |
m.shape, m.T |
| Heterogeneous container | list() |
dict, list |
| Categorical | factor(), levels() |
pd.Categorical(), .categories |
| Frequency table | table(x) |
Series.value_counts() |
| Data frame | data.frame() |
pd.DataFrame() |
| Column | df$id |
df["id"] |
| Row subset | df[df$passed, ] |
df[df["passed"]], df.query() |
| Vectorized if | ifelse() |
np.where() |
| Apply over elements | sapply(), lapply() |
list comprehension, map() |
| Apply over columns, rows | apply(m, 2, f), apply(m, 1, f) |
axis=0, axis=1 |
| Function | f <- function(x) { } |
def f(x): |
| Pipe | x \|> f() \|> g() |
x.f().g(), .pipe(f) |
| dplyr verbs | filter, select, mutate, arrange |
query, [[ ]], assign, sort_values |
| Group and summarize | group_by() \|> summarize() |
groupby().agg() |
| Reshape | pivot_longer(), pivot_wider() |
melt(), pivot() |
| List the environment | ls() |
%whos |
| Help | ?mean |
help(np.mean), np.mean? |