Python Fundamentals

Review Fundamentals

A working refresher on the Python language, followed by a first look at the data stack

  • Part 2: data structures
  • Part 3: control flow and functions
  • Part 4: NumPy and pandas

This notebook mirrors the R fundamentals notebook section by section, so each idea appears next to the R idiom it replaces.

Run every cell as we go. The fastest way to learn Python is to break things and read the traceback.

Aside: Data driven narrative

https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-APOCALYPSE/egpbykdxjvq/

Formatting

R prints every top-level expression in a cell. Python prints only the last one. The following cell restores the R behavior so the two notebooks produce comparable output; don’t worry about the details.

Code
# Show the value of every top-level expression, not just the last one
from IPython.core.interactiveshell import InteractiveShell

InteractiveShell.ast_node_interactivity = "all"

Part 1: The Basics

Why Python

  • General purpose language: data work lives in the same language as scripting, web services, and production code
  • Single values are the native unit of computation; vectorization is provided by NumPy, not by the language
  • Huge package ecosystem on PyPI and conda-forge, with strong tooling for notebooks and reproducible environments
  • Two dialects coexist: base Python (this notebook, Part 1 to 3) and the array/table stack (Part 4)

Assignment and printing

  • = is the only assignment operator; there is no <-
  • # starts a comment
  • A bare object name prints itself as the last line of a cell; print() does it explicitly
  • An f-string is the closest analogue of cat()
Code
x = 5
y = 3
z = x + y

z                        # auto-print
print(z)                 # explicit print
print("z is", z)         # print: space separated, newline added for you
print(f"z is {z}")       # f-string: any expression inside the braces
8
8
z is 8
z is 8

Arithmetic operators

  • + - * / behave as expected, but exponentiation is **, not ^
  • ^ is legal Python and means bitwise exclusive or, which makes it a silent R-to-Python bug
  • % is modulo (remainder), // is floor division
  • / always returns a float, even between two integers
Code
import math

7 + 2
7 / 2        # 3.5, always a float
7 % 2        # remainder, R's %%
7 // 2       # floor division, R's %/%
2 ** 10      # R's 2^10
math.sqrt(16)
round(3.14159, 2)
9
3.5
1
3
1024
4.0
3.14

Comparison and logical operators

  • Comparison: == != < > <= >=
  • Logical on single values: and, or, not
  • Logical element by element on arrays and Series: &, |, ~, with each side in parentheses
  • Comparisons return True or False, which are themselves a data type
  • Comparisons can be chained, which R cannot do
Code
5 > 3
5 == 5
5 != 5

(5 > 3) and (2 > 1)      # both must be True
(5 > 3) or (2 > 10)      # at least one True
not (5 > 3)              # negation

3 < 5 < 10               # chained comparison
True
True
False
True
True
False
True

The fundamental data types

Python has a small set of built-in types. Everything else is built from them.

  • float: real numbers, the default for anything with a decimal point, R’s double
  • int: whole numbers, unlimited precision, no L suffix needed
  • str: text, in single or double quotes, R’s character
  • bool: True and False, capitalized, R’s logical
  • complex and bytes: rarely needed in an intro course
Code
a = 3.14       # float
b = 3          # int
d = "hello"    # str
e = True       # bool

type(a)
type(b)
type(d)
type(e)
float
int
str
bool

Inspecting an object

Three questions worth asking about any object you did not create yourself.

  • type(): what is this, covering both R’s typeof() and class()
  • isinstance(): the analogue of is.numeric() and friends
  • dir(): every attribute and method the object carries, the usual way to discover what you can do next

A number in Python is a scalar. Unlike R, it is not a length-1 vector, and len() on it is an error.

Code
x = 3.14

type(x)
isinstance(x, float)
isinstance(x, str)

len("hello")             # len() works on containers, not on numbers
dir(x)[-8:]              # the last few methods available on a float
float
True
False
5
['__trunc__',
 'as_integer_ratio',
 'conjugate',
 'fromhex',
 'hex',
 'imag',
 'is_integer',
 'real']

Type conversion

  • int(), float(), str(), bool() convert explicitly
  • Failed conversions raise a ValueError and stop execution; R returns NA with a warning
  • int() truncates toward zero; it does not round
  • For the R-style behavior on a whole column, use pd.to_numeric(..., errors="coerce"), which yields missing values instead of raising
Code
int(3.99)         # 3, truncated
str(3.14)         # '3.14'
float("42")       # 42.0
bool(0)           # False
float(True)       # 1.0

try:
    float("banana")
except ValueError as err:
    print("ValueError:", err)
3
'3.14'
42.0
False
1.0
ValueError: could not convert string to float: 'banana'

Special values

  • None: the empty object, R’s NULL
  • float("nan") or np.nan: missing or undefined, covering both R’s NA and NaN
  • np.inf and -np.inf: overflow or division by zero
  • Test with pd.isna() or math.isnan(), never with == np.nan, because nan is not equal to itself

NumPy propagates missingness the way R does. pandas goes the other way and skips missing values by default, which is the single most common source of quiet disagreement between an R script and its Python translation.

Code
import numpy as np
import pandas as pd

v = np.array([1, 2, np.nan, 4])

v.mean()                # nan: missingness propagates
np.nanmean(v)           # drop the nan first, R's na.rm = TRUE
pd.Series(v).mean()     # pandas drops it without being asked

pd.isna(v)              # element-wise test, R's is.na()
np.nan == np.nan        # False: never test for missingness with ==

with np.errstate(divide="ignore", invalid="ignore"):
    print(np.float64(0) / np.float64(0))   # nan
    print(np.float64(1) / np.float64(0))   # inf
np.float64(nan)
np.float64(2.3333333333333335)
np.float64(2.3333333333333335)
array([False, False,  True, False])
False
nan
inf

Part 2: Data Structures

Lists

The base Python workhorse. A list is an ordered, mutable sequence that can mix types freely.

  • [ ] builds a list literal, the closest analogue of c()
  • range() builds an integer sequence, but it is lazy; wrap it in list() to see the values
  • A list does no arithmetic: x * 2 repeats the list, it does not double each element
  • For numeric work, reach for a NumPy array instead
Code
x = [10, 15, 20, 25, 30, 35]
x

list(range(1, 11))       # R's 1:10, the upper bound is excluded
["a", "b"] * 3           # R's rep(c("a", "b"), times = 3)
len(x)

x + [40]                 # concatenation, not addition
x * 2                    # repetition, not element-wise doubling
[10, 15, 20, 25, 30, 35]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
['a', 'b', 'a', 'b', 'a', 'b']
6
[10, 15, 20, 25, 30, 35, 40]
[10, 15, 20, 25, 30, 35, 10, 15, 20, 25, 30, 35]

NumPy arrays

The true analogue of an R vector: one type, one fixed length, arithmetic applied element by element.

  • np.array() converts a list; np.arange() is R’s seq(by = )
  • np.linspace() is R’s seq(length.out = )
  • .dtype reports the single stored type, .shape and .size report the extent
Code
x = np.array([10, 15, 20, 25, 30, 35])
x
x.dtype

np.arange(0, 1.25, 0.25)      # seq(0, 1, by = 0.25), upper bound excluded
np.linspace(0, 10, 5)         # seq(0, 10, length.out = 5)
np.tile(["a", "b"], 3)        # rep(..., times = 3)
np.repeat(["a", "b"], 3)      # rep(..., each = 3)
x.size
array([10, 15, 20, 25, 30, 35])
dtype('int64')
array([0.  , 0.25, 0.5 , 0.75, 1.  ])
array([ 0. ,  2.5,  5. ,  7.5, 10. ])
array(['a', 'b', 'a', 'b', 'a', 'b'], dtype='<U1')
array(['a', 'a', 'a', 'b', 'b', 'b'], dtype='<U1')
6

Indexing

  • Python indexes from 0, not from 1
  • A slice a:b includes position a and excludes position b
  • A negative index counts backward from the end; it does not mean “everything except”, which is the R meaning
  • A boolean array selects the positions that are True, exactly as in R
Code
x = np.array([10, 15, 20, 25, 30, 35])

x[0]                # first element, R's x[1]
x[-1]               # last element, R's x[length(x)]
x[1:4]              # a contiguous slice, R's x[2:4]
x[[0, 4]]           # arbitrary positions, R's x[c(1, 5)]
np.delete(x, 0)     # drop the first element, R's x[-1]
x[x > 20]           # boolean subsetting: the key idiom, identical in both languages
np.int64(10)
np.int64(35)
array([15, 20, 25])
array([10, 30])
array([15, 20, 25, 30, 35])
array([25, 30, 35])

Vectorized operations

Operations apply element by element with no loop required. This is the central habit to build in NumPy.

  • Arrays of different shapes are combined by broadcasting, which stretches length-1 dimensions only
  • Unlike R recycling, a genuine length mismatch raises an error instead of failing quietly
  • np.std() defaults to the population formula; R’s sd() is the sample formula, so pass ddof=1 to match
Code
x = np.array([1, 2, 3, 4])

x * 2               # every element doubled
x + x
x ** 2
np.sqrt(x)

x.sum(), x.mean(), x.max(), x.std(ddof=1)

try:
    x + np.array([1, 2, 3])      # R would recycle and warn; NumPy refuses
except ValueError as err:
    print("ValueError:", err)
array([2, 4, 6, 8])
array([2, 4, 6, 8])
array([ 1,  4,  9, 16])
array([1.        , 1.41421356, 1.73205081, 2.        ])
(np.int64(10), np.float64(2.5), np.int64(4), np.float64(1.2909944487358056))
ValueError: operands could not be broadcast together with shapes (4,) (3,) 

Labelled data and coercion

  • A pandas Series is an array plus an index of labels: R’s named vector
  • A plain dict is the other analogue, though it supports no arithmetic
  • An array holds exactly one type; mixing types silently coerces everything upward
  • The coercion ladder: bool, then int, then float, then string
Code
scores = pd.Series({"math": 90, "art": 75, "bio": 88})
scores
scores["art"]

np.array([1, 2, True])           # bool becomes int
np.array([1, 2, "three"])        # everything becomes a string
math    90
art     75
bio     88
dtype: int64
np.int64(75)
array([1, 2, 1])
array(['1', '2', 'three'], dtype='<U21')

Matrices

A matrix is a two-dimensional NumPy array.

  • reshape() fills row-first by default; pass order="F" to match R, which fills column-first
  • Indexed as [row, column]; a bare : keeps that whole dimension
Code
m = np.arange(1, 21).reshape(5, 4, order="F")    # matrix(1:20, nrow = 5, ncol = 4)
m

m.shape        # R's dim()
m[2, 1]        # single element, R's m[3, 2]
m[2, :]        # third row, R's m[3, ]
m[:, 1]        # second column, R's m[, 2]
m.T            # transpose, R's t()
array([[ 1,  6, 11, 16],
       [ 2,  7, 12, 17],
       [ 3,  8, 13, 18],
       [ 4,  9, 14, 19],
       [ 5, 10, 15, 20]])
(5, 4)
np.int64(8)
array([ 3,  8, 13, 18])
array([ 6,  7,  8,  9, 10])
array([[ 1,  2,  3,  4,  5],
       [ 6,  7,  8,  9, 10],
       [11, 12, 13, 14, 15],
       [16, 17, 18, 19, 20]])

Dictionaries

A dict holds elements of different types and different lengths: the analogue of an R list().

  • Keys are usually strings; values can be anything, including other dicts
  • R’s $ and [[ ]] both become [ ], with no partial matching of names
  • .get() supplies a default instead of raising a KeyError
Code
person = {"name": "Ada", "age": 36, "scores": [90, 95, 99]}

person                                   # the str() analogue for a small object
person.keys()
person["name"]
person["scores"]
person["scores"][1]                      # second score, R's person$scores[2]
person.get("height", "not recorded")     # safe lookup with a fallback
{'name': 'Ada', 'age': 36, 'scores': [90, 95, 99]}
dict_keys(['name', 'age', 'scores'])
'Ada'
[90, 95, 99]
95
'not recorded'

Categoricals

A pandas Categorical represents categorical data with a fixed set of possible values, called categories.

  • Stored as integer codes plus a label lookup table, exactly like an R factor
  • Category order controls the order categories appear in tables and plots
  • The codes start at 0, where R’s factor codes start at 1
Code
sizes = pd.Categorical(
    ["small", "large", "small", "medium"],
    categories=["small", "medium", "large"],
    ordered=True,
)

sizes
sizes.categories                                # R's levels()
pd.Series(sizes).value_counts(sort=False)       # R's table()
sizes.codes                                     # the underlying codes
['small', 'large', 'small', 'medium']
Categories (3, object): ['small' < 'medium' < 'large']
Index(['small', 'medium', 'large'], dtype='object')
small     2
medium    1
large     1
Name: count, dtype: int64
array([0, 2, 0, 1], dtype=int8)

Data frames

A pandas DataFrame is a dict of equal-length columns displayed as a table: rows are observations, columns are variables.

  • Each column has its own dtype
  • Every row also carries an index label, which a base R data frame does not have
  • This is the structure nearly all real analysis uses
Code
df = pd.DataFrame({
    "id":     [10, 20, 30, 40],
    "color":  ["red", "white", "red", None],
    "passed": [True, True, True, False],
})

df
df.info()      # R's str()
df.shape       # R's dim()
list(df.columns)   # R's names()
id color passed
0 10 red True
1 20 white True
2 30 red True
3 40 None False
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   id      4 non-null      int64 
 1   color   3 non-null      object
 2   passed  4 non-null      bool  
dtypes: bool(1), int64(1), object(1)
memory usage: 200.0+ bytes
(4, 3)
['id', 'color', 'passed']

Working with data frames

  • df["id"] pulls a single column out as a Series
  • df.loc[rows, columns] subsets by label, df.iloc[rows, columns] by position
  • head(), describe(), and info() are the first three things to run on any new dataset

Python ships no datasets with the language, so the cell below embeds R’s mtcars directly. For real work, seaborn.load_dataset() and statsmodels.api.datasets.get_rdataset() download the familiar R datasets on demand.

Code
from io import StringIO

MTCARS_CSV = """\
model,mpg,cyl,disp,hp,drat,wt,qsec,vs,am,gear,carb
Mazda RX4,21,6,160,110,3.9,2.62,16.46,0,1,4,4
Mazda RX4 Wag,21,6,160,110,3.9,2.875,17.02,0,1,4,4
Datsun 710,22.8,4,108,93,3.85,2.32,18.61,1,1,4,1
Hornet 4 Drive,21.4,6,258,110,3.08,3.215,19.44,1,0,3,1
Hornet Sportabout,18.7,8,360,175,3.15,3.44,17.02,0,0,3,2
Valiant,18.1,6,225,105,2.76,3.46,20.22,1,0,3,1
Duster 360,14.3,8,360,245,3.21,3.57,15.84,0,0,3,4
Merc 240D,24.4,4,146.7,62,3.69,3.19,20,1,0,4,2
Merc 230,22.8,4,140.8,95,3.92,3.15,22.9,1,0,4,2
Merc 280,19.2,6,167.6,123,3.92,3.44,18.3,1,0,4,4
Merc 280C,17.8,6,167.6,123,3.92,3.44,18.9,1,0,4,4
Merc 450SE,16.4,8,275.8,180,3.07,4.07,17.4,0,0,3,3
Merc 450SL,17.3,8,275.8,180,3.07,3.73,17.6,0,0,3,3
Merc 450SLC,15.2,8,275.8,180,3.07,3.78,18,0,0,3,3
Cadillac Fleetwood,10.4,8,472,205,2.93,5.25,17.98,0,0,3,4
Lincoln Continental,10.4,8,460,215,3,5.424,17.82,0,0,3,4
Chrysler Imperial,14.7,8,440,230,3.23,5.345,17.42,0,0,3,4
Fiat 128,32.4,4,78.7,66,4.08,2.2,19.47,1,1,4,1
Honda Civic,30.4,4,75.7,52,4.93,1.615,18.52,1,1,4,2
Toyota Corolla,33.9,4,71.1,65,4.22,1.835,19.9,1,1,4,1
Toyota Corona,21.5,4,120.1,97,3.7,2.465,20.01,1,0,3,1
Dodge Challenger,15.5,8,318,150,2.76,3.52,16.87,0,0,3,2
AMC Javelin,15.2,8,304,150,3.15,3.435,17.3,0,0,3,2
Camaro Z28,13.3,8,350,245,3.73,3.84,15.41,0,0,3,4
Pontiac Firebird,19.2,8,400,175,3.08,3.845,17.05,0,0,3,2
Fiat X1-9,27.3,4,79,66,4.08,1.935,18.9,1,1,4,1
Porsche 914-2,26,4,120.3,91,4.43,2.14,16.7,0,1,5,2
Lotus Europa,30.4,4,95.1,113,3.77,1.513,16.9,1,1,5,2
Ford Pantera L,15.8,8,351,264,4.22,3.17,14.5,0,1,5,4
Ferrari Dino,19.7,6,145,175,3.62,2.77,15.5,0,1,5,6
Maserati Bora,15,8,301,335,3.54,3.57,14.6,0,1,5,8
Volvo 142E,21.4,4,121,109,4.11,2.78,18.6,1,1,4,2
"""

mtcars = pd.read_csv(StringIO(MTCARS_CSV), index_col="model")
mtcars.head()
mpg cyl disp hp drat wt qsec vs am gear carb
model
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
Code
df["id"]
df[df["passed"]]                        # rows where passed is True
df.loc[df["passed"], ["id", "color"]]   # rows and columns at once
df[["id", "color"]]                     # selected columns

mtcars.head()
mtcars["mpg"].describe()                # R's summary(mtcars$mpg)
0    10
1    20
2    30
3    40
Name: id, dtype: int64
id color passed
0 10 red True
1 20 white True
2 30 red True
id color
0 10 red
1 20 white
2 30 red
id color
0 10 red
1 20 white
2 30 red
3 40 None
mpg cyl disp hp drat wt qsec vs am gear carb
model
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
count    32.000000
mean     20.090625
std       6.026948
min      10.400000
25%      15.425000
50%      19.200000
75%      22.800000
max      33.900000
Name: mpg, dtype: float64

Part 3: Control Flow and Functions

Conditionals

  • if / elif / else takes a single True or False
  • The header line ends in a colon and the body is indented; there are no braces
  • np.where() is the vectorized version, R’s ifelse()
Code
score = 85

if score >= 90:
    print("A")
elif score >= 80:
    print("B")
else:
    print("C")

scores = np.array([95, 85, 60])
np.where(scores >= 80, "pass", "fail")     # vectorized
B
array(['pass', 'pass', 'fail'], dtype='<U4')

For loops

  • for item in sequence: iterates over any list, array, string, or dict
  • enumerate() yields position and value together, the safer analogue of seq_along()
  • A list comprehension replaces most short loops and is the idiomatic form
  • Appending to a list is cheap, so preallocation is not the concern it is in R
Code
for i in range(1, 6):
    print(f"i = {i}  i^2 = {i ** 2}")

# grow a list
squares = []
for i in range(1, 6):
    squares.append(i ** 2)
squares

[i ** 2 for i in range(1, 6)]          # list comprehension, same result

for pos, val in enumerate(["a", "b", "c"]):
    print(pos, val)
i = 1  i^2 = 1
i = 2  i^2 = 4
i = 3  i^2 = 9
i = 4  i^2 = 16
i = 5  i^2 = 25
[1, 4, 9, 16, 25]
[1, 4, 9, 16, 25]
0 a
1 b
2 c

While loops

  • Repeats while a condition stays True
  • Use when the number of iterations is not known in advance
  • i += 1 is shorthand for i = i + 1; make sure something inside can flip the condition
Code
i = 1
while i <= 5:
    print(i)
    i += 1
1
2
3
4
5

Writing functions

  • def name(arguments): followed by an indented body
  • return is required. Unlike R, the last expression is not returned automatically, and a function without return hands back None
  • Arguments can carry default values
  • The triple-quoted docstring on the first line is what help() displays
Code
def summarize_vec(x, digits=2):
    """Return the mean, standard deviation, and length of x."""
    x = np.asarray(x)
    return pd.Series({
        "mean": x.mean(),
        "sd": x.std(ddof=1),
        "n": len(x),
    }).round(digits)


v = [4, 8, 15, 16, 23, 42]
summarize_vec(v)
summarize_vec(v, digits=4)
mean    18.00
sd      13.49
n        6.00
dtype: float64
mean    18.0000
sd      13.4907
n        6.0000
dtype: float64

The apply family

Idiomatic Python replaces most loops with a comprehension or with a vectorized method call.

  • List comprehension, or map(): R’s sapply() and lapply()
  • An array method with axis=: R’s apply()
  • Watch the axis: axis=0 collapses down the rows and returns one value per column, which is R’s apply(m, 2, ...), so the numbers are swapped relative to R
  • df.apply() runs a function column by column across a data frame
Code
[i ** 2 for i in range(1, 6)]                  # sapply(1:5, function(i) i^2)
list(map(lambda i: i ** 2, range(1, 6)))       # the map() form

m.sum(axis=0)      # column sums, R's apply(m, 2, sum)
m.max(axis=1)      # row maxima, R's apply(m, 1, max)

mtcars[["mpg", "hp", "wt"]].mean()
mtcars[["mpg", "hp", "wt"]].apply(lambda col: col.max() - col.min())
[1, 4, 9, 16, 25]
[1, 4, 9, 16, 25]
array([15, 40, 65, 90])
array([16, 17, 18, 19, 20])
mpg     20.090625
hp     146.687500
wt       3.217250
dtype: float64
mpg     23.500
hp     283.000
wt       3.911
dtype: float64

Getting unstuck

  • help(np.mean), or np.mean? in Jupyter, opens the documentation
  • type(x) when a result surprises you: the problem is almost always the type
  • dir(x) lists what the object can do
  • %whos is the IPython magic closest to R’s ls()
  • Read the traceback from the bottom up: the last line is the actual error, the lines above are the call path
Code
a = 1
b = "two"

%whos             # everything defined right now

# help(np.mean)   # uncomment to open the docs
mtcars.info()     # R's str() on a data frame
No variables match your requested type.
<class 'pandas.core.frame.DataFrame'>
Index: 32 entries, Mazda RX4 to Volvo 142E
Data columns (total 11 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   mpg     32 non-null     float64
 1   cyl     32 non-null     int64  
 2   disp    32 non-null     float64
 3   hp      32 non-null     int64  
 4   drat    32 non-null     float64
 5   wt      32 non-null     float64
 6   qsec    32 non-null     float64
 7   vs      32 non-null     int64  
 8   am      32 non-null     int64  
 9   gear    32 non-null     int64  
 10  carb    32 non-null     int64  
dtypes: float64(5), int64(6)
memory usage: 3.0+ KB

Part 4: NumPy and pandas

What the stack is

Python has no single tidyverse. A handful of independent packages divide the same work, each imported on its own.

  • pandas: data manipulation and reshaping, covering dplyr and tidyr
  • numpy: arrays and numerical computation, the layer pandas is built on
  • matplotlib, seaborn, plotnine: visualization, where plotnine is a direct port of ggplot2
  • pandas.read_csv and pyarrow: file import, covering readr
  • scipy, statsmodels, scikit-learn: statistics, models, and machine learning

Install once with conda or pip, import every session.

Code
import numpy as np
import pandas as pd

np.__version__
pd.__version__
'2.4.2'
'2.3.3'

Tidy data

The same rule governs everything downstream.

  • Each variable is a column
  • Each observation is a row
  • Each cell is a single value

pandas calls this long form. Data in this shape works directly with groupby, melt, and every plotting library.

Code
mtcars.head()
mtcars.info()      # the pandas counterpart to glimpse()
mpg cyl disp hp drat wt qsec vs am gear carb
model
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
<class 'pandas.core.frame.DataFrame'>
Index: 32 entries, Mazda RX4 to Volvo 142E
Data columns (total 11 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   mpg     32 non-null     float64
 1   cyl     32 non-null     int64  
 2   disp    32 non-null     float64
 3   hp      32 non-null     int64  
 4   drat    32 non-null     float64
 5   wt      32 non-null     float64
 6   qsec    32 non-null     float64
 7   vs      32 non-null     int64  
 8   am      32 non-null     int64  
 9   gear    32 non-null     int64  
 10  carb    32 non-null     int64  
dtypes: float64(5), int64(6)
memory usage: 3.0+ KB

Method chaining

Python has no pipe operator. Methods return a new object, so calls chain left to right instead.

  • Wrap the whole chain in parentheses to spread it over several lines
  • .pipe(f) inserts a function that is not a method, which is the nearest thing to |>
  • Each step returns a copy, so the original frame is untouched
Code
# without chaining, read from the inside out
mtcars[mtcars["cyl"] == 4].head(3)

# with chaining: read it as "take mtcars, then query, then take the head"
(
    mtcars
    .query("cyl == 4")
    .head(3)
)
mpg cyl disp hp drat wt qsec vs am gear carb
model
Datsun 710 22.8 4 108.0 93 3.85 2.32 18.61 1 1 4 1
Merc 240D 24.4 4 146.7 62 3.69 3.19 20.00 1 0 4 2
Merc 230 22.8 4 140.8 95 3.92 3.15 22.90 1 0 4 2
mpg cyl disp hp drat wt qsec vs am gear carb
model
Datsun 710 22.8 4 108.0 93 3.85 2.32 18.61 1 1 4 1
Merc 240D 24.4 4 146.7 62 3.69 3.19 20.00 1 0 4 2
Merc 230 22.8 4 140.8 95 3.92 3.15 22.90 1 0 4 2

The core pandas verbs

Five operations cover most day-to-day data manipulation.

  • query(), or a boolean mask: filter()
  • [[ ]], or .filter(items = ): select()
  • assign(): mutate()
  • sort_values(ascending = False): arrange() with desc()
  • agg(): summarize()

One false friend to watch: the pandas method named filter() selects columns, where the dplyr verb of the same name selects rows.

Code
(
    mtcars
    .query("cyl == 4")
    .filter(["mpg", "hp", "wt"])
    .assign(km_per_liter=lambda d: d["mpg"] * 0.425)
    .sort_values("mpg", ascending=False)
    .head(5)
)
mpg hp wt km_per_liter
model
Toyota Corolla 33.9 65 1.835 14.4075
Fiat 128 32.4 66 2.200 13.7700
Honda Civic 30.4 52 1.615 12.9200
Lotus Europa 30.4 113 1.513 12.9200
Fiat X1-9 27.3 66 1.935 11.6025

Group and summarize

groupby() splits the data, agg() collapses each group, and the result is one row per group.

  • This single pattern replaces most of what pivot tables do
  • "size" counts rows within the current group, R’s n()
  • The group labels become the index, so reset_index() turns them back into an ordinary column
Code
(
    mtcars
    .groupby("cyl")
    .agg(
        n=("mpg", "size"),
        mean_mpg=("mpg", "mean"),
        mean_hp=("hp", "mean"),
    )
    .sort_values("mean_mpg", ascending=False)
    .reset_index()
)
cyl n mean_mpg mean_hp
0 4 11 26.663636 82.636364
1 6 7 19.742857 122.285714
2 8 14 15.100000 209.214286

Reshaping

  • melt() turns wide data into long data, which is what plotting functions expect: R’s pivot_longer()
  • pivot() and pivot_table() reverse it: R’s pivot_wider()
Code
long = (
    mtcars
    .reset_index()
    .filter(["model", "mpg", "hp"])
    .melt(id_vars="model", var_name="measure", value_name="value")
)

long.head()

long.pivot(index="model", columns="measure", values="value").head()
model measure value
0 Mazda RX4 mpg 21.0
1 Mazda RX4 Wag mpg 21.0
2 Datsun 710 mpg 22.8
3 Hornet 4 Drive mpg 21.4
4 Hornet Sportabout mpg 18.7
measure hp mpg
model
AMC Javelin 150.0 15.2
Cadillac Fleetwood 205.0 10.4
Camaro Z28 245.0 13.3
Chrysler Imperial 230.0 14.7
Datsun 710 93.0 22.8

Quick reference

Task R Python
Assign x <- 5 x = 5
Exponent 2^10 2 ** 10
Integer division, remainder %/%, %% //, %
Storage type typeof(x), class(x) type(x)
Structure summary str(x) df.info(), type(x), dir(x)
Combine values c(1, 2, 3) [1, 2, 3], np.array([1, 2, 3])
Integer sequence 1:10 range(1, 11), np.arange(1, 11)
Sequence by step, by length seq(by =), seq(length.out =) np.arange(), np.linspace()
Repeat rep(x, times =), rep(x, each =) np.tile(), np.repeat()
First element x[1] x[0]
Last element x[length(x)] x[-1]
Drop first element x[-1] np.delete(x, 0), x[1:]
Boolean subset x[x > 20] x[x > 20]
Missing value NA, is.na() np.nan, pd.isna()
Empty object NULL None
Ignore missing mean(x, na.rm = TRUE) np.nanmean(x), Series.mean()
Sample standard deviation sd(x) x.std(ddof=1)
Matrix matrix(1:20, nrow = 5) np.arange(1, 21).reshape(5, 4, order="F")
Dimensions, transpose dim(m), t(m) m.shape, m.T
Heterogeneous container list() dict, list
Categorical factor(), levels() pd.Categorical(), .categories
Frequency table table(x) Series.value_counts()
Data frame data.frame() pd.DataFrame()
Column df$id df["id"]
Row subset df[df$passed, ] df[df["passed"]], df.query()
Vectorized if ifelse() np.where()
Apply over elements sapply(), lapply() list comprehension, map()
Apply over columns, rows apply(m, 2, f), apply(m, 1, f) axis=0, axis=1
Function f <- function(x) { } def f(x):
Pipe x \|> f() \|> g() x.f().g(), .pipe(f)
dplyr verbs filter, select, mutate, arrange query, [[ ]], assign, sort_values
Group and summarize group_by() \|> summarize() groupby().agg()
Reshape pivot_longer(), pivot_wider() melt(), pivot()
List the environment ls() %whos
Help ?mean help(np.mean), np.mean?