{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Python Fundamentals\n",
        "\n",
        "# Review Fundamentals\n",
        "\n",
        "A working refresher on the Python language, followed by a first look at\n",
        "the data stack\n",
        "\n",
        "-   Part 2: data structures\n",
        "-   Part 3: control flow and functions\n",
        "-   Part 4: NumPy and pandas\n",
        "\n",
        "This notebook mirrors the R fundamentals notebook section by section, so\n",
        "each idea appears next to the R idiom it replaces.\n",
        "\n",
        "Run every cell as we go. The fastest way to learn Python is to break\n",
        "things and read the traceback.\n",
        "\n",
        "**Aside**: Data driven narrative\n",
        "\n",
        "https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-APOCALYPSE/egpbykdxjvq/\n",
        "\n",
        "## Formatting\n",
        "\n",
        "R prints every top-level expression in a cell. Python prints only the\n",
        "last one. The following cell restores the R behavior so the two\n",
        "notebooks produce comparable output; don’t worry about the details."
      ],
      "id": "519e3755-60a3-4e97-8ee2-73d98e7ed597"
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Show the value of every top-level expression, not just the last one\n",
        "from IPython.core.interactiveshell import InteractiveShell\n",
        "\n",
        "InteractiveShell.ast_node_interactivity = \"all\""
      ],
      "id": "cell-3"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Part 1: The Basics\n",
        "\n",
        "## Why Python\n",
        "\n",
        "-   General purpose language: data work lives in the same language as\n",
        "    scripting, web services, and production code\n",
        "-   Single values are the native unit of computation; vectorization is\n",
        "    provided by NumPy, not by the language\n",
        "-   Huge package ecosystem on PyPI and conda-forge, with strong tooling\n",
        "    for notebooks and reproducible environments\n",
        "-   Two dialects coexist: base Python (this notebook, Part 1 to 3) and\n",
        "    the array/table stack (Part 4)\n",
        "\n",
        "## Assignment and printing\n",
        "\n",
        "-   `=` is the only assignment operator; there is no `<-`\n",
        "-   `#` starts a comment\n",
        "-   A bare object name prints itself as the last line of a cell;\n",
        "    `print()` does it explicitly\n",
        "-   An f-string is the closest analogue of `cat()`"
      ],
      "id": "cc29db03-d5f5-40d6-9221-b29858d3a480"
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "8"
            ]
          }
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "8\n",
            "z is 8\n",
            "z is 8"
          ]
        }
      ],
      "source": [
        "x = 5\n",
        "y = 3\n",
        "z = x + y\n",
        "\n",
        "z                        # auto-print\n",
        "print(z)                 # explicit print\n",
        "print(\"z is\", z)         # print: space separated, newline added for you\n",
        "print(f\"z is {z}\")       # f-string: any expression inside the braces"
      ],
      "id": "cell-7"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Arithmetic operators\n",
        "\n",
        "-   `+ - * /` behave as expected, but exponentiation is `**`, not `^`\n",
        "-   `^` is legal Python and means bitwise exclusive or, which makes it a\n",
        "    silent R-to-Python bug\n",
        "-   `%` is modulo (remainder), `//` is floor division\n",
        "-   `/` always returns a float, even between two integers"
      ],
      "id": "1c7c6072-8a8f-40f8-a8d9-77fa04577b32"
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "9"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "3.5"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "1"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "3"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "1024"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "4.0"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "3.14"
            ]
          }
        }
      ],
      "source": [
        "import math\n",
        "\n",
        "7 + 2\n",
        "7 / 2        # 3.5, always a float\n",
        "7 % 2        # remainder, R's %%\n",
        "7 // 2       # floor division, R's %/%\n",
        "2 ** 10      # R's 2^10\n",
        "math.sqrt(16)\n",
        "round(3.14159, 2)"
      ],
      "id": "cell-9"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Comparison and logical operators\n",
        "\n",
        "-   Comparison: `== != < > <= >=`\n",
        "-   Logical on single values: `and`, `or`, `not`\n",
        "-   Logical element by element on arrays and Series: `&`, `|`, `~`, with\n",
        "    each side in parentheses\n",
        "-   Comparisons return `True` or `False`, which are themselves a data\n",
        "    type\n",
        "-   Comparisons can be chained, which R cannot do"
      ],
      "id": "133cc4e2-16af-4e32-9d28-35376899dc1f"
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "True"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "True"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "False"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "True"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "True"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "False"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "True"
            ]
          }
        }
      ],
      "source": [
        "5 > 3\n",
        "5 == 5\n",
        "5 != 5\n",
        "\n",
        "(5 > 3) and (2 > 1)      # both must be True\n",
        "(5 > 3) or (2 > 10)      # at least one True\n",
        "not (5 > 3)              # negation\n",
        "\n",
        "3 < 5 < 10               # chained comparison"
      ],
      "id": "cell-11"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## The fundamental data types\n",
        "\n",
        "Python has a small set of built-in types. Everything else is built from\n",
        "them.\n",
        "\n",
        "-   `float`: real numbers, the default for anything with a decimal\n",
        "    point, R’s `double`\n",
        "-   `int`: whole numbers, unlimited precision, no `L` suffix needed\n",
        "-   `str`: text, in single or double quotes, R’s `character`\n",
        "-   `bool`: `True` and `False`, capitalized, R’s `logical`\n",
        "-   `complex` and `bytes`: rarely needed in an intro course"
      ],
      "id": "f2f36538-e314-434a-876b-033fe52265e3"
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "float"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "int"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "str"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "bool"
            ]
          }
        }
      ],
      "source": [
        "a = 3.14       # float\n",
        "b = 3          # int\n",
        "d = \"hello\"    # str\n",
        "e = True       # bool\n",
        "\n",
        "type(a)\n",
        "type(b)\n",
        "type(d)\n",
        "type(e)"
      ],
      "id": "cell-13"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Inspecting an object\n",
        "\n",
        "Three questions worth asking about any object you did not create\n",
        "yourself.\n",
        "\n",
        "-   `type()`: what is this, covering both R’s `typeof()` and `class()`\n",
        "-   `isinstance()`: the analogue of `is.numeric()` and friends\n",
        "-   `dir()`: every attribute and method the object carries, the usual\n",
        "    way to discover what you can do next\n",
        "\n",
        "A number in Python is a scalar. Unlike R, it is not a length-1 vector,\n",
        "and `len()` on it is an error."
      ],
      "id": "f36866d1-7306-4c34-92d0-0328714ca237"
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "float"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "True"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "False"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "5"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "['__trunc__',\n",
              " 'as_integer_ratio',\n",
              " 'conjugate',\n",
              " 'fromhex',\n",
              " 'hex',\n",
              " 'imag',\n",
              " 'is_integer',\n",
              " 'real']"
            ]
          }
        }
      ],
      "source": [
        "x = 3.14\n",
        "\n",
        "type(x)\n",
        "isinstance(x, float)\n",
        "isinstance(x, str)\n",
        "\n",
        "len(\"hello\")             # len() works on containers, not on numbers\n",
        "dir(x)[-8:]              # the last few methods available on a float"
      ],
      "id": "cell-15"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Type conversion\n",
        "\n",
        "-   `int()`, `float()`, `str()`, `bool()` convert explicitly\n",
        "-   Failed conversions raise a `ValueError` and stop execution; R\n",
        "    returns `NA` with a warning\n",
        "-   `int()` truncates toward zero; it does not round\n",
        "-   For the R-style behavior on a whole column, use\n",
        "    `pd.to_numeric(..., errors=\"coerce\")`, which yields missing values\n",
        "    instead of raising"
      ],
      "id": "bc8acc85-27be-4314-bee7-da647ca75fbc"
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "3"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "'3.14'"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "42.0"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "False"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "1.0"
            ]
          }
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ValueError: could not convert string to float: 'banana'"
          ]
        }
      ],
      "source": [
        "int(3.99)         # 3, truncated\n",
        "str(3.14)         # '3.14'\n",
        "float(\"42\")       # 42.0\n",
        "bool(0)           # False\n",
        "float(True)       # 1.0\n",
        "\n",
        "try:\n",
        "    float(\"banana\")\n",
        "except ValueError as err:\n",
        "    print(\"ValueError:\", err)"
      ],
      "id": "cell-17"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Special values\n",
        "\n",
        "-   `None`: the empty object, R’s `NULL`\n",
        "-   `float(\"nan\")` or `np.nan`: missing or undefined, covering both R’s\n",
        "    `NA` and `NaN`\n",
        "-   `np.inf` and `-np.inf`: overflow or division by zero\n",
        "-   Test with `pd.isna()` or `math.isnan()`, never with `== np.nan`,\n",
        "    because nan is not equal to itself\n",
        "\n",
        "NumPy propagates missingness the way R does. pandas goes the other way\n",
        "and skips missing values by default, which is the single most common\n",
        "source of quiet disagreement between an R script and its Python\n",
        "translation."
      ],
      "id": "4b7306a4-bf50-45fc-9445-4e394d92ed78"
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "np.float64(nan)"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "np.float64(2.3333333333333335)"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "np.float64(2.3333333333333335)"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([False, False,  True, False])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "False"
            ]
          }
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "nan\n",
            "inf"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "v = np.array([1, 2, np.nan, 4])\n",
        "\n",
        "v.mean()                # nan: missingness propagates\n",
        "np.nanmean(v)           # drop the nan first, R's na.rm = TRUE\n",
        "pd.Series(v).mean()     # pandas drops it without being asked\n",
        "\n",
        "pd.isna(v)              # element-wise test, R's is.na()\n",
        "np.nan == np.nan        # False: never test for missingness with ==\n",
        "\n",
        "with np.errstate(divide=\"ignore\", invalid=\"ignore\"):\n",
        "    print(np.float64(0) / np.float64(0))   # nan\n",
        "    print(np.float64(1) / np.float64(0))   # inf"
      ],
      "id": "cell-19"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Part 2: Data Structures\n",
        "\n",
        "## Lists\n",
        "\n",
        "The base Python workhorse. A list is an ordered, mutable sequence that\n",
        "can mix types freely.\n",
        "\n",
        "-   `[ ]` builds a list literal, the closest analogue of `c()`\n",
        "-   `range()` builds an integer sequence, but it is lazy; wrap it in\n",
        "    `list()` to see the values\n",
        "-   A list does no arithmetic: `x * 2` repeats the list, it does not\n",
        "    double each element\n",
        "-   For numeric work, reach for a NumPy array instead"
      ],
      "id": "0b052e29-11eb-44ae-9358-5b8d2eccb079"
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "[10, 15, 20, 25, 30, 35]"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "['a', 'b', 'a', 'b', 'a', 'b']"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "6"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "[10, 15, 20, 25, 30, 35, 40]"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "[10, 15, 20, 25, 30, 35, 10, 15, 20, 25, 30, 35]"
            ]
          }
        }
      ],
      "source": [
        "x = [10, 15, 20, 25, 30, 35]\n",
        "x\n",
        "\n",
        "list(range(1, 11))       # R's 1:10, the upper bound is excluded\n",
        "[\"a\", \"b\"] * 3           # R's rep(c(\"a\", \"b\"), times = 3)\n",
        "len(x)\n",
        "\n",
        "x + [40]                 # concatenation, not addition\n",
        "x * 2                    # repetition, not element-wise doubling"
      ],
      "id": "cell-22"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## NumPy arrays\n",
        "\n",
        "The true analogue of an R vector: one type, one fixed length, arithmetic\n",
        "applied element by element.\n",
        "\n",
        "-   `np.array()` converts a list; `np.arange()` is R’s `seq(by = )`\n",
        "-   `np.linspace()` is R’s `seq(length.out = )`\n",
        "-   `.dtype` reports the single stored type, `.shape` and `.size` report\n",
        "    the extent"
      ],
      "id": "0f59d9ea-2475-4a22-8bff-94a0d9d12a35"
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([10, 15, 20, 25, 30, 35])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "dtype('int64')"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([0.  , 0.25, 0.5 , 0.75, 1.  ])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([ 0. ,  2.5,  5. ,  7.5, 10. ])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array(['a', 'b', 'a', 'b', 'a', 'b'], dtype='<U1')"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array(['a', 'a', 'a', 'b', 'b', 'b'], dtype='<U1')"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "6"
            ]
          }
        }
      ],
      "source": [
        "x = np.array([10, 15, 20, 25, 30, 35])\n",
        "x\n",
        "x.dtype\n",
        "\n",
        "np.arange(0, 1.25, 0.25)      # seq(0, 1, by = 0.25), upper bound excluded\n",
        "np.linspace(0, 10, 5)         # seq(0, 10, length.out = 5)\n",
        "np.tile([\"a\", \"b\"], 3)        # rep(..., times = 3)\n",
        "np.repeat([\"a\", \"b\"], 3)      # rep(..., each = 3)\n",
        "x.size"
      ],
      "id": "cell-24"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Indexing\n",
        "\n",
        "-   Python indexes from 0, not from 1\n",
        "-   A slice `a:b` includes position `a` and excludes position `b`\n",
        "-   A negative index counts backward from the end; it does not mean\n",
        "    “everything except”, which is the R meaning\n",
        "-   A boolean array selects the positions that are `True`, exactly as in\n",
        "    R"
      ],
      "id": "d5945fea-16f9-47a9-ae8d-80a6c2e9d237"
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "np.int64(10)"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "np.int64(35)"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([15, 20, 25])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([10, 30])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([15, 20, 25, 30, 35])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([25, 30, 35])"
            ]
          }
        }
      ],
      "source": [
        "x = np.array([10, 15, 20, 25, 30, 35])\n",
        "\n",
        "x[0]                # first element, R's x[1]\n",
        "x[-1]               # last element, R's x[length(x)]\n",
        "x[1:4]              # a contiguous slice, R's x[2:4]\n",
        "x[[0, 4]]           # arbitrary positions, R's x[c(1, 5)]\n",
        "np.delete(x, 0)     # drop the first element, R's x[-1]\n",
        "x[x > 20]           # boolean subsetting: the key idiom, identical in both languages"
      ],
      "id": "cell-26"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Vectorized operations\n",
        "\n",
        "Operations apply element by element with no loop required. This is the\n",
        "central habit to build in NumPy.\n",
        "\n",
        "-   Arrays of different shapes are combined by broadcasting, which\n",
        "    stretches length-1 dimensions only\n",
        "-   Unlike R recycling, a genuine length mismatch raises an error\n",
        "    instead of failing quietly\n",
        "-   `np.std()` defaults to the population formula; R’s `sd()` is the\n",
        "    sample formula, so pass `ddof=1` to match"
      ],
      "id": "4fbac50c-880b-4b83-8177-5fd6867f3793"
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([2, 4, 6, 8])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([2, 4, 6, 8])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([ 1,  4,  9, 16])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([1.        , 1.41421356, 1.73205081, 2.        ])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "(np.int64(10), np.float64(2.5), np.int64(4), np.float64(1.2909944487358056))"
            ]
          }
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ValueError: operands could not be broadcast together with shapes (4,) (3,) "
          ]
        }
      ],
      "source": [
        "x = np.array([1, 2, 3, 4])\n",
        "\n",
        "x * 2               # every element doubled\n",
        "x + x\n",
        "x ** 2\n",
        "np.sqrt(x)\n",
        "\n",
        "x.sum(), x.mean(), x.max(), x.std(ddof=1)\n",
        "\n",
        "try:\n",
        "    x + np.array([1, 2, 3])      # R would recycle and warn; NumPy refuses\n",
        "except ValueError as err:\n",
        "    print(\"ValueError:\", err)"
      ],
      "id": "cell-28"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Labelled data and coercion\n",
        "\n",
        "-   A pandas `Series` is an array plus an index of labels: R’s named\n",
        "    vector\n",
        "-   A plain `dict` is the other analogue, though it supports no\n",
        "    arithmetic\n",
        "-   An array holds exactly one type; mixing types silently coerces\n",
        "    everything upward\n",
        "-   The coercion ladder: bool, then int, then float, then string"
      ],
      "id": "589640a7-6054-4b1e-98da-8424962e9e00"
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "math    90\n",
              "art     75\n",
              "bio     88\n",
              "dtype: int64"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "np.int64(75)"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([1, 2, 1])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array(['1', '2', 'three'], dtype='<U21')"
            ]
          }
        }
      ],
      "source": [
        "scores = pd.Series({\"math\": 90, \"art\": 75, \"bio\": 88})\n",
        "scores\n",
        "scores[\"art\"]\n",
        "\n",
        "np.array([1, 2, True])           # bool becomes int\n",
        "np.array([1, 2, \"three\"])        # everything becomes a string"
      ],
      "id": "cell-30"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Matrices\n",
        "\n",
        "A matrix is a two-dimensional NumPy array.\n",
        "\n",
        "-   `reshape()` fills row-first by default; pass `order=\"F\"` to match R,\n",
        "    which fills column-first\n",
        "-   Indexed as `[row, column]`; a bare `:` keeps that whole dimension"
      ],
      "id": "2244ae90-579e-403c-8c63-3cb3b210e772"
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([[ 1,  6, 11, 16],\n",
              "       [ 2,  7, 12, 17],\n",
              "       [ 3,  8, 13, 18],\n",
              "       [ 4,  9, 14, 19],\n",
              "       [ 5, 10, 15, 20]])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "(5, 4)"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "np.int64(8)"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([ 3,  8, 13, 18])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([ 6,  7,  8,  9, 10])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([[ 1,  2,  3,  4,  5],\n",
              "       [ 6,  7,  8,  9, 10],\n",
              "       [11, 12, 13, 14, 15],\n",
              "       [16, 17, 18, 19, 20]])"
            ]
          }
        }
      ],
      "source": [
        "m = np.arange(1, 21).reshape(5, 4, order=\"F\")    # matrix(1:20, nrow = 5, ncol = 4)\n",
        "m\n",
        "\n",
        "m.shape        # R's dim()\n",
        "m[2, 1]        # single element, R's m[3, 2]\n",
        "m[2, :]        # third row, R's m[3, ]\n",
        "m[:, 1]        # second column, R's m[, 2]\n",
        "m.T            # transpose, R's t()"
      ],
      "id": "cell-32"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Dictionaries\n",
        "\n",
        "A dict holds elements of different types and different lengths: the\n",
        "analogue of an R `list()`.\n",
        "\n",
        "-   Keys are usually strings; values can be anything, including other\n",
        "    dicts\n",
        "-   R’s `$` and `[[ ]]` both become `[ ]`, with no partial matching of\n",
        "    names\n",
        "-   `.get()` supplies a default instead of raising a `KeyError`"
      ],
      "id": "e530bc39-9bcd-471b-9399-e862be68325e"
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "{'name': 'Ada', 'age': 36, 'scores': [90, 95, 99]}"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "dict_keys(['name', 'age', 'scores'])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "'Ada'"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "[90, 95, 99]"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "95"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "'not recorded'"
            ]
          }
        }
      ],
      "source": [
        "person = {\"name\": \"Ada\", \"age\": 36, \"scores\": [90, 95, 99]}\n",
        "\n",
        "person                                   # the str() analogue for a small object\n",
        "person.keys()\n",
        "person[\"name\"]\n",
        "person[\"scores\"]\n",
        "person[\"scores\"][1]                      # second score, R's person$scores[2]\n",
        "person.get(\"height\", \"not recorded\")     # safe lookup with a fallback"
      ],
      "id": "cell-34"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Categoricals\n",
        "\n",
        "A pandas `Categorical` represents categorical data with a fixed set of\n",
        "possible values, called categories.\n",
        "\n",
        "-   Stored as integer codes plus a label lookup table, exactly like an R\n",
        "    factor\n",
        "-   Category order controls the order categories appear in tables and\n",
        "    plots\n",
        "-   The codes start at 0, where R’s factor codes start at 1"
      ],
      "id": "d6bbdee0-fc4e-4e0d-8c4b-a89cfef2185e"
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "['small', 'large', 'small', 'medium']\n",
              "Categories (3, object): ['small' < 'medium' < 'large']"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "Index(['small', 'medium', 'large'], dtype='object')"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "small     2\n",
              "medium    1\n",
              "large     1\n",
              "Name: count, dtype: int64"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([0, 2, 0, 1], dtype=int8)"
            ]
          }
        }
      ],
      "source": [
        "sizes = pd.Categorical(\n",
        "    [\"small\", \"large\", \"small\", \"medium\"],\n",
        "    categories=[\"small\", \"medium\", \"large\"],\n",
        "    ordered=True,\n",
        ")\n",
        "\n",
        "sizes\n",
        "sizes.categories                                # R's levels()\n",
        "pd.Series(sizes).value_counts(sort=False)       # R's table()\n",
        "sizes.codes                                     # the underlying codes"
      ],
      "id": "cell-36"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Data frames\n",
        "\n",
        "A pandas `DataFrame` is a dict of equal-length columns displayed as a\n",
        "table: rows are observations, columns are variables.\n",
        "\n",
        "-   Each column has its own dtype\n",
        "-   Every row also carries an index label, which a base R data frame\n",
        "    does not have\n",
        "-   This is the structure nearly all real analysis uses"
      ],
      "id": "cf2a051c-3af4-4920-af92-e6067784c6ae"
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 4 entries, 0 to 3\n",
            "Data columns (total 3 columns):\n",
            " #   Column  Non-Null Count  Dtype \n",
            "---  ------  --------------  ----- \n",
            " 0   id      4 non-null      int64 \n",
            " 1   color   3 non-null      object\n",
            " 2   passed  4 non-null      bool  \n",
            "dtypes: bool(1), int64(1), object(1)\n",
            "memory usage: 200.0+ bytes"
          ]
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "(4, 3)"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "['id', 'color', 'passed']"
            ]
          }
        }
      ],
      "source": [
        "df = pd.DataFrame({\n",
        "    \"id\":     [10, 20, 30, 40],\n",
        "    \"color\":  [\"red\", \"white\", \"red\", None],\n",
        "    \"passed\": [True, True, True, False],\n",
        "})\n",
        "\n",
        "df\n",
        "df.info()      # R's str()\n",
        "df.shape       # R's dim()\n",
        "list(df.columns)   # R's names()"
      ],
      "id": "cell-38"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Working with data frames\n",
        "\n",
        "-   `df[\"id\"]` pulls a single column out as a Series\n",
        "-   `df.loc[rows, columns]` subsets by label, `df.iloc[rows, columns]`\n",
        "    by position\n",
        "-   `head()`, `describe()`, and `info()` are the first three things to\n",
        "    run on any new dataset\n",
        "\n",
        "Python ships no datasets with the language, so the cell below embeds R’s\n",
        "`mtcars` directly. For real work, `seaborn.load_dataset()` and\n",
        "`statsmodels.api.datasets.get_rdataset()` download the familiar R\n",
        "datasets on demand."
      ],
      "id": "b6df09c6-dad4-4f2f-81cd-274a18b2be2a"
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        }
      ],
      "source": [
        "from io import StringIO\n",
        "\n",
        "MTCARS_CSV = \"\"\"\\\n",
        "model,mpg,cyl,disp,hp,drat,wt,qsec,vs,am,gear,carb\n",
        "Mazda RX4,21,6,160,110,3.9,2.62,16.46,0,1,4,4\n",
        "Mazda RX4 Wag,21,6,160,110,3.9,2.875,17.02,0,1,4,4\n",
        "Datsun 710,22.8,4,108,93,3.85,2.32,18.61,1,1,4,1\n",
        "Hornet 4 Drive,21.4,6,258,110,3.08,3.215,19.44,1,0,3,1\n",
        "Hornet Sportabout,18.7,8,360,175,3.15,3.44,17.02,0,0,3,2\n",
        "Valiant,18.1,6,225,105,2.76,3.46,20.22,1,0,3,1\n",
        "Duster 360,14.3,8,360,245,3.21,3.57,15.84,0,0,3,4\n",
        "Merc 240D,24.4,4,146.7,62,3.69,3.19,20,1,0,4,2\n",
        "Merc 230,22.8,4,140.8,95,3.92,3.15,22.9,1,0,4,2\n",
        "Merc 280,19.2,6,167.6,123,3.92,3.44,18.3,1,0,4,4\n",
        "Merc 280C,17.8,6,167.6,123,3.92,3.44,18.9,1,0,4,4\n",
        "Merc 450SE,16.4,8,275.8,180,3.07,4.07,17.4,0,0,3,3\n",
        "Merc 450SL,17.3,8,275.8,180,3.07,3.73,17.6,0,0,3,3\n",
        "Merc 450SLC,15.2,8,275.8,180,3.07,3.78,18,0,0,3,3\n",
        "Cadillac Fleetwood,10.4,8,472,205,2.93,5.25,17.98,0,0,3,4\n",
        "Lincoln Continental,10.4,8,460,215,3,5.424,17.82,0,0,3,4\n",
        "Chrysler Imperial,14.7,8,440,230,3.23,5.345,17.42,0,0,3,4\n",
        "Fiat 128,32.4,4,78.7,66,4.08,2.2,19.47,1,1,4,1\n",
        "Honda Civic,30.4,4,75.7,52,4.93,1.615,18.52,1,1,4,2\n",
        "Toyota Corolla,33.9,4,71.1,65,4.22,1.835,19.9,1,1,4,1\n",
        "Toyota Corona,21.5,4,120.1,97,3.7,2.465,20.01,1,0,3,1\n",
        "Dodge Challenger,15.5,8,318,150,2.76,3.52,16.87,0,0,3,2\n",
        "AMC Javelin,15.2,8,304,150,3.15,3.435,17.3,0,0,3,2\n",
        "Camaro Z28,13.3,8,350,245,3.73,3.84,15.41,0,0,3,4\n",
        "Pontiac Firebird,19.2,8,400,175,3.08,3.845,17.05,0,0,3,2\n",
        "Fiat X1-9,27.3,4,79,66,4.08,1.935,18.9,1,1,4,1\n",
        "Porsche 914-2,26,4,120.3,91,4.43,2.14,16.7,0,1,5,2\n",
        "Lotus Europa,30.4,4,95.1,113,3.77,1.513,16.9,1,1,5,2\n",
        "Ford Pantera L,15.8,8,351,264,4.22,3.17,14.5,0,1,5,4\n",
        "Ferrari Dino,19.7,6,145,175,3.62,2.77,15.5,0,1,5,6\n",
        "Maserati Bora,15,8,301,335,3.54,3.57,14.6,0,1,5,8\n",
        "Volvo 142E,21.4,4,121,109,4.11,2.78,18.6,1,1,4,2\n",
        "\"\"\"\n",
        "\n",
        "mtcars = pd.read_csv(StringIO(MTCARS_CSV), index_col=\"model\")\n",
        "mtcars.head()"
      ],
      "id": "cell-40"
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "0    10\n",
              "1    20\n",
              "2    30\n",
              "3    40\n",
              "Name: id, dtype: int64"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "count    32.000000\n",
              "mean     20.090625\n",
              "std       6.026948\n",
              "min      10.400000\n",
              "25%      15.425000\n",
              "50%      19.200000\n",
              "75%      22.800000\n",
              "max      33.900000\n",
              "Name: mpg, dtype: float64"
            ]
          }
        }
      ],
      "source": [
        "df[\"id\"]\n",
        "df[df[\"passed\"]]                        # rows where passed is True\n",
        "df.loc[df[\"passed\"], [\"id\", \"color\"]]   # rows and columns at once\n",
        "df[[\"id\", \"color\"]]                     # selected columns\n",
        "\n",
        "mtcars.head()\n",
        "mtcars[\"mpg\"].describe()                # R's summary(mtcars$mpg)"
      ],
      "id": "cell-41"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Part 3: Control Flow and Functions\n",
        "\n",
        "## Conditionals\n",
        "\n",
        "-   `if` / `elif` / `else` takes a single `True` or `False`\n",
        "-   The header line ends in a colon and the body is indented; there are\n",
        "    no braces\n",
        "-   `np.where()` is the vectorized version, R’s `ifelse()`"
      ],
      "id": "caee6a3f-7961-4747-a669-d1502fd70d54"
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {},
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "B"
          ]
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array(['pass', 'pass', 'fail'], dtype='<U4')"
            ]
          }
        }
      ],
      "source": [
        "score = 85\n",
        "\n",
        "if score >= 90:\n",
        "    print(\"A\")\n",
        "elif score >= 80:\n",
        "    print(\"B\")\n",
        "else:\n",
        "    print(\"C\")\n",
        "\n",
        "scores = np.array([95, 85, 60])\n",
        "np.where(scores >= 80, \"pass\", \"fail\")     # vectorized"
      ],
      "id": "cell-44"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## For loops\n",
        "\n",
        "-   `for item in sequence:` iterates over any list, array, string, or\n",
        "    dict\n",
        "-   `enumerate()` yields position and value together, the safer analogue\n",
        "    of `seq_along()`\n",
        "-   A list comprehension replaces most short loops and is the idiomatic\n",
        "    form\n",
        "-   Appending to a list is cheap, so preallocation is not the concern it\n",
        "    is in R"
      ],
      "id": "7882713a-a0e2-495f-bdec-7bd0683cb12b"
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {},
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "i = 1  i^2 = 1\n",
            "i = 2  i^2 = 4\n",
            "i = 3  i^2 = 9\n",
            "i = 4  i^2 = 16\n",
            "i = 5  i^2 = 25"
          ]
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "[1, 4, 9, 16, 25]"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "[1, 4, 9, 16, 25]"
            ]
          }
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0 a\n",
            "1 b\n",
            "2 c"
          ]
        }
      ],
      "source": [
        "for i in range(1, 6):\n",
        "    print(f\"i = {i}  i^2 = {i ** 2}\")\n",
        "\n",
        "# grow a list\n",
        "squares = []\n",
        "for i in range(1, 6):\n",
        "    squares.append(i ** 2)\n",
        "squares\n",
        "\n",
        "[i ** 2 for i in range(1, 6)]          # list comprehension, same result\n",
        "\n",
        "for pos, val in enumerate([\"a\", \"b\", \"c\"]):\n",
        "    print(pos, val)"
      ],
      "id": "cell-46"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## While loops\n",
        "\n",
        "-   Repeats while a condition stays `True`\n",
        "-   Use when the number of iterations is not known in advance\n",
        "-   `i += 1` is shorthand for `i = i + 1`; make sure something inside\n",
        "    can flip the condition"
      ],
      "id": "a48af17b-dd57-4649-9bef-4d49e18aff54"
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {},
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1\n",
            "2\n",
            "3\n",
            "4\n",
            "5"
          ]
        }
      ],
      "source": [
        "i = 1\n",
        "while i <= 5:\n",
        "    print(i)\n",
        "    i += 1"
      ],
      "id": "cell-48"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Writing functions\n",
        "\n",
        "-   `def name(arguments):` followed by an indented body\n",
        "-   `return` is required. Unlike R, the last expression is not returned\n",
        "    automatically, and a function without `return` hands back `None`\n",
        "-   Arguments can carry default values\n",
        "-   The triple-quoted docstring on the first line is what `help()`\n",
        "    displays"
      ],
      "id": "b426b02a-52df-4f8c-83af-7cdc26a03284"
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "mean    18.00\n",
              "sd      13.49\n",
              "n        6.00\n",
              "dtype: float64"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "mean    18.0000\n",
              "sd      13.4907\n",
              "n        6.0000\n",
              "dtype: float64"
            ]
          }
        }
      ],
      "source": [
        "def summarize_vec(x, digits=2):\n",
        "    \"\"\"Return the mean, standard deviation, and length of x.\"\"\"\n",
        "    x = np.asarray(x)\n",
        "    return pd.Series({\n",
        "        \"mean\": x.mean(),\n",
        "        \"sd\": x.std(ddof=1),\n",
        "        \"n\": len(x),\n",
        "    }).round(digits)\n",
        "\n",
        "\n",
        "v = [4, 8, 15, 16, 23, 42]\n",
        "summarize_vec(v)\n",
        "summarize_vec(v, digits=4)"
      ],
      "id": "cell-50"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## The apply family\n",
        "\n",
        "Idiomatic Python replaces most loops with a comprehension or with a\n",
        "vectorized method call.\n",
        "\n",
        "-   List comprehension, or `map()`: R’s `sapply()` and `lapply()`\n",
        "-   An array method with `axis=`: R’s `apply()`\n",
        "-   Watch the axis: `axis=0` collapses down the rows and returns one\n",
        "    value per column, which is R’s `apply(m, 2, ...)`, so the numbers\n",
        "    are swapped relative to R\n",
        "-   `df.apply()` runs a function column by column across a data frame"
      ],
      "id": "37d40327-f39c-4d5c-8ab2-25e8345db0e4"
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "[1, 4, 9, 16, 25]"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "[1, 4, 9, 16, 25]"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([15, 40, 65, 90])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "array([16, 17, 18, 19, 20])"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "mpg     20.090625\n",
              "hp     146.687500\n",
              "wt       3.217250\n",
              "dtype: float64"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "mpg     23.500\n",
              "hp     283.000\n",
              "wt       3.911\n",
              "dtype: float64"
            ]
          }
        }
      ],
      "source": [
        "[i ** 2 for i in range(1, 6)]                  # sapply(1:5, function(i) i^2)\n",
        "list(map(lambda i: i ** 2, range(1, 6)))       # the map() form\n",
        "\n",
        "m.sum(axis=0)      # column sums, R's apply(m, 2, sum)\n",
        "m.max(axis=1)      # row maxima, R's apply(m, 1, max)\n",
        "\n",
        "mtcars[[\"mpg\", \"hp\", \"wt\"]].mean()\n",
        "mtcars[[\"mpg\", \"hp\", \"wt\"]].apply(lambda col: col.max() - col.min())"
      ],
      "id": "cell-52"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Getting unstuck\n",
        "\n",
        "-   `help(np.mean)`, or `np.mean?` in Jupyter, opens the documentation\n",
        "-   `type(x)` when a result surprises you: the problem is almost always\n",
        "    the type\n",
        "-   `dir(x)` lists what the object can do\n",
        "-   `%whos` is the IPython magic closest to R’s `ls()`\n",
        "-   Read the traceback from the bottom up: the last line is the actual\n",
        "    error, the lines above are the call path"
      ],
      "id": "7a86f2ba-b231-4243-8f00-dd34a57fa647"
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {},
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "No variables match your requested type.\n",
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Index: 32 entries, Mazda RX4 to Volvo 142E\n",
            "Data columns (total 11 columns):\n",
            " #   Column  Non-Null Count  Dtype  \n",
            "---  ------  --------------  -----  \n",
            " 0   mpg     32 non-null     float64\n",
            " 1   cyl     32 non-null     int64  \n",
            " 2   disp    32 non-null     float64\n",
            " 3   hp      32 non-null     int64  \n",
            " 4   drat    32 non-null     float64\n",
            " 5   wt      32 non-null     float64\n",
            " 6   qsec    32 non-null     float64\n",
            " 7   vs      32 non-null     int64  \n",
            " 8   am      32 non-null     int64  \n",
            " 9   gear    32 non-null     int64  \n",
            " 10  carb    32 non-null     int64  \n",
            "dtypes: float64(5), int64(6)\n",
            "memory usage: 3.0+ KB"
          ]
        }
      ],
      "source": [
        "a = 1\n",
        "b = \"two\"\n",
        "\n",
        "%whos             # everything defined right now\n",
        "\n",
        "# help(np.mean)   # uncomment to open the docs\n",
        "mtcars.info()     # R's str() on a data frame"
      ],
      "id": "cell-54"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Part 4: NumPy and pandas\n",
        "\n",
        "## What the stack is\n",
        "\n",
        "Python has no single tidyverse. A handful of independent packages divide\n",
        "the same work, each imported on its own.\n",
        "\n",
        "-   `pandas`: data manipulation and reshaping, covering `dplyr` and\n",
        "    `tidyr`\n",
        "-   `numpy`: arrays and numerical computation, the layer pandas is built\n",
        "    on\n",
        "-   `matplotlib`, `seaborn`, `plotnine`: visualization, where `plotnine`\n",
        "    is a direct port of ggplot2\n",
        "-   `pandas.read_csv` and `pyarrow`: file import, covering `readr`\n",
        "-   `scipy`, `statsmodels`, `scikit-learn`: statistics, models, and\n",
        "    machine learning\n",
        "\n",
        "Install once with conda or pip, import every session."
      ],
      "id": "292bf407-5848-4204-8c95-d3fb57f6ffb9"
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "'2.4.2'"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/plain": [
              "'2.3.3'"
            ]
          }
        }
      ],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "np.__version__\n",
        "pd.__version__"
      ],
      "id": "cell-57"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Tidy data\n",
        "\n",
        "The same rule governs everything downstream.\n",
        "\n",
        "-   Each variable is a column\n",
        "-   Each observation is a row\n",
        "-   Each cell is a single value\n",
        "\n",
        "pandas calls this long form. Data in this shape works directly with\n",
        "`groupby`, `melt`, and every plotting library."
      ],
      "id": "0a197da4-6db7-426f-817f-8aa4255ee2e0"
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Index: 32 entries, Mazda RX4 to Volvo 142E\n",
            "Data columns (total 11 columns):\n",
            " #   Column  Non-Null Count  Dtype  \n",
            "---  ------  --------------  -----  \n",
            " 0   mpg     32 non-null     float64\n",
            " 1   cyl     32 non-null     int64  \n",
            " 2   disp    32 non-null     float64\n",
            " 3   hp      32 non-null     int64  \n",
            " 4   drat    32 non-null     float64\n",
            " 5   wt      32 non-null     float64\n",
            " 6   qsec    32 non-null     float64\n",
            " 7   vs      32 non-null     int64  \n",
            " 8   am      32 non-null     int64  \n",
            " 9   gear    32 non-null     int64  \n",
            " 10  carb    32 non-null     int64  \n",
            "dtypes: float64(5), int64(6)\n",
            "memory usage: 3.0+ KB"
          ]
        }
      ],
      "source": [
        "mtcars.head()\n",
        "mtcars.info()      # the pandas counterpart to glimpse()"
      ],
      "id": "cell-59"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Method chaining\n",
        "\n",
        "Python has no pipe operator. Methods return a new object, so calls chain\n",
        "left to right instead.\n",
        "\n",
        "-   Wrap the whole chain in parentheses to spread it over several lines\n",
        "-   `.pipe(f)` inserts a function that is not a method, which is the\n",
        "    nearest thing to `|>`\n",
        "-   Each step returns a copy, so the original frame is untouched"
      ],
      "id": "2c4a18a5-3361-4d35-a619-0effae906373"
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        }
      ],
      "source": [
        "# without chaining, read from the inside out\n",
        "mtcars[mtcars[\"cyl\"] == 4].head(3)\n",
        "\n",
        "# with chaining: read it as \"take mtcars, then query, then take the head\"\n",
        "(\n",
        "    mtcars\n",
        "    .query(\"cyl == 4\")\n",
        "    .head(3)\n",
        ")"
      ],
      "id": "cell-61"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## The core pandas verbs\n",
        "\n",
        "Five operations cover most day-to-day data manipulation.\n",
        "\n",
        "-   `query()`, or a boolean mask: `filter()`\n",
        "-   `[[ ]]`, or `.filter(items = )`: `select()`\n",
        "-   `assign()`: `mutate()`\n",
        "-   `sort_values(ascending = False)`: `arrange()` with `desc()`\n",
        "-   `agg()`: `summarize()`\n",
        "\n",
        "One false friend to watch: the pandas method named `filter()` selects\n",
        "columns, where the dplyr verb of the same name selects rows."
      ],
      "id": "500d5d74-1442-48df-8dba-f981a980ff5d"
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        }
      ],
      "source": [
        "(\n",
        "    mtcars\n",
        "    .query(\"cyl == 4\")\n",
        "    .filter([\"mpg\", \"hp\", \"wt\"])\n",
        "    .assign(km_per_liter=lambda d: d[\"mpg\"] * 0.425)\n",
        "    .sort_values(\"mpg\", ascending=False)\n",
        "    .head(5)\n",
        ")"
      ],
      "id": "cell-63"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Group and summarize\n",
        "\n",
        "`groupby()` splits the data, `agg()` collapses each group, and the\n",
        "result is one row per group.\n",
        "\n",
        "-   This single pattern replaces most of what pivot tables do\n",
        "-   `\"size\"` counts rows within the current group, R’s `n()`\n",
        "-   The group labels become the index, so `reset_index()` turns them\n",
        "    back into an ordinary column"
      ],
      "id": "9c934b1d-2478-4e22-ae3d-65e3a63906f7"
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        }
      ],
      "source": [
        "(\n",
        "    mtcars\n",
        "    .groupby(\"cyl\")\n",
        "    .agg(\n",
        "        n=(\"mpg\", \"size\"),\n",
        "        mean_mpg=(\"mpg\", \"mean\"),\n",
        "        mean_hp=(\"hp\", \"mean\"),\n",
        "    )\n",
        "    .sort_values(\"mean_mpg\", ascending=False)\n",
        "    .reset_index()\n",
        ")"
      ],
      "id": "cell-65"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Reshaping\n",
        "\n",
        "-   `melt()` turns wide data into long data, which is what plotting\n",
        "    functions expect: R’s `pivot_longer()`\n",
        "-   `pivot()` and `pivot_table()` reverse it: R’s `pivot_wider()`"
      ],
      "id": "889cf0b5-decf-437f-b099-c6a01b3e5ae8"
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {},
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        },
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "text/html": [
              "\n",
              "</div>"
            ]
          }
        }
      ],
      "source": [
        "long = (\n",
        "    mtcars\n",
        "    .reset_index()\n",
        "    .filter([\"model\", \"mpg\", \"hp\"])\n",
        "    .melt(id_vars=\"model\", var_name=\"measure\", value_name=\"value\")\n",
        ")\n",
        "\n",
        "long.head()\n",
        "\n",
        "long.pivot(index=\"model\", columns=\"measure\", values=\"value\").head()"
      ],
      "id": "cell-67"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Quick reference\n",
        "\n",
        "| Task | R | Python |\n",
        "|------------------------|------------------------|------------------------|\n",
        "| Assign | `x <- 5` | `x = 5` |\n",
        "| Exponent | `2^10` | `2 ** 10` |\n",
        "| Integer division, remainder | `%/%`, `%%` | `//`, `%` |\n",
        "| Storage type | `typeof(x)`, `class(x)` | `type(x)` |\n",
        "| Structure summary | `str(x)` | `df.info()`, `type(x)`, `dir(x)` |\n",
        "| Combine values | `c(1, 2, 3)` | `[1, 2, 3]`, `np.array([1, 2, 3])` |\n",
        "| Integer sequence | `1:10` | `range(1, 11)`, `np.arange(1, 11)` |\n",
        "| Sequence by step, by length | `seq(by =)`, `seq(length.out =)` | `np.arange()`, `np.linspace()` |\n",
        "| Repeat | `rep(x, times =)`, `rep(x, each =)` | `np.tile()`, `np.repeat()` |\n",
        "| First element | `x[1]` | `x[0]` |\n",
        "| Last element | `x[length(x)]` | `x[-1]` |\n",
        "| Drop first element | `x[-1]` | `np.delete(x, 0)`, `x[1:]` |\n",
        "| Boolean subset | `x[x > 20]` | `x[x > 20]` |\n",
        "| Missing value | `NA`, `is.na()` | `np.nan`, `pd.isna()` |\n",
        "| Empty object | `NULL` | `None` |\n",
        "| Ignore missing | `mean(x, na.rm = TRUE)` | `np.nanmean(x)`, `Series.mean()` |\n",
        "| Sample standard deviation | `sd(x)` | `x.std(ddof=1)` |\n",
        "| Matrix | `matrix(1:20, nrow = 5)` | `np.arange(1, 21).reshape(5, 4, order=\"F\")` |\n",
        "| Dimensions, transpose | `dim(m)`, `t(m)` | `m.shape`, `m.T` |\n",
        "| Heterogeneous container | `list()` | `dict`, `list` |\n",
        "| Categorical | `factor()`, `levels()` | `pd.Categorical()`, `.categories` |\n",
        "| Frequency table | `table(x)` | `Series.value_counts()` |\n",
        "| Data frame | `data.frame()` | `pd.DataFrame()` |\n",
        "| Column | `df$id` | `df[\"id\"]` |\n",
        "| Row subset | `df[df$passed, ]` | `df[df[\"passed\"]]`, `df.query()` |\n",
        "| Vectorized if | `ifelse()` | `np.where()` |\n",
        "| Apply over elements | `sapply()`, `lapply()` | list comprehension, `map()` |\n",
        "| Apply over columns, rows | `apply(m, 2, f)`, `apply(m, 1, f)` | `axis=0`, `axis=1` |\n",
        "| Function | `f <- function(x) { }` | `def f(x):` |\n",
        "| Pipe | `x \\|> f() \\|> g()` | `x.f().g()`, `.pipe(f)` |\n",
        "| dplyr verbs | `filter`, `select`, `mutate`, `arrange` | `query`, `[[ ]]`, `assign`, `sort_values` |\n",
        "| Group and summarize | `group_by() \\|> summarize()` | `groupby().agg()` |\n",
        "| Reshape | `pivot_longer()`, `pivot_wider()` | `melt()`, `pivot()` |\n",
        "| List the environment | `ls()` | `%whos` |\n",
        "| Help | `?mean` | `help(np.mean)`, `np.mean?` |"
      ],
      "id": "6ff187e6-a9bb-4d58-b8d4-89fec93b137e"
    }
  ],
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "5200",
      "language": "python"
    },
    "language_info": {
      "name": "python",
      "codemirror_mode": {
        "name": "ipython",
        "version": "3"
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.14"
    }
  }
}