{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hands-on 01a: Introduction\n",
    "\n",
    "Some helpful links:\n",
    " * Python (https://docs.python.org/3.9/tutorial/index.html): an introduction to the Python programming language\n",
    " * Google Colab (https://colab.research.google.com/): for Python development in your web-browser\n",
    " * Miniconda (https://docs.conda.io/en/latest/miniconda.html): a free minimal installer for conda\n",
    " * Numpy (https://numpy.org/doc/stable/user/quickstart.html): a widely used library for mathematical operations in Python\n",
    " * Keras (https://keras.io/): a beginner-friendly deep learning library used in these exercises\n",
    " * TensorFlow (https://www.tensorflow.org/): a useful backend for deep learning development\n",
    " * Scikit-learn (https://scikit-learn.org/stable/): helpful machine learning library \n",
    " * Seaborn (https://seaborn.pydata.org/): a library for creating nice looking graphs and figures"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numpy array basics\n",
    "\n",
    "The basic data strucutre is a numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Numpy\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a 1D array (i.e. a vector)\n",
    "x = np.array([1, 2, 3, 4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the \"shape\" of the array\n",
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a 2D array (i.e. a matrix)\n",
    "w = np.array([[1, 2, 3, 4], [1, 2, 3, 4]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the shape of w\n",
    "w.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do a matrix multiplication\n",
    "out = np.matmul(w, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note, this is the same\n",
    "out = w @ x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading data with Numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.loadtxt(fname=\"data/inflammation-01.csv\", delimiter=\",\")\n",
    "\n",
    "print(data)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expression `np.loadtxt(...)` is a function call that asks Python to run the function] `loadtxt` that\n",
    "belongs to the NumPy library.\n",
    "The dot notation in Python is used most of all as an object attribute/property specifier or for invoking its method. \n",
    "`object.property` will give you the object.property value, `object_name.method()` will invoke on object_name method.\n",
    "\n",
    "`np.loadtxt` has two parameters: the name of the file we want to read and the delimiter that separates values on a line. \n",
    "These both need to be character strings (or strings for short), so we put them in quotes.\n",
    "\n",
    "By default, only a few rows and columns are shown (with `...` to omit elements when displaying big arrays).\n",
    "Note that, to save space when displaying NumPy arrays, Python does not show us trailing zeros, so `1.0` becomes `1.`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's print out some features of this data\n",
    "print(type(data))\n",
    "print(data.dtype)\n",
    "print(data.shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output tells us that the `data` array variable contains 60 rows and 40 columns. \n",
    "When we created the variable `data` to store our arthritis data, we did not only create the array; we also\n",
    "created information about the array, called members or attributes. \n",
    "`data.shape` is an attribute of `data` which describes the dimensions of `data`.\n",
    "\n",
    "If we want to get a single number from the array, we must provide an index in square brackets after the variable name, just as we do in math when referring to an element of a matrix.  \n",
    "\n",
    "\n",
    "Our inflammation data has two dimensions, so we will need to use two indices to refer to one specific value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"first value in data: {data[0, 0]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"middle value in data: {data[30, 20]}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expression `data[30, 20]` accesses the element at row 30, column 20, while `data[0, 0]` accesses the element at row 0, column 0.\n",
    "Languages in the C family (including C++, Java, Perl, and Python) count from 0 because it represents an offset from the first value in the array (the second value is offset by one index from the first value).\n",
    "As a result, if we have an $M\\times N$ array in Python, its indices go from $0$ to $M-1$ on the first axis and $0$ to $N-1$ on the second.\n",
    "\n",
    "![\"data\" is a 3 by 3 numpy array containing row 0: ['A', 'B', 'C'], row 1: ['D', 'E', 'F'], and\n",
    "row 2: ['G', 'H', 'I']. Starting in the upper left hand corner, data[0, 0] = 'A', data[0, 1] = 'B', data[0, 2] = 'C', data[1, 0] = 'D', data[1, 1] = 'E', data[1, 2] = 'F', data[2, 0] = 'G', data[2, 1] = 'H', and data[2, 2] = 'I', in the bottom right hand corner.](images/python-zero-index.svg)\n",
    "\n",
    "When Python displays an array, it shows the element with index `[0, 0]` in the upper left corner rather than the lower left.\n",
    "This is consistent with the way mathematicians draw matrices but different from the Cartesian coordinates.\n",
    "The indices are (row, column) instead of (column, row) for the same reason, which can be confusing when plotting data."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slicing data\n",
    "An index like `[30, 20]` selects a single element of an array, but we can select whole sections as well.\n",
    "For example, we can select the first ten days (columns) of values for the first four patients (rows) like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data[0:4, 0:10])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The slice `0:4` means, “Start at index 0 and go up to, but not including, index 4”. \n",
    "The difference between the upper and lower bounds is the number of values in the slice.\n",
    "\n",
    "We don’t have to start slices at 0:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data[5:10, 0:10])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also don't have to include the upper and lower bound on the slice. \n",
    "If we don't include the lower bound, Python uses `0` by default; if we don't include the upper, the slice runs to the end of the axis, and if we don’t include either (i.e., if we use `:` on its own), the slice includes everything:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "small = data[:3, 36:]\n",
    "print(f\"small is:\\n{small}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above example selects rows 0 through 2 and columns 36 through to the end of the array."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing data\n",
    "\n",
    "NumPy has several useful functions that take an array as input to perform operations on its values.\n",
    "If we want to find the average inflammation for all patients on\n",
    "all days, for example, we can ask NumPy to compute `data`'s mean value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_over_patients_and_days = np.mean(data)\n",
    "print(mean_over_patients_and_days)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When analyzing data, though, we often want to look at variations in statistical values, such as the maximum inflammation per patient or the average inflammation per day.\n",
    "One way to do this is to create a new temporary array of the data we want, then ask it to do the calculation:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we need the maximum inflammation for each patient over all days (as in the next diagram on the left) or the average for each day (as in the diagram on the right)? As the diagram below shows, we want to perform the operation across an axis:\n",
    "\n",
    "![Per-patient maximum inflammation is computed row-wise across all columns using\n",
    "numpy.max(data, axis=1). Per-day average inflammation is computed column-wise across all rows using\n",
    "numpy.mean(data, axis=0).](images/python-operations-across-axes.png)\n",
    "\n",
    "To support this functionality, most array functions allow us to specify the axis we want to work on.\n",
    "If we ask for the average across axis 0 (rows in our 2D example), we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_over_patients = np.mean(data, axis=0)\n",
    "print(mean_over_patients)\n",
    "print(mean_over_patients.shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The expression `(40,)` tells us we have a $40\\times 1$ vector, so this is the average inflammation per day for all patients.\n",
    "\n",
    "If we average across axis 1 (columns in our 2D example), we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_over_days = np.mean(data, axis=1)\n",
    "print(mean_over_days)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which is the average inflammation per patient across all days."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing data\n",
    "Visualization deserves an entire lecture of its own, but we can explore a few features of Python's `matplotlib` library here.\n",
    "While there is no official plotting library, `matplotlib` is the _de facto_ standard.\n",
    "First, we will import the `pyplot` module from `matplotlib` and use two of its functions to create and display a heat map of our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Note: origin=\"lower\" keyword argument (\"upper\" is the default)\n",
    "image = plt.imshow(data, origin=\"lower\")\n",
    "cbar = plt.colorbar()\n",
    "# always add axis labels (and units, if appropriate)\n",
    "plt.ylabel(\"Patient\")\n",
    "plt.xlabel(\"Day\")\n",
    "cbar.set_label(\"Inflammation\")\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row in the heat map corresponds to a patient in the clinical trial dataset, and each column corresponds to a day in the dataset.\n",
    "Blue pixels in this heat map represent low values, while yellow pixels represent high values.\n",
    "As we can see, the general number of inflammation flare-ups for the patients rises and falls over a 40-day period.\n",
    "\n",
    "So far so good as this is in line with our knowledge of the clinical trial and Dr. Maverick's\n",
    "claims:\n",
    "\n",
    "* the patients take their medication once their inflammation flare-ups begin\n",
    "* it takes around 3 weeks for the medication to take effect and begin reducing flare-ups\n",
    "* and flare-ups appear to drop to zero by the end of the clinical trial.\n",
    "\n",
    "Now let's take a look at the average inflammation over time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ave_inflammation = np.mean(data, axis=0)\n",
    "# Note: drawstyle=\"steps-mid\" to reflect discrete nature of data\n",
    "ave_plot = plt.plot(ave_inflammation, drawstyle=\"steps-mid\")\n",
    "plt.ylabel(\"Average inflammation\")\n",
    "plt.xlabel(\"Day\")\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we have put the average inflammation per day across all patients in the variable `ave_inflammation`, then asked `matplotlib.pyplot` to create and display a line graph of those values.\n",
    "The result is a reasonably linear rise and fall, in line with Dr. Maverick's claim that the medication takes 3 weeks to take effect.\n",
    "But a good data scientist doesn't just consider the average of a dataset, so let's have a look at two other statistics:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_plot = plt.plot(np.max(data, axis=0), drawstyle=\"steps-mid\")\n",
    "plt.xlabel(\"Day\")\n",
    "plt.ylabel(\"Maximum inflammation\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_plot = plt.plot(np.min(data, axis=0), drawstyle=\"steps-mid\")\n",
    "plt.xlabel(\"Day\")\n",
    "plt.ylabel(\"Minimum inflammation\")\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The maximum value rises and falls linearly, while the minimum seems to be a step function.\n",
    "Neither trend seems particularly likely, so either there's a mistake in our calculations or\n",
    "something is wrong with our data. This insight would have been difficult to reach by examining\n",
    "the numbers themselves without visualization tools."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make your own plot\n",
    "\n",
    "Create a plot showing the standard deviation (`numpy.std`)\n",
    "of the inflammation data for each day across all patients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "phys139",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "d0ea348b636367bcdf67fd2d6d24251712b38670f61fdee14f28eb58fe74f081"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}