3 Linear regression.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Thinking in tensors, writing in PyTorch\n",
    "\n",
    "A hands-on course by [Piotr Migdał](https://p.migdal.pl) (2019-2022).\n",
    "This notebook prepared by [Weronika Ormaniec](https://github.com/werkaaa) and Piotr Migdał.\n",
    "\n",
    "## Notebook 3: Linear regression\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/stared/thinking-in-tensors-writing-in-pytorch/blob/master/3%20Linear%20regression.ipynb\"  target=\"_parent\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/>\n",
    "</a>\n",
    "\n",
    "\n",
    "[Linear regression](https://en.wikipedia.org/wiki/Linear_regression) is one of the most common predictive models. In plain words, we fit a straight line that fits to the data. Mathematically speaking, we use linear combination of input variables to predict the output variable.\n",
    "\n",
    "$$y = a_1 x_1 + \\ldots + a_n x_n + b$$\n",
    "\n",
    "Before moving any further, try to experience it viscerally with [Ordinary Least Squares Regression\n",
    "Explained Visually - Visually Explained](http://setosa.io/ev/ordinary-least-squares-regression/):\n",
    "\n",
    "![http://setosa.io/ev/ordinary-least-squares-regression/](imgs/linreg_setosa.png)\n",
    "\n",
    "However, it occurs that lots of dependencies in the actual world can be described just by fitting a linear equation to the observed data. That's what we are going to do now!\n",
    "\n",
    "In Python we typically use [LinearRegression from scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html). Here we use PyTorch to show everything step-by-step. Moreover, linear regression is a building block of any regression with deep learning - so it is good to understand it well!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "from livelossplot import PlotLosses\n",
    "from livelossplot.outputs import MatplotlibPlot\n",
    "from ipywidgets import interact, fixed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "\n",
    "Have you ever wondered what is the relation between brain and body weights among various animal species?\n",
    "\n",
    "Let's try a [Brain to Body Weight Dataset](http://wiki.stat.ucla.edu/socr/index.php/SOCR_Data_Brain2BodyWeight)!\n",
    "\n",
    "> These data represent the relation between weights of the body and brain of various species. It may be used to discuss bivariate exploratory and quantitative data analyses in the case of allometric relationships. Brain-to-body weight ratio is assumed to be related to species intelligence. The encephalization quotient is a more complex measurement that takes into account allometric effects of widely divergent body sizes across several taxa. The brain-to-body mass ratio is a simpler measure of encephalization within species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# locally, \"data/Animals.csv\" suffice\n",
    "data = pd.read_csv(\"https://raw.githubusercontent.com/stared/thinking-in-tensors-writing-in-pytorch/master/data/Animals.csv\",\n",
    "                   index_col='Species')\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or sorted in a different way\n",
    "data.sort_values(by=\"BrainWeight(kg)\", ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, which is the smartest one? \n",
    "\n",
    "If we go by brain to body weight proportion, humans are on the top, but so are hamsters (one can argue that there are smarter creatures on the list).\n",
    "\n",
    "If we go by brain weight, it favors bigger animals. Sure, whales and elephants are smart - but are they THAT smarter than humans?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's make a scatter plot\n",
    "data.plot.scatter(x=\"BodyWeight(kg)\", y=\"BrainWeight(kg)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At first glance it does not resemble any particular dependance. However, if we change the scale something interesting can be spotted with [logarithmic scaling](https://simple.wikipedia.org/wiki/Logarithmic_scale):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.plot.scatter(x=\"BodyWeight(kg)\", y=\"BrainWeight(kg)\", logx=True, logy=True);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see a clear dependence that the bigger body weight (on average) the bigger brain weight.\n",
    "Let's investigate that!\n",
    "\n",
    "First of all, we need to prepare the data. We see some dependence on the scatter plot only after scaling both x and y axes logarithmically. That is why, if we want to see the same relationship in the data itself, we take natural logarithm of brain and body weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.log(data['BodyWeight(kg)']).astype('float32')\n",
    "Y = np.log(data['BrainWeight(kg)']).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A toy example\n",
    "\n",
    "\n",
    "At the beginning let's take a look only on a few points from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_less = X[::6]\n",
    "X_less"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_less = Y[::6]\n",
    "Y_less"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(5, 6))\n",
    "ax.scatter(X_less, Y_less, color='r')\n",
    "ax.set_xlabel('x')\n",
    "ax.set_ylabel('y')\n",
    "ax.set_ylim([-6, 4]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the scatterplot it can be seen that the relationship between presented data is almost linear. We will try to apply the equation:\n",
    "\n",
    "$$ y = ax+b$$\n",
    "\n",
    "to the analysed dataset. The only problem is how to find $a$ and $b$.\n",
    "Try to find a proper line manually!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_model(a, b, X, Y):\n",
    "    X = np.sort(X)\n",
    "    Y_pred = a * X + b\n",
    "    fig, ax = plt.subplots(figsize=(5, 6))\n",
    "    ax.plot(X, Y_pred)\n",
    "    ax.scatter(X, Y, color='r')\n",
    "    ax.set_xlabel('x')\n",
    "    ax.set_ylabel('y')\n",
    "    ax.set_ylim([-6, 4])\n",
    "    \n",
    "interact(plot_model, \n",
    "         a=(-4.0, 4.0), \n",
    "         b=(-6.0, 6.0),\n",
    "         X=fixed(X_less),\n",
    "         Y=fixed(Y_less)\n",
    "        );"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss function\n",
    "\n",
    "We will try to somehow measure if the coefficients in the equation are good enough to describe our problem. In order to do it we will define a loss function - an equation that will tell us how much our approximation differs from the expected output. \n",
    "\n",
    "The loss function should:\n",
    "\n",
    "* depend only on the coefficients of the model, expected output and our approximation,\n",
    "* shrink if our approximation is becoming better and grow if it gets worse.\n",
    "\n",
    "When it comes to linear regression the most common approach is the [least-squares loss function](https://en.wikipedia.org/wiki/Least_squares). We will calculate the average square of the vertical deviations from each data point to the line. Since we first square the deviations, it does not matter if the data point is above or below the line. \n",
    "\n",
    "\n",
    "$$y^{ pred}_{i} = ax_{i}+b $$\n",
    "$$L=\\frac{1}{N}\\sum_{i=0}^{N-1}( y^{pred}_{i} - y_{i})^2 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa_ = np.linspace(-2, 3, num=100)\n",
    "bb_ = np.linspace(-8, 8, num=100)\n",
    "aa, bb = np.meshgrid(aa_, bb_)\n",
    "\n",
    "def loss_numpy(aa, bb, X, Y):\n",
    "    loss = np.zeros_like(aa)\n",
    "    for i in range(len(loss)):\n",
    "        for j in range(len(loss[0])):\n",
    "            loss[i][j] = ((aa[i,j] * X + bb[i,j] - Y)**2).sum()\n",
    "    return loss\n",
    "\n",
    "cs = plt.contour(aa, bb, np.sqrt(loss_numpy(aa, bb, X_less, Y_less)), cmap='coolwarm')\n",
    "plt.clabel(cs, inline=1, fontsize=10)\n",
    "plt.title(\"Loss\")\n",
    "plt.xlabel('a')\n",
    "plt.ylabel('b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to use PyTorch in our model we need tensors!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_less = torch.tensor(X_less)\n",
    "X_less"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_less = torch.tensor(Y_less)\n",
    "Y_less"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Y_pred(A, B, X):\n",
    "    return A * X + B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss(Y_pred, Y):\n",
    "    return (Y_pred - Y).pow(2).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Minimizing loss function\n",
    "\n",
    "Since we have defined the loss function, we should minimize it. \n",
    "There is an explicit formula for the coefficients that will guarantee the best fit of the line for particular data:\n",
    "\n",
    "$$a=\\frac{\\sum_{i=0}^{N-1}{x_i} \\cdot \\sum_{i=0}^{N-1}{y_i}  - \\sum_{i=0}^{N-1}{x_iy_i} }{(\\sum_{i=0}^{N-1}{x_i})^2-\\sum_{i=0}^{N-1}{x_i^2} }$$\n",
    "$$b=\\frac{1}{N}\\left(\\sum_{i=0}^{N-1}{y_i} -a\\sum_{i=0}^{N-1}{x_i}\\right) $$\n",
    "\n",
    "You can see also [expected value](https://en.wikipedia.org/wiki/Expected_value) notation, were $\\mathbb{E}[x]$ means an average of $x$, that is $\\sum_{i=0}^{N-1}{x_i}/N$.\n",
    "\n",
    "$$a = \\frac{\\mathbb{E}[x]\\mathbb{E}[y] - \\mathbb{E}[xy]}{(\\mathbb{E}[x])^2 - \\mathbb{E}[x^2]}$$\n",
    "$$b = \\mathbb{E}[y] - a E[x] $$\n",
    "\n",
    "However, for didactic purpose we will minimize the loss function differently, using **gradient descent** (discussed in the previous notebook).\n",
    "\n",
    "By doing so we will step by step rotate and move the line, so it will reflect the actual location of data points. In order to do it we need to repeatedly shift the weights till we find a minimum of the loss function. What we need is a mathematical operation that will tell us how the loss function change, if we increase or decrease $a$ and $b$. The operation we are looking for is partial derivative:\n",
    "\n",
    "$$\\dfrac{\\partial L}{\\partial a}  = \\frac{2}{N}\\sum_{i=0}^{N-1} (y^{pred}_{i} -y_{i}) \\cdot x_{i}$$ \n",
    "\n",
    "$$\\dfrac{\\partial L}{\\partial b} = \\frac{2}{N}\\sum_{i=0}^{N-1} (y^{pred}_{i} -y_{i})$$\n",
    "\n",
    "Let's write them as code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dL_da(A, B, X, Y):\n",
    "    Y_prediction = Y_pred(A, B, X)\n",
    "    return 2 * ((Y_prediction - Y) * X).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dL_db(A, B, X, Y):\n",
    "    Y_prediction = Y_pred(A, B, X)\n",
    "    return 2 * (Y_prediction - Y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to specify two more things:\n",
    "\n",
    "- **learning\\_rate** - hyperparameter that will define how much the value of the derivative will influence the change of $a$ and $b$,\n",
    "- **num\\_epochs** - hyperparameter defining how many iterations it will take to sufficiently minimize the loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model_manually(X, Y, A, B, learning_rate, num_epochs):\n",
    "    logs = {}\n",
    "    epoch_loss = 0.0\n",
    "    \n",
    "    def extra_plot(*args):\n",
    "        plt.plot(X.numpy(), Y.numpy(), 'r.', label=\"Ground truth\")\n",
    "        plt.plot(X.numpy(), Y_pred(A, B, X).numpy(), '-', label=\"Model\")\n",
    "        plt.title(\"Prediction\")\n",
    "        plt.legend(loc='lower right')\n",
    "        \n",
    "    liveloss = PlotLosses()\n",
    "    \n",
    "    for i in range(num_epochs):\n",
    "        A -= learning_rate * dL_da(A, B, X, Y)\n",
    "        B -= learning_rate * dL_db(A, B, X, Y)\n",
    "        \n",
    "        epoch_loss = loss(Y_pred(A, B, X), Y)\n",
    "        \n",
    "        avg_loss = epoch_loss / len(X)\n",
    "        \n",
    "\n",
    "        liveloss.update({\n",
    "            'loss': avg_loss,\n",
    "        })\n",
    "        liveloss.draw()\n",
    "        \n",
    "        print(\"y = {:.3f}x{:+.3f}\".format(A.item(), B.item()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last thing we need to do before trying to minimalize the loss function is to initialize the coefficients with some random values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = torch.randn(1)\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "B = torch.randn(1)\n",
    "B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_model_manually(X_less, Y_less, A, B, learning_rate=0.025, num_epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that is how we find the propper line!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear regression using PyTorch\n",
    "\n",
    "Knowing how linear regression works, let's come back to the relation between body and brain weights. This time we will use built-in PyTorch functions.\n",
    "\n",
    "Firstly, we need to prepare the data. PyTorch built-in models have specified shapes of the input data. It is all specified in [PyTorch documentation](https://pytorch.org/docs/stable/nn.html#linear). Below, we are changing the shape of our data into format **(number_of_datapoints, in_features)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.tensor(X).view(-1, 1)\n",
    "X.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = torch.tensor(Y).view(-1, 1)\n",
    "Y.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of initializing the coefficients manually, we can define the model using a built in class. Since both input and output in the analyzed problem have only one dimension we set **(in_features=1, out_features=1)** as arguments of **nn.Linear**. That is also specified in [PyTorch documentation](https://pytorch.org/docs/stable/nn.html#linear)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_model = nn.Linear(in_features=1, out_features=1)\n",
    "print(linear_model.weight)\n",
    "print(linear_model.bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of **gradient\\_step** function, we will define an **optimizer** with learning rate and built-in **loss function**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optim = torch.optim.SGD(linear_model.parameters(), lr=0.03)\n",
    "loss_function = nn.MSELoss()\n",
    "loss = loss_function(linear_model(X), Y)\n",
    "print(loss)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before training the model, let's see what does the line with random coefficients look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_animals_to_print = {'Elephant', 'Adult_Human', 'Alligator',\n",
    "                            'Owl', 'Cat', 'Chimpanzee',\n",
    "                            'Green_Lizard', 'Hamster', 'Cow'}\n",
    "\n",
    "def plot_model_annotate(X, Y, A, B, labels,\n",
    "                        animals_to_print=default_animals_to_print):\n",
    "    fig, ax = plt.subplots()\n",
    "    ax.scatter(X, Y, color='red')\n",
    "    y_pred = A * X + B\n",
    "    ax.plot(X, y_pred)\n",
    "    ax.set_xlabel(\"LogBodyWeight(kg)\")\n",
    "    ax.set_ylabel(\"LogBrainWeight(kg)\")\n",
    "    ax.set_ylim([-8, 4])\n",
    "    \n",
    "    for i, label in enumerate(labels):\n",
    "        if animals_to_print is None or label in animals_to_print: \n",
    "            ax.annotate(label, (X[i], Y[i]))\n",
    "   \n",
    "    print(\"LogBodyWeight(kg) = {:.3f}*LogBrainWeight(kg){:+.3f}\".format(A, B))\n",
    "    \n",
    "plot_model_annotate(X.view(-1).numpy(), Y.view(-1).numpy(),\n",
    "                    linear_model.weight.item(), linear_model.bias.item(),\n",
    "                    data.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can train the model - minimise the loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(X, Y, model, loss_function, optim, num_epochs):\n",
    "    loss_history = []\n",
    "    \n",
    "    def extra_plot(*args):\n",
    "        plt.plot(X.numpy(), Y.numpy(), 'r.', label=\"Ground truth\")\n",
    "        plt.plot(X.numpy(), linear_model(X).detach().numpy(), '-', label=\"Model\")\n",
    "        plt.title(\"Prediction\")\n",
    "        plt.legend(loc='lower right')\n",
    "    \n",
    "    liveloss = PlotLosses()\n",
    "\n",
    "    for epoch in range(num_epochs):\n",
    "        \n",
    "        epoch_loss = 0.0\n",
    "        \n",
    "        Y_pred = model(X)\n",
    "        loss = loss_function(Y_pred, Y)\n",
    "        \n",
    "        loss.backward()\n",
    "        optim.step()\n",
    "        optim.zero_grad()\n",
    "                \n",
    "        epoch_loss = loss.data.item()\n",
    "        \n",
    "        avg_loss = epoch_loss / len(X)\n",
    "\n",
    "        liveloss.update({\n",
    "            'loss': avg_loss,\n",
    "           #'a': model.weight[0][0].item(),\n",
    "           #'b': model.bias[0].item()\n",
    "        })\n",
    "        liveloss.draw()\n",
    "        print(\"y = {:.3f}x{:+.3f}\".format(model.weight[0][0].item(), model.bias[0].item()))\n",
    "        \n",
    "\n",
    "train_model(X, Y, linear_model, loss_function, optim, num_epochs=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if we fitted the final line properly!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_model_annotate(X.view(-1).numpy(), Y.view(-1).numpy(),\n",
    "                    linear_model.weight.item(), linear_model.bias.item(),\n",
    "                    data.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It fits the data much better than at the begining. We have found the relation between brain and body weights among various animal species:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"log(PredictedBrainWeight) = {a:.3f} * log(BodyWeight) {b:.3f}\".format(\n",
    "    a=linear_model.weight[0][0].item(),\n",
    "    b=linear_model.bias[0].item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be transformed into an explicit formula:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"PredictedBrainWeight =  {e_b:.3f} * (BodyWeight)^{e_a:.3f}\".format(\n",
    "    e_a=linear_model.weight[0][0].exp().item(),\n",
    "    e_b=linear_model.bias[0].exp().item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's attach data\n",
    "data['PredictedBrainWeight(kg)'] = linear_model(X).exp().detach().squeeze().numpy()\n",
    "data['PredictedActualRatio'] = data['PredictedBrainWeight(kg)'] / data['BrainWeight(kg)']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now compare predicted brain weights with actual data. What does it mean that the actual brain weight is bigger than predicted one? Is an animal more clever in that case?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.sort_values(by='PredictedActualRatio').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is it a good guess? Well, for me it sounds reasonable. At the same time, we don't have a simple way to compare intelligence of species operating in different environments, see [Cat vs. Squid by Wumo](http://wumo.com/wumo/2013/02/25):\n",
    "\n",
    "![Cat vs. Squid | Wum](http://wumo.com/img/wumo/2013/02/25.png)\n",
    "\n",
    "For food for though, crows are wicked smart, vide [Causal understanding of water displacement by a crow](https://www.youtube.com/watch?v=ZerUbHmuY04). [So are octopodes](https://www.nytimes.com/2018/11/30/science/animal-intelligence-octopus-cephalopods.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "If you want to practice linear regression, here is another dataset. It describes the relation between weight and average heart rate of various animals. \n",
    "\n",
    "(Tip: try to scale the data, by taking logarithm of both values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# locally: \"data/Heart_rate_and_weight.csv\"\n",
    "heart_rate_dataset = pd.read_csv(\"https://raw.githubusercontent.com/stared/thinking-in-tensors-writing-in-pytorch/master/data/Heart_rate_and_weight.csv\",\n",
    "                                 index_col=0)\n",
    "heart_rate_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra\n",
    "\n",
    "Here are some interesting websites on the subject of linear regression:\n",
    "\n",
    "\n",
    "* [Linear regression](http://www.stat.yale.edu/Courses/1997-98/101/linreg.htm)\n",
    "* [Ordinary Least Squares Regression-Explained Visually](http://setosa.io/ev/ordinary-least-squares-regression/)\n",
    "* [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient)\n",
    "\n",
    "\n",
    "Beware, even if there is some correlation, it may be not that sound:\n",
    "\n",
    "![https://imgs.xkcd.com/comics/linear_regression.png](https://imgs.xkcd.com/comics/linear_regression.png)\n",
    "\n",
    "![https://imgs.xkcd.com/comics/extrapolating.png](https://imgs.xkcd.com/comics/extrapolating.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch-macos-m1-gpu",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "notify_time": "30",
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  },
  "vscode": {
   "interpreter": {
    "hash": "032715a8d60757555a2623145b534a689e8b5634c5398e8bbec1014c4a8ede12"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}