Pytorch Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.9594  0.2891\n",
       " 0.5366  0.0986\n",
       " 0.6576  0.0606\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.rand(3,2)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 1  1\n",
       " 1  1\n",
       " 1  1\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = torch.ones(x.size())\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 1.9594  1.2891\n",
       " 1.5366  1.0986\n",
       " 1.6576  1.0606\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = x+y\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 1.2891\n",
       " 1.0986\n",
       " 1.0606\n",
       "[torch.FloatTensor of size 3]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## We can slice them like numpy \n",
    "z[:,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors typically have two forms of methods, one method that returns another tensor and another method that performs the operation in place. That is, the values in memory for that tensor are changed without creating a new tensor. In place functions are always followed by an underscore, for example `z.add()` and `z.add_()`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 2.9594  2.2891\n",
       " 2.5366  2.0986\n",
       " 2.6576  2.0606\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.add(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 1.9594  1.2891\n",
       " 1.5366  1.0986\n",
       " 1.6576  1.0606\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 2.9594  2.2891\n",
       " 2.5366  2.0986\n",
       " 2.6576  2.0606\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.add_(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 2.9594  2.2891\n",
       " 2.5366  2.0986\n",
       " 2.6576  2.0606\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reshaping\n",
    "Reshaping tensors is really common operation. First to get the size and shape of tensor use `.size()`.Then, to reshape a tensor, use `.resize_()`.Notice the undersocre, reshaping is an in-place operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 2])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 2.9594  2.2891  2.5366\n",
       " 2.0986  2.6576  2.0606\n",
       "[torch.FloatTensor of size 2x3]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.resize_(2,3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 2.9594  2.2891  2.5366\n",
       " 2.0986  2.6576  2.0606\n",
       "[torch.FloatTensor of size 2x3]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numpy to Torch and back\n",
    "Converting between Numpy arrays and Torch tensors is super simple and useful. To create a tensor form Numpy array, use `torch.from_numpy()`. To convert a tensor to a Numpy array, use the `.numpy()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.69308973, 0.09646615, 0.97728747],\n",
       "       [0.72016418, 0.5282418 , 0.97438102],\n",
       "       [0.48658814, 0.61781038, 0.26487553],\n",
       "       [0.84180575, 0.57768569, 0.75021959]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a  = np.random.rand(4,3)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.6931  0.0965  0.9773\n",
       " 0.7202  0.5282  0.9744\n",
       " 0.4866  0.6178  0.2649\n",
       " 0.8418  0.5777  0.7502\n",
       "[torch.DoubleTensor of size 4x3]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = torch.from_numpy(a)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.69308973, 0.09646615, 0.97728747],\n",
       "       [0.72016418, 0.5282418 , 0.97438102],\n",
       "       [0.48658814, 0.61781038, 0.26487553],\n",
       "       [0.84180575, 0.57768569, 0.75021959]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The memory is shared between the Numpy array and Torch tensor, so if we change the value in-place of one object, the other will change as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 1.3862  0.1929  1.9546\n",
       " 1.4403  1.0565  1.9488\n",
       " 0.9732  1.2356  0.5298\n",
       " 1.6836  1.1554  1.5004\n",
       "[torch.DoubleTensor of size 4x3]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.mul_(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.38617947, 0.19293231, 1.95457494],\n",
       "       [1.44032836, 1.0564836 , 1.94876203],\n",
       "       [0.97317629, 1.23562076, 0.52975106],\n",
       "       [1.68361149, 1.15537138, 1.50043918]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural networks with PyTorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "#import helper\n",
    "import matplotlib.pyplot as plt\n",
    "from torchvision import datasets, transforms\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a transform to normalize the data\n",
    "transform = transforms.Compose([transforms.ToTensor(),\n",
    "                               transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5)),\n",
    "                               ])\n",
    "# Download and load the training data\n",
    "trainset = datasets.MNIST('MNIST_data/',download=True, train=True,\n",
    "                         transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=64,shuffle=True)\n",
    "\n",
    "# Download and load the test data\n",
    "testset = datasets.MNIST('MNIST_data/', download=True, train=False, transform=transforms)\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataiter = iter(trainloader)\n",
    "images, labels = dataiter.next()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have the training data loaded into `trainloader` and we make that na iterator with `iter(trainloader)`. We'd use this to loop through the dataset for training, but here I'm just grabbing the first batch sow we can check out the data.We can see below that `images` is just tensor with size (64, 1, 28, 28). So, 64 images per batch, 1 color channel, and 28x28 images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([64, 1, 28, 28])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images.size()\n",
    "# 64 is batchsize\n",
    "# 1 channel size so black and white image\n",
    "# 28x28 is width x length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADQRJREFUeJzt3V+oXfWZxvHniaYotmAkxgSb0U4J45QIaTlKo0NQ1JiRYuxFJV7UFMqkQgNT6YV/bqoXhSBtOt6keEpDU2hsI2k0F2WmEqqZwliMf2jSxrZazyQZQ07FamLAhJO8c3FWymk857f32Xvttfbx/X7gcPZe71p7v+zkOb+199pr/RwRApDPvLYbANAOwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+IKkLm3wy23ydEBiwiHA36/U18tteY/sPtl+3/WA/jwWgWe71u/22L5D0R0m3SToi6UVJ90TE7wvbMPIDA9bEyH+9pNcj4s8RcVrSTyWt7ePxADSon/BfKenwlPtHqmV/x/YG2/ts7+vjuQDUrJ8P/KbbtfjQbn1EjEoaldjtB4ZJPyP/EUlLp9z/pKS3+msHQFP6Cf+LkpbZ/pTtj0laJ2l3PW0BGLSed/sjYsL2Rkn/JekCSVsj4ne1dQZgoHo+1NfTk/GeHxi4Rr7kA2DuIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiCpnqfoliTbY5JOSDojaSIiRupoCsDg9RX+ys0R8XYNjwOgQez2A0n1G/6Q9EvbL9neUEdDAJrR727/jRHxlu1Fkp61/VpE7J26QvVHgT8MwJBxRNTzQPYjkt6PiO8U1qnnyQDMKCLczXo97/bbvsT2J87dlrRa0oFeHw9As/rZ7b9C0i7b5x5ne0T8Zy1dARi42nb7u3oydvuBgRv4bj+AuY3wA0kRfiApwg8kRfiBpAg/kFQdZ/Wlt3nz5mL9/vvvL9ZPnz5drG/cuLFYP3To0Iy1F154objte++9V6z368ILZ/4vdssttxS3ff7554v1Dz74oKeeMImRH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeS4jh/DSYmJor1TqdNz58/v1h/4oknZt3TOfv37y/WT548WayfOXOmWH/jjTeK9TVr1sxYW7RoUXHbt98uXxR6y5Ytxfqjjz46Y63JU9mHFSM/kBThB5Ii/EBShB9IivADSRF+ICnCDyTFpbtrcPHFFxfrr732WrG+dOnSOttBZcmSJTPWjh071mAnzeLS3QCKCD+QFOEHkiL8QFKEH0iK8ANJEX4gqY7n89veKukLksYjYnm17DJJP5N0taQxSXdHxF8H1+Zwu/zyy4v1xYsXF+unTp0q1rdv316sL1y4cMbarbfeWty2TfPmlceeiy66qFjnnPz+dDPy/0jS+VdkeFDSnohYJmlPdR/AHNIx/BGxV9I75y1eK2lbdXubpLtq7gvAgPX6nv+KiDgqSdXv8vWYAAydgV/Dz/YGSRsG/TwAZqfXkf+Y7SWSVP0en2nFiBiNiJGIGOnxuQAMQK/h3y1pfXV7vaRn6mkHQFM6ht/2k5L+R9I/2T5i+6uSNkm6zfafJN1W3Qcwh3A+fw1uvvnmYn3Pnj3F+vHjx4v1Sy+9dNY9zQUPPPBAsb5pU3lMefPNN4v1a665Zsba6dOni9vOZZzPD6CI8ANJEX4gKcIPJEX4gaQIP5AUh/pq0OnU1FdeeaVYX758ebH+2GOPFesPPfRQsd6m0mXNOx2q6zSF93333Vesj46OFusfVRzqA1BE+IGkCD+QFOEHkiL8QFKEH0iK8ANJDfwyXhmcPXu2WN+xY0exfu211xbrq1atmnVPw6J0nL/TcfyJiYlivdPU5yhj5AeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpDjO34DDhw8X6ydOnCjWV69eXWc7jdq1a1fP23a6DsLevXt7fmww8gNpEX4gKcIPJEX4gaQIP5AU4QeSIvxAUh2P89veKukLksYjYnm17BFJ/ybpL9VqD0fELwbV5Efd448/XqyfPHmyoU5m74YbbijWV65c2fNjD/N8BB8F3Yz8P5K0Zprl34uIFdUPwQfmmI7hj4i9kt5poBcADernPf9G27+1vdX2gto6AtCIXsP/fUmflrRC0lFJ351pRdsbbO+zva/H5wIwAD2FPyKORcSZiDgr6QeSri+sOxoRIxEx0muTAOrXU/htL5ly94uSDtTTDoCmdHOo70lJN0laaPuIpG9Jusn2CkkhaUzS1wbYI4ABcEQ092R2c082RObNK+9gdfo3aPLfaLbuvPPOYv3pp5+esTY+Pl7cdtmyZcV6p+sgZBUR7mY9vuEHJEX4gaQIP5AU4QeSIvxAUoQfSIpLdzeg0xTec9m6det63va5554r1jmUN1iM/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFMf5UXTVVVcV67fffnuxXjod+amnnuqpJ9SDkR9IivADSRF+ICnCDyRF+IGkCD+QFOEHkuI4P4ruvffeYn3BgvI0jadOnZqxtnPnzp56Qj0Y+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gqY7H+W0vlfRjSYslnZU0GhGP275M0s8kXS1pTNLdEfHXwbWKNnQ6n7+Td999t6ZOULduRv4JSd+MiH+W9HlJX7f9GUkPStoTEcsk7anuA5gjOoY/Io5GxMvV7ROSDkq6UtJaSduq1bZJumtQTQKo36ze89u+WtJnJf1G0hURcVSa/AMhaVHdzQEYnK6/22/745J2SvpGRBy33e12GyRt6K09AIPS1chve74mg/+TiPh5tfiY7SVVfYmk8em2jYjRiBiJiJE6GgZQj47h9+QQ/0NJByNi85TSbknrq9vrJT1Tf3sABqWb3f4bJX1Z0n7br1bLHpa0SdIO21+VdEjSlwbTIgZp/vz5xfodd9xRrHd6+7d79+5Z94RmdAx/RPxa0kz/wrfU2w6ApvANPyApwg8kRfiBpAg/kBThB5Ii/EBSXLo7uZUrVxbrixcvLtZLU3BL0pYtW2bdE5rByA8kRfiBpAg/kBThB5Ii/EBShB9IivADSXGcP7lVq1b1tf3Y2FixfuDAgb4eH4PDyA8kRfiBpAg/kBThB5Ii/EBShB9IivADSXGcP7nrrruur+3PnDnTVx3tYeQHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQ6ht/2Utu/sn3Q9u9s/3u1/BHb/2f71eqnPJE7hpLt4g8+urr5ks+EpG9GxMu2PyHpJdvPVrXvRcR3BtcegEHpGP6IOCrpaHX7hO2Dkq4cdGMABmtW7/ltXy3ps5J+Uy3aaPu3trfaXjDDNhts77O9r69OAdSq6/Db/riknZK+ERHHJX1f0qclrdDknsF3p9suIkYjYiQiRmroF0BNugq/7fmaDP5PIuLnkhQRxyLiTESclfQDSdcPrk0Adevm035L+qGkgxGxecryJVNW+6IkLtMKzCHdfNp/o6QvS9pv+9Vq2cOS7rG9QlJIGpP0tYF0iIHqNMX2oLdHe7r5tP/XkqY74PuL+tsB0BS+4QckRfiBpAg/kBThB5Ii/EBShB9Iyk0ep7XNQWFgwCKiq3OxGfmBpAg/kBThB5Ii/EBShB9IivADSRF+IKmmp+h+W9L/Trm/sFo2jIa1t2HtS6K3XtXZ21Xdrtjol3w+9OT2vmG9tt+w9jasfUn01qu2emO3H0iK8ANJtR3+0Zafv2RYexvWviR661UrvbX6nh9Ae9oe+QG0pJXw215j+w+2X7f9YBs9zMT2mO391czDrU4xVk2DNm77wJRll9l+1vafqt/TTpPWUm9DMXNzYWbpVl+7YZvxuvHdftsXSPqjpNskHZH0oqR7IuL3jTYyA9tjkkYiovVjwrZXSXpf0o8jYnm17DFJ70TEpuoP54KIeGBIentE0vttz9xcTSizZOrM0pLukvQVtfjaFfq6Wy28bm2M/NdLej0i/hwRpyX9VNLaFvoYehGxV9I75y1eK2lbdXubJv/zNG6G3oZCRByNiJer2ycknZtZutXXrtBXK9oI/5WSDk+5f0TDNeV3SPql7Zdsb2i7mWlcUU2bfm769EUt93O+jjM3N+m8maWH5rXrZcbrurUR/ukuMTRMhxxujIjPSfpXSV+vdm/Rna5mbm7KNDNLD4VeZ7yuWxvhPyJp6ZT7n5T0Vgt9TCsi3qp+j0vapeGbffjYuUlSq9/jLffzN8M0c/N0M0trCF67YZrxuo3wvyhpme1P2f6YpHWSdrfQx4fYvqT6IEa2L5G0WsM3+/BuSeur2+slPdNiL39nWGZunmlmabX82g3bjNetfMmnOpTxH5IukLQ1Ir7deBPTsP2PmhztpckzHre32ZvtJyXdpMmzvo5J+pakpyXtkPQPkg5J+lJENP7B2wy93aTJXde/zdx87j12w739i6T/lrRf0tlq8cOafH/d2mtX6OsetfC68Q0/ICm+4QckRfiBpAg/kBThB5Ii/EBShB9IivADSRF+IKn/B7v+1FCTlqkdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(images[1].numpy().squeeze(),cmap='Greys_r');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building networks with Pytorch\n",
    "\n",
    "Here we'll use PyTroch to build a simple feedforward network to classify the MNIST images. That is, the network will receive a digit image as input and predict the digit in the images. \n",
    "\n",
    "### Network summary\n",
    "* inputLayer(784) -> HiddenLayer1(128) -> HiddenLayer2(64) -> OutputLayer(10) -> LossLayer(cross entropy)\n",
    "\n",
    "\n",
    "\n",
    "To build a neural network with PyTorch, we use the `torch.nn` module. The network itself is **class inheriting** from `torch.nn.Module`. We define each of the operations separately, like `nn.Linear(784, 128)` for a fully connected linear layer with 784 inputs and 128 units.\n",
    "\n",
    "The class needs to inculde a `forward` method that implements the forward pass through the network. In this method, we pass some input tensor `x` through each of the operations we defined earlier. The `torch.nn` module also has functional equivalent for things like ReLUs in `torch.nn.functional`. This module is uaually imported as `F`. THen to use ReLU activation on some layer (which is just tensor),we'd do `F.relu(x)`.\n",
    "<br>$tanh(x)=\\frac{2}{1 + e^{-2x}}-1$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Network(\n",
       "  (fc1): Linear(in_features=784, out_features=128, bias=True)\n",
       "  (fc2): Linear(in_features=128, out_features=64, bias=True)\n",
       "  (fc3): Linear(in_features=64, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Network(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__() ## calls the init method of nn.Module class\n",
    "        \n",
    "        ## Defining architecture of our NN\n",
    "        self.fc1 = nn.Linear(784,128)\n",
    "        self.fc2 = nn.Linear(128,64)\n",
    "        self.fc3 = nn.Linear(64,10)\n",
    "        \n",
    "    def forward(self,x): ## x is a PyTorch tensor\n",
    "        x = self.fc1(x)\n",
    "        x = F.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        x = F.relu(x)\n",
    "        x = self.fc3(x)\n",
    "        x = F.softmax(x,dim=1) ## x dimension (batch_size,output_layer)\n",
    "        \n",
    "        return x\n",
    "    \n",
    "model = Network()\n",
    "model \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initializing weights and biases \n",
    "\n",
    "The weight and such are automatically initialized of us, but it's possible to customize how they are intitialized. The weights and biases are tensors attached to the layer we defined, we can get them with `model.fc1.weight` for instance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameter containing:\n",
      "1.00000e-02 *\n",
      " 2.0380 -0.9702 -0.6042  ...  -3.3627  3.3756 -3.1681\n",
      " 3.4554 -1.8584  2.0858  ...   1.7871  3.4589  3.0426\n",
      "-2.0556  1.2204  1.7049  ...  -0.2535  3.1532 -0.8945\n",
      "          ...             ⋱             ...          \n",
      "-0.6080  2.8535  2.5105  ...  -1.3106 -0.2657 -1.9641\n",
      " 0.6631  3.4396  1.8102  ...   0.2219 -3.0908 -2.1933\n",
      "-0.3851  0.7717  1.9081  ...  -3.2981  3.4775 -1.9514\n",
      "[torch.FloatTensor of size 128x784]\n",
      "\n",
      "Parameter containing:\n",
      "1.00000e-02 *\n",
      "  1.5002\n",
      "  2.1226\n",
      "  3.4332\n",
      " -1.9844\n",
      " -2.4903\n",
      " -2.2037\n",
      " -2.3775\n",
      " -1.4231\n",
      " -1.3111\n",
      " -0.6829\n",
      " -3.5619\n",
      "  0.3586\n",
      " -3.5230\n",
      " -2.9998\n",
      " -0.9786\n",
      "  0.6031\n",
      "  1.8653\n",
      "  2.6148\n",
      "  1.4130\n",
      " -2.4119\n",
      "  3.5293\n",
      "  0.9330\n",
      " -1.5652\n",
      " -0.6715\n",
      "  1.3817\n",
      "  1.7881\n",
      "  0.8383\n",
      " -3.3728\n",
      " -2.0068\n",
      "  1.8602\n",
      "  0.3209\n",
      " -0.4366\n",
      " -1.9947\n",
      "  0.5025\n",
      "  2.2558\n",
      "  2.5077\n",
      " -2.2163\n",
      "  1.7415\n",
      "  3.5629\n",
      " -2.9644\n",
      " -1.4071\n",
      "  1.0051\n",
      "  3.2749\n",
      " -0.6647\n",
      " -2.2725\n",
      "  3.3667\n",
      "  3.5495\n",
      "  2.2987\n",
      " -3.3816\n",
      " -1.1975\n",
      " -1.9135\n",
      " -2.1939\n",
      " -3.1940\n",
      " -2.6132\n",
      " -0.4266\n",
      " -1.7732\n",
      "  1.0596\n",
      "  1.2387\n",
      " -2.7693\n",
      " -1.4323\n",
      "  2.8941\n",
      " -2.3335\n",
      " -3.3372\n",
      "  2.4929\n",
      " -2.9799\n",
      "  0.4054\n",
      " -0.1961\n",
      "  0.1414\n",
      " -1.2943\n",
      "  0.5288\n",
      "  2.8554\n",
      "  2.6912\n",
      " -2.9487\n",
      " -1.1784\n",
      " -1.5017\n",
      "  0.8349\n",
      " -1.2940\n",
      "  2.1595\n",
      " -1.7891\n",
      " -2.9663\n",
      " -0.3310\n",
      "  3.3682\n",
      " -0.3583\n",
      " -1.9143\n",
      "  2.8665\n",
      "  2.8360\n",
      "  0.4677\n",
      " -1.5841\n",
      " -0.2131\n",
      "  0.7874\n",
      "  2.8704\n",
      "  0.0699\n",
      " -0.4338\n",
      "  0.4080\n",
      " -1.5466\n",
      " -1.9285\n",
      "  1.5572\n",
      "  0.7658\n",
      "  0.6844\n",
      "  3.3350\n",
      "  3.3034\n",
      "  2.3234\n",
      " -2.2628\n",
      " -0.9898\n",
      " -2.1181\n",
      "  1.2798\n",
      "  0.4476\n",
      "  0.1301\n",
      "  1.4005\n",
      " -2.2982\n",
      " -2.2027\n",
      " -3.3535\n",
      " -3.0150\n",
      " -0.7325\n",
      " -0.7818\n",
      " -2.7029\n",
      "  0.8196\n",
      " -2.7412\n",
      "  1.6572\n",
      " -0.3148\n",
      " -3.3405\n",
      " -3.4581\n",
      "  1.9011\n",
      " -1.1338\n",
      " -0.2256\n",
      " -1.1221\n",
      "  1.6591\n",
      " -1.0454\n",
      "[torch.FloatTensor of size 128]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(model.fc1.weight)\n",
    "print(model.fc1.bias)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "-1.4444e-01  2.1468e-01  2.5262e-02  ...  -1.2799e-01 -4.5069e-02 -4.1398e-02\n",
       "-1.5676e-01  1.1333e-01 -7.2867e-02  ...   2.6565e-01  1.3103e-01 -1.1353e-02\n",
       "-1.6840e-01 -1.4185e-01  1.6091e-01  ...   6.1927e-02 -4.3640e-03  5.9997e-02\n",
       "                ...                   ⋱                   ...                \n",
       " 2.0832e-01  2.5102e-02 -1.2298e-01  ...  -3.4854e-02 -9.7913e-02  2.0798e-01\n",
       " 9.0667e-02  7.2290e-02 -6.2925e-02  ...  -1.7155e-01 -4.7951e-02  8.6481e-02\n",
       " 1.1777e-01  1.1152e-01 -1.9407e-01  ...  -8.3053e-03  1.5506e-02 -2.9949e-02\n",
       "[torch.FloatTensor of size 128x784]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fc1.weight.data.normal_(std=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.0922\n",
       "-0.0768\n",
       " 0.0460\n",
       "-0.0001\n",
       " 0.0463\n",
       "-0.0021\n",
       " 0.0358\n",
       "-0.0577\n",
       "-0.0378\n",
       "-0.0726\n",
       "-0.0806\n",
       " 0.1048\n",
       "-0.0804\n",
       " 0.0425\n",
       "-0.0332\n",
       "-0.0551\n",
       "-0.2615\n",
       "-0.1452\n",
       " 0.0209\n",
       " 0.0612\n",
       "-0.1499\n",
       "-0.0161\n",
       "-0.0437\n",
       "-0.0264\n",
       "-0.0248\n",
       "-0.0871\n",
       " 0.0978\n",
       "-0.0055\n",
       "-0.0223\n",
       "-0.1226\n",
       "-0.1224\n",
       "-0.0278\n",
       " 0.1323\n",
       "-0.0517\n",
       "-0.0352\n",
       "-0.0408\n",
       " 0.1584\n",
       " 0.1351\n",
       " 0.0506\n",
       " 0.1626\n",
       " 0.0689\n",
       " 0.0930\n",
       " 0.0315\n",
       "-0.0174\n",
       "-0.2004\n",
       " 0.0749\n",
       " 0.0286\n",
       " 0.0813\n",
       " 0.1335\n",
       " 0.0349\n",
       "-0.1159\n",
       "-0.0224\n",
       "-0.1096\n",
       "-0.0250\n",
       "-0.0088\n",
       "-0.1269\n",
       "-0.2541\n",
       " 0.1841\n",
       " 0.1876\n",
       " 0.0636\n",
       " 0.1291\n",
       " 0.1518\n",
       "-0.0509\n",
       " 0.0713\n",
       " 0.0746\n",
       " 0.1796\n",
       " 0.1056\n",
       "-0.1231\n",
       "-0.0601\n",
       "-0.1616\n",
       " 0.0893\n",
       "-0.0255\n",
       " 0.0420\n",
       "-0.1269\n",
       " 0.0735\n",
       " 0.0427\n",
       "-0.0081\n",
       "-0.0686\n",
       " 0.0126\n",
       " 0.0181\n",
       "-0.1094\n",
       "-0.0862\n",
       " 0.0426\n",
       " 0.1013\n",
       " 0.0799\n",
       " 0.0796\n",
       " 0.0537\n",
       "-0.1756\n",
       " 0.0828\n",
       " 0.0480\n",
       "-0.0553\n",
       " 0.1216\n",
       "-0.2561\n",
       "-0.0195\n",
       " 0.0262\n",
       " 0.0153\n",
       " 0.0164\n",
       " 0.0830\n",
       "-0.0494\n",
       " 0.0443\n",
       "-0.0269\n",
       " 0.0362\n",
       " 0.0360\n",
       " 0.1588\n",
       "-0.0180\n",
       "-0.0975\n",
       " 0.2635\n",
       "-0.0461\n",
       " 0.0398\n",
       " 0.0757\n",
       "-0.0587\n",
       " 0.1534\n",
       " 0.0938\n",
       " 0.0608\n",
       " 0.1282\n",
       " 0.1200\n",
       " 0.0782\n",
       "-0.0333\n",
       "-0.0404\n",
       " 0.0132\n",
       "-0.0093\n",
       "-0.0178\n",
       " 0.0601\n",
       " 0.1256\n",
       " 0.0698\n",
       " 0.2388\n",
       "-0.0266\n",
       " 0.0448\n",
       "[torch.FloatTensor of size 128]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fc1.bias.data.normal_(std=0.1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Forward pass\n",
    "\n",
    "Now that we have a network, let's see what happens when we pass in an image. This is called forward pass. We're going to covert the image data into a tensor, then pass it through the operation defined by the network architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0845,  0.1116,  0.0993,  0.1095,  0.0841,  0.1077,  0.1070,\n",
       "          0.1104,  0.1043,  0.0816]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images, labels = next(iter(trainloader)) ## trainloader is a generator\n",
    "\n",
    "images.resize_(64,1,784)\n",
    "# print(images.size())\n",
    "# images.resize_(,1,784)\n",
    "ps = model.forward(images[0,:])\n",
    "ps\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch provides a convient way to build networks like where a tensor is passed sequentially through operations, `nn.Sequential`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Linear(in_features=784, out_features=128, bias=True)\n",
      "  (1): ReLU()\n",
      "  (2): Linear(in_features=128, out_features=64, bias=True)\n",
      "  (3): ReLU()\n",
      "  (4): Linear(in_features=64, out_features=10, bias=True)\n",
      "  (5): Softmax()\n",
      ")\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0816,  0.0966,  0.1195,  0.0930,  0.0960,  0.0979,  0.0959,\n",
       "          0.1202,  0.1154,  0.0839]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_size = 784\n",
    "hidden_size = [128, 64]\n",
    "output_size = 10\n",
    "model = nn.Sequential(nn.Linear(input_size,hidden_size[0]),nn.ReLU(),\n",
    "                     nn.Linear(hidden_size[0],hidden_size[1]),nn.ReLU(),\n",
    "                      nn.Linear(hidden_size[1],output_size),nn.Softmax(dim=1))\n",
    "print(model)\n",
    "\n",
    "## Forward pass through the network and display the output\n",
    "images, labels = next(iter(trainloader))\n",
    "images.resize_(64,1,784)\n",
    "ps = model.forward(images[0,:])\n",
    "ps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also pass in an `OrderedDict` to name the individual layers and operations. Note that a dictionary keys must be unique, so each operation must have a different name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (fc1): Linear(in_features=784, out_features=128, bias=True)\n",
       "  (relu1): ReLU()\n",
       "  (fc2): Linear(in_features=128, out_features=64, bias=True)\n",
       "  (relu2): ReLU()\n",
       "  (fc3): Linear(in_features=64, out_features=10, bias=True)\n",
       "  (softmax): Softmax()\n",
       ")"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import OrderedDict\n",
    "model = nn.Sequential(OrderedDict([\n",
    "    ('fc1',nn.Linear(input_size,hidden_size[0])),\n",
    "    ('relu1',nn.ReLU()),\n",
    "    ('fc2',nn.Linear(hidden_size[0],hidden_size[1])),\n",
    "    ('relu2',nn.ReLU()),\n",
    "    ('fc3',nn.Linear(hidden_size[1],output_size)),\n",
    "    ('softmax',nn.Softmax(dim=1))\n",
    "]))\n",
    "model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Exercise**: Build a network to classify the MNIST images with three hidden layers. Use 400 units in the first hidden layer, 200 units in the second layer, and 100 units in the third layer. Each hidden layer should have a ReLU activation function, and use softmax on the output layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0979,  0.0946,  0.0980,  0.0974,  0.0942,  0.0998,  0.1046,\n",
       "          0.1053,  0.1034,  0.1046]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_size = 784\n",
    "output_size = 10\n",
    "hidden_size = [400,200,100]\n",
    "\n",
    "model = nn.Sequential(OrderedDict([\n",
    "    ('fc1',nn.Linear(input_size,hidden_size[0])),\n",
    "    ('relu1',nn.ReLU()),\n",
    "    ('fc2',nn.Linear(hidden_size[0],hidden_size[1])),\n",
    "    ('relu2',nn.ReLU()),\n",
    "    ('fc3',nn.Linear(hidden_size[1],hidden_size[2])),\n",
    "    ('relu3',nn.ReLU()),\n",
    "    ('fc4',nn.Linear(hidden_size[2],output_size)),\n",
    "    ('softmax',nn.Softmax(dim=1))\n",
    "]))\n",
    "model \n",
    "\n",
    "images.resize_(images.shape[0], 1, 784)\n",
    "ps = model.forward(images[0,:])\n",
    "ps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from torch import nn, optim\n",
    "from torch.autograd import Variable\n",
    "\n",
    "\n",
    "def test_network(net, trainloader):\n",
    "\n",
    "    criterion = nn.MSELoss()\n",
    "    optimizer = optim.Adam(net.parameters(), lr=0.001)\n",
    "\n",
    "    dataiter = iter(trainloader)\n",
    "    images, labels = dataiter.next()\n",
    "\n",
    "    # Create Variables for the inputs and targets\n",
    "    inputs = Variable(images)\n",
    "    targets = Variable(images)\n",
    "\n",
    "    # Clear the gradients from all Variables\n",
    "    optimizer.zero_grad()\n",
    "\n",
    "    # Forward pass, then backward pass, then update weights\n",
    "    output = net.forward(inputs)\n",
    "    loss = criterion(output, targets)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    return True\n",
    "\n",
    "\n",
    "def imshow(image, ax=None, title=None, normalize=True):\n",
    "    \"\"\"Imshow for Tensor.\"\"\"\n",
    "    if ax is None:\n",
    "        fig, ax = plt.subplots()\n",
    "    image = image.numpy().transpose((1, 2, 0))\n",
    "\n",
    "    if normalize:\n",
    "        mean = np.array([0.485, 0.456, 0.406])\n",
    "        std = np.array([0.229, 0.224, 0.225])\n",
    "        image = std * image + mean\n",
    "        image = np.clip(image, 0, 1)\n",
    "\n",
    "    ax.imshow(image)\n",
    "    ax.spines['top'].set_visible(False)\n",
    "    ax.spines['right'].set_visible(False)\n",
    "    ax.spines['left'].set_visible(False)\n",
    "    ax.spines['bottom'].set_visible(False)\n",
    "    ax.tick_params(axis='both', length=0)\n",
    "    ax.set_xticklabels('')\n",
    "    ax.set_yticklabels('')\n",
    "\n",
    "    return ax\n",
    "\n",
    "\n",
    "def view_recon(img, recon):\n",
    "    ''' Function for displaying an image (as a PyTorch Tensor) and its\n",
    "        reconstruction also a PyTorch Tensor\n",
    "    '''\n",
    "\n",
    "    fig, axes = plt.subplots(ncols=2, sharex=True, sharey=True)\n",
    "    axes[0].imshow(img.numpy().squeeze())\n",
    "    axes[1].imshow(recon.data.numpy().squeeze())\n",
    "    for ax in axes:\n",
    "        ax.axis('off')\n",
    "        ax.set_adjustable('box-forced')\n",
    "\n",
    "def view_classify(img, ps, version=\"MNIST\"):\n",
    "    ''' Function for viewing an image and it's predicted classes.\n",
    "    '''\n",
    "    ps = ps.data.numpy().squeeze()\n",
    "\n",
    "    fig, (ax1, ax2) = plt.subplots(figsize=(6,9), ncols=2)\n",
    "    ax1.imshow(img.resize_(1, 28, 28).numpy().squeeze())\n",
    "    ax1.axis('off')\n",
    "    ax2.barh(np.arange(10), ps)\n",
    "    ax2.set_aspect(0.1)\n",
    "    ax2.set_yticks(np.arange(10))\n",
    "    if version == \"MNIST\":\n",
    "        ax2.set_yticklabels(np.arange(10))\n",
    "    elif version == \"Fashion\":\n",
    "        ax2.set_yticklabels(['T-shirt/top',\n",
    "                            'Trouser',\n",
    "                            'Pullover',\n",
    "                            'Dress',\n",
    "                            'Coat',\n",
    "                            'Sandal',\n",
    "                            'Shirt',\n",
    "                            'Sneaker',\n",
    "                            'Bag',\n",
    "                            'Ankle Boot'], size='small');\n",
    "    ax2.set_title('Class Probability')\n",
    "    ax2.set_xlim(0, 1.1)\n",
    "\n",
    "    plt.tight_layout()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing we need to do for training is define our loss function. In Pytorch, we'll usually see this as `criterion`. Here we're using softmax ouput, so we want to use `criterion = nn.CrossEntropyLoss()` as our loss. Later when training, we use `loss = Criterion(output, targets)` to calculate the actual loss.\n",
    "\n",
    "We also need to define the optimizer we're using, SGD or Adam, or something along those lines. Here I'll just use SGD with `torch.optim.SGD`, passing in the network parameters and the learning rate.\n",
    "\n",
    "## Autograd\n",
    "Torch provide a module, `autograd`, for automatically calculating the gradient of tensors. It does this by keeping track of operations performed on tensor. To make sure PyTorch keeps track of operations on a tensor and calculate the gradients, we need to set `require_grad` on a tensor. We can do this at creation with the `require_grad` keyword, or at any time with `x.requires_grad_(True)`.\n",
    "\n",
    "\n",
    "We can turn off gradients for a block of code with the `torch.no_grad()` content:\n",
    "```python\n",
    "x = torch.zeros(1, requires_grad=True)\n",
    ">>> with torch.no_grad():\n",
    "...     y = x * 2\n",
    ">>> y.requires_grad\n",
    "False\n",
    "```\n",
    "\n",
    "Also, we can trun on or off gradients altogether with `torch.set_grad_enabled(True|False)`\n",
    "\n",
    "The gradient are computed with respect to some variable `z` with `z.backward()`. This does a backward pass through the operations that created `z`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.8221, -0.0212, -0.2925],\n",
       "        [-0.1392,  1.3154,  0.7319],\n",
       "        [-1.6575,  0.0801,  1.0192]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.randn(3,3, requires_grad=True)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 3.3201,  0.0004,  0.0855],\n",
       "        [ 0.0194,  1.7302,  0.5357],\n",
       "        [ 2.7473,  0.0064,  1.0387]])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = x**2\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<PowBackward0 object at 0x118b3a710>\n"
     ]
    }
   ],
   "source": [
    "print(y.grad_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.0538)\n"
     ]
    }
   ],
   "source": [
    "z = y.mean()\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the gradient for `x` and `y` but they are empty currently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None None\n"
     ]
    }
   ],
   "source": [
    "print(x.grad,y.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate the gradients, you need to run the `.backward` method on a Variable, `z` for example. This will calculate the gradient for `z` with respect to `x`\n",
    "\n",
    "$$\n",
    "\\frac{\\partial z}{\\partial x} = \\frac{\\partial}{\\partial x}\\left[\\frac{1}{n}\\sum_i^n x_i^2\\right] = \\frac{x}{2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.4049, -0.0047, -0.0650],\n",
      "        [-0.0309,  0.2923,  0.1626],\n",
      "        [-0.3683,  0.0178,  0.2265]])\n",
      "tensor([[ 0.4049, -0.0047, -0.0650],\n",
      "        [-0.0309,  0.2923,  0.1626],\n",
      "        [-0.3683,  0.0178,  0.2265]])\n"
     ]
    }
   ],
   "source": [
    "z.backward()\n",
    "print(x.grad)\n",
    "print(x*(2/9))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These gradients calculations are particularly useful for neural networks. For training we need the gradients of the weights with respect to the cost. With PyTorch, we run data forward through the network to calculate the cost, then, go backwards to calculate the gradients with respect to the cost. Once we have the gradients we can make a gradient descent step. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll build a network with `nn.Sequential` here. Only difference from the last part is I'm not actually using softmax on the output, but instead just using the raw output from the lost layer. This is because the output from softmax is probability distribution. Often, the ouptut will have values really closed to zero or really close to close to one. Due to inaccuracies with representation numbers as floating points, computation with a softmax output can lose accuracy and become unstable. To get around this, we'll use the raw output, called **logits**, to calcualte the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (fc1): Linear(in_features=784, out_features=128, bias=True)\n",
       "  (relu1): ReLU()\n",
       "  (fc2): Linear(in_features=128, out_features=64, bias=True)\n",
       "  (relu2): ReLU()\n",
       "  (logits): Linear(in_features=64, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Hyperparameters for our network\n",
    "input_size = 784\n",
    "hidden_size = [128,64]\n",
    "output_size = 10\n",
    "\n",
    "model = nn.Sequential(OrderedDict([\n",
    "    ('fc1',nn.Linear(input_size,hidden_size[0])),\n",
    "    ('relu1',nn.ReLU()),\n",
    "    ('fc2',nn.Linear(hidden_size[0],hidden_size[1])),\n",
    "    ('relu2',nn.ReLU()),\n",
    "    ('logits',nn.Linear(hidden_size[1],output_size))\n",
    "]))\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the network!\n",
    "The first thing we need to do for training is define our loss function. In Pytorch, we'll usually see this as `criterion`. Here we're using softmax output, so we want to use `criterion = nn.CrossEntropyLoss()` as our loss. Later when training, we use `loss = criterion(output, targets)` to calculate the actual loss.\n",
    "\n",
    "We also need to define optimizer we're using, SGD or Adam, or something along those lines. Here I'll just use SGD with `torch.optim.SGD`, passing in the network parameters and the learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(),lr=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's consider just one learning step before looping through all the data.\n",
    "The general process with PyTorch:\n",
    "* Make a forward pass through the network to get the logits\n",
    "* Use the logits to calculate the loss\n",
    "* Perform a backward pass through the network with `loss.backward()` to calculate the gradients\n",
    "* Take a step with the the optimizer to update weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial weights -  Parameter containing:\n",
      "tensor([[-2.7111e-02,  1.3836e-02,  2.5886e-02,  ..., -1.4867e-02,\n",
      "         -9.8428e-03, -6.2038e-03],\n",
      "        [-3.0883e-02,  1.7498e-02,  3.0558e-02,  ..., -2.0247e-02,\n",
      "         -1.9177e-02, -1.9072e-02],\n",
      "        [ 8.7670e-03,  3.5633e-02,  9.7913e-03,  ..., -1.8948e-02,\n",
      "         -2.0131e-03,  8.2791e-03],\n",
      "        ...,\n",
      "        [ 4.2191e-04, -3.3190e-02,  6.7878e-03,  ...,  3.1593e-02,\n",
      "          6.6738e-03,  2.1985e-02],\n",
      "        [ 3.7708e-03,  1.5728e-02, -8.7492e-05,  ...,  3.3355e-02,\n",
      "          7.0147e-03, -2.1429e-02],\n",
      "        [-7.7849e-03, -3.3800e-02, -2.5990e-02,  ...,  1.4680e-02,\n",
      "         -2.1666e-02, -3.1533e-02]])\n",
      "Gradient -  tensor(1.00000e-02 *\n",
      "       [[-0.0120, -0.0120, -0.0120,  ..., -0.0120, -0.0120, -0.0120],\n",
      "        [ 0.0084,  0.0084,  0.0084,  ...,  0.0084,  0.0084,  0.0084],\n",
      "        [ 0.0970,  0.0970,  0.0970,  ...,  0.0970,  0.0970,  0.0970],\n",
      "        ...,\n",
      "        [-0.0852, -0.0852, -0.0852,  ..., -0.0852, -0.0852, -0.0852],\n",
      "        [ 0.0329,  0.0329,  0.0329,  ...,  0.0329,  0.0329,  0.0329],\n",
      "        [-0.2054, -0.2054, -0.2054,  ..., -0.2054, -0.2054, -0.2054]])\n"
     ]
    }
   ],
   "source": [
    "print('Initial weights - ', model.fc1.weight)\n",
    "\n",
    "images, labels = next(iter(trainloader))\n",
    "images.resize_(64,784)\n",
    "\n",
    "# Clear the gradients, do this because gradients are accumulated \n",
    "optimizer.zero_grad()\n",
    "\n",
    "# Forward pass, then backward pass, then update weights \n",
    "output = model.forward(images)\n",
    "loss = criterion(output,labels)\n",
    "loss.backward()\n",
    "print('Gradient - ', model.fc1.weight.grad)\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated weights -  Parameter containing:\n",
      "tensor([[-2.7109e-02,  1.3837e-02,  2.5887e-02,  ..., -1.4866e-02,\n",
      "         -9.8416e-03, -6.2026e-03],\n",
      "        [-3.0884e-02,  1.7497e-02,  3.0557e-02,  ..., -2.0248e-02,\n",
      "         -1.9178e-02, -1.9073e-02],\n",
      "        [ 8.7573e-03,  3.5623e-02,  9.7816e-03,  ..., -1.8958e-02,\n",
      "         -2.0228e-03,  8.2694e-03],\n",
      "        ...,\n",
      "        [ 4.3043e-04, -3.3182e-02,  6.7964e-03,  ...,  3.1601e-02,\n",
      "          6.6823e-03,  2.1993e-02],\n",
      "        [ 3.7675e-03,  1.5725e-02, -9.0786e-05,  ...,  3.3352e-02,\n",
      "          7.0114e-03, -2.1432e-02],\n",
      "        [-7.7643e-03, -3.3780e-02, -2.5969e-02,  ...,  1.4701e-02,\n",
      "         -2.1645e-02, -3.1512e-02]])\n"
     ]
    }
   ],
   "source": [
    "print('Updated weights - ', model.fc1.weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training for real\n",
    "\n",
    "Now we'll put this algorithm into a loop so we can go through all the images. This is fairly straightforward. We'll loop through the mini-batches in our dataset, pass the data through the network to calculate the losses, get the gradients, then run the optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = optim.SGD(model.parameters(),lr=0.003)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/3... Loss: 2.2717\n",
      "Epoch: 1/3... Loss: 2.2491\n",
      "Epoch: 1/3... Loss: 2.2272\n",
      "Epoch: 1/3... Loss: 2.1962\n",
      "Epoch: 1/3... Loss: 2.1744\n",
      "Epoch: 1/3... Loss: 2.1394\n",
      "Epoch: 1/3... Loss: 2.1013\n",
      "Epoch: 1/3... Loss: 2.0586\n",
      "Epoch: 1/3... Loss: 2.0268\n",
      "Epoch: 1/3... Loss: 1.9713\n",
      "Epoch: 1/3... Loss: 1.9392\n",
      "Epoch: 1/3... Loss: 1.8778\n",
      "Epoch: 1/3... Loss: 1.8074\n",
      "Epoch: 1/3... Loss: 1.7575\n",
      "Epoch: 1/3... Loss: 1.6892\n",
      "Epoch: 1/3... Loss: 1.6108\n",
      "Epoch: 1/3... Loss: 1.5460\n",
      "Epoch: 1/3... Loss: 1.4690\n",
      "Epoch: 1/3... Loss: 1.3896\n",
      "Epoch: 1/3... Loss: 1.3345\n",
      "Epoch: 1/3... Loss: 1.2560\n",
      "Epoch: 1/3... Loss: 1.1769\n",
      "Epoch: 1/3... Loss: 1.1698\n",
      "Epoch: 2/3... Loss: 0.6008\n",
      "Epoch: 2/3... Loss: 1.0611\n",
      "Epoch: 2/3... Loss: 1.0194\n",
      "Epoch: 2/3... Loss: 0.9602\n",
      "Epoch: 2/3... Loss: 0.9266\n",
      "Epoch: 2/3... Loss: 0.8932\n",
      "Epoch: 2/3... Loss: 0.8745\n",
      "Epoch: 2/3... Loss: 0.8479\n",
      "Epoch: 2/3... Loss: 0.8158\n",
      "Epoch: 2/3... Loss: 0.7938\n",
      "Epoch: 2/3... Loss: 0.7799\n",
      "Epoch: 2/3... Loss: 0.7713\n",
      "Epoch: 2/3... Loss: 0.7248\n",
      "Epoch: 2/3... Loss: 0.7046\n",
      "Epoch: 2/3... Loss: 0.7017\n",
      "Epoch: 2/3... Loss: 0.6802\n",
      "Epoch: 2/3... Loss: 0.6511\n",
      "Epoch: 2/3... Loss: 0.6280\n",
      "Epoch: 2/3... Loss: 0.6647\n",
      "Epoch: 2/3... Loss: 0.6104\n",
      "Epoch: 2/3... Loss: 0.6087\n",
      "Epoch: 2/3... Loss: 0.6178\n",
      "Epoch: 2/3... Loss: 0.5788\n",
      "Epoch: 3/3... Loss: 0.0670\n",
      "Epoch: 3/3... Loss: 0.5990\n",
      "Epoch: 3/3... Loss: 0.5627\n",
      "Epoch: 3/3... Loss: 0.5416\n",
      "Epoch: 3/3... Loss: 0.5409\n",
      "Epoch: 3/3... Loss: 0.5601\n",
      "Epoch: 3/3... Loss: 0.5349\n",
      "Epoch: 3/3... Loss: 0.5524\n",
      "Epoch: 3/3... Loss: 0.5215\n",
      "Epoch: 3/3... Loss: 0.5358\n",
      "Epoch: 3/3... Loss: 0.5285\n",
      "Epoch: 3/3... Loss: 0.5127\n",
      "Epoch: 3/3... Loss: 0.5212\n",
      "Epoch: 3/3... Loss: 0.5109\n",
      "Epoch: 3/3... Loss: 0.4893\n",
      "Epoch: 3/3... Loss: 0.4899\n",
      "Epoch: 3/3... Loss: 0.4530\n",
      "Epoch: 3/3... Loss: 0.4650\n",
      "Epoch: 3/3... Loss: 0.4692\n",
      "Epoch: 3/3... Loss: 0.4611\n",
      "Epoch: 3/3... Loss: 0.4539\n",
      "Epoch: 3/3... Loss: 0.4625\n",
      "Epoch: 3/3... Loss: 0.4732\n",
      "Epoch: 3/3... Loss: 0.4345\n"
     ]
    }
   ],
   "source": [
    "epoch = 3\n",
    "print_every = 40\n",
    "steps = 0\n",
    "for e in range(epoch):\n",
    "    running_loss = 0\n",
    "    for images,labels in iter(trainloader):\n",
    "        steps+=1\n",
    "        ## Flatten the MNIST images into a 784 long vector\n",
    "        images.resize_(images.size()[0],784)\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        ## Forward and backward passes\n",
    "        output = model.forward(images)\n",
    "        loss = criterion(output,labels)\n",
    "        loss.backward() ##calculating gradient\n",
    "        optimizer.step() ## gradient decent \n",
    "        \n",
    "        running_loss += loss.item()\n",
    "        \n",
    "        if steps % print_every == 0:\n",
    "            print('Epoch: {}/{}...'.format(e+1,epoch),\n",
    "                 'Loss: {:.4f}'.format(running_loss/print_every))\n",
    "            \n",
    "            running_loss = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "with the network trained, we can check out it's predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "images, labels = next(iter(trainloader))\n",
    "img = images[0].view(1,784)\n",
    "## Turn off gradients to speed up this part\n",
    "with torch.no_grad():\n",
    "    logits = model.forward(img)\n",
    "    \n",
    "#Output the network are logits, neeed to take softmax for proabailities\n",
    "ps = F.softmax(logits,dim=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAADhCAYAAACdkiHQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFqdJREFUeJzt3XuclVW9x/HvlwEhMhEBy7g4auQlOV4iyzxaKpaBSWUXNDvWqzRL7WKnsux00VMvj5Vpr26SmmYlCWblraQMrQwUvKGggogOkOIFUITAGX/nj/1Q2+l5hgFm9lrDfN6v136x57eetfdvP8r+zVrP4lmOCAEAkJs+qRMAAKAMBQoAkCUKFAAgSxQoAECWKFAAgCxRoAAAWaJAAWgI21+1/bPUeWwO25fa/t/N7Nvh57Z9n+03tz/W9ijbq203bVbSWwEKFIAuY/s427OLL9a/277B9n8myiVsP1fkstT2eTl+2UfEayJiRkn80YjYNiLaJMn2DNsfaXiCCVGgAHQJ26dLOl/SNyS9XNIoST+QNDFhWvtExLaSDpd0nKQT2x9gu2/Ds0KnUKAAbDHbgySdJemUiPhVRDwXEc9HxDUR8dmKPlNtP2Z7le1bbL+mrm287Xm2ny1GP/9dxIfavtb2SttP2/6z7Y1+j0XE/ZL+LGnv4nUW2/687XskPWe7r+09i1HKymLa7eh2LzPU9vQip5tt71yX7wW2W2w/Y3uO7YPb9R1g+5dF3zts71PXd7HtcSXnp7kYBfa1/XVJB0v6XjEi/J7t79v+drs+19j+1MbOR09BgQLQFQ6UNEDS1ZvQ5wZJoyXtKOkOST+va7tY0kcj4mWqFZWbivhnJC2RNEy1UdoXJW30fm2291LtC/7OuvCxkiZI2l6SJV0j6cYin9Mk/dz27nXHv1/S2ZKGSrqrXb63S9pX0g6SfiFpqu0Bde0TJU2ta/+17X4by3uDiDhTtQJ7ajHtd6qkyyQdu6FA2x6q2kjxis6+bu4oUAC6whBJT0ZEa2c7RMQlEfFsRKyT9FVJ+xQjMUl6XtJetreLiBURcUddfCdJOxcjtD9HxzcUvcP2CtWKz0WSflLX9t2IaImItZLeIGlbSedExPqIuEnStaoVsQ2ui4hbinzPlHSg7ZHFZ/lZRDwVEa0R8W1J/SXVF7c5ETEtIp6XdJ5qxfwNnT1XZSLiNkmrVCtKkjRJ0oyIeHxLXjcnFCgAXeEp1abAOnU9x3aT7XNsP2T7GUmLi6ahxZ/HSBov6ZFiOu3AIv5NSQsl3Wh7ke0zNvJW+0fE4IjYLSK+FBEv1LW11D1/paSWdu2PSBpednxErJb0dNFPtj9je34xXblS0qC6z9K+7wuqjQJfuZHcO+MySccXz4+XdHkXvGY2KFAAusLfJP1D0js6efxxqk17jVPty7y5iFuSIuL2iJio2nTbryVdWcSfjYjPRMSukt4u6XTbh2vz1I+8lkka2e561ihJS+t+Hrnhie1tVZuuW1Zcb/q8pPdKGhwR26s2snFF3z6SRhTvubn5bvAzSROLa1p7qnauthoUKABbLCJWSfqypO/bfoftgbb72X6b7XNLurxM0jrVRl4DVVv5J0myvY3t99seVEyJPSNpw1Lro2y/yrbr4m1d8BFmSXpO0ueKvN+sWgGcUnfMeNv/aXsb1a5FzYqIluKztEp6QlJf21+WtF2713+t7XcVI8xPFZ995ibm+LikXesDEbFEtetfl0u6qpiu3GpQoAB0iYg4T9Lpkr6k2pd1i6RTVf5b/U9Vm0JbKmme/v3L+gOSFhfTfyfrX9NYoyX9QdJq1UZtPyj7N0Sbkft6SUdLepukJ1VbHv9fxeq/DX4h6SuqTe29VrVFE5L0e9UWfDxYfKZ/6MXTh5L0G0nvk7Si+GzvKorvprhA0rttr7D93br4ZZLGaCub3pMks2EhAPRctg9Rbaqvud01tB6PERQA9FDFUvVPSrpoaytOEgUKAHok23tKWqnasvvzE6fTLZjiAwBkqaH3oDqiz3uohthqTH9hqjd+FIDNxRQfACBL3MUX6AGGDh0azc3NqdMAusScOXOejIhhGzuOAgX0AM3NzZo9e3bqNIAuYfuRzhzHFB8AIEsUKABAlihQAIAsUaAAAFmiQAEAskSBAgBkiQIF9ABzl65KnQLQcBQoAECWKFAAgCxRoIBEbH/S9r2277P9qdT5ALmhQAEJ2N5b0omSDpC0j6SjbI9OmxWQFwoUkMaekmZGxJqIaJV0s6R3Js4JyAoFCkjjXkmH2B5ie6Ck8ZJG1h9g+yTbs23PblvDKj70PtzNHEggIubb/j9J0yWtlnS3pNZ2x0yWNFmS+u80ms0+0eswggISiYiLI2L/iDhE0tOSFqTOCcgJIyggEds7RsRy26MkvUvSgalzAnJCgQLSucr2EEnPSzolIlakTgjICQUKSCQiDk6dA5AzrkEBALJEgQJ6gDHDB6VOAWg4ChQAIEsUKABAllgkAfQAc5euUvMZ16VOAz3E4nMmpE6hSzCCAgBkiQIFAMgSBQpIxPani72g7rV9he0BqXMCckKBAhKwPVzSJySNjYi9JTVJmpQ2KyAvFCggnb6SXmK7r6SBkpYlzgfICqv4cmGXhte/dWxllyFnPlwan7bbH7okJUlaF89Xth185idK44Mv/VuXvf/WKiKW2v6WpEclrZV0Y0TcmDgtICuMoIAEbA+WNFHSLpJeKemlto9vdwwbFqJXo0ABaYyT9HBEPBERz0v6laQ31h8QEZMjYmxEjG0ayK2O0PtQoIA0HpX0BtsDbVvS4ZLmJ84JyAoFCkggImZJmibpDklzVfu7ODlpUkBmWCQBJBIRX5H0ldR5ALliBAUAyBIjqAbqu9MrKtvmfW1kaXzhhAsr+6yN9aXxP6596aYlJuncxW8rjV86ekpln1EfXlAaf/bSTX57bMSY4YM0eyu5ASjQWYygAABZokABALJEgQIAZIkCBQDIEgUKAJAlVvF1g6bty29L89m/VN8L9OABraXxj7S8qbLPXZePKY3v+P1bO8iuXJ99tyuNt1zdv7LPeTv/ujR+8qtPqOzT9uBDm5bYVsr27pJ+WRfaVdKXI+L8RCkB2aFAAQlExAOS9pUk202Slkq6OmlSQGaY4gPSO1zSQxHxSOpEgJxQoID0Jkm6InUSQG4oUEBCtreRdLSkqSVt/9wP6oknnmh8ckBiFCggrbdJuiMiHm/fUL8f1LBhwxKkBqTFIonucHX5vfAOGlC9ffqBd00qjQ99//LKPjuu3PTVelUePKF8Fd9rt2mq7HPasnHlDcuf7IqUeotjxfQeUIoRFJCI7YGSjlBtN10A7TCCAhKJiDWShqTOA8gVIygAQJYoUACALFGgAABZokABALLEIonN1Oc/9qhsu3i3i0rjf/1H+VJuSRo8oXz79LZNS2uzDb3T5Q3vre5z44Lyc7Dbyru6ICMAvR0jKKAHmLt0VeoUgIajQAEAskSBAgBkiQIFJGJ7e9vTbN9ve77tA1PnBOSERRJAOhdI+l1EvLu4q/nA1AkBOaFAbab7T65ekbdjU/n3zBeXHdTBKz67hRltmTWvqFjFh25heztJh0j6oCRFxHpJ61PmBOSGKT4gjV0lPSHpJ7bvtH2R7RfdBr9+P6i2NaziQ+9DgQLS6Ctpf0k/jIj9JD0n6Yz6A+r3g2oaOChFjkBSFCggjSWSlkTErOLnaaoVLAAFChSQQEQ8JqnF9u5F6HBJ8xKmBGSHRRJAOqdJ+nmxgm+RpA8lzgfICgUKSCQi7pI0NnUeQK4oUJtpr28+Xtl2//h1pfGLRt5c2efH80aWxs/9w1GVffY8d0lpvHXpstL4M5NeX/lafzztm6XxKatHVfZ59VfLl8Y36ga3ALZuXIMCeoAxw1nFh96HAgUAyBIFCgCQJa5BAT3A3KWr1HzGdZ06dvE5E7o5G6AxGEEBALLECGoztT78SGXb595Uvk/6wnMHV/a57aALS+MnHvPDyj4zJvQrjZ90zYml8Qff84PK1/r40nGl8ZbjXlHZp23hQ5VtALClKFBAIrYXq3Yb+zZJrRHBv4kC6lCggLQOjYgnUycB5IhrUACALFGggHRC0o2259g+KXUyQG6Y4gPSOSgiltneUdJ02/dHxC0bGouidZIkNW03LFWOQDKMoIBEImJZ8edySVdLOqBdOxsWoldjBNUNWh9pKY03v688LknHHHpKafzTP/5FZZ8jX7KmNF61nPx3awdWvlbLCSNK420LF1T2weYrtnfvExHPFs/fIumsxGkBWaFAAWm8XNLVtqXa38NfRMTv0qYE5IUCBSQQEYsk7ZM6DyBnXIMCAGSJERTQA4wZPkizuQksehlGUACALDGCysQ2LStK43v06+guONWr8sqsbHtpZZtXr92k1wKA7sYICgCQJQoU0APMXboqdQpAw1GgAABZokABCdlusn2n7WtT5wLkhgIFpPVJSfNTJwHkiFV8DeS+1af7dVc9WBrfoU/17xD7fufU0ni8sfx6xd2vv7zytb505itL468+eUllH2wZ2yMkTZD0dUmnJ04HyA4jKCCd8yV9TtILqRMBckSBAhKwfZSk5RExp4NjTrI92/bstjWs4kPvQ4EC0jhI0tG2F0uaIukw2z+rP4D9oNDbUaCABCLiCxExIiKaJU2SdFNEHJ84LSArFCgAQJZYxQckFhEzJM1InAaQHQpUAz1w4b6VbdcNnVwa3/vWj1b2GfWtW0vjft2Y0viKX1XfEPbW8eeVxj+yy/sr+7Q+/EhlGwBsKab4AABZokABPcCY4aziQ+9DgQIAZIkCBQDIEoskgB5g7tJVaj7jusr2xedMaGA2QGNQoBpo1lsuqGz7e1uUxnf9/LOVfVor4nH73NL4628uv7msJD146MWl8YePH17ZZ+TZrOID0H2Y4gMAZIkCBSRge4Dt22zfbfs+219LnROQG6b4gDTWSTosIlbb7ifpL7ZviIiZqRMDckGBAhKIiJC0uvixX/EovxAJ9FJM8QGJ2G6yfZek5ZKmR8Ssdu3sB4VejQIFJBIRbRGxr6QRkg6wvXe7dvaDQq/GFF83aHrVLqXx/v5bZZ+W1qbSeOuixV2RkiRpp6u3qW48tDx84IR7KrssOXsLE4IkKSJW2p4h6UhJ9yZOB8gGIyggAdvDbG9fPH+JpHGS7k+bFZAXRlBAGjtJusx2k2q/KF4ZEdcmzgnICgUKSCAi7pG0X+o8gJwxxQcAyBIjKKAHGDN8kGZzQ1j0MhSobrDigJeXxrd1/8o+77vq46Xx3dR1NxYYNLOlsm3u+udL4+eNmF7Z59jdP1Aab3tg4aYlBgAlmOIDAGSJAgX0ABv2g+poTyhga0OBAgBkiQIFAMgSBQpIwPZI23+yPb/YD+qTqXMCcsMqPiCNVkmfiYg7bL9M0hzb0yNiXurEgFxQoLrBdgufK42vjfWVfV7Ysbqtq7QuXVbZ9q5bPlYaXzDuoso+yw8eVhofwjLzjYqIv0v6e/H8WdvzJQ2XRIECCkzxAYnZblbttkezOj4S6F0oUEBCtreVdJWkT0XEM+3a2LAQvRoFCkjEdj/VitPPI+JX7dvZsBC9HQUKSMC2JV0saX5EnJc6HyBHFCggjYMkfUDSYbbvKh7jUycF5IRVfN3htrml4dOWHFHZ5a9v/m5p/KiPfrayz9ALq7eQ31TD/lhxI9tx1X1WHra2ND6keuEfChHxF0lOnQeQM0ZQAIAsMYICegD2g0JvxAgKAJAlChQAIEsUKABAlrgG1UDLPtFc2fbEleX/KW7+n+9U9jnmve8ujT927ajS+OD7y7d1l6TW9zxV2VblhRXVW9ija81dyp0k0PswggIAZIkCBSRg+xLby23fmzoXIFcUKCCNSyUdmToJIGcUKCCBiLhF0tOp8wByRoECAGSJAgVkiv2g0NuxzLyRKm4iK0lnHHFcaXzCb26v7HP9Hr8tb9hjk7Lq0PS1L6ls2+N75TNUbV339r1aREyWNFmS+u80OhKnAzQcIygAQJYoUEACtq+Q9DdJu9teYvvDqXMCcsMUH5BARBybOgcgd4ygAABZokABALLEFF8m2hYsKo1fd8irK/v84KS3l8YHH/JYaXz37ZdXvtajzw0ujT81dURln2Hzu27LeXRszPBBqVMAGo4RFAAgSxQoAECWKFBAD8B+UOiNKFAAgCxRoAAAWaJAAYnYPtL2A7YX2j4jdT5Ablhmnrm2J5+qbBvxjVvLG75RHl7Swfv00erS+DC1dNALm8t2k6TvSzpCtf80t9v+bUTMS5sZkA9GUEAaB0haGBGLImK9pCmSJibOCcgKBQpIY7j0ouHpkiL2T+wHhd6OAgWk4ZLYi/Z8iojJETE2IsY2DeROEuh9KFBAGkskjaz7eYSkZYlyAbJEgQLSuF3SaNu72N5G0iRJFVskA70Tq/iABCKi1fapkn4vqUnSJRFxX+K0gKxQoIBEIuJ6SdenzgPIFVN8AIAsUaCAHoD9oNAbUaAAAFmiQAEAskSBAgBkiQIFAMgSBQoAkCUKFAAgSxQoAECWuJME0APMmTNnte0HUuexEUMlPZk6iY0gx66xpTnu3JmDKFBAz/BARIxNnURHbM8mxy1Hjv/S0AI1/YWpZXvgAADwb7gGBQDIEgUK6Bkmp06gE8ixa5BjwRGx8aMAAGgwRlAAgCxRoIDEbB9p+wHbC22fUdLe3/Yvi/ZZtpvr2r5QxB+w/daEOZ5ue57te2z/0fbOdW1ttu8qHt22rX0ncvyg7SfqcvlIXdsJthcUjxMS5fedutwetL2yrq1R5/AS28tt31vRbtvfLT7DPbb3r2vr+nMYETx48Ej0UG2794ck7SppG0l3S9qr3TEfl/Sj4vkkSb8snu9VHN9f0i7F6zQlyvFQSQOL5x/bkGPx8+pMzuMHJX2vpO8OkhYVfw4ung9udH7tjj9N0iWNPIfF+xwiaX9J91a0j5d0gyRLeoOkWd15DhlBAWkdIGlhRCyKiPWSpkia2O6YiZIuK55Pk3S4bRfxKRGxLiIelrSweL2G5xgRf4qINcWPMyWN6IY8tijHDrxV0vSIeDoiVkiaLunIxPkdK+mKLs5hoyLiFklPd3DIREk/jZqZkra3vZO66RxSoIC0hktqqft5SRErPSYiWiWtkjSkk30blWO9D6v2W/YGA2zPtj3T9ju6IT+p8zkeU0xNTbM9chP7NiI/FdOju0i6qS7ciHPYGVWfo1vOIXeSANIq+8fr7ZfWVh3Tmb5dodPvY/t4SWMlvakuPCoiltneVdJNtudGxEMJcrxG0hURsc72yaqNSg/rZN9G5LfBJEnTIqKtLtaIc9gZDf1/kREUkNYSSSPrfh4haVnVMbb7Shqk2jRMZ/o2KkfZHifpTElHR8S6DfGIWFb8uUjSDEn7pcgxIp6qy+vHkl7b2b6NyK/OJLWb3mvQOeyMqs/RPeewERfeePDgUf5QbRZjkWpTOhsunr+m3TGn6MWLJK4snr9GL14ksUjds0iiMznup9oigNHt4oMl9S+eD5W0QB0sDujmHHeqe/5OSTOL5ztIerjIdXDxfIdG51cct7ukxSr+jWojz2Hd+zWrepHEBL14kcRt3XkOmeIDEoqIVtunSvq9aiu9LomI+2yfJWl2RPxW0sWSLre9ULWR06Si7322r5Q0T1KrpFPixdNCjczxm5K2lTS1tn5Dj0bE0ZL2lHSh7RdUm7E5JyLmJcrxE7aPVu1cPa3aqj5FxNO2z5Z0e/FyZ0VERwsFuis/qbY4YkoU3/qFhpxDSbJ9haQ3Sxpqe4mkr0jqV3yGH0m6XrWVfAslrZH0oaKtW84hd5IAAGSJa1AAgCxRoAAAWaJAAQCyRIECAGSJAgUAyBIFCgCQJQoUACBLFCgAQJYoUACALFGgAABZ+n9aYDNBOq/wggAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x648 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "view_classify(img.view(1,28,28),ps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our network is brilliant. It can accurately predict the digits in our images. Next up you'll write the code for training a neural network on a more complex dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference and Validation\n",
    "Now that we have trained network, we can use it for making predictions. This is typically called **inference**, a term borrowed from statistics. However, neural network have tendency of perform too well on the training data and aren't able to generalize to data that hasn't been seen before. This is called **overfitting** and it impairs inference performance. To test for overfitting while training, we measure the performance on data not in the training set called the **validation** dataset. We avoid overfitting through regularization such as dropout while monitoring the validation performance during training. In this notebook.\n",
    "\n",
    "First off, I'll implement my own feedforward network for the exercise you worked on in part 4 using the Fashion-MNIST dataset.\n",
    "\n",
    "As usual, let's start by loading the dataset through torchvision. You'll learn more about torchvision and loading data in a later part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz\n",
      "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz\n",
      "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz\n",
      "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz\n",
      "Processing...\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "#Define a transform to normalize the data\n",
    "trasform = transforms.Compose([transforms.ToTensor(),\n",
    "                              transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))])\n",
    "# Download and load the training data\n",
    "trainset = datasets.FashionMNIST('F_MNIST_data/', download=True, train=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle = True)\n",
    "\n",
    "# Download and load the test data\n",
    "testset = datasets.FashionMNIST('F_MNIST_data/',download=True, train=False,transform=transform)\n",
    "testloader = torch.utils.data.DataLoader(testset,batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the network\n",
    "\n",
    "As with MNIST, each image in Fashion-MNIST is 28x28 which is a total of 784 pixels, and there are 10 classes. I'm going to get a bit more advanced here, I want to be able to build a network with an arbitrary number of hidden layers. That is, I want to pass in a parameter like `hidden_layers = [512, 256, 128]` and the network is constructed with three hidden layers have 512, 256, and 128 units respectively. To do this, I'll use `nn.ModuleList` to allow for an arbitrary number of hidden layers. Using `nn.ModuleList` works pretty much the same as a normal Python list, except that it registers each hidden layer `Linear` module properly so the model is aware of the layers.\n",
    "\n",
    "The issue here is I need a way to define each `nn.Linear` module with the appropriate layer sizes. Since each `nn.Linear` operation needs an size and an output size,\n",
    "\n",
    "```python\n",
    "# Create ModuleList and add input layer\n",
    "hidden_layers = nn.ModuleList([nn.Linear(input_size,hidden_layers[0])])\n",
    "# Add hidden layers to the Module List\n",
    "hidden_layers.extend([nn.Linear(h1,h2) for h1,h2 in layer_sizes])\n",
    "```\n",
    "\n",
    "Getting these pairs of input and output sizes can be done with a handy trick using `zip`.\n",
    "\n",
    "```python\n",
    "hidden_layer = [512, 256, 128, 64]\n",
    "layer_sizes = zip(hidden_layers[:-1],hidden_layers[1:])\n",
    "for each in layer_sizes:\n",
    "    print(each)\n",
    "\n",
    ">> (512, 256)\n",
    ">> (256, 128)\n",
    ">> (128, 64)\n",
    "```\n",
    "\n",
    "I also have the `forward` method returning the log-softmax for the output. Since softmax is a probability distribution over the classes, the log-softmax is a log probability which comes with a [lot of benefits](https://en.wikipedia.org/wiki/Log_probability). Using the log probability, computations are often faster and more accurate. To get the class probabilities later, I'll need to take the exponential (`torch.exp`) of the output. \n",
    "\n",
    "$$ \\large{e^{\\ln{x}} = x }$$\n",
    "\n",
    "We can include dropout in our network with [`nn.Dropout`](http://pytorch.org/docs/master/nn.html#dropout). This works similar to other modules such as `nn.Linear`. It also takes the dropout probability as an input which we can pass as an input to the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Network(\n",
       "  (hidden_layers): ModuleList(\n",
       "    (0): Linear(in_features=784, out_features=516, bias=True)\n",
       "    (1): Linear(in_features=516, out_features=256, bias=True)\n",
       "  )\n",
       "  (output): Linear(in_features=256, out_features=10, bias=True)\n",
       "  (dropout): Dropout(p=0.5)\n",
       ")"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Network(nn.Module):\n",
    "    def __init__(self,input_size,output_size, hidden_layers, drop_p = 0.5):\n",
    "        \"\"\"Builds a feedforward network with arbitrary hidden layers.\n",
    "        \n",
    "        Arguments\n",
    "        ---------\n",
    "        input_size: integer, size of the input\n",
    "        output_size: integer, size of the output layer\n",
    "        hidden_layers: list of integers, the sizes of the hidden layers\n",
    "        drop_p: float between 0 and 1, dropout probability\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        # Add the first layer, input to a hidden layer\n",
    "        self.hidden_layers = nn.ModuleList([nn.Linear(input_size,hidden_layers[0])])\n",
    "        \n",
    "        # Add a variable number of more hidden layers\n",
    "        layer_sizes = zip(hidden_layers[:-1],hidden_layers[1:])\n",
    "        self.hidden_layers.extend([nn.Linear(h1,h2) for h1,h2 in layer_sizes])\n",
    "        \n",
    "        self.output = nn.Linear(hidden_layers[-1],output_size)\n",
    "        self.dropout = nn.Dropout(p=drop_p)\n",
    "        \n",
    "    def forward(self,x):\n",
    "        \"\"\"Forward pass through the network, returns the output logits\"\"\"\n",
    "        \n",
    "        # Forward through each layers in `hidden_layers`, with ReLU activation and dropout\n",
    "        for linear in self.hidden_layers:\n",
    "            x = F.relu(linear(x))\n",
    "            x = self.dropout(x)\n",
    "        x = self.output(x)\n",
    "        return F.log_softmax(x, dim=1)\n",
    "model = Network(784,10,[516,256])\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train the network\n",
    "\n",
    "Since the model's forward method returns the log-softmax, I used the [negative log loss](http://pytorch.org/docs/master/nn.html#nllloss) as my criterion, `nn.NLLLoss()`. I also chose to use the [Adam optimizer](http://pytorch.org/docs/master/optim.html#torch.optim.Adam). This is a variant of stochastic gradient descent which includes momentum and in general trains faster than your basic SGD.\n",
    "\n",
    "I've also included a block to measure the validation loss and accuracy. Since I'm using dropout in the network, I need to turn it off during inference. Otherwise, the network will appear to perform poorly because many of the connections are turned off. PyTorch allows you to set a model in \"training\" or \"evaluation\" modes with `model.train()` and `model.eval()`, respectively. In training mode, dropout is turned on, while in evaluation mode, dropout is turned off. This effects other modules as well that should be on during training but off during inference.\n",
    "\n",
    "The validation code consists of a forward pass through the validation set (also split into batches). With the log-softmax output, I calculate the loss on the validation set, as well as the prediction accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the network, define the criterion and optimizer\n",
    "model = Network(784, 10 , [516,256],drop_p=0.5)\n",
    "criterion = nn.NLLLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "model(x).detach()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Implement a function for the validation pass\n",
    "def validation(model, testloader, criterion):\n",
    "    test_loss = 0\n",
    "    accuracy = 0\n",
    "    for images, labels in testloader:\n",
    "        \n",
    "        images.resize_(images.shape[0],784)\n",
    "        \n",
    "        output = model.forward(images)\n",
    "        test_loss += criterion(output, labels).item()\n",
    "        \n",
    "        ps = torch.exp(output)\n",
    "        equality = (labels.data == ps.max(dim=1)[1])\n",
    "        accuracy += equality.type(torch.FloatTensor).mean()\n",
    "    return test_loss, accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/2..  Training Loss: 1.282..  Test Loss: 0.809..  Test Accuracy: 0.697\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-97-72e7ab1f947f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     27\u001b[0m             \u001b[0;31m# Trun off gradients for validation, saves memory and computations\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 29\u001b[0;31m                 \u001b[0mtest_loss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccuracy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvalidation\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtestloader\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcriterion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     30\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     31\u001b[0m             print(\"Epoch: {}/{}.. \".format(e+1, epochs),\n",
      "\u001b[0;32m<ipython-input-71-6962c9251772>\u001b[0m in \u001b[0;36mvalidation\u001b[0;34m(model, testloader, criterion)\u001b[0m\n\u001b[1;32m      3\u001b[0m     \u001b[0mtest_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0maccuracy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m     \u001b[0;32mfor\u001b[0m \u001b[0mimages\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtestloader\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m         \u001b[0mimages\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresize_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mimages\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m784\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py\u001b[0m in \u001b[0;36m__next__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    262\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_workers\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# same-process loading\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    263\u001b[0m             \u001b[0mindices\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msample_iter\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# may raise StopIteration\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 264\u001b[0;31m             \u001b[0mbatch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcollate_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mindices\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    265\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpin_memory\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    266\u001b[0m                 \u001b[0mbatch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpin_memory_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m    262\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_workers\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# same-process loading\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    263\u001b[0m             \u001b[0mindices\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msample_iter\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# may raise StopIteration\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 264\u001b[0;31m             \u001b[0mbatch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcollate_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mindices\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    265\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpin_memory\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    266\u001b[0m                 \u001b[0mbatch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpin_memory_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/datasets/mnist.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, index)\u001b[0m\n\u001b[1;32m     72\u001b[0m         \u001b[0;31m# doing this so that it is consistent with all other datasets\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     73\u001b[0m         \u001b[0;31m# to return a PIL Image\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 74\u001b[0;31m         \u001b[0mimg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mImage\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfromarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mimg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'L'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     75\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     76\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtransform\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/lib/python3.6/site-packages/PIL/Image.py\u001b[0m in \u001b[0;36mfromarray\u001b[0;34m(obj, mode)\u001b[0m\n\u001b[1;32m   2448\u001b[0m             \u001b[0mobj\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtostring\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2449\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2450\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mfrombuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msize\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"raw\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrawmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2451\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2452\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/lib/python3.6/site-packages/PIL/Image.py\u001b[0m in \u001b[0;36mfrombuffer\u001b[0;34m(mode, size, data, decoder_name, *args)\u001b[0m\n\u001b[1;32m   2394\u001b[0m             \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m  \u001b[0;31m# may change to (mode, 0, 1) post-1.1.6\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2395\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32min\u001b[0m \u001b[0m_MAPMODES\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2396\u001b[0;31m             \u001b[0mim\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnew\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2397\u001b[0m             im = im._new(\n\u001b[1;32m   2398\u001b[0m                 \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmap_buffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msize\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecoder_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/lib/python3.6/site-packages/PIL/Image.py\u001b[0m in \u001b[0;36mnew\u001b[0;34m(mode, size, color)\u001b[0m\n\u001b[1;32m   2297\u001b[0m         \u001b[0mcolor\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mImageColor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetcolor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcolor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2298\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2299\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mImage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_new\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfill\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msize\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcolor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2300\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2301\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/lib/python3.6/site-packages/PIL/Image.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    526\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    527\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpalette\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 528\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    529\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcategory\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNORMAL\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    530\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadonly\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "epochs = 2\n",
    "steps = 0\n",
    "running_loss = 0\n",
    "print_every = 40\n",
    "\n",
    "for e in range(epochs):\n",
    "    model.train()\n",
    "    for images, labels in trainloader:\n",
    "        steps+=1\n",
    "        \n",
    "        #Flatten images into a 784 long vector\n",
    "        images.resize_(images.size()[0],784)\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        output = model.forward(images)\n",
    "        loss = criterion(output,labels)\n",
    "        loss.backward(labels.grad)\n",
    "        optimizer.step()\n",
    "        \n",
    "        running_loss += loss.item()\n",
    "        \n",
    "        if steps%print_every == 0:\n",
    "            # Make sure network is in eval mode for inference\n",
    "            model.eval()\n",
    "            \n",
    "            # Trun off gradients for validation, saves memory and computations\n",
    "            with torch.no_grad():\n",
    "                test_loss, accuracy = validation(model,testloader,criterion)\n",
    "                \n",
    "            print(\"Epoch: {}/{}.. \".format(e+1, epochs),\n",
    "                  \"Training Loss: {:.3f}.. \".format(running_loss/print_every),\n",
    "                  \"Test Loss: {:.3f}.. \".format(test_loss/len(testloader)),\n",
    "                  \"Test Accuracy: {:.3f}\".format(accuracy/len(testloader)))\n",
    "            running_loss = 0\n",
    "            \n",
    "            # Make sure trainign is back on\n",
    "            model.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference \n",
    "Now that the model is trained, we can use it for inference. We've done this before, but now we need to remember to set the model in inference mode with `model.eval()`.We'll also want to turn off autograd with `torch.no_grad()` context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 7.7985e-12,  1.0000e+00,  4.8038e-13,  2.8291e-09,  1.3598e-12,\n",
      "          5.7526e-18,  2.7550e-14,  6.9943e-17,  4.6248e-17,  2.4475e-18]])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAADNCAYAAADt/OSdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAH+xJREFUeJzt3XmYXFW19/HvrzudSSABg8iQECYR0CtgUPBliAyCgAYVvUwqvlej4oAXZ7lXuchFHEBUnHgVUQRlcGKUKRJAJJA4oCAxMSaEOWSADGTo7vX+cXZrUbUrqaSrqk+H3+d5+umqdfY5taoCvWrv2rW3IgIzM7Oy6RjoBMzMzHJcoMzMrJRcoMzMrJRcoMzMrJRcoMzMrJRcoMzMrJRcoMysVCSdIenHA53HhpB0saSzNvDctT5vSfdLmljdVtI4ScskdW5Q0iXmAmVmbSfpBEnT0x/WxyTdIGn/AcolJC1PuTwi6bwy/rGPiD0i4rZM/KGI2CQiegAk3Sbp3W1PsAVcoMysrSSdBpwPnA1sBYwDvgVMGsC0XhERmwCHACcA76luIGlI27N6nnOBMrO2kTQKOBP4QET8PCKWR8SaiLgmIj5e55wrJT0u6WlJt0vao+LYkZIekLQ09X4+luJjJF0raYmkRZLukLTOv3cR8SBwB/CydJ25kj4p6T5guaQhknZLvZQladjtjVWXGSPp5pTTVEnbV+T7NUnzJT0jaYakA6rOHS7p8nTu7yW9ouLcuZIOzbw+41MvcIik/wUOAC5IPcILJH1T0rlV51wj6SPrej0GmguUmbXTfsBw4Bfrcc4NwC7Ai4DfA5dWHPs+8N6I2JSiqExJ8Y8CDwNbUvTSPgOsc103SbtT/IH/Q0X4eOAoYDQg4BrgppTPh4BLJe1a0f5E4PPAGOCPVfneC+wJbAFcBlwpaXjF8UnAlRXHfympa11594mI0ykK7AfTsN8HgR8Cx/cVaEljKHqKP2n0ugPFBcrM2umFwFMR0d3oCRFxUUQsjYhVwBnAK1JPDGANsLukzSJicUT8viK+NbB96qHdEWtfePT3khZTFJ/vAT+oOPb1iJgfEc8C+wKbAOdExOqImAJcS1HE+lwXEbenfE8H9pM0Nj2XH0fEwojojohzgWFAZXGbERFXRcQa4DyKYr5vo69VTkTcAzxNUZQAjgNui4gn+nPddnCBMrN2WkgxBNbQ5zmSOiWdI+nvkp4B5qZDY9LvtwBHAvPScNp+Kf5lYDZwk6Q5kj61jofaOyI2j4idIuK/IqK34tj8itvbAPOrjs8Dts21j4hlwKJ0HpI+KumvabhyCTCq4rlUn9tL0QvcZh25N+KHwEnp9knAJU24Zsu5QJlZO/0OWAkc02D7EyiGvQ6l+GM+PsUFEBH3RsQkiuG2XwJXpPjSiPhoROwIvAE4TdIhbJjKntejwNiqz7PGAY9U3B/bd0PSJhTDdY+mz5s+CbwN2DwiRlP0bFTn3A5gu/SYG5pvnx8Dk9JnWrtRvFal5wJlZm0TEU8DnwW+KekYSSMldUl6vaQvZU7ZFFhF0fMaSTHzDwBJQyWdKGlUGhJ7Buiban20pJ0lqSLe04SnMA1YDnwi5T2RogD+tKLNkZL2lzSU4rOoaRExPz2XbmABMETSZ4HNqq7/SklvTj3Mj6Tnfvd65vgEsGNlICIepvj86xLgZ2m4svRcoMysrSLiPOA04L8o/ljPBz5I/l39jyiG0B4BHqD2j/Xbgblp+O99/GsYaxfgFmAZRa/tW7nvEG1A7quBNwKvB56imB7/jjT7r89lwOcohvZeSTFpAuBGigkff0vPaSXPHT4E+BXw78Di9NzenIrv+vgacKykxZK+XhH/IfByBsnwHoC8YaGZ2cZP0oEUQ33jqz5DKy33oMzMNnJpqvqpwPcGS3ECFygzs42apN2AJRTT7s8f4HTWi4f4zMyslNq6ttRhHW/d+Kvhrdtlw92R76wuWPaCbHyLkbWTbBYuH5ltu9mIldn48CH570KuWFP7xfSe3nx+mx81KxsfrG7uvVLrbmVmZeDFD82eh8aMGRPjx48f6DTseWrGjBlPRcSW62rnAmX2PDR+/HimT58+0GnY85SkeY208yQJMzMrJRcoMzMrJQ/x9UPHppvWxG7Y7dps2/tW5ycy7Dwk/x6hK7Oh58Pd+dVJlkb+n3FsZ/7rDn9avUlN7P8Mz39Z/Zjt80umdc+r/gK8mVlzuQdlZmal5AJlZmal5AJlZmal5AJlZmal5EkS/bDkDXvUxB7rviHb9sHV22bjs1bnF9cY3bm8JrYy8qtOLOiu3lKm8PiQxdn4A6tqc5kw7MFMS3jkjWOz8a2+4UkSZtZa7kGZmVkpuUCZNUjSXZI+vY424yVdVRWbKOkrDT7GLEm3SfqdpHM3IMfJ63uOWVm5QJk1QNJYil1QD2nxQz0dERMjYj9gT0n5seH6XKBso+ECZdaYYyl2I50jaScASWdIulTSDZJul/TP5eYldUj6rqQTKy8i6QhJd6Te2PH1HkxSJ9AFrJQ0RNJlkqZKul7SFqnNVyXdmXpcO0h6P7Brun9Q5pqTJU2XNH3BggVNeVHMWskFyqwxhwA3AT+hKFZ9ZkbE64E7gENTrBP4HnBzRFza11BSB/DZdK39gfelQlRplKTbgL8A8yJiIfAm4KGIOAi4HPiQpH2ArSNif+BzwGcj4tspn4kRMbX6CUTEhRExISImbLnlOheSNhtwnsXXD49P7KmJdSq/3VBvnf2g6u29PFy1Sw/Vm63XUecqS3tHZONdqs37L6tr94gC6H7t0/kEv5EPb4wkbQf8G3ANxZu6EcAX0+E/pN/zgc3T7VcD90bEcz6LAsYAu1AUur77WwKPV7R5OiImpsf9pqQDgJ2Ae9PxaRSFsDp21oY/Q7Nycg/KbN2OBU6NiCMi4nXATEk7pGOV3xPoe3dyF/AbSV+ous5TwF+Bw1IR2jMiHqe+JcAWwGxgnxR7NTCrTqw6H7NBzT0os3V7CzCp4v4UnjvMVyMizpf035I+Q1GwiIheSf8L3CKpF1gAvK3q1L4hPtLx/6HoaL9Z0u3AcuDEiFgk6TFJdwLdwLvSOTMl/Qz4ckTcvYHP16wUXKDM1iEiDqi6f1mmzXcq7h6bYp+viN2WYjcCN67lsXapc+iETNv/zMROqndts8HGQ3xmZlZKLlBmZlZKHuLrh23HLayJLe3Nf0a9pGdkNr5pZ34jwyW9te1HdqzKtu2I/Ay8LnXXidfO4nu8Z1S27fKnh2fjZmat5h6UmZmVkguUmZmVkguUmZmVkguUWYml1dEXpPX1pks6bqBzMmsXFyiz8puaVp44EPjEAOdi1jaexdcPE7Z8qCa2tLfejLramXNFPD/TrjO3Yo3ya+51Rn7mYG49v3p27Hoqf425wxq+hrXcSGCFpMOATwObAD+PiHMkjQauoFh14hFgfkScMWCZmjWBe1Bm5XdQWv7oPuAHwG8j4mCKNfiOkTQCeA9wVUQcATyWu4i327DBxgXKrPz6hvjGAycDe0m6hWL5pB2BF1Gsbj4jtb+35gp4uw0bfFygzAaJiFhNsWL6WcCHgdcCD6XY34G9UtNXDkiCZk3mz6DMyq9viG8YcC3FZ0yXA3+mWN0cig0Sr5T0VuBJ4MEByNOsqVyg+uHgzR6oiS3MLFEE8FT3ptl4vUkSewyt3SbopV35CQsPdecnOCzozbfvXV27qeLS3qHZtiOe8PZCAyki5lJsaljt4so7abfewyOiR9JZFPtFmQ1qLlBmG4cRwK8lCXgCOHOA8zHrNxcos41ARCwHDlhnQ7NBxJMkzMyslFygzMyslFygzMyslPwZVD+8Znjtt/FvXDEu23ZlnSWQdhr6ZDa+sHdETWy3O96Rbfu3A3+Ujc95tjMb78m8L8kurQRs+kh+iSYzs1ZzD8rMzErJBcqsySRtJumatEXGPZLe0M/rTZT0lWblZzZYeIjPrPneDvw6Ir6Zvpc0qt0JSOqIiPzy92aDhHtQZs23AniVpK2isETSXyVdKukPkt4OIGlHSTemntZXU+zlkqZIukvSBZUXlTRc0lWSDk63f5zaXp16beMl3SHpSuBj1Ul5NXMbbFygzJrvEmAmcGMqNLsALwbeT/Fl2lNSuy8Cp6SVyodImkCxRNEhEfEaYJt0LhR7QV0GnB8RU4B3A1PSths/BCandtsAJ0bEl6qT8mrmNth4iK8fNu+onWm3pCe/Ft+ynvy6eGOHrMjGr1m2W01sxy/WmVF3YD5cb5PEF3SsqomNysQAhi1anb+41RUR3cDZwNmSXkux7NCciHgGIA37AewKfD/d3RS4lWLx1/MkjQR2oCg4AJMoNie8M93fHdhH0juALuCOFP9TWvXcbNBzgTJrMknbA4+lQvEkxUhFbh7/TOBjETEvFa1O4KvANyLiekk/p9hKA+AnQKek90XEdyhWK/9dRFySHrML2JZiR12zjYKH+Mya7+XA7WmLjG8Cn6/T7pPAdyRNAW6m6C1dA3xZ0s8oClal04C9JZ0EXAgclj6DmgK8rvlPw2xguQdl1mQRcS3Fvk2VJlQc3zf9ngO8vqrdQ8Aemcveln5Projlvrl97PrkalZm7kGZmVkpuUCZmVkpeYivAUO23SYb79Qfa2Kr6qy5N7IzP7Fq687amYAAVz/2itrgH+6vk2Hemsivxbeoe5Oa2II6OwEPuf8f2bhX6DOzVnMPyszMSskFyszMSskFyszMSskFyqxEciuhS5qeafcpSTtk4idLGtqebM1ay5MkGvDsHvlJEjn1lhcaM2RZnfb5iQwP3V678eE4Hm44j7XlkjO6Y2U23rPk6fV6TOu3hlZCj4hzqmOSOoCTgasAL3dkg557UGblUrMSOvCCzEroF0t6Wdor6lpJVwP/DewJ3CDp1AF8DmZN4R6UWblcAmxNsRL6CuCd/Gsl9F6KJZEuqTpnM+CgiIi0OO3REVHTZZc0mbQSxbhxtT10s7JxD8qsRCKiOyLOjog9gdOpWAk9FR1lTpseEbnFaKuv7e02bFBxgTIrEUnbV0xyWNtK6JUqVzBfQ+0is2aDkguUWbk0uhJ6PVcDV0j6j2YnZtZu/gyqAc+MzS9f1BO1W+/0ZEdgoHOdb4Kfa8eL59fEuuu0zeUBsLTO8kXDOtbUxGatflHDuVnrrMdK6CdXHL+t4vg3gG+0LkOz9nEPyszMSskFyszMSskFyszMSskFyszMSskFyszMSsmz+BqwdMd8vDczM6838jV/m6ELs/HFPSuy8e55tbP46lnY+2w23kl+M8RcjvNWj2n48czM2sE9KDMzKyUXKLM2yW2lsYHXeZ+kk9dyvGZ7DrPByEN8Zu3T0FYaZlZwD8qsfWq20pD049SjulPSOABJv5f0bUnTJH06xcalNtcDB6ZYh6Sb0vk3S9psbQ8uabKk6ZKmL1iwoNXP1azfXKDM2ucSYCbFVhp3SdoFmBwRE4EvAe9N7UYD5wD7Acel2CeAMyPiSNLisRHRC0xK518D/PvaHtyrmdtg4yG+Bgx56TPZ+LLeVQ1f4yVdT2bjb/rrCdn4MOY2fO3lvfl1/rqUX71vRW/tjuBrwgtgt1pEdANnA2enfZvOBJ6StCcwDLg/NV0cEfMAJPVN0dwZmJFu35OOvQD4bup5jQZ+1pYnYtYm7kGZtUlmK40xwFYRcQBwFv/a6yn3jmM2sFe63bd47BHAoxFxIPA98ntFmQ1a7kGZtc/LgcslrUz3TwUukHQz8MA6zv0ScJmkjwFLUuxu4HRJ1wGPAQ+3IGezAeMCZdYmdbbSOCDTLre9xkPA/pnL7r22880GMw/xmZlZKbkH1YB9t52Xjd+7qvZrLPUmG4zKbBII8MRd22Tj49ZjkkRXnU8eeuosu7S0Z3htHqvqzVBe3nAeZmbN5B6UmZmVkguUmZmVkguUmZmVkguUmZmVkguUWclIOiCtr3e7pFslvazB80ZLelur8zNrF8/ia8Cq3vzLNHdN7Xpm9WbO9da59vbX5pdRyi9elPdET+3SRQAroysb71JPTWz+8tF1ru5ZfO0k6YXAt4DDIuLxdD8/1bPWaOBtwBWtys+sndyDMiuXo4CfR8TjABGxEHgo7SM1VdIVkoZK2krSLamXdZWkTuD9wEGp97XrQD4Js2ZwgTIrl62BR6tik4HrIuIgigVljwcWA0ekdfgeAg4Gvg1MjYiJETGz+sLebsMGGxcos3J5FNi2KrYTcG+6PY1iZfMtgKskTQWOpoFhQG+3YYONC5RZuVwHvEnSiwEkbUGxCOw+6firgVnAicBNqVd1LcVK5msA75tiGw0XKLMSiYhFwCkUq55PBS6n2IzwaEm3A3sAPwVuBd4v6VfAi9PpjwEj0mdSO7Y/e7Pm8iy+Bvz2vpdk46cfcX1N7Bdr9sq0hIW9w7LxjnlPZOO18+zq66mzDdCSnpHZ+KadK2tiK7vzM/7yWVsrRcQdwEFV4aOr7v+RYvuOake0JCmzAeAelJmZlZILlJmZlZILlJmZlZILlJmZlZILlJmZlZJn8TXghTPyXy3Z5uja2XOjhqzItu2st7pe7/rM18sbWmelv3Fdi/IPmXlf0t2bf6/iWXxmNlDcgzIzs1JyD8psAEkaT7GM0Z8pVoO4HTgrItYMYFpmpeAelNnAmxoRB1Ms+NoBfKTvgCT/P2rPW+5BmZVERISks4BbJR0P3AWMkvQe4HsUC8IuA04CXgRcAqwC/hYRkyVdTLGwbADviIi57X8WZs3jAtWAF/3ib9n4yDNqNwrctmtxtu1Wnauz8VWv2CEbHzIlP8Eh58meTbLxsUOezsZzGxk+8ugW2bYvYU7DeVj/RcQqScOAzYHzI2K2pA8CUyLiIklvodh+YzFwaURcIKlDUhewG7BvKnQ1PS9Jk9O5jBs3rm3PyWxDefjArEQkDQVWA4sjYnYK706xMOxtwGnAGIpdc7eT9CPgpPSZ1deAiySdD9QsxOjtNmywcQ/KrFw+A/yKYuv2Pg8Cv4uISwBSb2lIRHwq3X9A0qXAlRFxmaTPAG8GftTe1M2aywXKbOAdJGkKxYjGncD5PLdAXQhcKOld6f65wCZp6G8Y8GtgU+DqNLTXCxzXruTNWsUFymwApYkMufG2CRVtVgLvyLS5vOr+gc3LzGzg+TMoMzMrJfegGtDz1MJsfHFv7cZ/m3bUzuwDeEGdr7PMPzTffocpDSYHLO0dkY0PVX5G4ZadS2tiXU/mNyw0Mxso7kGZmVkpuUCZmVkpuUCZmVkpuUCZmVkpuUCZmVkpeRZfP0ye8+aa2FfH/yzbdlF+T0FG7L6k33n0ULtxItR/9zG6o3ZdwEzI+qFiG437gS7gnRVLF1W2mx4REySdAUyPiGvbmadZmbkHZdY6UyNiInAe8Ml2PrC36bCNgf8jNmu9vwAnSfoKgKSXpq0xsiR9VdKdkm6TtIOkt0r6RDq2maSb0+3PSJoq6XZJL0+x30u6APhh5rqTJU2XNH3BggUteJpmzeUCZdZ6BwAzG2koaR9g64jYH/gc8FngWuCo1OQY4JepIO0aEQdRrNt3Zjret03H26uv7dXMbbBxgTJrnYPSFhlHAqdWxPMfGhZ2ovjsCmAasHNEPAs8Imln4C3AVRR7P70mXf+nwGbpnMW5z7rMBiNPkjBrnakRcSyApH8Dxqb4K9dyzmyKXhLAq4FZ6fblwHspttl4QtKD6frvTtfvW6uqznQcs8HHBaofZt24U01sy1PyL+nsNfk3ze/ceVo2fuM/3xCv2xNrRmfjXcMfycZzf8GGLlnbm3prgj8DwyXdQlGEsiJiuqTHJN0JdAN9W2z8GrgI+ERqd5+kWZKmUvyT3gyc3conYNZuLlBmLZC20Ti24n4AkzLtJqTfZ1TE/jPTbhXwwqrYF4Ev5q5ntjHwZ1BmZlZKLlBmZlZKLlBmZlZK/gyqH8be+HRNbNH7urNtV0Z+U8EPbT4rG79xrRO9nmvuyhdm48M3y098WNobNbERT9XGbOP150eeZvynrhvoNGyQm3vOUetu1A/uQZmZWSm5QJmZWSl5iM+sRSQNBW5Kd18JzEi3j46IZQOTldng4QJl1iIRsRqYCP/cVmNi5XFJHRHRkpUfJCnl4A8XbdDyEJ9ZG0k6VNLVkq4GTpB0mKS7JU2T9PbU5seSXppuf0XS/unnnrTC+efSsaPSSuZ3SXpbxbnfAm4FNh2gp2nWFO5B9UPMuL8mdsez22fb7jH00Wz8tyvzs/uaobPOmqSdqn1TrczMPmuZTYBDIiIk3QscDiwHpkm6os45RwGfjYhfS+qQ1Al8hqKH1gv8RtKVqe20iDil+gKSJgOTATo382rmVn7uQZm13/SKobeIiEVpKaPZwIuByncLfe8yvgEcKelHwOuArYBdKNbgu5ViGaS+7xvcS0bldhudI0c19QmZtYJ7UGbtV/m5kyRtASwDdgYeBxYDYyXNBPYGfkGxjcaHJQ2n2IZjb+BB4LCIWCOpK/2uvr7ZoOUCZTawTgduSLfPjYhVki6i2BF3LrAyHTtF0iTgBcAPIqJH0jnALZJ6KQrb8e1N3ay1XKDM2qBi1fJbgFsq4jfxr6nofbE/Aq+ousSdwLlV7a4Hrq+KndS8rM0Glj+DMjOzUnIPqsm+99D++fhLLs3Gt+pck40/O+lVNbERv7on23ZVb1c2vob8zLyVUTu7b8hKz+J7Pnn5tqOY3uJ11Mz6yz0oMzMrJRcoMzMrJRcoMzMrJRcoMzMrJRcoMzMrJc/ia4Tya9qRWSh67pwXZZt2vCR/iZ46D7lw99p/mu1+lW/7TPewbHx5nfX1Hu/ZpCY2/Kn8bEJ7rg3ZQiOtZD6hKnYyMDMiflcVPwa4KyKeTPd3A/4v8NvKuNnzgQuU2XpY1xYa63Gdi6tjkjqAYyjW5OsrREdQrDTxjqq42UbPQ3xmTSRpv7QtxlRJZ6Zwh6Rvpy01Pp3anSHpaEnjJd2RViL/OEVB+oGkL6Rz9wceqYxLGiXpmvQYV0gaKmmipOvSzz2SdsnkNlnSdEnTFyxY0PoXw6yf3IMya64jgTMj4trUIwIYDZwDzAf+AHyh6pxtKLbfWJ2G9L4SEX+RNAJYExEzJf26Iv5x4LqI+E7aG+p4YB4wCjgA2I9ijb+TKx8kIi4ELgSYMGGCv5ltpecelFk/STotbSR4GvBN4LC0LcYRqcniiJiXds99NnOJP6Whw2oTgamZ+E78a0uNaRSroAP8IW3jMSO1MRvU3INqQMeI/KaCvStW1MS2vzp/jXFHj1yvx9zh8H/UxNZUv+9ODt+iduNEgB26aidDAHSq9rP8jtX1pmvYukTEecB5AJJGRMSpaTLFDIrFXNfVW6ncHmMN0JluHw58LROfDeyTrv9qYFaK75m2et8L+PsGPyGzknCBMmuu90p6M8W2GBdvwPk3AOdLuhHYISL+kYl/G7hU0gkU22x8AXgNsBS4DhgDnNivZ2FWAi5QZhuoeup4ip0PnF+vXUTsm36fUdHk2IrjPwd+LmkYxYaEz4lXnHN05WOkjQofiIiPbcBTMSslFyizEkpbwP9yoPMwG0guUGYbgYi4DbhtgNMwayrP4jMzs1JyD6oBsTo3Azhv2HX3ZuO7/OL9+WsP7c3Gt/xt7T/N5jyWbXvB/7w1G//8rvn3H8OW1MZefNdd2bZmZgPFPSgzMyslFygzMyslFygzMyslfwZl9jw0Y8aMZZJmDnQeFcYATw10ElXKllPZ8oENz2n7Rhq5QJk9P83MfdF4oOT2zBpoZcupbPlA63NSZDbdM7ONW9n+2JUtHyhfTmXLB1qfkz+DMjOzUnKBMnt+unCgE6hStnygfDmVLR9ocU4e4jMzs1JyD8rMzErJBcrMzErJBcpsIyPpCEkzJc2W9KnM8WGSLk/Hp0kaX3Hs0yk+U9LhbcrnNEkPSLpP0q2Stq841iPpj+mnzn7VTc/nZEkLKh733RXH3ilpVvp5ZzPyaTCnr1bk8zdJSyqOteI1ukjSk5L+Uue4JH095XufpL0rjjXvNYoI//jHPxvJD8W28H8HdgSGAn8Cdq9qcwrwnXT7OODydHv31H4YsEO6Tmcb8nktMDLdfn9fPun+sgF4fU4GLsicuwUwJ/3ePN3evB05VbX/EHBRq16jdM0Dgb2Bv9Q5fiTFLs8C9gWmteI1cg/KbOPyKmB2RMyJiNXAT4FJVW0mAT9Mt68CDlGxJe8k4KcRsSqKreZnp+u1NJ+I+E1ErEh37wa26+dj9iuftTgcuDkiFkXEYuBm4IgByOl44CdNeNy6IuJ2YNFamkwCfhSFu4HRkramya+RC5TZxmVbYH7F/YdTLNsmIrqBp4EXNnhuK/Kp9B8U78z7DJc0XdLdko7pZy7rk89b0tDVVZLGrue5rcqJNPy5AzClItzs16gR9XJu6mvkpY7MNi7KxKq/S1KvTSPntiKfoqF0EjABOKgiPC4iHpW0IzBF0p8j4u8tzuca4CcRsUrS+yh6mwc3eG6rcupzHHBVRPRUxJr9GjWiLf8NuQdltnF5GBhbcX874NF6bSQNAUZRDOc0cm4r8kHSocDpwBsjYlVfPCIeTb/nUGxpv1er84mIhRU5/D/glY2e26qcKhxH1fBeC16jRtTLubmvUbM/XPOPf/wzcD8UoyJzKIaB+j5w36OqzQd47iSJK9LtPXjuJIk59H+SRCP57EUxSWCXqvjmwLB0ewwwi7VMHmhiPltX3H4TcHe6vQXwj5TX5un2Fu34N0vtdgXmkhZYaNVrVHHt8dSfJHEUz50kcU8rXiMP8ZltRCKiW9IHgRspZoddFBH3SzoTmB4RVwPfBy6RNJui53RcOvd+SVcADwDdwAfiuUNJrcrny8AmwJXFXA0eiog3ArsB35XUSzHac05EPNCGfD4s6Y0Ur8Eiill9RMQiSZ8H7k2XOzMi1jaRoJk5QTE54qeRKkHS9NcIQNJPgInAGEkPA58DulK+3wGup5jJNxtYAbwrHWvqa+SljszMrJT8GZSZmZWSC5SZmZWSC5SZmZWSC5SZmZWSC5SZmZWSC5SZmZWSC5SZmZWSC5SZmZWSC5SZmZXS/wcrixAL4hROHwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x648 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Test out our network\n",
    "model.eval()\n",
    "\n",
    "dataiter = iter(testloader)\n",
    "images, labels = dataiter.next()\n",
    "img = images[1]\n",
    "#Convert 2D image to 1D vector\n",
    "img = img.view(1, 784)\n",
    "\n",
    "#Calculate the class probability (softmax) for img\n",
    "with torch.no_grad():\n",
    "    output = model.forward(img)\n",
    "\n",
    "ps = torch.exp(output)\n",
    "print(ps)\n",
    "#Plot the image and probabilities\n",
    "view_classify(img.view(1,28,28),ps,version='Fashion')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving and Loading Trained Networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Network(\n",
       "  (hidden_layers): ModuleList(\n",
       "    (0): Linear(in_features=784, out_features=516, bias=True)\n",
       "    (1): Linear(in_features=516, out_features=256, bias=True)\n",
       "  )\n",
       "  (output): Linear(in_features=256, out_features=10, bias=True)\n",
       "  (dropout): Dropout(p=0.5)\n",
       ")"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAABvZJREFUeJzt3c1uVfcVxuF9/IEBx8HG2KIQhIBO0kShs3TQD6kX0CjqIOq1tb2VKEpoFEVKVWYljdKKKh2UAY1tCNg+53SUUbXXOrID5q2eZ7qy7W3sX/Zg6b/PZD6fD8Crb+msbwBYjFghhFghhFghhFghhFghhFghxMoi/9Gvf37XMhZesA/v3Z9Uc09WCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCLFy1jfA2di5cqWc37x5c3T2+sZGee2HH310klt6JSxNJuV8Np+/pDv5X56sEEKsEEKsEEKsEEKsEEKsEEKsEOL/fs86KfZm8zPcmXUuvX6pnH+79205/+3775fzzUub5Xw6m47Ozp07V1772eefl/OnT5+W87N0mj3qa+vr5fzgyZMTf+1h8GSFGGKFEGKFEGKFEGKFEGKFEGKFEC9lz1qdEey2Wqfdhb7Ku9Qf37kzOtvZ2SmvvfvOO+X86PConE9ns3o+Hd+zHh8fl9f+7oMPyvmfPv20nP/1wYNy/iItL9XPr/d+897obHt7u7z293/8w4nu6XuerBBCrBBCrBBCrBBCrBBCrBDipaxuzvL1jVtbW6Ozx48fl9devny5nF+/dq2c7zbrl+p1n9165Jtv/lXOH/7zYTnvvv6d2+Nrpd3d+ueaNWuhX/3il+X8J2++OTp79ux5ee3e3l45X1+/WM67ldnm5vjRwoOD+gjcysrpcvNkhRBihRBihRBihRBihRBihRBihRAvZc9avQ70xhtvlNfeunWrnG8Ve69hGIZrPxrfhX5875Py2p+9+245398/KOfLy/X/C4+Oxo+xPfjyy/La7nWet2/dLuc7O/VHPpavG23W5vPmP5jP6nm1y5wM9Ucydn9P7b01P9uT4nWia2v1K1qvX79ef/GGJyuEECuEECuEECuEECuEECuEECuEWGjPWu1Jh6F/9WS12jpqzlXOio8eHIZhWFpaLueP/zN+ZvXtt94ur612asPQ79WOj+t7r/asP717t7x247WNcj6d1v+uR0f1/PDwcHTWnVddWq5/J92idnVl9cTf+7j5uWfNjrd7dW319zib1ve21nxUZseTFUKIFUKIFUKIFUKIFUKIFUKIFUIstGddaj4G7+NP7pXzGzfGzxju7uyW165frN/zunZ+rZxX78edNnuxznGzq5ws1fvp5WIf2Z1Xff68fn9u97N1H224vNLtSsfN5/X37vb21S512uzduz3qUvM7WZrU/y6rxbt/V5s9avX7XoQnK4QQK4QQK4QQK4QQK4QQK4QQK4RYaM+6tlbvMr/7rt4JfvHFn0dnx8fjZzqHYRiuNJ+X2b0n9urVq6OzixculNdeaOblu3UXmC8XZ3G7c5ur58bPfA5D/27e7txndf2s2aN2Z0K7fWP179Jdu7pa/0l37wWuzhgPwzDsH+yPzmbPnpXXPnr0qP7mDU9WCCFWCCFWCCFWCCFWCCFWCLHQ6qY7rjVtXie6uXlp/AZW6xXEYXMU7G9ffVXO/3L//uisWhEMQ/+q0W5F0R2ZWimOW3XHyLp5p1sNVV+/+97TaX2MrduflNPm2klz9K+79+7o4PPiFa0bG/XrYc+fP1/OO56sEEKsEEKsEEKsEEKsEEKsEEKsEGKhPWun2j0NwzD8+xRHg6pd5DD0x/eqV5lub18pr93a2izn3XGqbg9b7SO7j4usXrE6DP1HZXb31s0rk2a/3L3a9jT75+7vpdPtn6vfWXft1w8fnuievufJCiHECiHECiHECiHECiHECiHECiF+kD3ri9TtE7t5ZW9//LWSwzAMf//Hib80/OA8WSGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCHEZD6fn/U9AAvwZIUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQ/wVGzFQZbRtq8wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "image, label = next(iter(trainloader))\n",
    "imshow(image[0,:]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving and loading networks\n",
    "As you imagine, it's impractical to train a network every time you need to use it, Instead, we can save the trained networks then load them later to train more or use them for prediction.\n",
    "\n",
    "The parameters for PyTorch networks are stored in model's `state_dict`. We can see the state dict contains the weight and bias matrices for each of our layers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Our model: \n",
      "\n",
      " Network(\n",
      "  (hidden_layers): ModuleList(\n",
      "    (0): Linear(in_features=784, out_features=516, bias=True)\n",
      "    (1): Linear(in_features=516, out_features=256, bias=True)\n",
      "  )\n",
      "  (output): Linear(in_features=256, out_features=10, bias=True)\n",
      "  (dropout): Dropout(p=0.5)\n",
      ") \n",
      "\n",
      "The state dict keys: \n",
      "\n",
      " odict_keys(['hidden_layers.0.weight', 'hidden_layers.0.bias', 'hidden_layers.1.weight', 'hidden_layers.1.bias', 'output.weight', 'output.bias'])\n"
     ]
    }
   ],
   "source": [
    "print(\"Our model: \\n\\n\", model,\"\\n\")\n",
    "print(\"The state dict keys: \\n\\n\", model.state_dict().keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest thing to do is simply save the state dict with `torch.save`. For example, we can save it to a file `checkpoint.pth`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model.state_dict(), 'checkpoint.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can load the state dict with `torch.load`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "odict_keys(['hidden_layers.0.weight', 'hidden_layers.0.bias', 'hidden_layers.1.weight', 'hidden_layers.1.bias', 'output.weight', 'output.bias'])\n"
     ]
    }
   ],
   "source": [
    "state_dict = torch.load('checkpoint.pth')\n",
    "print(state_dict.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And to load the state dict in to the network, we do\n",
    "`model.load_state_dict(state_dict)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.load_state_dict(state_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seems pretty straightforward, but as usual it's a bit more complicated. Loading the state dict works only if the model architecture is exactly the same as the checkpoint architecture. If I create a model with a different arhitecture, this fails."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint = {'input_size':784,\n",
    "             'output_size':10,\n",
    "             'hidden_layers':[each.out_features for each in model.hidden_layers],\n",
    "             'state_dict':model.state_dict()}\n",
    "torch.save(checkpoint,'checkpoint.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the checkpoint has all the necessary information to rebuild the trained model. You can easily make that a function if you want. Similarly, we can write a function to load checkpoints. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_checkpoint(filepath):\n",
    "    checkpoint = torch.load(filepath)\n",
    "    model = Network(checkpoint['input_size'],\n",
    "                             checkpoint['output_size'],\n",
    "                             checkpoint['hidden_layers'])\n",
    "    model.load_state_dict(checkpoint['state_dict'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network(\n",
      "  (hidden_layers): ModuleList(\n",
      "    (0): Linear(in_features=784, out_features=516, bias=True)\n",
      "    (1): Linear(in_features=516, out_features=256, bias=True)\n",
      "  )\n",
      "  (output): Linear(in_features=256, out_features=10, bias=True)\n",
      "  (dropout): Dropout(p=0.5)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "model = load_checkpoint('checkpoint.pth')\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading data sets with Torchvision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far we've been working with fairly artificial datasets that we wouldn't typically be using in real projects. Instead, we'll likely dealing with full-sized images like we'd get from smart phone cameras. In this notebook, we'll look at how to load images and use them to train networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easiest way to load image data is with `datasets.ImageFolder` form `torchvision`. In general we'll use `ImageFolder` like so:\n",
    "```python\n",
    "dataset = datasets.ImageFolder('path/to/data',transform=transforms)\n",
    "```\n",
    "where `path/to/data` is the file path to the data directory and `transforms` a list of processing steps built with `transforms` module from `torchvision`.ImageFolder expects the file and directories to be constructed like so:\n",
    "\n",
    "```\n",
    "root/dog/xxx.png\n",
    "root/dog/xxy.png\n",
    "root/dog/xxz.png\n",
    "\n",
    "root/cat/123.png\n",
    "root/cat/nsdf3.png\n",
    "root/cat/asd932_.png\n",
    "```\n",
    "where each class has it's own directory (`cat` and `dog`) for the images. The images are then labeled with the class taken from the directory name. So here, the image `123.png` would be loaded with the class label `cat`. We can download the dataset already structured like this  [from here](https://s3.amazonaws.com/content.udacity-data.com/nd089/Cat_Dog_data.zip). I've also split it into a training set and test set.\n",
    "\n",
    "\n",
    "## Transforms\n",
    "When we load in the data with `ImageFolder`, we'll need to define some transforms. For example, the images are different sizes but we'll need them to all be the same soze of trainign. We can either resize them with `transforms.Resize()` or crop with `transforms.CenterCrop()`, `transforms.RandomResizedCrop()`, etc. We'll also need to convert the images to Pytorch tensor with `transforms.ToTensor()`. Typically we'll combine these transforms into a pipeline with `transform.Compose()`, which accepts a list fo transforms and runs them in sequence. It looks something like this to scale, then convert to a tensor:\n",
    "```python\n",
    "transforms = transforms.Compose([transfroms.Resize(255),\n",
    "                                 transforms.CenterCrop(244),\n",
    "                                 transforms.ToTensor()])\n",
    "```\n",
    "Ther are plenty of transforms available, I'll cover more in a bit and we can read through the documentation.\n",
    "\n",
    "\n",
    "## Data Loaders\n",
    "With the `ImageFolder` loaded, we have to pass it to a DataLoader. The `DataLoader` takes a dataset (such as we would get from `ImageFolder`) and returns batches of images and the corresponding labels. We can set various parameters like the batch size and if the data is shuffled after each epoch.\n",
    "```python\n",
    "dataloader = torch.utils.data.DataLoader(dataset, batch_size = 32, shuffle = True)\n",
    "```\n",
    "Here `dataloader` is a generator. To get data out of it, we need to loop through it or convert it to an iterator and call `next()`.\n",
    "\n",
    "```python\n",
    "# Looping through it, get a batch on each loop\n",
    "for images, labels in dataloader:\n",
    "    pass\n",
    "\n",
    "# Get one batch\n",
    "images, labels = next(iter(dataloader))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Augmentation\n",
    "A common strategy for training neural networks is to introduce randomness in the input data itself. For example, we can randomly rotate, mirror, scale, and/or crop our images during training. This will help our network generalize as it's seeing the same images bu in different locations, with different sizes, in different orientations, etc.\n",
    "\n",
    "To randomly rotate, scale and crop, then flip our images we would define our transform like this:\n",
    "```python\n",
    "\n",
    "train_transforms = transforms.Compose([transforms.RandomRotation(30),\n",
    "                                       transforms.RandomResizedCrop(100), transforms.RandomHorizontalFlip(),\n",
    "                                       transforms.ToTensor(),\n",
    "                                       transforms.Normailize([0.5]*3,[0.5]*3)\n",
    "```\n",
    "We'll also typically want to normalize images with the `transforms.Normalize`. We pass in a list of means and list of standard deviations, then the color channels are normalized like so\n",
    "`input[channel] = (input[channel] - mean[channel])/std[channel]`\n",
    "Substracting `mean` centers the data around zero and dividing by `std` squishes the values to be betweeb -1 and 1. Normalizing helps keep the network work weights near zero which in turn makes backpropagation more stable. Without normalizing, networks will tend to fail to learn.\n",
    "\n",
    "* When we're testing however, we'll want to use images that aren't altered (except we'll need to normalize the same way). So far validation/test images, we typically just resize and crop.\n",
    "\n",
    "\n",
    "```python\n",
    "train_transforms = transforms.Compose([\n",
    "    #transforms.Resize(255),\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.RandomResizedCrop(128),\n",
    "    transforms.RandomRotation(10),\n",
    "    transforms.ToTensor(),\n",
    "   transforms.Normalize([0.5,0.5,0.5],[0.5,0.5,0.5]),\n",
    "    \n",
    "])\n",
    "\n",
    "test_transforms = transforms.Compose([\n",
    "    transforms.RandomResizedCrop(128),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize([0.5,0.5,0.5],[0.5,0.5,0.5]),\n",
    "    \n",
    "])\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transfer Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24576"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}