5. contrib.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# High-Level Frameworks : contrib\n",
    "\n",
    "There are a number of good high-level Neural Network & Machine Learning framework based on TensorFlow.\n",
    "\n",
    "One of the easiest one you can pick right away is [**tensorflow.contrib**](https://www.tensorflow.org/get_started/tflearn) module.\n",
    "\n",
    "This one is built-in module like *Theano*'s *theano.nn* module. That is, it provide a decent abstraction for building DNN architecture, on the behalf of fast implementation and testing cycle of NN research.\n",
    "\n",
    "Some of high-level abstraction of contrib.learn has similar interface with [scikit-learn](http://scikit-learn.org/), for maximum simplicity and usability. So if you already have some experience with scikit-learn, it is really easy to get used to **contrib.learn**\n",
    "\n",
    "\n",
    "## 1. Training MLP w/ tf.contrlib.learn\n",
    "\n",
    "In this session, we will do the same thing with the previous one, but at this time we will use ***tf.contrib***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x7f7db2983650>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEolJREFUeJzt3X2wXHV9x/H3JwEUCZhArjFGICg4SNVivM2IIKUaIVIa\nsDMFEx9iB4mtMKMWO1qKlRar2Cqog09BMAHkyUJKHNEK0ZESLHJRCgFaojFA4k1yY0ACgiHw7R/n\nd+lhubt3s09nk9/nNXPmnj2/8/Dds/vZ87R7jyICM8vPhKoLMLNqOPxmmXL4zTLl8JtlyuE3y5TD\nb5ap7MMvaaKkxyQd0MVl3CLpfd2afzf0Yr1USdLBklq6zt3OtP1kpwt/ekOOds9IeqL0+F07Or+I\neDoiJkXEgy3W80JJn5X0YKrlfklnSlIr86tK7QdUu+ulyWUeKun3kpa0MY85ktZ2rqruk/RpSask\nbZd0dlV17FbVglsVEZNG+9OL/v6IuKne+JJ2i4jt3aglBfxaYD9gLnA/MBu4DJgB/E03llunlq49\nzy76CvDTqouowP3AR4Ezqixip9vyj0fSpyRdLelKSVuBd0s6QtJ/SXpE0rCkL0naPY2/m6SQNDM9\nvjy1f0/SVkk/kXRQncUdC7wF+POIuDcitkfErcB7gQ/VTHeIpCFJv5W0TNKUtLwXSbpC0m9SfT+V\nNDW1TZb0zVTzOkn/JGlCanu/pJtTrVuAf5b0qKRDS+vipWlvZL/U3SBpRNLDkr4jaUYa77PAEcDX\n0h7UF8ZYL5PTuhmRtFbS343u3aRafizpgvQc1kg6dpzX6d3ABuDHzb2yO07SPEl3pvXyoKRPjDHO\naZJ+nbqPlIZPkHSWpF9K2izpqtHXrF0RsSQivg881on5tWqXC3/yDuAK4MXA1cB24EPAVOBIiq30\nBxpMvwD4BLAv8CBwbp3x3gbcGhG/Lg+MiJUUb+y3lAa/N3UvAwRckIb/JfAi4OUUexAfBJ5MbZcB\nTwCvBN4A/Gkaf9SbgPuAAeAc4N+B+aX2U4AVEfEbitf6IuAA4EDgKeCLqd6PAT8B/irt6n94jOf6\nlVTnK9LzOjU9n3Itd6fncAFw8RjzAIoPEuCTFFu/bnoMeBcwGfgzig/kE2rGORo4GHg7cLakY9Lw\nj1Cs76MpXpvHgC81s1BJX5fU1LiVioidtgPWAnNqhn0K+OE4030U+Hbq3w0IYGZ6fDnwtdK484BV\ndeazBLi8TtsQ8LHUfwvwqVLb6ygCLmBRan9tzfQzKIL/gtKw9wA3pv73A2tqppkL3F96fBuwoE59\ng8BI6fEtwPtKj59dL8DuFB+gryq1nw7cVKrlf0pt+6Rpp9ZZ9peBM0uv15I23gNzgLVNjnsh8K+p\n/+BU48Gl9vOBr6f+1cAfl9r2T6/ZhNFpO/D+vQo4u5sZadTtdMf8TXqo/CDtCn+eYuv5Ioo39m0N\npt9Q6v8dMKnOeJuB19dpm57ax6rpAeAFFHsWSyj2Bq6RtA/F1v5siq3zC4CNpXOHEyg+8MaaJ8BN\nwGRJbwAeAQ4DrgeQNAn4AsWhyuQ0/t51aq/1EmBiqrv8HGaUHteuMyjWW3kdIGmQYms61t4FNeNO\nBH5bGvSqqNnLGmf6I4DPAH8A7EGxPq+sGa32dZmT+g8AviPpmZrxX9Ls8vvdrrrbX3sZ5uvAKopP\n+X2Af6DY6rbrJuBNkl5WHijpSOClwI9Kg/cv9R8A/B7YEhHbIuKciHg1cBTFIcu7KN6UvwP2jYjJ\nqdsnIl5Xms9znmcUJ/y+TbHrvwBYHhGPp+a/BQ4CZqd1UD4ked68amwCnqb4QCo/h/UNpqnnmFTH\nQ5I2UHwInCLp9toR4/+vOIx2TQc/uYrihOz+EfFi4Bs8/3WvfV1Gl7EOeFtp3U+OiBdGxAZ2Ebtq\n+GvtTbEFeVzSq2l8vL8j/gO4GbhO0mHpJNkRwKXAhRGxpjTue9Olrb2AfwSuiYiQ9BZJr0kn8h6l\nOBZ/JiIeojgZ9jlJ+6QTUAdLOnqcmq6gONZfkPpH7U3xYfKwpP0oPgDLNlIczz9PRDwF/BvwaUmT\n0onMj1AcIu2or1DsNh+euouA5cDxLcxrlFRcci13onjOWyLiSUlvBN45xrSfkLSnpNcCCynOEQF8\njeL5HpAW8BJJ89qosVzs7pJeSJG/3VK9Pc9iLuE/k+KF3UqxF3B149GbE8WB20nAfwI/SPO/lOKN\nU7tbexlFWIYpdqFH218GXEcR/Hso9iZGQ/tuYC/gXuBhiq36S8cp61aK4/OBVNOo8ylOgP4mjfO9\nmum+AMxPZ+vPH2O+HwS2URx2/BhYmp7rDomI30XEhtEOeBx4IiJGdnReJQdQnB8pdwcCfw18RsVV\nn7OAa8aY9hZgDcW6+kxE/DANPx/4PrAiTX8r8EfNFCPpG5IubDDKN1ONf0Fx4vMJig/rnlI68WBm\nmclly29mNRx+s0w5/GaZcvjNMtXTL/lMnTo1Zs6c2ctFmmVl7dq1bN68uanvsLQVfklzKb4fPhH4\nRkSc12j8mTNnMjQ01M4izayBwcHBpsdtebc/ffXyyxQ/iDiM4hrxYa3Oz8x6q51j/tnALyJiTURs\no/gq5YmdKcvMuq2d8M/guT+KWMdzf+gBgKRF6XfsQyMj7XyJy8w6qetn+yNicUQMRsTgwMBAtxdn\nZk1qJ/zree4vol5Oa7/yMrMKtBP+2yn+NdVBkvag+MXU8s6UZWbd1vKlvojYLukMip+1TgQuiYh7\nOlaZmXVVW9f5I+IG4IYO1WJmPeSv95plyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8Jtl\nyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/WaZ6eotu2/mc\ndtppDdsvv/zyhu0rV66s2zZr1qyWarLO8JbfLFMOv1mmHH6zTDn8Zply+M0y5fCbZcrhN8uUr/Nb\nQwceeGDD9ieffLJh++rVq+u2+Tp/tdoKv6S1wFbgaWB7RAx2oigz675ObPn/JCI2d2A+ZtZDPuY3\ny1S74Q/gB5LukLRorBEkLZI0JGloZGSkzcWZWae0G/6jImIW8HbgdElH144QEYsjYjAiBgcGBtpc\nnJl1Slvhj4j16e8mYBkwuxNFmVn3tRx+SXtJ2nu0HzgWWNWpwsysu9o52z8NWCZpdD5XRMT3O1KV\n9Y3xrvOPZ+nSpXXbTjnllLbmbe1pOfwRsQb4ww7WYmY95Et9Zply+M0y5fCbZcrhN8uUw2+WKf+k\n17pqjz32qLoEq8NbfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sU77Obw0tW7asrennz5/foUqs\n07zlN8uUw2+WKYffLFMOv1mmHH6zTDn8Zply+M0y5ev8mfv5z3/esP273/1uw/apU6c2bJ83b94O\n12S94S2/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5YpX+fP3LZt29pqnzCh8fZjzz333OGarDfG\n3fJLukTSJkmrSsP2lXSjpNXp75TulmlmndbMbv8SYG7NsI8DKyLiEGBFemxmO5Fxwx8RNwNbagaf\nCCxN/UuBkzpcl5l1Wasn/KZFxHDq3wBMqzeipEWShiQNjYyMtLg4M+u0ts/2R0QA0aB9cUQMRsTg\nwMBAu4szsw5pNfwbJU0HSH83da4kM+uFVsO/HFiY+hcC13emHDPrlXGv80u6EjgGmCppHfBJ4Dzg\nGkmnAg8AJ3ezSOuea6+9tuoSrCLjhj8i6t114a0drsXMeshf7zXLlMNvlimH3yxTDr9Zphx+s0z5\nJ72ZGx4eHn8k2yV5y2+WKYffLFMOv1mmHH6zTDn8Zply+M0y5fCbZcrhN8uUw2+WKYffLFMOv1mm\nHH6zTDn8Zply+M0y5fCbZcq/59/FjXeL7V/96ldtzf/QQw9ta3qrjrf8Zply+M0y5fCbZcrhN8uU\nw2+WKYffLFMOv1mmfJ1/F/f44483bF+5cmVb858zZ05b01t1xt3yS7pE0iZJq0rDzpG0XtKdqTu+\nu2WaWac1s9u/BJg7xvALIuLw1N3Q2bLMrNvGDX9E3Axs6UEtZtZD7ZzwO0PSXemwYEq9kSQtkjQk\naWhkZKSNxZlZJ7Ua/q8CrwQOB4aBz9cbMSIWR8RgRAwODAy0uDgz67SWwh8RGyPi6Yh4BrgImN3Z\nssys21oKv6TppYfvAFbVG9fM+tO41/klXQkcA0yVtA74JHCMpMOBANYCH+hijdaG4eHhrs5/7tyx\nLgTZzmDc8EfE/DEGX9yFWsysh/z1XrNMOfxmmXL4zTLl8JtlyuE3y5R/0ruLO/fcc9ua/oQTTmjY\nPmvWrLbmb9Xxlt8sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5Sv8+/iVqxY0db0U6bU/Q9tAEyc\nOLGt+Vt1vOU3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl6/y7gI0bN9Zte+qpp3pYie1MvOU3\ny5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLVzC269wcuBaZR3JJ7cUR8UdK+wNXATIrbdJ8cEQ93\nr1SrZ9GiRXXbHnnkkbbmvWDBgramt/7VzJZ/O3BmRBwGvBE4XdJhwMeBFRFxCLAiPTazncS44Y+I\n4Yj4WerfCtwHzABOBJam0ZYCJ3WrSDPrvB065pc0E3g9cBswLSKGU9MGisMCM9tJNB1+SZOAa4EP\nR8Sj5baICIrzAWNNt0jSkKShkZGRtoo1s85pKvySdqcI/rci4ro0eKOk6al9OrBprGkjYnFEDEbE\n4MDAQCdqNrMOGDf8kgRcDNwXEeeXmpYDC1P/QuD6zpdnZt3SzE96jwTeA9wt6c407CzgPOAaSacC\nDwAnd6dEW7duXcP2O+64o+V5z5kzp2H7cccd1/K8rb+NG/6IuAVQnea3drYcM+sVf8PPLFMOv1mm\nHH6zTDn8Zply+M0y5fCbZcr/unsnsGnTmF+efNb69etbnvfChQsbthff8bJdkbf8Zply+M0y5fCb\nZcrhN8uUw2+WKYffLFMOv1mmfJ1/F/fmN7+5Yfu8efN6VIn1G2/5zTLl8JtlyuE3y5TDb5Yph98s\nUw6/WaYcfrNM+Tr/TmDWrFkN24u7pZntGG/5zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMjRt+\nSftL+pGkeyXdI+lDafg5ktZLujN1x3e/XDPrlGa+5LMdODMifiZpb+AOSTemtgsi4nPdK8/MumXc\n8EfEMDCc+rdKug+Y0e3CzKy7duiYX9JM4PXAbWnQGZLuknSJpCl1plkkaUjS0MjISFvFmlnnNB1+\nSZOAa4EPR8SjwFeBVwKHU+wZfH6s6SJicUQMRsTgwMBAB0o2s05oKvySdqcI/rci4jqAiNgYEU9H\nxDPARcDs7pVpZp3WzNl+ARcD90XE+aXh00ujvQNY1fnyzKxbmjnbfyTwHuBuSXemYWcB8yUdDgSw\nFvhAVyo0s65o5mz/LcBYN2m/ofPlmFmv+Bt+Zply+M0y5fCbZcrhN8uUw2+WKYffLFMOv1mmHH6z\nTDn8Zply+M0y5fCbZcrhN8uUw2+WKYffLFPq5e2dJY0AD5QGTQU296yAHdOvtfVrXeDaWtXJ2g6M\niKb+X15Pw/+8hUtDETFYWQEN9Gtt/VoXuLZWVVWbd/vNMuXwm2Wq6vAvrnj5jfRrbf1aF7i2VlVS\nW6XH/GZWnaq3/GZWEYffLFOVhF/SXEn/K+kXkj5eRQ31SFor6e502/Ghimu5RNImSatKw/aVdKOk\n1envmPdIrKi2vrhte4Pbyle67vrtdvc9P+aXNBG4H3gbsA64HZgfEff2tJA6JK0FBiOi8i+ESDoa\neAy4NCJek4b9C7AlIs5LH5xTIuJjfVLbOcBjVd+2Pd1Nanr5tvLAScD7qHDdNajrZCpYb1Vs+WcD\nv4iINRGxDbgKOLGCOvpeRNwMbKkZfCKwNPUvpXjz9Fyd2vpCRAxHxM9S/1Zg9Lbyla67BnVVoorw\nzwAeKj1eR4UrYAwB/EDSHZIWVV3MGKZFxHDq3wBMq7KYMYx72/ZeqrmtfN+su1Zud99pPuH3fEdF\nxCzg7cDpafe2L0VxzNZP12qbum17r4xxW/lnVbnuWr3dfadVEf71wP6lxy9Pw/pCRKxPfzcBy+i/\nW49vHL1Dcvq7qeJ6ntVPt20f67by9MG666fb3VcR/tuBQyQdJGkP4J3A8grqeB5Je6UTMUjaCziW\n/rv1+HJgYepfCFxfYS3P0S+3ba93W3kqXnd9d7v7iOh5BxxPccb/l8DfV1FDnbpeAfx36u6pujbg\nSordwKcozo2cCuwHrABWAzcB+/ZRbZcBdwN3UQRtekW1HUWxS38XcGfqjq963TWoq5L15q/3mmXK\nJ/zMMuXwm2XK4TfLlMNvlimH3yxTDr9Zphx+s0z9H4+8vlSZyOOwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7de37dd710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# load libraries\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as mtick\n",
    "import numpy as np\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "from tensorflow import nn\n",
    "from tensorflow.contrib.learn.python.learn.estimators import model_fn as model_fn_lib\n",
    "\n",
    "\n",
    "# load data\n",
    "# with one_hot=True, categorical labels (e.g. {1,2,3...}) coded into binary vector form (e.g.{[1,0,0],[0,1,0]...})\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)\n",
    "\n",
    "# visual check for the loaded dataset\n",
    "disp_target = 4\n",
    "plt.imshow(mnist.train.images[disp_target].reshape((28,28)),cmap='gray_r')\n",
    "plt.title(\n",
    "    'Train Observation {:d} - Label : {:d}'.format(\n",
    "        disp_target,\n",
    "        np.argmax(mnist.train.labels[disp_target])\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Build MLP Architecture From Scratch\n",
    "\n",
    "Now we will build a Multi-Layer Perceptron with 3 Fully-Connected (Dense) Layer. The structure is as below equations:\n",
    "\n",
    "\\begin{align}\n",
    "h_1 & = relu(W_1 \\cdot x + b_1) \\\\\n",
    "h_2 & = relu(W_2 \\cdot h_1 + b_2) \\\\\n",
    "h_3 & = relu(W_3 \\cdot h_2 + b_3) \\\\\n",
    "o & = W_o \\cdot h_3 + b_o \\\\\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# make a inference expression w/ Variables\n",
    "x = tf.placeholder(tf.float32, shape=[None,28*28])\n",
    "y = tf.placeholder(tf.float32, shape=[None,10])\n",
    "\n",
    "# 3 hidden layers\n",
    "h1 = tf.contrib.layers.relu(x,100)\n",
    "h2 = tf.contrib.layers.relu(h1,100)\n",
    "h3 = tf.contrib.layers.relu(h2,100)\n",
    "\n",
    "# read out layer\n",
    "o = tf.contrib.layers.linear(h3,10)\n",
    "\n",
    "# EVEN MORE SIMPLER VERSION! (!CAUTION! over simplified)\n",
    "# relu_lyr = tf.contrib.layers.relu # alias\n",
    "# lin_lyr = tf.contrib.layers.linear # alias\n",
    "# \n",
    "# o = lin_lyr(relu_lyr(relu_lyr(relu_lyr(x,100),100),100),10)\n",
    "#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3. Setup Cost Function & Get Train Op\n",
    "\n",
    "In this section, we will define cost function for training. We will at first get a probability distribution over categories from logits by applying Softmax function. After then, it is compared with target distribution (one-hot distribution of labels). We will minimize the distance between the two distributions (inferences and labels) by categorical cross-entropy, so we will finally have a classification model for handwritten digits!\n",
    "\n",
    "\\begin{align}\n",
    "\\underset{W,b}{\\text{min}}\\frac{1}{|M|}\\sum_{x,y\\in M}crossentropy(y,y')\\\\\n",
    "y' = softmax(o)\\\\\n",
    "\\end{align}\n",
    "\n",
    "($M$ here means mini-batch)\n",
    "\n",
    "After we compile cost function, we can have symbolic expression for its gradient. With the computation graph of tensorflow automatically calculate partial derivatives of the cost w.r.t the parametes. Then we can simply plug them in update rule, which is [Stochastic Gradient Descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) in our case.\n",
    "\n",
    "$$ W \\leftarrow W - \\alpha \\cdot \\frac{\\partial{C}}{\\partial{W}}$$\n",
    "\n",
    "$$ b \\leftarrow b - \\alpha \\cdot \\frac{\\partial{C}}{\\partial{b}}$$\n",
    "\n",
    "(we ommit the sub letters indicating layer here)\n",
    "\n",
    "($\\alpha$ is learning rate, which is usually set with small number (e.g. 0.01))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# set cross_entropy loss function with y (target) and o (logits)\n",
    "cross_entropy = tf.reduce_mean(\n",
    "    nn.softmax_cross_entropy_with_logits(\n",
    "        labels=y,\n",
    "        logits=o\n",
    "    )\n",
    ")\n",
    "\n",
    "# then we'll get update rule (train op) here\n",
    "train_step = tf.train.GradientDescentOptimizer(learning_rate=0.1).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Traning\n",
    "\n",
    "Now, we have every parts we need. Let's try the first run of DNN training.\n",
    "\n",
    "Before get into running, we will prepare simple evaluation function to check it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# binary accuracy calculation\n",
    "correct_prediction = tf.equal(tf.argmax(o,1), tf.argmax(y,1))\n",
    "# ...and its mean per feed\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's show time!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0th Iteration - Train Acc: 12.00% - Test Acc: 10.14%\n",
      "100th Iteration - Train Acc: 94.00% - Test Acc: 87.15%\n",
      "200th Iteration - Train Acc: 95.00% - Test Acc: 89.58%\n",
      "300th Iteration - Train Acc: 97.00% - Test Acc: 91.91%\n",
      "400th Iteration - Train Acc: 93.00% - Test Acc: 92.19%\n",
      "500th Iteration - Train Acc: 97.00% - Test Acc: 93.34%\n",
      "600th Iteration - Train Acc: 97.00% - Test Acc: 94.32%\n",
      "700th Iteration - Train Acc: 99.00% - Test Acc: 94.28%\n",
      "800th Iteration - Train Acc: 97.00% - Test Acc: 94.58%\n",
      "900th Iteration - Train Acc: 99.00% - Test Acc: 94.72%\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f7db06bce50>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9+P/Xe7JvZIUQCBBkERDZxA2whlIXWgvVYl2w\nXvXr12qvW3tr622trV5vv7bX21+rtbbUqrVVcKtir6J1yxV3RQF1BiRAgJAJkIVM9mRmPr8/zslk\nJhBIwkwmk3k/H488Zs6Zc86856Oc9zmf9zmfI8YYlFJKKQBHtANQSik1dGhSUEopFaBJQSmlVIAm\nBaWUUgGaFJRSSgVoUlBKKRWgSUHFBRFJEJEmERkfzmVjhYhUikhptONQQ58mBTUk2Tvlrj+/iLQG\nTa/s7/aMMT5jTKYxZnc4l+0vEblLRDrt33FQRN4WkVPC/T19iOGRwfxOFTs0Kaghyd4pZxpjMoHd\nwNeD5j3Wc3kRSRz8KAfsMft3jQTWA09FOR6lAjQpqJhkH+0+ISKrRaQRuExETheR9+wjcLeI3Csi\nSfbyiSJiRKTEnv6b/fk6EWkUkXdFZGJ/l7U/XyoiX4hIg4jcZx/9X3G032CM6QQeB8aLSG7Q9paJ\nyCb7d7wlIjODPvuxiFSJiEdEtnR1Cdkx/jxoua+ISMVh2u084IfASvtsZUMfm1zFCU0KKpadj7VT\nzQaeALzATUABsBA4F/jOEda/FPgpkId1NvIf/V1WREYBTwK32N+7E+hTd5CIpACXAwcAjz3vZOBP\nwNVAPvAQsFZEkkXkBPv3zDPGjACW2rH0mTHmf4BfYZ+tGGNO6s/6avjTpKBi2VvGmH8YY/zGmFZj\nzIfGmPeNMV5jzA5gFXDmEdZ/2hjzkX3E/hgwZwDLngdsNMastT/7/4Cao8R9qYgcBFqAfwFWGGN8\n9mfXAL+3f4vPGPOQPf9krKSXCpwgIonGmJ3271QqbDQpqFi2J3hCRKaJyAsiUi0iHuBOrKP33lQH\nvW8BMgew7JjgOIw1wmTlUeJ+3BiTA4wGtgJzgz6bAPzI7jo6aCePImCsMWYr8G9Yv2u/3XU2+ijf\npVS/aFJQsaznEL9/BD4DJtvdK7cDEuEY3EBx14SICDC2LysaYw5gnRncJSKF9uw9wB3GmJygv3Rj\nzJP2On8zxiwEJgIJwP+z12sG0oM2f6RkoUMjq15pUlDDSRbQADSLyHSOXE8Il/8B5onI1+0roG7C\nuqqoT4wxTuA14Af2rD8B/yoiJ4sl0952hohMF5HFdi2i1f7z2+ttBL4mIrkiUgTceISv3QeU2AlM\nqRCaFNRw8m9YffSNWGcNT0T6C40x+4CLgF8DtcAk4BOgvR+b+S/gOhEpMMa8B1wHPADUA18Al9nL\npWAViWuwurNygZ/Ynz0CuIBdwEvAmiN83xNAMlAnIh/0I04VB0QfsqNU+IhIAlCFVTxeH+14lOov\nPVNQ6hiJyLkikmN36/wU6AT0CFzFJE0KSh27RcAOrPsNzgHON8b0p/tIqSFDu4+UUkoF6JmCUkqp\ngFgaRAyAgoICU1JSMqB1m5ubycjICG9AMUzbI5S2Rzdti1DDoT02bNhQY4w56uXSMZcUSkpK+Oij\njwa0bllZGaWlpeENKIZpe4TS9uimbRFqOLSHiOzqy3LafaSUUipAk4JSSqkATQpKKaUCIpYUROQh\nEdkvIp/18rnYDy4pF5HNIjIvUrEopZTqm0ieKTyC9ZCT3iwFpth/12CN9aKUUiqKIpYUjDFvAnVH\nWGQ58KixvAfk2KM7KqWUipJoXpI6ltCHpFTa89w9FxSRa7DOJigsLKSsrGxAX9jU1DTgdYcjbY9Q\n2h7dtC1CxVN7xMR9CsaYVViPVmT+/PlmoNcLD4drjcNJ2yOUtke3aLVFY1sne+pa2V3XQmV9C35j\nGJ+Xzjj7b0Rq0qDHBNFpD7/fcKCpnT11Ley2/5ZMK+TE4uyIfm80k8JeYFzQdLE9Tyk1THX6/LgP\ntrGnvntHt7uuhUr7tb6l84jr56QnWUki10oSVsJIY3xeOmNy0khKiK0LKpvaveypawns+AOv9a3s\nqWuh3esPWT4/M2VYJ4XngetFZA1wKtBgjDmk60gpFTuMMdQ1d7CnvjWwk+ve0bVQdbANn797EM5E\nh1Ccm8a4vHSWnlgU2OGPt3f4AHvqW0KOlvfUt+J0e/ins5pOX/e2HAJjctK6189Ppzg3LbCtvIxk\nBvthc16fH3dDW0gb7O46E6proba5I2T5zJRExuelM2lkBouPH8n4vHSK7fjH5qSRmpQQ8ZgjlhRE\nZDVQChSISCXwMyAJwBjzB+BF4KtAOdaD0K+MVCxKqfBp6/RRWd8S6ObZ3eNIt7nDF7J8QWYy4/LS\nmTc+l+Wz04O6g9Ioyk4jwXHkHXV2ejYzxx56dOzzG/Z52g4529hd18JrW/ZT0xQ6enl6ckL3d+em\nMz4vjfH53WcdA9nhGmM42NIZtMPvSoRW21QdbMXbIwmOybES1dknjA6c5XQlw5z0pEFPXD1FLCkY\nYy45yucG+NdIfb+KDR1eP0kJEvV/CENFc7sX3xAYzr6uzc8HO+sOe7S/zxO6s01NcgSOzk87Lj/Q\nrTM+zzpSz0iJzG4mwd7BjslJ47Tj8g/5vKXDS2V9K7trQ3fYu2tbeGtbDa2doclrVFZKUJdU147a\nOotxN/l5Y+v+kMSzp87q4mls94ZsJz8jmeK8dGaPy+Hrs4sCbTMuL52i7FQSh3gXV0wUmlXsCj6a\nC925WEdSBxrbKcpOZeHkAhZOzmfhpAJGjUiNdtiDpr65g3d31PJ2eQ3vbK9lZ01ztEPqVvYuACJQ\nNCKVcXnpnDFlZPeRbZ61wxyZmTIkk3p6ciJTC7OYWph1yGfGGGqaOrq7poISxwc763hu414Oyc1v\nfQhASqIjkDROLskNSSTj8tLJjFASHCyxHb0aEhpaOw85mtxtH0XtrW+lw9ddLHMIFGWnMS4vjcXH\nj2R0dhrl+xt51bWPpzdUAjBlVCYLJxewYFI+p03Kj9oVJ5HQ2uHjw4o63i6v4e3tNXxe5cEYqy/5\n1Il5rDipmJTE6B9J7t65nSWnzWFcbhpjc9NISYx8X/ZgEhFGZqUwMiuFeeNzD/m8w+un6mBr4P/n\nivIvOGfhSYzPS6cgMwXHUbq8YpkmBXVUPf+B9Ow3bWgNvWIkO826QmRG0QjOPqEwpM90TE4ayYfZ\n6fn9BqfbY+8sa1nz4W4eeacCh8Cs4hzrLGJyAfPG5w5KsS1cvD4/myobeKe8hrfKa/hk90E6fFaX\n2bzxuXzvK1NZOLmAWcXZQ+rKmTLfbs6cetSh94et5EQHJQUZlBRYz1Aoa93J/JK8KEc1ODQpKIwx\n1DZ3hPS5du/8W3E3tBJUKyM5wRG4YmT2uOyQSwTH5aWTndb/I3uHQ5g51ioofufMSbR7fXyy+2Bg\nZ/qH/93B/W9sJyXRwckleYHuphPGZB+1UDmYjDFs29/EW9tqeGd7De/tqKOp3YsIzCgawZULS1gw\nuYCTS3JJT9Z/fmro0f8rhyG/39DU4aWhpZOG1k48rfZrm/Xa9ff5jjb+3ydvsruupdei2ykT8+yr\nNdICl/kVZqVG/PQ5JTGB047L57Tj8vn+2cfT2NbJBzvreKu8hnfKa/nlS1sA66zk9OPyA2cSEwsy\nBr1/e+/BVusMx64LHGi0CrEl+eksmzOGhZMKOH1SPnkZyYMal1IDoUlhiPL5Ta87864/T6s3sEzw\nX2NbZ8iRfU8JDiE7LYl08TN9fDoLJxeEXJ5XnJtOWvLQ6qLJSk1iyfRClkwvBGB/Yxvvbq+1d8a1\nvPR5NQBF2aksmFTAoimRK1r3VhwuyEwJFMsXTM6nODc97N8dccZAeyPJ7fXQuM+qMgMgvbwndH5g\nOkzvjR98neDrAL/Xeu/vtF+Dp71B83tOe495Gycc2AdVf7DiMX4wvu73fn8v831Wex52vt/+7HDz\ng9bz+0K3/bX/hvlXhe0/9+FoUhhklfUtvLFl/2F38N3vOw+5zK2n5AQHI9KSGJGWSHZaEvmZyRw3\nMoPstKTA34i0JEakdk9np1uvGckJiIh96/78Qfrl4TUqK5Xlc8ayfM5YjDHsqm3h7e3W0fprW/bx\nzMdW0XryqEwWHWPR+mjF4W+fNoGFkwuYWpg5NK7C8XZAWwO0HbReWw/a74One34eNG38LAB4N9o/\nZJCIAxxJkJAEjkT7NQkSEgPz01rbwNMCkmAtLw5w9HyfdOh8SbCS22Hnd63r6GV+0Lpd80fPjnhz\naFIYZL940cWLn1pHtalJDmvnbe+4i7JTmTY6ixFBO/aunXt2j7/UJMfQ2AENASISKAquPHXCIUXr\nJz7cc2jRelIB8yYcvmgd9eKw3w8djfaOug878Z7TnS1H3n5CCqTlQGo2pOZAxkgomNI9nZrNFxV7\nmTp1irV88LWZxgCml/dY08f0nu73AIi1c05IPmRHHboD732HHjqdfOhnjqP/N/wojsbF0qQwyD7b\n6+HsGYXcd+ncYXeZ31BxpKL129trDylaL5icz5xxOfyzopO//eVD3t9RR2O4isPeDmitg5Y6aKnt\n8b7eem2ps+fb79s9VndBrwRSRwR24KTlQMHk0OnUnB7T2d07/aSjd6lVdZYx9eTS/v1WNSxoUhhE\njW3W7fAXnTxOE8IgCilaQ6Bo/Xa5VRf41UtbA8uW5Dfx9d6Kwx0t0FB9mB18j5164H29dcTfm6QM\nSM+DtFxIz4ec8ZCWF3oUH7JTt19TRvTp6FapgdCkMIi2VFs7iOlFh95hqQbA7w8tGAYXDf3e7kJh\nj/dZvk6WJHpZMqUTJnnxNLdSWXOQjuotzJlQYO3Ut9bCJ8E7/DrwtvYeS0o2pNs79/QCKJhqvU/L\ns3b86Xn2+/zu9304YldqsGlSGEQutweAGUWRHfp2yDIGmvZBzTao3QY15VBbDh3NoVd9+H2Hf++z\nd/xdyxKeMYJGADO6JnZJ95F7eh5kF0PRbGuHH7xTD97hp+Va/dNKDQOaFAaRs8pDbnoShSNSoh1K\nZHW2Qd12qPnC3vFvsxNBudVf3iUxDfInWV0iiamQkmUX/hJ6FBATQouJwQXF/iybYE/3suxbHztZ\ntORr2jWj4pomhUHkcnuYMWbE8LhqyBhodIce9dd8Yb0/uIeQo/gRYyF/Msy6yLrKJX+y9TqieEjt\ngL1Je4ZUPEpFgyaFQeL1+dlS3ci3T5sQ7VD6p6Ol96P+jqbu5ZLSrZ198ckw+1Jrp9+VAJIzohe/\nUqpfNCkMkoraZtq9fqYXjYh2KIcyBjx7u3f2Ndvso/5yaNgTumz2OGtHP2dl6FF/1hg9ylZqGNCk\nMEg+r7KLzGOimBS6+voPbIWaL5jufAu2/BRqt0Nn0Dj+yZnWzn78aZD/besa+IKpkDcJkmNw6Aal\nVJ9pUhgkLncjSQnCpJGZkf+ytgY48AXUbA0kAA5shYO7gm6KEkakjoSxJ8KEBfZRv93lk1UUOraN\nUipuaFIYJE63hymjsg77LIEB6bq8M3inX7PVSgZN1d3LOZKso/6iWXDihTDyeOuoP38y77/zQdzc\nuq+U6htNCoPE5fbwpSkDeGiJ3wf1FUE7/q7XbdDe0L1cchaMnAqTvmy9Ftg7/9wS61JMpZTqA91b\nDIIDje0caGw/cj2hs80u8m4N6vqxi72+oAelZ4yyjvZPXNF91D/yeO3yUUqFhSaFQdB1J/P0oixr\ngDT3xtDunpqtUL+LkJEhcydYR/uTv2y9jjze6u9PO/R5skopFS6aFAaBMzC8xQhY90PY8LD1QUKK\n1d8/Zi7Mutja6Y883pqXlBbFiJVS8UqTwiBwuT2MyU4lJz0Zdr8LExbCsvus/n6HjpaqlBo69G6j\nQdA1vAUdzVahuOQMa8wfTQhKqSFGk0KEtXX62H6g2bqTufoz6z6Bosg/Uk8ppQZCk0KEbdvXhM9v\nrHqCe6M1c8yc6AallFK90KQQYU63dS/B9KIR4N5kPQ83qyjKUSml1OFpUogwl7uRjOQExuelQ9VG\nKJqj9xMopYYsTQoR5qzyMK1oBA5fGxzYovUEpdSQpkkhgowxuNwe66a1fZ+D8Wk9QSk1pGlSiKDK\n+lYa273WM5mrPrFmFmlSUEoNXZoUIsgZPLyFe5P1oPfs4ihHpZRSvdOkEEHOKg8OgWmj7ctRx2iR\nWSk1tGlSiCCX20NJQQZpDi/sd2mRWSk15EU0KYjIuSKyVUTKReTWw3w+XkTeEJFPRGSziHw1kvEM\nNqfbY920tu9z8Hu1nqCUGvIilhREJAG4H1gKzAAuEZEZPRa7DXjSGDMXuBj4faTiGWyetk4q61u7\nb1oDPVNQSg15kTxTOAUoN8bsMMZ0AGuA5T2WMUDXk2eygaoIxjOotrgbAayB8NwbITXbGhVVKaWG\nsEgmhbHAnqDpSntesJ8Dl4lIJfAicEME4xlUzipreIsZXWcKRbO1yKyUGvKi/TyFS4BHjDH/LSKn\nA38VkZnGGH/wQiJyDXANQGFhIWVlZQP6sqampgGv21+vf9ZOVjK4PnyTke5PqSz+OjsG6bv7ajDb\nIxZoe3TTtggVT+0RyaSwFxgXNF1szwv2f4BzAYwx74pIKlAA7A9eyBizClgFMH/+fFNaWjqggMrK\nyhjouv3135++xezxSZTOSIP1Xsafch7jTxyc7+6rwWyPWKDt0U3bIlQ8tUcku48+BKaIyEQRScYq\nJD/fY5ndwBIAEZkOpAIHIhjToPD6/Gzd12jftNY1XPbc6AallFJ9ELGkYIzxAtcDLwMurKuMPheR\nO0Vkmb3YvwH/V0Q2AauBK4wx5vBbjB07aprp8PrtIvMmSM6C3InRDksppY4qojUFY8yLWAXk4Hm3\nB713AgsjGUM0uALDW4yAjzZaRWaH3ieolBr6dE8VAc4qD8kJDiblp8K+z3RkVKVUzNCkEAFOt4cp\nhZkk1W0Db5vetKaUihmaFCLA1TW8RZVdZNbhLZRSMUKTQpjtb2yjpqmje3iLpAzInxTtsJRSqk80\nKYSZs8oqMgeGtyiaBY6EKEellFJ9o0khzFz2mEfTR2VA9adaT1BKxRRNCmHmdHsYm5NGdksFdLZo\nPUEpFVM0KYSZy+3R4bKVUjFLk0IYtXX62HGgiRldw1skpkHB1GiHpZRSfaZJIYy2VjfiN3QPbzF6\nJiREeyBapZTqO00KYRQY3mJ0Jrg3az1BKRVzNCmEkdPtITMlkXGmGjoatZ6glIo5mhTCyOX2MG10\nFo5qu8isYx4ppWKMJoUw8fsNLndj901rCSkwclq0w1JKqX7RpBAmlfWtNLV7rctRqzZC4QmQkBTt\nsJRSql80KYSJ090AwIzRWVaRWbuOlFIxSJNCmDjdjTgEpqXWQnuDFpmVUjFJk0KYOKs8HDcyk5T9\nm60ZejmqUioGaVIIk5DhLRxJMGp6tENSSql+06QQBg0tnew92Go9WMe9EQpnQGJKtMNSSql+06QQ\nBq7q4DuZN2k9QSkVszQphEHX8BYzMxqgtV7rCUqpmKVJIQycVR4KMpPJb3RZMzQpKKVilCaFMHBV\nW0VmcW8CSbBuXFNKqRikSeEYdfr8fFHdZBeZN1lXHSWlRjsspZQaEE0Kx2jHgWY6fH6mj86yhrfQ\nriOlVAzTpHCMuoa3mJXdDC01OryFUiqmaVI4Ri53I8mJDia0b7Nm6OWoSqkYpknhGDmrPBxfmEXC\nvs0gDiicGe2QlFJqwDQpHANjjD28hV1PKDgektOjHZZSSg2YJoVjsL+xndrmju4rj7SeoJSKcUdN\nCiJyg4jkDkYwscZp38k8K7sNmqq1nqCUinl9OVMoBD4UkSdF5FwRkUgHFSucVVZSmMZ2a4ZejqqU\ninFHTQrGmNuAKcCfgSuAbSLyCxGZFOHYhjyX20NxbhrpNZ8DAqNPjHZISil1TPpUUzDGGKDa/vMC\nucDTIvKrCMY25DkDz1DYCAVTICUz2iEppdQx6UtN4SYR2QD8CngbONEYcx1wEvDNCMc3ZLV2+Kio\nae4uMms9QSk1DPTlTCEPuMAYc44x5iljTCeAMcYPnHekFe0axFYRKReRW3tZ5lsi4hSRz0Xk8X7/\ngijZuq8Rv4HZuZ3g2av1BKXUsJDYh2XWAXVdEyIyAphujHnfGOPqbSURSQDuB84CKrGK1c8bY5xB\ny0wB/h1YaIypF5FRA/wdg66ryHyiY6c1Q88UlFLDQF/OFB4AmoKmm+x5R3MKUG6M2WGM6QDWAMt7\nLPN/gfuNMfUAxpj9fdjukOBye8hKSaQg8AyFWdENSCmlwqAvZwpiF5oBq9tIRPqy3lhgT9B0JXBq\nj2WmAojI20AC8HNjzEuHBCByDXANQGFhIWVlZX34+kM1NTUNeN2e3t3SSlE61Hz6GhlpY/jgvU/C\nst3BFM72GA60PbppW4SKp/boy859h4jcSPfZwXeBHWH8/ilAKVAMvCkiJxpjDgYvZIxZBawCmD9/\nviktLR3Ql5WVlTHQdYP5/Qb36y+z4qRiRu7YC8edFpbtDrZwtcdwoe3RTdsiVDy1R1+6j64FFgB7\n6T7av6YP6+0FxgVNF9vzglUCzxtjOo0xO4EvsJLEkLa7roXmDh9zCvzQsFuHt1BKDRtHPVOw+/kv\nHsC2PwSmiMhErGRwMXBpj2WeAy4BHhaRAqzupHCdhUSMyx7eYnbCLmuGFpmVUsPEUZOCiKQC/wc4\nAQg8Z9IYc9WR1jPGeEXkeuBlrHrBQ8aYz0XkTuAjY8zz9mdni4gT8AG3GGNqB/xrBonT7SHBIYzT\nZygopYaZvtQU/gpsAc4B7gRWAr1eihrMGPMi8GKPebcHvTfA9+2/mOFyeziuIIOkfZsgZwKk6XiB\nSqnhoS81hcnGmJ8CzcaYvwBf49CriOKKs8rDjDE6XLZSavjpS1LotF8PishMIBuImZvMwu1gSwdV\nDW3MLhCo36ldR0qpYaUv3Uer7Ocp3AY8D2QCP41oVENY1zMUTkrZbc3Q4S2UUsPIEZOCiDgAj33H\n8ZvAcYMS1RDmcjcCMKmz3JqhSUEpNYwcsfvIHvTuh4MUS0xwuT0UZKaQWfcZZI+DjPxoh6SUUmHT\nl5rCqyLyAxEZJyJ5XX8Rj2yICikyaz1BKTXM9KWmcJH9+q9B8wxx2JXU4fVTvr+JJcelwe5ymDWQ\ne/qUUmro6ssdzRMHI5BYsP1AEx0+P6em2aN16OWoSqlhpi93NF9+uPnGmEfDH87Q1jW8xfH+7dYM\n7T5SSg0zfek+OjnofSqwBPgYiLuk4KzykJLoIL/RBVljIDNub9dQSg1Tfek+uiF4WkRysB6YE3dc\n1R6OH52FQ4vMSqlhqi9XH/XUDMRdncEYg7PKw+zCJKj5QusJSqlhqS81hX9gXW0EVhKZATwZyaCG\non2edupbOjk9vQoweqaglBqW+lJTuCfovRfYZYypjFA8Q5bT3QDACY6d1gy9k1kpNQz1JSnsBtzG\nmDYAEUkTkRJjTEVEIxtiuoa3GNO8FTJGQdboKEeklFLh15eawlOAP2jaZ8+LK84qD+Py0kja/6lV\nTxCJdkhKKRV2fUkKicaYjq4J+31y5EIamlxuD7MLU+DAFq0nKKWGrb4khQMisqxrQkSWAzWRC2no\naenwsrO2mUUj9oHxaT1BKTVs9aWmcC3wmIj8zp6uBA57l/NwtaW6EWNgdqDIrGcKSqnhqS83r20H\nThORTHu6KeJRDTFdw1uMa98G6fmQXRzliJRSKjKO2n0kIr8QkRxjTJMxpklEckXkrsEIbqhwVnnI\nSk0ko+4zq+tIi8xKqWGqLzWFpcaYg10T9lPYvhq5kIYel9vDrNGpyH6Xdh0ppYa1viSFBBFJ6ZoQ\nkTQg5QjLDyt+v2FLdSNnZh8Av1eHt1BKDWt9KTQ/BrwmIg8DAlwB/CWSQQ0lu+paaOnwMS9plzVD\nzxSUUsNYXwrNvxSRTcBXsMZAehmYEOnAhgpnlVVkPs5bDqk5kBM3P10pFYf6OkrqPqyEcCHwZcAV\nsYiGGJfbQ4JDyDnotM4StMislBrGej1TEJGpwCX2Xw3wBCDGmMWDFNuQ4HR7OL4gGccBJ5x2XbTD\nUUqpiDrSmcIWrLOC84wxi4wx92GNexRXXG4Pi/PqwNeh9QSl1LB3pKRwAeAG3hCRP4nIEqxCc9yo\nb+7A3dDGySl7rBk6vIVSapjrNSkYY54zxlwMTAPeAG4GRonIAyJy9mAFGE1ddzJP9ZdDygjIjbsH\nziml4sxRC83GmGZjzOPGmK8DxcAnwI8iHtkQ4LSTwsimrVbXkWMgTy9VSqnY0a+9nDGm3hizyhiz\nJFIBDSVOt4eizESSDnyu9QSlVFzQQ98jcLkb+XLBQfC2aT1BKRUXNCn0osPrp3x/IwvS7SKzDm+h\nlIoDmhR6Ub6/iU6fYTo7IDkT8iZFOySllIq4iCYFETlXRLaKSLmI3HqE5b4pIkZE5kcynv7oKjIX\ntWyF0bO0yKyUigsR29OJSAJwP7AUmAFcIiIzDrNcFnAT8H6kYhkIl9tDehKk1jq1yKyUihuRPPw9\nBSg3xuwwxnQAa4Dlh1nuP4BfAm0RjKXfnFUevlzgQTpbtJ6glIobfRk6e6DGAnuCpiuBU4MXEJF5\nwDhjzAsicktvGxKRa4BrAAoLCykrKxtQQE1NTX1a1xjD5j0t3Jj9GQAfVHbQUj+w7xzK+toe8ULb\no5u2Rah4ao9IJoUjEhEH8Gus5zMckTFmFbAKYP78+aa0tHRA31lWVkZf1q062Erzy69zRv5BaEvj\nlKUrwZEwoO8cyvraHvFC26ObtkWoeGqPSHYf7QXGBU0X2/O6ZAEzgTIRqQBOA54fCsXmruEtxrVt\nhdEnDsuEoJRShxPJpPAhMEVEJopIMnAx8HzXh8aYBmNMgTGmxBhTArwHLDPGfBTBmPrEWeVB8JNZ\n79R6glIqrkQsKRhjvMD1WE9qcwFPGmM+F5E7RWRZpL43HFzVHhbmNiAdzXrlkVIqrkS0pmCMeRF4\nsce823tZtjSSsfSHs8rDZVlV0IoOb6GUiit6R1YPTe1edtW1MC9pFySkwMjjox2SUkoNGk0KPWyt\n9mAMlHSK10DNAAAXSUlEQVSWw+iZkJAU7ZCUUmrQaFLoweluBAw5B53adaSUijuaFHpwuT3MSK3F\n0eHRIrNSKu5oUujBWeXh7Nxqa0IvR1VKxRlNCkF8fsPW6kZOTt4FCckwcnq0Q1JKqUGlSSFIRW0z\nrZ0+Jvu2w6gZkJgc7ZCUUmpQaVIIYg1vYchvdGk9QSkVlzQpBHFWeZjgqCWxvUHrCUqpuKRJIYjL\n7eErOVXWhJ4pKKXikCaFIE63h9PSKsGRCKNOiHY4Sik16DQp2Gqb2tnnaWea2WFddZSUGu2QlFJq\n0GlSsLnsO5kLm7fAGO06UkrFJ00KNpfbQxF1JLfX6fAWSqm4pUnB5nR7WJRZaU1oUlBKxSlNCjaX\n28OijEoQBxRqkVkpFZ80KQDtXh/l+5uYKRUwchokp0c7JKWUigpNCsC2fU14/YaxrVu160gpFdc0\nKWB1HY2kntT2Gr1pTSkV1zQpYBWZ5yftsiZ0eAulVBzTpIB1pvClrL2AQOHMaIejlFJRE/dJwRiD\ns8rD7MRdUDAVUjKjHZJSSkVN3CeFqoY2PG1eJrR/ofUEpVTci/uk4KzyUEADGe37tZ6glIp7cZ8U\nXG4PMxN2WhN6pqCUinNxnxScVR4Wpe+1JkbPim4wSikVZXGfFFzVHuYn74K8SZA6ItrhKKVUVMV1\nUmhs62RXbQvHebdrPUEppYjzpLC1upEcGhnR7tZ6glJKEedJwen2MNNRYU3omEdKKRXfScHl9nBy\nij28hZ4pKKVUfCcFp7uRU1P2QG4JpOVEOxyllIq6uE0KPr9ha7WHqf4d2nWklFK2xGgHEC07a5pJ\n7vSQl7AXiq6OdjhK9VtnZyeVlZW0tbWFfdvZ2dm4XK6wbzdWxVJ7pKamUlxcTFJS0oDWj9uk4HR7\nOMGhw2Wr2FVZWUlWVhYlJSWISFi33djYSFZWVli3GctipT2MMdTW1lJZWcnEiRMHtI247T5yuT3M\nCQxvoUlBxZ62tjby8/PDnhBU7BIR8vPzj+nsMaJJQUTOFZGtIlIuIrce5vPvi4hTRDaLyGsiMiGS\n8QRzVnk4NbUSssdDet5gfa1SYaUJQfV0rP9PRCwpiEgCcD+wFJgBXCIiM3os9gkw3xgzC3ga+FWk\n4unJ5fYwQ3ZAkY53pJRSXSJ5pnAKUG6M2WGM6QDWAMuDFzDGvGGMabEn3wOKIxhPQE1TOy2N9Yzq\nqNR6glIDUFtby5w5c5gzZw6jR49m7NixgemOjo4+bePKK69k69atR1zm/vvv57HHHgtHyADs27eP\nxMREHnzwwbBtc7gRY0xkNiyyAjjXGHO1Pf1t4FRjzPW9LP87oNoYc9dhPrsGuAagsLDwpDVr1gwo\npqamJjIzM/msxsf6DZ/wRMp/sPnE26nLP2lA24t1Xe2hLLHWHtnZ2UyePDki2/b5fCQkJPRp2V/8\n4hdkZmZy4403hsw3xmCMweEYOqXLP/zhDzz77LMkJyfzj3/8o8/r9ac9ALxeL4mJ0buOp7y8nIaG\nhpB5ixcv3mCMmX+0dYfE1UcichkwHzjzcJ8bY1YBqwDmz59vSktLB/Q9ZWVllJaWsvV/tzPT8XcA\nZp3zbcgcNaDtxbqu9lCWWGsPl8sVuCLmjn98jrPKE7Zt+3w+ThyXy8++fsJRl01JSSElJYWsrCzK\ny8tZtmwZc+fO5ZNPPuGVV17hjjvu4OOPP6a1tZWLLrqI22+/HYBFixbxu9/9jpkzZ1JQUMC1117L\nunXrSE9PZ+3atYwaNYrbbruNgoICbr75ZhYtWsSiRYt4/fXXaWho4OGHH2bBggU0Nzdz+eWX43K5\nmDFjBhUVFTz44IPMmXNoL8Czzz7Lfffdx4oVK2hqaqKoqAiAF154gZ/+9Kf4fD4KCwv55z//SWNj\nI9dffz2ffPIJfr+fu+66i/POO4+CggIOHjwIwJo1a3j11Vd58MEHueyyy8jKymLDhg2UlpZywQUX\n8L3vfY+2tjbS09N55JFHmDJlCl6vl1tuuYVXXnkFh8PBtddey+TJk1m1ahVPP/00AOvWreOhhx7i\nqaeeGtB/v9TUVObOnTugdSOZFPYC44Kmi+15IUTkK8BPgDONMe0RjCfA5fZwbspuyBgTtwlBqUjZ\nsmULjz76KPPnWweld999N3l5eXi9XhYvXsyKFSuYMSO0vNjQ0MCZZ57J3Xffzfe//30eeughbr31\nkGtTMMbwwQcf8Pzzz3PnnXfy0ksvcd999zF69GieeeYZNm3axLx58w4bV0VFBXV1dZx00klceOGF\nPPnkk9x0001UV1dz3XXXsX79eiZMmEBdXR0AP//5zxk5ciSbN2/G4/Hg8/mO+tvdbjfvvfceDoeD\nhoYG1q9fT2JiIi+99BK33XYbTzzxBA888ABVVVVs2rSJhIQE6urqyMnJ4frrr6e2tpb8/Hwefvhh\nrrrqqv42fVhEMil8CEwRkYlYyeBi4NLgBURkLvBHrG6m/RGMJYTT7eGHCRVaT1DDRl+O6PvjWK7L\nnzRpUiAhAKxevZo///nPeL1eqqqqcDqdhySFtLQ0li5dCsBJJ53E+vXrD7vtCy64ILBMRUUFAG+9\n9RY/+tGPAJg9ezYnnHD4tlizZg0XXXQRABdffDHf/e53uemmm3j33XdZvHgxEyZYFz/m5VlXI776\n6qs899xzgHVFT25uLl6v94i//cILLwx0lx08eJDLL7+c7du3hyzz6quvcvPNNwe6o7q+b+XKlTz+\n+OOsXLmSDRs2sHr16iN+V6RELCkYY7wicj3wMpAAPGSM+VxE7gQ+MsY8D/wXkAk8ZV9GtdsYsyxS\nMQG0dfpwH6ilKHkPFF0Wya9SKi5lZGQE3m/bto3f/va3fPDBB+Tk5HDZZZcd9hr65OTkwPuEhIRe\nd74pKSlHXaY3q1evpqamhr/85S8AVFVVsWPHjn5tw+FwEFyH7flbgn/7T37yE8455xy++93vUl5e\nzrnnnnvEbV911VV885vfBOCiiy7qVw0jnCJaATLGvGiMmWqMmWSM+U973u12QsAY8xVjTKExZo79\nF9GEAFC+v4mppgLB6MioSkWYx+MhKyuLESNG4Ha7efnll8P+HQsXLuTJJ58E4NNPP8XpdB6yjNPp\nxOv1snfvXioqKqioqOCWW25hzZo1LFiwgDfeeINdu6wRDrq6j8466yzuv/9+wOq2qq+vx+FwkJub\ny7Zt2/D7/Tz77LO9xtXQ0MDYsWMBeOSRRwLzzzrrLP7whz8EuqO6vm/cuHEUFBRw9913c8UVVxxb\noxyDoXNZwCBxVnk40WHfyazdR0pF1Lx585gxYwbTpk3j8ssvZ+HChWH/jhtuuIG9e/cyY8YM7rjj\nDmbMmEF2dnbIMqtXr+b8888PmffNb36T1atXU1hYyAMPPMDy5cuZPXs2K1euBOBnP/sZ+/btY+bM\nmSxcuDDQpfXLX/6Sc845hwULFlBc3PtV9D/60Y+45ZZbmDdvXsjZxXe+8x1Gjx7NrFmzmD17diCh\nAVx66aVMnDiRqVOnHnO7DFTELkmNlPnz55uPPvpoQOuWlZVR5hnJ7A3/zjeytiA/+CLM0cWWWLva\nJtJirT1cLhfTp0+PyLZjZawfsC7/9Hq9pKamsm3bNs4++2y2bdsW1ktCB6s9rr32Wk4//XT+5V/+\n5Zi2c7j/N0Qkdi5JHUxOt4crEysQHe9IqWGhqamJJUuW4PV6Mcbwxz/+Mar3CAzUnDlzyM3N5d57\n741qHLHXcsfAGMNO9wHGsQeKLop2OEqpMMjJyWHDhg3RDuOYbdy4MdohAHFWU6hpNRS378CBX+sJ\nSil1GHGVFPY0+pnZVWTWK4+UUuoQcZUUdjf6memowKQXwIix0Q5HKaWGnLhKCnsa/ZyUtAspmg06\nDr1SSh0irpJCdUMbE/27tZ6g1DFavHjxITei/eY3v+G666474npdo9BWVVWxYsWKwy5TWlrK0S47\n/81vfkNLS0tg+qtf/WpgkLpwmDNnDhdffHHYthdL4iYpeNo6yW/fTQI+rScodYwuueQSeg5hv2bN\nGi655JI+rT9mzJjAiKAD0TMpvPjii+Tk5Ax4e8FcLhc+n4/169fT3Nwclm0eTn+H6RgscXNJ6hZ3\nIzMdFdaE3qOghpt1t0L1p2HbXJrPC2PnwtK7D/v5ihUruO222+jo6CA5OZmKigqqqqo444wzaGpq\nYvny5dTX19PZ2cldd93F8uUhz9eioqKC8847j88++4zW1lauvPJKNm3axLRp02htbQ0sd9111/Hh\nhx/S2trKihUruOOOO7j33nupqqpi8eLFFBQU8MYbb1BSUsJHH31EQUEBv/71r3nooYcAuPrqq7n5\n5pupqKhg6dKlLFq0iHfeeYexY8eydu1a0tLSDvltq1ev5tvf/jYul4u1a9dy6aXWOJ7l5eVce+21\nHDhwgISEBJ566ikmTZrEL3/5S/72t7/hcDhYunQpd999N6Wlpdxzzz3Mnz+fmpoa5s+fT0VFBY88\n8gh///vfaWpqwufz8cILL/TaVo8++ij33HMPIsKsWbP4/e9/z6xZs/jiiy9ISkrC4/Ewe/bswHS4\nxE1ScLk9zJQd+FNzceSMj3Y4SsW0vLw8TjnlFNatW8fy5ctZs2YN3/rWtxARUlNTefbZZxkxYgQ1\nNTWcdtppLFu2rNdnBz/wwAOkp6fjcrnYvHlzyNDX//mf/0leXh4+n48lS5awefNmbrzxRn7961/z\nxhtvUFBQELKtDRs28PDDD/P+++9jjOHUU0/lzDPPDIxXtHr1av70pz/xrW99i2eeeYbLLjt0UMwn\nnniCV155hS1btnDfffcFksLKlSu59dZbOf/882lra8Pv97Nu3TrWrl3L+++/T3p6emAcoyP5+OOP\n2bx5c2A48cO1ldPp5K677uKdd96hoKCAuro6srKyKC0t5YUXXuAb3/gGa9as4YILLghrQoA4SgrT\nRmcxOnUXMkaLzGoY6uWIfqBa+zCsQ1cXUldS+POf/wxYN4n++Mc/5s0338ThcLB371727dvH6NGj\nD7udN998M/DUtlmzZjFrVvdz05988klWrVqF1+vF7XbjdDpDPu/prbfe4vzzzw+MVnrBBRewfv16\nli1bxsSJEwMP3gkeejtY19nG+PHjGTt2LFdddRV1dXW0tbWxd+/ewPhJqampgDUM9pVXXkl6ejrQ\nPQz2kZx11lmB5Xprq9dff50LL7wwkPS6lr/66qv51a9+xTe+8Q0efvhh/vSnPx31+/orbmoKp47P\nYpxvtw5voVSYLF++nNdee42PP/6YlpYWTjrJeqztY489xoEDB9iwYQMbN26ksLDwsMNlH83OnTu5\n5557eO2119i8eTNf+9rXBrSdLl3DbkPvQ2+vXr2aLVu2UFJSwqRJk/B4PDzzzDP9/q7ExET8fj9w\n5OG1+9tWCxcupKKigrKyMnw+HzNnzux3bEcTN0mBAy4cxqtFZqXCJDMzk8WLF3PVVVeFFJgbGhoY\nNWoUSUlJIUNS9+ZLX/oSjz/+OACfffYZmzdvBqxhtzMyMsjOzmbfvn2sW7cusE5WVhaNjY2HbOuM\nM87gueeeo6WlhebmZp599lnOOOOMPv0ev9/Pk08+yaeffhoYXnvt2rWsXr2arKwsiouLAw/daW9v\np6WlhbPOOouHH344UPTu6j4qKSkJDL1xpIJ6b2315S9/maeeeora2tqQ7QJcfvnlXHrppVx55ZV9\n+l39FT9JocoeV0QvR1UqbC655BI2bdoUkhRWrlzJRx99xIknnsijjz7KtGnTjriN6667jqamJqZP\nn87tt98eOOOYPXs2c+fOZdq0aVx66aUhw25fc801nHvuuSxevDhkW/PmzeOKK67glFNO4dRTT+Xq\nq6/u87OK169fz9ixYxkzZkxg3pe+9CWcTifV1dX89a9/5d5772XWrFksWLCA6upqzj33XJYtW8b8\n+fOZM2cO99xzDwA/+MEPeOCBB5g7dy41NTW9fmdvbXXCCSfwk5/8hDPPPJPZs2fz/e9/P2Sd+vr6\nPl/p1V/xM3T2lheoeeW3FFz/stYUbLE2VHSkxVp76NDZg2cotcfTTz/N2rVr+etf/9rrMjp0dl9M\n+xqfVWdQqglBKRWjbrjhBtatW8eLL74Yse+In6SglFIx7r777ov4d8RPTUGpYSjWun9V5B3r/xOa\nFJSKUampqdTW1mpiUAHGGGprawP3UQyEdh8pFaOKi4uprKzkwIEDYd92W1vbMe1YhptYao/U1FSK\ni4sHvL4mBaViVFJSEhMnTozItsvKyvp8KWc8iKf20O4jpZRSAZoUlFJKBWhSUEopFRBzdzSLyAHg\nyIOp9K4A6P2e8/ij7RFK26ObtkWo4dAeE4wxI4+2UMwlhWMhIh/15TbveKHtEUrbo5u2Rah4ag/t\nPlJKKRWgSUEppVRAvCWFVdEOYIjR9gil7dFN2yJU3LRHXNUUlFJKHVm8nSkopZQ6Ak0KSimlAuIm\nKYjIuSKyVUTKReTWaMcTLSIyTkTeEBGniHwuIjdFO6ahQEQSROQTEfmfaMcSbSKSIyJPi8gWEXGJ\nyOnRjilaROR79r+Tz0RktYjExqh4xyAukoKIJAD3A0uBGcAlIjIjulFFjRf4N2PMDOA04F/juC2C\n3QS4oh3EEPFb4CVjzDRgNnHaLiIyFrgRmG+MmQkkABdHN6rIi4ukAJwClBtjdhhjOoA1wPIoxxQV\nxhi3MeZj+30j1j/4sdGNKrpEpBj4GvBgtGOJNhHJBr4E/BnAGNNhjDkY3aiiKhFIE5FEIB2oinI8\nERcvSWEssCdoupI43xECiEgJMBd4P7qRRN1vgB8C/mgHMgRMBA4AD9vdaQ+KSEa0g4oGY8xe4B5g\nN+AGGowx/4xuVJEXL0lB9SAimcAzwM3GGE+044kWETkP2G+M2RDtWIaIRGAe8IAxZi7QDMRlDU5E\ncrF6FCYCY4AMEbksulFFXrwkhb3AuKDpYnteXBKRJKyE8Jgx5u/RjifKFgLLRKQCq1vxyyLyt+iG\nFFWVQKUxpuvs8WmsJBGPvgLsNMYcMMZ0An8HFkQ5poiLl6TwITBFRCaKSDJWsej5KMcUFSIiWP3F\nLmPMr6MdT7QZY/7dGFNsjCnB+v/idWPMsD8a7I0xphrYIyLH27OWAM4ohhRNu4HTRCTd/nezhDgo\nusfF4ziNMV4RuR54GesKgoeMMZ9HOaxoWQh8G/hURDba835sjHkxijGpoeUG4DH7AGoHcGWU44kK\nY8z7IvI08DHWVXufEAfDXegwF0oppQLipftIKaVUH2hSUEopFaBJQSmlVIAmBaWUUgGaFJRSSgVo\nUlBxS0Sa7NcSEbk0zNv+cY/pd8K5faUiRZOCUlAC9Csp2AOkHUlIUjDGDPs7YdXwoElBKbgbOENE\nNtrj5yeIyH+JyIcisllEvgMgIqUisl5Ense+y1dEnhORDfaY+9fY8+7GGllzo4g8Zs/rOisRe9uf\nicinInJR0LbLgp5j8Jh9F61Sgyou7mhW6ihuBX5gjDkPwN65NxhjThaRFOBtEekaHXMeMNMYs9Oe\nvsoYUyciacCHIvKMMeZWEbneGDPnMN91ATAH6zkFBfY6b9qfzQVOwBqe+W2su8/fCv/PVap3eqag\n1KHOBi63hwF5H8gHptiffRCUEABuFJFNwHtYgy5O4cgWAauNMT5jzD7gf4GTg7ZdaYzxAxuxurWU\nGlR6pqDUoQS4wRjzcshMkVKsoaSDp78CnG6MaRGRMuBYHtfYHvTeh/77VFGgZwpKQSOQFTT9MnCd\nPcQ4IjK1lwfNZAP1dkKYhvV40y6dXev3sB64yK5bjMR6ytkHYfkVSoWBHokoBZsBn90N9AjWM4pL\ngI/tYu8B4BuHWe8l4FoRcQFbsbqQuqwCNovIx8aYlUHznwVOBzYBBvihMabaTipKRZ2OkqqUUipA\nu4+UUkoFaFJQSikVoElBKaVUgCYFpZRSAZoUlFJKBWhSUEopFaBJQSmlVMD/D92u7iEpwtP5AAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7e105d8d50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "m = 100 # batch size\n",
    "n = 1000 # number of batches to train\n",
    "acc_tr = []\n",
    "acc_vl = []\n",
    "\n",
    "# Assume that you have 2GB of GPU memory and want to allocate 512MB:\n",
    "gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.25)\n",
    "\n",
    "# open an session\n",
    "with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:\n",
    "    \n",
    "    # Initialize Variables\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "    for j in xrange(1000):\n",
    "        # here we fetch a batch from dataset\n",
    "        batch = mnist.train.next_batch(m)\n",
    "        # update is conducted by tf\n",
    "        # feed dict is key-value dictionary which\n",
    "        # each keys are placeholder and\n",
    "        # each values are actual value for placeholders\n",
    "        sess.run(train_step,feed_dict={x:batch[0], y:batch[1]})\n",
    "        \n",
    "        if j%100==0:\n",
    "            acc_tr.append(accuracy.eval(feed_dict={x:batch[0], y:batch[1]}))\n",
    "            acc_vl.append(accuracy.eval(feed_dict={x:mnist.test.images, y:mnist.test.labels}))\n",
    "\n",
    "            print(\n",
    "                '{:d}th Iteration - Train Acc: {:.2%} - Test Acc: {:.2%}'.format(\n",
    "                    j,acc_tr[-1],acc_vl[-1]\n",
    "                )\n",
    "            )\n",
    "\n",
    "\n",
    "fig,ax = plt.subplots()\n",
    "ax.plot(acc_tr, label='Training Accuracy')\n",
    "ax.plot(acc_vl, label='Validation Accuracy')\n",
    "ax.grid(True)\n",
    "ax.set_ylabel('Accuracy')\n",
    "ax.set_xlabel('Iteration')\n",
    "ax.set_title('Training Result')\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Want higher level framework?\n",
    "\n",
    "Pleas look at [Keras](https://keras.io/). this is one of the most mature high-level framework for NN.\n",
    "\n",
    "Its aim is to make NN easier and simpler, as well as provide full functionality of its backend engine (tensorflow / theano)\n",
    "\n",
    "Or what you want is just to build simple MLP structure as part of some application, [sklearn.neural_network](http://scikit-learn.org/stable/modules/neural_networks_supervised.html) is also very quick and fast solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "TensorFlowTutorial",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}