Part01_prerequisites.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0. Foreword\n",
    "\n",
    "<center><a href=\"https://twitter.com/emilymbender/status/1067108757488848896?ref_src=twsrc%5Etfw\"><img src=\"img/bender.png\" /></a></blockquote></center>\n",
    "\n",
    "\n",
    "### 1. All datasets are in German\n",
    "### 2. All links are to open access journals _only_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Prerequisites\n",
    "\n",
    "Neural networks have four building blocks that define how the network operates and how it is trained. As the network architectures have become increasingly complex the details of how the networks are trained have similarly increased, how large the mini batches are, what optimizer to use etc. For instance, see [_RoBERTa: A Robustly Optimized BERT Pretraining Approach_](https://arxiv.org/abs/1907.11692).\n",
    "\n",
    "The four main components of a trained neural network are\n",
    "\n",
    "- the network architecture\n",
    "    - feed forward, recurrent, transformer?\n",
    "    - how many layers?\n",
    "    - how wide are the layers?\n",
    "    \n",
    "    \n",
    "- the optimizer\n",
    "    - SGD, SGD with momentum\n",
    "    - Adam, AdamW, RAdam\n",
    "    \n",
    "    \n",
    "- the scheduler\n",
    "    - constant\n",
    "    - slanted learning rates\n",
    "    \n",
    "    \n",
    "- the learning objective and loss function\n",
    "    - this is problem dependent\n",
    "    - masked language model (MLM)\n",
    "    - causal language model (CLM)\n",
    "    - translation language model (TLM)\n",
    "    - classification\n",
    "\n",
    "https://pytorch.org/tutorials/beginner/deep_learning_nlp_tutorial.html\n",
    "\n",
    "\n",
    "## 1.1 Network Architecture\n",
    "\n",
    "The networks architecture is perhaps the most visible part of the four. It defines how the individual (artificial) neurons work and how they connect together. At its simplest the network is a fully connected feed forward network with sigmoid as the activation function for the neurons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "class FeedForward(nn.Module):\n",
    "    def __init__(self, layers, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        \n",
    "        # create feed forward layers\n",
    "        self.layers = nn.ModuleList([nn.Linear(in_features=in_features, out_features=out_features)\n",
    "                                     for (in_features, out_features) in layers])\n",
    "        \n",
    "        # TODO: initialise weights\n",
    "        \n",
    "        # create activation functions for each layer\n",
    "        self.activation = torch.nn.Sigmoid()\n",
    "\n",
    "    def forward(self, X):\n",
    "        output = None\n",
    "        for i_layer, layer in enumerate(self.layers):\n",
    "            if output is None:\n",
    "                output = self.activation(layer(X))\n",
    "            else:\n",
    "                output = self.activation(layer(output))\n",
    "        # do softmax\n",
    "        log_prob = F.log_softmax(output, dim=1)\n",
    "        return log_prob\n",
    "\n",
    "\n",
    "n_classes = 2\n",
    "n_features = 50\n",
    "layer_spec = [(n_features, 25), (25, n_classes)]\n",
    "\n",
    "ff = FeedForward(layers=layer_spec)\n",
    "inputs = torch.randn((10, n_features), dtype=torch.float32)\n",
    "ff(inputs).exp().detach().numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "\n",
    "## 1.2 Optimizer and Scheduler\n",
    "\n",
    "- Sebastian Ruder: [An overview of gradient descent optimization algorithms](http://ruder.io/optimizing-gradient-descent/) January 2016\n",
    "\n",
    "### Stochastic Gradient Descent\n",
    "\n",
    "- update network parameters by a fraction _learning rate_ after each example\n",
    "\n",
    "\n",
    "### Mini-batch Gradient Descent\n",
    "\n",
    "- update network parameters by a fraction _learning rate_ after each mini-batch of size $ n>1 $\n",
    "\n",
    "### Adaptive Learning Rate Optimizers\n",
    "\n",
    "- Adagrad, RMSprop, Adam, RAdam, AdamW\n",
    "- Liu et al.: [On the Variance of the Adaptive Learning Rate and Beyond](https://arxiv.org/abs/1908.03265)\n",
    "> The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate (i.e., it has problematically large variance in the early stage), suggest warmup works as a variance reduction technique, and provide both empirical and theoretical evidence to verify our hypothesis. We further propose RAdam, a new variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate ...\n",
    "- Loshchilov et al.: [Decoupled Weight Decay Regularization.](https://openreview.net/forum?id=Bkg6RiCqY7) ICLR (Poster) 2019\n",
    "> L2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is _not_ the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L2 regularization (often calling it \"weight decay\" in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by _decoupling_ the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Training the Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.datasets import make_blobs\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from torch.optim.lr_scheduler import LambdaLR\n",
    "from torch.utils.data import TensorDataset, RandomSampler, DataLoader\n",
    "\n",
    "X, y = make_blobs(n_samples=500, n_features=n_features, centers=2)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
    "\n",
    "train_dataset = TensorDataset(torch.FloatTensor(X_train), torch.LongTensor(y_train))\n",
    "test_dataset = TensorDataset(torch.FloatTensor(X_test), torch.LongTensor(y_test))\n",
    "sampler = RandomSampler(train_dataset)\n",
    "\n",
    "# Utilize the data loader from pytorch to create batches of data \n",
    "train_data = DataLoader(train_dataset, sampler=sampler, batch_size=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.1 A Learning Function\n",
    "Define a training function that loops through the training data some number of times and updates the model weights as well as the learning rate schedule.\n",
    "\n",
    "__Note__: each iteration over the inner for loop takes _one batch_ of data from the `pytorch` data loader class. The batch size determines how much data is used to compute the gradient. ___This will become important later!___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model: nn.Module,\n",
    "                data: DataLoader,\n",
    "                criterion: nn.modules.loss._Loss,\n",
    "                optimizer: torch.optim.Optimizer,\n",
    "                scheduler: torch.optim.lr_scheduler._LRScheduler,\n",
    "                n_epochs: int = 1):\n",
    "    model.train()\n",
    "    losses = []\n",
    "    # for each epoch\n",
    "    for i_epoch in range(n_epochs):\n",
    "        optimizer.zero_grad()\n",
    "        train_loss = 0\n",
    "        # go through the training data one batch at a time\n",
    "        for x_, y_ in data:\n",
    "            # Forward pass\n",
    "            y_pred = model(x_)\n",
    "\n",
    "            # Compute Loss\n",
    "            loss = criterion(y_pred.squeeze(), y_)\n",
    "            train_loss += loss.item() / i_step\n",
    "            \n",
    "            # Backprop loss\n",
    "            loss.backward()\n",
    "            \n",
    "            # Update weights\n",
    "            optimizer.step()\n",
    "            \n",
    "            # Update learning rate schedule\n",
    "            scheduler.step()\n",
    "            \n",
    "            optimizer.zero_grad()\n",
    "        losses.append(train_loss)\n",
    "    return model, losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "model = FeedForward(layer_spec)\n",
    "criterion = torch.nn.NLLLoss()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr = 0.001)\n",
    "scheduler = LambdaLR(optimizer, lambda epoch: 1e-3)\n",
    "\n",
    "trained_model, losses_sgd = train_model(model, train_data, criterion, optimizer, scheduler, n_epochs=50)\n",
    "\n",
    "model = FeedForward(layer_spec)\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr = 0.01, momentum=0.9)\n",
    "trained_model, losses_sgd_momentum = train_model(model, train_data, criterion, optimizer, scheduler, n_epochs=50)\n",
    "\n",
    "model = FeedForward(layer_spec)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr = 0.01)\n",
    "trained_model, losses_adam = train_model(model, train_data, criterion, optimizer, scheduler, n_epochs=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The choice of optimizer makes quite a difference. As the network becomes larger (more parameters) and the problems more complex the optimizer becomes increasingly important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "plt.plot(losses_sgd, label='SGD');\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "plt.plot(losses_sgd_momentum, label='SGD with momentum');\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(losses_adam, label='Adam');\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# 1.4 Natural Language (text) to Linear Algebra\n",
    "\n",
    "The dummy data set we used in the previous section was already vectors so that no work was required to transform the data into something (a vector) a neural network can consume. Natural langauge text, or `python` source code for that matter, does not come in this readily available form so some amount of preprocessing is required.\n",
    "\n",
    "The great promise of artificial neural networks for NLP is that a lot of the feature engineering work that has in the past been done by arduous PhD students can be offloaded to the combination of neural network machinery, gradient descent and the loss function to be minimised. Optimising a loss function given some training data may lead to problems down the line in the networks ability to [generalise to other kinds of data](https://thegradient.pub/frontiers-of-generalization-in-natural-language-processing/), but we'll forego that concern here.\n",
    "\n",
    "## 1.4.1 Representing Natural Language - a Computer Science Perspective\n",
    "\n",
    "Natural languages are ambiguous, complex and highly expressive ways of encoding information, not just semantics but also a lot of cultural details like class hierarchies and history. The Viking conquests are present in the everyday usage of english. We grow _cows_ _pigs_ and _lamb_ but eat _beef_ _pork_ and _mutton_, at least the non-vegetarians do. In finnish _vasta_ and _vihta_ refer to a collection of young leafy birch branches used when bathing (sauna), however _vihta_ is only ever used in western Finland and _vasta_ only in eastern Finland. Whether or not these etymological tidbits and subtle differences in the semantics of the words are necessary or even helpful in solving some task remains debatable.\n",
    "\n",
    "Word vectors, also known as _embeddings_, aim to create word representations that capture distributional similarity between words as vectors. Specifically, the word vectors should act in such a way that algebraic modifications maintain the semantic coherence. By capturing distributional similarity they end up capturing semantic information about words and their relations and thus allow for a somewhat coherent way of encoding natural language text into vectors.\n",
    "\n",
    "These vectors can be used to encode documents for neural networks.\n",
    "\n",
    "## 1.4.2 A Text Classifier with Embeddings\n",
    "\n",
    "Below, we'll use the fastText word embeddings to create a document classifier for the \"_10k German News Articles Dataset_\" (GNAD) data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -O https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.de.300.vec.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -O https://raw.githubusercontent.com/tblock/10kGNAD/master/train.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -O https://raw.githubusercontent.com/tblock/10kGNAD/master/test.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m spacy download de_core_news_sm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "ft_vec = KeyedVectors.load_word2vec_format('cc.de.300.vec.SMALL.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much has been said about word vectors, and as a result I won't go into a lot of detail here, but instead simply refer to the existing excellent resources that are already available.\n",
    "\n",
    "- Noah Smith: [Contextual Word Representations: A Contextual Introduction](https://arxiv.org/abs/1902.06006)\n",
    "\n",
    "\n",
    "- Mikolov et al.: [_Efficient Estimation of Word Representations in Vector Space._](http://arxiv.org/abs/1301.3781) ICLR (Workshop Poster) 2013\n",
    "\n",
    "  > \"_We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities._\"\n",
    "\n",
    "- Mikolov et al. [_Distributed Representations of Words and Phrases and their Compositionality._](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality) NIPS 2013: 3111-3119\n",
    "\n",
    "- Le et al. [_Distributed Representations of Sentences and Documents._](http://proceedings.mlr.press/v32/le14.html) ICML 2014: 1188-1196\n",
    "\n",
    "\n",
    "- Chris Olah [Deep Learning, NLP, and Representations](https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/) (July 7, 2014)\n",
    "\n",
    "- Arora et al. [_A Latent Variable Model Approach to PMI-based Word Embeddings._](https://transacl.org/ojs/index.php/tacl/article/view/742) TACL 4: 385-399 (2016)\n",
    "  > Semantic word embeddings represent the meaning of a word via a vector, and are created by diverse methods. Many use nonlinear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting methods.\n",
    "\n",
    "  > This paper proposes a new generative model, a dynamic version of the log-linear topic model of Mnih and Hinton (2007). The methodological novelty is to use the prior to compute closed form expressions for word statistics. This provides a theoretical justification for nonlinear models like PMI, word2vec, and GloVe, as well as some hyperparameter choices. It also helps explain why low-dimensional semantic embeddings contain linear algebraic structure that allows solution of word analogies, as shown by Mikolov et al. (2013a) and many subsequent papers.\n",
    "\n",
    "  - Sanjeev Arora [Semantic Word Embeddings](http://www.offconvex.org/2015/12/12/word-embeddings-1/) (Dec 12, 2015)\n",
    "  - Sanjeev Arora [Word Embeddings: Explaining their properties](https://www.offconvex.org/2016/02/14/word-embeddings-2/) (February 14, 2016)\n",
    "\n",
    "---\n",
    "\n",
    "![word2vec](./img/word2vec.png)\n",
    "\n",
    "<sub>From: Mikolov et al.: [_Efficient Estimation of Word Representations in Vector Space._](http://arxiv.org/abs/1301.3781) ICLR (Workshop Poster) 2013</sub>\n",
    "\n",
    "In short word vectors are a way of capturing and representing information about distributional semantics in a way that is convenient for computers to deal with. More to the point, they allow arithmetic operations that are, at least to an extent, semantically coherent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_vec.most_similar('Berlin')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_vec_en = KeyedVectors.load_word2vec_format('wiki-news-300d-1M.vec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_vec_en.most_similar('Berlin')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_vec.most_similar('Python')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_vec_en.most_similar('Python')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mystery = ft_vec['Berlin'] - ft_vec['Germany'] + ft_vec['France']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_vec.most_similar([mystery])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mystery = ft_vec_en['Berlin'] - ft_vec_en['Germany'] + ft_vec_en['France']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_vec_en.most_similar([mystery])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extend the feed forward network from before by adding an embedding layer\n",
    "\n",
    "The embedding layer is just a lookup table that maps a word index to a vector. The embeddings can be pre-trained embeddings, like fastText or word2vec or glove, or you can learn them from scratch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "\n",
    "\n",
    "class FeedForwardWithEmbeddings(nn.Module):\n",
    "    def __init__(self, layers, pretrained_embeddings=None,\n",
    "                 num_embeddings=None, embedding_dim=50,\n",
    "                 *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        \n",
    "        if pretrained_embeddings is not None:\n",
    "            self.embedding = nn.Embedding.from_pretrained(pretrained_embeddings)\n",
    "        elif num_embeddings is not None and num_embeddings > 0 and embedding_dim > 0:\n",
    "            self.embedding = nn.Embedding(num_embeddings, embedding_dim)\n",
    "        else:\n",
    "            raise Exception('You must either define pretrained embeddings or both the number and dimensionality of embeddings.')\n",
    "        \n",
    "        # create feed forward layers\n",
    "        self.layers = nn.ModuleList([nn.Linear(in_features=in_features, out_features=out_features)\n",
    "                                     for (in_features, out_features) in layers])\n",
    "        \n",
    "        # TODO: initialise weights\n",
    "        \n",
    "        # create activation functions for each layer\n",
    "        self.activation = torch.nn.Sigmoid()\n",
    "\n",
    "    def forward(self, X, labels=None):\n",
    "        output = None\n",
    "        emb = self.embedding(X).view(X.size()[0], -1)\n",
    "        for i_layer, layer in enumerate(self.layers):\n",
    "            if output is None:\n",
    "                output = self.activation(layer(emb))\n",
    "            else:\n",
    "                output = self.activation(layer(output))\n",
    "        # apply softmax\n",
    "        return F.log_softmax(output, dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import utils\n",
    "\n",
    "gnad_train, gnad_test = utils.load_gnad()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "spc = spacy.load('de_core_news_sm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "def tokenize(msg, truncate=100):\n",
    "    tokens = [t.text for t in spc.tokenizer(msg)]\n",
    "    token_idx = [word2idx[t] if t in ft_vec else 0 for t in tokens[:truncate]]\n",
    "    if len(token_idx) < truncate:\n",
    "        token_idx += [0] * (truncate - len(token_idx))\n",
    "    return token_idx\n",
    "\n",
    "word2idx = {w: idx+1 for idx, w in enumerate(ft_vec.index2word)}\n",
    "vectors = np.concatenate([np.random.rand(1, 300), ft_vec.vectors])\n",
    "X = [tokenize(x, truncate=25) for x in gnad_train['text']]\n",
    "\n",
    "label_encoder = LabelEncoder()\n",
    "y = label_encoder.fit_transform(gnad_train['category'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Note__: The precomputed fast text word embeddings contain a lot more entries than those in the GNAD data set. For practical purposes one _could_ leave only those words that are contained in the training data. This, however, comes at the cost of potentially losing information for future unseen words that _are_ in the pretrained fast text embeddings.\n",
    "\n",
    "__Note 2__: There are at least two other ways of handling the unknown words: 1) ignore them completely, 2) add an extra random embedding for each and let gradient descent adjust those embeddings during training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import RandomSampler, DataLoader, TensorDataset\n",
    "\n",
    "X = torch.LongTensor(X)\n",
    "y = torch.LongTensor(y)\n",
    "data = TensorDataset(X, y)\n",
    "sampler = RandomSampler(data)\n",
    "train_dataloader = DataLoader(data, batch_size=16, sampler=sampler)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_spec = [(300*25, 25), (25, len(label_encoder.classes_))]\n",
    "layer_spec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm_notebook as tqdmn\n",
    "from torch.nn.utils import clip_grad_norm_\n",
    "from torch.optim.lr_scheduler import LambdaLR\n",
    "from torch.nn import functional as F\n",
    "\n",
    "\n",
    "ffn = FeedForwardWithEmbeddings(layer_spec, torch.FloatTensor(ft_vec.vectors))\n",
    "optimizer = torch.optim.SGD(ffn.parameters(), lr = 0.01, momentum=0.9)\n",
    "scheduler = LambdaLR(optimizer, lambda epoch: 1e-3)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "num_epochs = 20\n",
    "\n",
    "for _ in range(num_epochs):\n",
    "    train_loss = 0\n",
    "    for i_step, batch in enumerate(train_dataloader, 1):\n",
    "        batch_X, batch_y = batch\n",
    "        output = ffn(batch_X)\n",
    "        loss = criterion(output, batch_y)\n",
    "        train_loss += loss.item() / i_step\n",
    "        loss.backward()\n",
    "        clip_grad_norm_(ffn.parameters(), 1.0)\n",
    "\n",
    "        optimizer.step()\n",
    "        scheduler.step()\n",
    "        optimizer.zero_grad()\n",
    "    print(train_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output.exp()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Bag of Tricks for Efficient Text Classification\n",
    "- Edouard Grave, Tomas Mikolov, Armand Joulin, Piotr Bojanowski: [Bag of Tricks for Efficient Text Classification.](https://www.aclweb.org/anthology/E17-2068/) EACL (2) 2017: 427-431\n",
    "\n",
    "\n",
    "| Model | AG | Sogou | DBP | Yelp P. | Yelp F. | Yah. A. | Amz. F. | Amz. P. |\n",
    "| ----- | -- | ----- | --- | ------- | ------- | ------- | ------- | ------- |\n",
    "| BoW (Zhang et al., 2015) | 88.8 | 92.9 | 96.6 | 92.2 | 58.0 | 68.9 | 54.6 | 90.4 |\n",
    "| ngrams (Zhang et al., 2015) | 92.0 | 97.1 | 98.6 | 95.6 | 56.3 | 68.5 | 54.3 | 92.0 |\n",
    "| ngrams TFIDF (Zhang et al., 2015) | 92.4 | 97.2 | 98.7 | 95.4 | 54.8 | 68.5 | 52.4 | 91.5 |\n",
    "| char-CNN (Zhang and LeCun, 2015) | 87.2 | 95.1 | 98.3 | 94.7 | 62.0 | 71.2 | 59.5 | 94.5 |\n",
    "| char-CRNN (Xiao and Cho, 2016) | 91.4 | 95.2 | 98.6 | 94.5 | 61.8 | 71.7 | 59.2 | 94.1 |\n",
    "| VDCNN (Conneau et al., 2016) | 91.3 | 96.8 | 98.7 | 95.7 | 64.7 | 73.4 | 63.0 | 95.7 |\n",
    "| |\n",
    "| fastText, h = 10 | 91.5 | 93.9 | 98.1 | 93.8 | 60.4 | 72.0 | 55.8 | 91.2 |\n",
    "| fastText, h = 10, bigram | 92.5 | 96.8 | 98.6 | 95.7 | 63.9 | 72.3 | 60.2 | 94.6 |"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}