Part03_transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Transformer and BERT (thanks 🤗)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Vaswani et al., [_Attention is All you Need._](https://papers.nips.cc/paper/7181-attention-is-all-you-need) NIPS 2017: 5998-6008\n",
    "- Devlin et al., [_BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding._](https://www.aclweb.org/anthology/N19-1423/) NAACL-HLT (1) 2019: 4171-4186\n",
    "\n",
    "---\n",
    "\n",
    "- [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/)\n",
    "- [The Illustrated BERT, ELMo ...](https://jalammar.github.io/illustrated-bert/)\n",
    "- [The Annotated Transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html)\n",
    "\n",
    "---\n",
    "\n",
    "# Model Compression\n",
    "\n",
    "- Liu et al.: [_RoBERTa: A Robustly Optimized BERT Pretraining Approach_](https://arxiv.org/abs/1907.11692)\n",
    "- Sam Sucik: [Compressing BERT for faster prediction](https://blog.rasa.com/compressing-bert-for-faster-prediction-2/amp/)\n",
    "\n",
    "---\n",
    "\n",
    "# Interpretability\n",
    "\n",
    "- Jain et al.: [Attention is not Explanation](https://arxiv.org/abs/1902.10186)\n",
    "- Alon Jacovi: [The Problem of Faithfulness in (Neural Network) NLP Interpretations](https://medium.com/@alonjacovi/the-problem-of-faithfulness-in-neural-network-nlp-interpretations-ee98d7027cbd)\n",
    "> An easy observation to make is that faithfulness is in a trade-off balance with another key problem of interpretability, which is readability: how easy-to-understand the explanation is by a human.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# Sub-word tokenistion using Byte Pair Encoding\n",
    "\n",
    "Tokenise words not based on whitespace, but based on frequency patterns learned from a corpus. The method trains an unsupervised tokeniser and does not, therefore, require any labelled data.\n",
    "\n",
    "- Sennrich et al.: [Neural Machine Translation of Rare Words with Subword Units.](https://www.aclweb.org/anthology/P16-1162/) ACL (1) 2016\n",
    "> Neural machine translation (NMT) models typically operate with a fixed vocabulary, but ___translation is an open-vocabulary problem.___ Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. ___This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations)___. [Emphasis mine]\n",
    "\n",
    "- Heinzerling et al.: [BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages.](http://www.lrec-conf.org/proceedings/lrec2018/pdf/1049.pdf) LREC 2018\n",
    "- [BPEmb: Subword Embeddings in 275 Languages](https://nlp.h-its.org/bpemb/)\n",
    "- Provlikov et al.: [BPE-Dropout: Simple and Effective Subword Regularization](https://arxiv.org/abs/1910.13267)\n",
    "> While multiple segmentations are possible even with the same vocabulary, BPE splits words into unique sequences; this may prevent a model from better learning the compositionality of words and being robust to segmentation errors.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import BertModel, BertTokenizer\n",
    "\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-cased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "token_ids = tokenizer.encode('The BERT tokenizers splits words into sub-word units.')\n",
    "tokenizer.convert_ids_to_tokens(token_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "token_ids = tokenizer.encode('The new Berlin airport (BER) will open soon.')\n",
    "tokenizer.convert_ids_to_tokens(token_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document Classification with BERT (thanks 🤗)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert = BertModel.from_pretrained('bert-base-multilingual-cased')\n",
    "bert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "import utils\n",
    "\n",
    "gnad_train, gnad_test = utils.load_gnad()\n",
    "label_encoder = LabelEncoder()\n",
    "\n",
    "# turn all the data into integer indices\n",
    "y_train = label_encoder.fit_transform(gnad_train.category)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer\n",
    "\n",
    "def doc2bert(doc, tokenizer):\n",
    "    tokens = tokenizer.tokenize(doc)[:510]  # NOTE: that's 510 SUBword tokens, not 510 tokens from the original document\n",
    "    token_ids = tokenizer.convert_tokens_to_ids(tokens)\n",
    "    \n",
    "    # pad everything to 512\n",
    "    if len(token_ids) < 510:\n",
    "        token_ids = token_ids + [0] * (510 - len(token_ids))\n",
    "        \n",
    "    return [tokenizer.cls_token_id] + token_ids + [tokenizer.sep_token_id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')\n",
    "\n",
    "token_ids = (doc2bert(x_, tokenizer) for x_ in gnad_train.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import TensorDataset, RandomSampler, DataLoader\n",
    "\n",
    "X_train = torch.LongTensor(list(token_ids))\n",
    "y_train = torch.LongTensor(y_train)\n",
    "\n",
    "batch_size = 8\n",
    "data_train = TensorDataset(X_train, y_train)\n",
    "sampler = RandomSampler(data_train)\n",
    "train_dataloader = DataLoader(data_train, sampler=sampler, batch_size=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import BertForSequenceClassification\n",
    "\n",
    "bert = BertForSequenceClassification.from_pretrained('bert-base-multilingual-cased', num_labels=len(torch.unique(y_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have quick look at what the output from looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output, *_ = bert(X_train[:4])\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "\n",
    "F.log_softmax(output, dim=1).exp()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's going on here?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import AdamW\n",
    "from transformers.optimization import WarmupLinearSchedule\n",
    "\n",
    "num_epochs = 5\n",
    "\n",
    "# pass the parameters of the classifier head ONLY to the optimizer\n",
    "params = [p for n, p in bert.named_parameters() if 'classifier.' in n]\n",
    "optimizer = AdamW(params, lr=3e-5, correct_bias=False)\n",
    "\n",
    "num_total_steps = num_epochs * (len(train_dataloader.sampler) // batch_size)\n",
    "num_warmup_steps = int(num_total_steps * 0.15)\n",
    "scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_total_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning Rate Schedule\n",
    "## Warmup Linear Schedule"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "def plot_learning_rate(num_warmup_steps):\n",
    "    scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_total_steps)\n",
    "    learning_rates = []\n",
    "    for i in range(num_total_steps):\n",
    "        learning_rates.append(scheduler.get_lr())\n",
    "        scheduler.step()\n",
    "    plt.plot(learning_rates);\n",
    "    plt.xlabel('Iteration');\n",
    "    plt.ylabel('Learning Rate');\n",
    "\n",
    "plot_learning_rate(int(num_total_steps * 0.05))\n",
    "plot_learning_rate(int(num_total_steps * 0.10))\n",
    "plot_learning_rate(int(num_total_steps * 0.15))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document Classification with BERT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import tqdm\n",
    "from torch.nn.utils import clip_grad_norm_\n",
    "\n",
    "# Use a GPU if one is available\n",
    "DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "bert.to(DEVICE)\n",
    "\n",
    "for _ in tqdm.trange(num_epochs, total=num_epochs, desc=\"Epoch\"):\n",
    "#     steps = tqdm.tqdm(train_dataloader,\n",
    "#                       total=X_train.size()[0] // train_dataloader.batch_size + 1,\n",
    "#                       desc='Mini-batch')\n",
    "    train_loss = 0\n",
    "    for i_step, batch in enumerate(train_dataloader):\n",
    "        batch_X, batch_y = (b.to(DEVICE) for b in batch)\n",
    "        loss, *_ = bert(batch_X, labels=batch_y)\n",
    "        train_loss += loss.item()\n",
    "        loss.backward()\n",
    "        clip_grad_norm_(bert.parameters(), 1.0)\n",
    "#         steps.set_postfix_str(f'avg. loss {train_loss / (i_step + 1):.4f}')\n",
    "        \n",
    "        optimizer.step()\n",
    "        scheduler.step()\n",
    "        optimizer.zero_grad()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "bert.to('cpu')\n",
    "torch.save(bert.to('cpu'), 'bert-GNADs-5epochs-HEAD.pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert = torch.load('bert-GNADs-5epochs-HEAD.pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "del batch_X, batch_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import SequentialSampler\n",
    "\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')\n",
    "\n",
    "token_ids = (doc2bert(x_, tokenizer) for x_ in gnad_test.text)\n",
    "X_test = torch.LongTensor(list(token_ids))\n",
    "y_test = label_encoder.transform(gnad_test.category)\n",
    "y_test = torch.LongTensor(y_test)\n",
    "\n",
    "batch_size = 8\n",
    "data_test = TensorDataset(X_test, y_test)\n",
    "sampler = SequentialSampler(data_test)\n",
    "test_dataloader = DataLoader(data_test, sampler=sampler, batch_size=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import SequentialSampler\n",
    "from torch.nn import functional as F\n",
    "\n",
    "\n",
    "DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "bert.eval()\n",
    "bert.to(DEVICE)\n",
    "pred = []\n",
    "for x, *_batch in test_dataloader:\n",
    "    x = x.to(DEVICE)\n",
    "    pred_, *_ = bert(x)\n",
    "    _, pred_ = F.log_softmax(pred_, dim=1).exp().max(dim=1)\n",
    "    pred.extend(pred_.cpu().numpy().tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "print(metrics.classification_report(y_test, pred, target_names=list(label_map.keys())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "                       BERT classifier head ONLY\n",
    "\n",
    "                   precision    recall  f1-score   support\n",
    " \n",
    "             Etat       0.00      0.00      0.00        67\n",
    "           Inland       0.00      0.00      0.00       102\n",
    "    International       0.29      0.03      0.06       151\n",
    "           Kultur       0.00      0.00      0.00        54\n",
    "         Panorama       0.18      0.61      0.28       168\n",
    "            Sport       0.82      0.07      0.14       120\n",
    "              Web       0.27      0.67      0.38       168\n",
    "       Wirtschaft       0.00      0.00      0.00       141\n",
    "     Wissenschaft       0.00      0.00      0.00        57\n",
    "\n",
    "         accuracy                           0.22      1028\n",
    "        macro avg       0.17      0.15      0.09      1028\n",
    "     weighted avg       0.21      0.22      0.13      1028\n",
    "\n",
    "\n",
    "---\n",
    "                 Linear SVM with parameter tuning\n",
    "\n",
    "                   precision    recall  f1-score   support\n",
    "\n",
    "             Etat       0.94      0.75      0.83        67\n",
    "           Inland       0.89      0.84      0.86       102\n",
    "    International       0.89      0.85      0.87       151\n",
    "           Kultur       0.89      0.89      0.89        54\n",
    "         Panorama       0.80      0.88      0.84       168\n",
    "            Sport       0.99      0.97      0.98       120\n",
    "              Web       0.92      0.90      0.91       168\n",
    "       Wirtschaft       0.82      0.88      0.85       141\n",
    "     Wissenschaft       0.89      0.96      0.92        57\n",
    "\n",
    "         accuracy                           0.88      1028\n",
    "        macro avg       0.89      0.88      0.89      1028\n",
    "     weighted avg       0.89      0.88      0.88      1028\n",
    "    \n",
    "---\n",
    "\n",
    "      BERT with fine-tuning the whole transformer stack\n",
    "\n",
    "                  precision    recall  f1-score   support\n",
    "\n",
    "             soc       0.55      0.70      0.61       398\n",
    "             rec       0.82      0.69      0.75      1590\n",
    "             alt       0.02      0.08      0.03       319\n",
    "             sci       0.21      0.09      0.12      1579\n",
    "            misc       0.81      0.55      0.65       390\n",
    "            talk       0.63      0.58      0.60      1301\n",
    "            comp       0.79      0.85      0.82      1955\n",
    "\n",
    "        accuracy                           0.55      7532\n",
    "       macro avg       0.54      0.51      0.51      7532\n",
    "    weighted avg       0.60      0.55      0.57      7532\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.nn.utils import clip_grad_norm_\n",
    "\n",
    "from transformers.optimization import WarmupLinearSchedule\n",
    "from transformers import BertForSequenceClassification\n",
    "from transformers import AdamW\n",
    "\n",
    "from tqdm import tqdm_notebook as tqdmn\n",
    "\n",
    "num_epochs = 5\n",
    "bert = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=len(torch.unique(y_train)))\n",
    "\n",
    "# include ALL the network parameters in the optimizer\n",
    "params = [p for n, p in bert.named_parameters()]  # if .classifier in n]\n",
    "optimizer = AdamW(params, lr=3e-5, correct_bias=False)\n",
    "\n",
    "num_total_steps = num_epochs * (len(train_dataloader.sampler)\n",
    "                              // batch_size)\n",
    "num_warmup_steps = int(num_total_steps * 0.15)\n",
    "scheduler = WarmupLinearSchedule(optimizer,\n",
    "                                 warmup_steps=num_warmup_steps,\n",
    "                                 t_total=num_total_steps)\n",
    "\n",
    "DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "bert.to(DEVICE)\n",
    "\n",
    "for _ in tqdmn(range(num_epochs), total=num_epochs, desc=\"Epoch\"):\n",
    "    steps = tqdmn(train_dataloader,\n",
    "                  total=X_train.size()[0] // train_dataloader.batch_size + 1,\n",
    "                  desc='Mini-batch')\n",
    "    train_loss = 0\n",
    "    for i_step, batch in enumerate(steps):\n",
    "        batch_X, batch_y = (b.to(DEVICE) for b in batch)\n",
    "        loss, *_ = bert(batch_X, labels=batch_y)\n",
    "        train_loss += loss.item()\n",
    "        loss.backward()\n",
    "        clip_grad_norm_(bert.parameters(), 1.0)\n",
    "        steps.set_postfix_str(f'avg. loss {train_loss / (i_step + 1):.4f}')\n",
    "        \n",
    "        optimizer.step()\n",
    "        scheduler.step()\n",
    "        optimizer.zero_grad()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "bert.to('cpu')\n",
    "torch.save(bert.to('cpu'), 'bert-GNAD-5epochs-ALL.pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import SequentialSampler\n",
    "from torch.nn import functional as F\n",
    "\n",
    "X_test = torch.LongTensor(X_test)\n",
    "\n",
    "batch_size = 4\n",
    "data_test = TensorDataset(X_test)\n",
    "sampler = SequentialSampler(data_test)\n",
    "test_dataloader = DataLoader(data_test, sampler=sampler, batch_size=batch_size)\n",
    "\n",
    "bert.eval()\n",
    "bert.to(DEVICE)\n",
    "pred = []\n",
    "for x, *_batch in test_dataloader:\n",
    "    x = x.to(DEVICE)\n",
    "    pred_, *_ = bert(x)\n",
    "    _, pred_ = F.log_softmax(pred_, dim=1).exp().max(dim=1)\n",
    "    pred.extend(pred_.cpu().numpy().tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "print(metrics.classification_report(y_test, pred, target_names=list(label_map.keys())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Author: Olivier Grisel <olivier.grisel@ensta.org>\n",
    "#         Peter Prettenhofer <peter.prettenhofer@gmail.com>\n",
    "#         Mathieu Blondel <mathieu@mblondel.org>\n",
    "# License: BSD 3 clause\n",
    "from pprint import pprint\n",
    "from time import time\n",
    "import logging\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('vect', CountVectorizer()),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', SGDClassifier(tol=1e-3)),\n",
    "])\n",
    "\n",
    "# uncommenting more parameters will give better exploring power but will\n",
    "# increase processing time in a combinatorial way\n",
    "parameters = {\n",
    "    'vect__max_df': (0.5, 0.75, 1.0),\n",
    "    # 'vect__max_features': (None, 5000, 10000, 50000),\n",
    "    'vect__ngram_range': ((1, 1), (1, 2)),  # unigrams or bigrams\n",
    "    # 'tfidf__use_idf': (True, False),\n",
    "    # 'tfidf__norm': ('l1', 'l2'),\n",
    "    'clf__max_iter': (20,),\n",
    "    'clf__alpha': (0.00001, 0.000001),\n",
    "    'clf__penalty': ('l2', 'elasticnet'),\n",
    "    # 'clf__max_iter': (10, 50, 80),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "grid_search = GridSearchCV(pipeline, parameters, cv=5, n_jobs=3, verbose=1)\n",
    "\n",
    "grid_search.fit(gnad_train.text, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "print(metrics.classification_report(y_test,\n",
    "                                    grid_search.best_estimator_.predict(gnad_test.text),\n",
    "                                    target_names=label_encoder.classes_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "rise": {
   "theme": "white"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}