Handling-Unbalanced-Dataset-in-Deep-Learning-with-Class-Weight.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling Unbalanced Dataset in Deep Learning with Class Weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What is unbalanced dataset?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the samples per classes in a dataset varies to different numbers, then the dataset is called unbalanced dataset. \n",
    "\n",
    "For example,\n",
    "Let's consider a image dataset has two classes, A and B.\n",
    "And the number of samples in these classes are as bellow- "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Class Name | Number of Samples |\n",
    "|------------------|---------------------------|\n",
    "| Class A | 25 |\n",
    "| Class B | 75 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, if you are already familiar with machine learning and deep learning, then its almost certain that any model trained on this dataset will not generalize for the problem. The model will be biased towards class B."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How can we solve the above problem of biasness?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two ways to solve the above problem-\n",
    "\n",
    "1. Increase the number of samples in class A, so that both the classes have almost equal number of data points\n",
    "2. Explicitly tell the model during training to put more focus on class A while optimizing the loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Class weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since class B has 3 times more data than class A, so class B will get 3 times more focus by the model during loss optimzation.\n",
    "\n",
    "So we can say that the classes have the following weights with the current amount of data- \n",
    "\n",
    "| Class Name | Current Weight |\n",
    "|-----------------|----------------------|\n",
    "| Class A | 1 |\n",
    "| Class B | 3 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can explicitly tell the model to put more focus on class A by adding more weights. But how much weight we should add is dependent on the current weight."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Class weight calculation\n",
    "\n",
    "So, how can we calculate the class weight?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = 25\n",
    "B = 75\n",
    "\n",
    "majority = max(A, B)\n",
    "\n",
    "A_weight = majority / A\n",
    "B_weight = majority / B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define a simple method that will help to update the class weight - "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "def get_class_weights(y):\n",
    "    counter = Counter(y)\n",
    "    #get maximum number of occurances of a class\n",
    "    majority = max(counter.values()) \n",
    "    return  {cls: float(majority/count) for cls, count in counter.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = [1, 1, 0, 0, 0, 2, 3, 1, 2, 5] #class labels for ground truth that express the occurances of a classes in the dataset\n",
    "weights = get_class_weights(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: 1.0, 0: 1.0, 2: 1.5, 3: 3.0, 5: 3.0}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can pass this dictionary formated class weights data as *class weight* parameter in keras fit method while training out model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have performed an experiment with the well know MNIST dataset and it seems that training with class weight gains signicant performance gain in the model accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "[Scikit learn class weight computation method](https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}