Working on M Cuturi paper

Author: gnthibault

OptimalTransportWasserteinDistance.ipynb

@@ -564,18 +564,41 @@
     "I would like to stop and mention that as we now interpret $P$ as a joint probability matrix, we can define its entropy, the marginal probabiilty entropy, and KL-divergence between two different transportation matrix. These takes the form of\n",
     "\n",
     "\\begin{align*}\n",
-    "    \\text{Entropy} &= H(P) &= -\\sum_{i,j} p_{i,j} log(p_{i,j}) & \\\\\n",
-    "    \\text{Marginal source distribution entropy r} &= H(r) &= -\\sum_{i} \\left( \\sum_{j} p_{i,j} \\right) log\\left( \\sum_{j} p_{i,j} \\right) &=  −\\sum_{i} r_i log(r_i)\\\\\n",
-    "    \\text{Marginal destination distribution entropy c} &= H(c) &= -\\sum_{j} \\left( \\sum_{i} p_{i,j} \\right) log\\left( \\sum_{i} p_{i,j} \\right) &= −\\sum_{i} c_i log(c_i)\\\\\n",
-    "    \\text{KL-divergence between P and Q transportation} &= KL(P\\|Q) &= \\sum_{i,j} p_{i,j} log\\left(\\frac{p_{i,j}}{{q}_{i,j}}\\right)  &\n",
+    "    \\text{Entropy} &= H(P) &= -\\sum_{i,j} p_{i,j} log(p_{i,j}) & \\tag{1.1} \\\\\n",
+    "    \\text{Marginal source distribution entropy r} &= H(r) &= -\\sum_{i} \\left( \\sum_{j} p_{i,j} \\right) log\\left( \\sum_{j} p_{i,j} \\right) &=  −\\sum_{i} r_i log(r_i) \\tag{1.2}\\\\\n",
+    "    \\text{Marginal destination distribution entropy c} &= H(c) &= -\\sum_{j} \\left( \\sum_{i} p_{i,j} \\right) log\\left( \\sum_{i} p_{i,j} \\right) &= −\\sum_{i} c_i log(c_i) \\tag{1.3}\\\\\n",
+    "    \\text{KL-divergence between P and Q transportation} &= KL(P\\|Q) &= \\sum_{i,j} p_{i,j} log\\left(\\frac{p_{i,j}}{{q}_{i,j}}\\right)  & \\tag{1.4}\n",
+    "\\end{align*}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can easily get the following inequality for joint distributions entropy (we recall that $\\Sigma_d$ is a probability simplex in dimension d, and $U(r,c)$ is the set of transport matrices on $(r,c)$) :\n",
+    "\\begin{align*} \\tag{1.5}\n",
+    "    \\forall r,c \\in \\Sigma_d, \\forall P \\in U(r,c), h(P) \\leq h(r) + h(c)\n",
+    "\\end{align*}\n",
+    "ie, by using log-sum inequality, proved in the notebook called InformationTheoryOptimization\n",
+    "\\begin{align*}\\tag{1.6}\n",
+    "    \\sum_{i=0}^{N-1} a_i log\\left(\\frac{a_i}{b_i}\\right) &\\geq \\left(\\sum_{i=0}^{N-1} a_i\\right) log\\left(\\frac{\\sum_{i=0}^{N-1}a_i}{\\sum_{i=0}^{N-1}b_i}\\right)\n",
     "\\end{align*}\n",
+    "Where $a_i,b_i \\in [\\mathbb{R}^{+} \\times \\mathbb{R}^{+}]$, $a=\\sum_{i=0}^{N-1} a_i$, $b=\\sum_{i=0}^{N-1} b_i = 1$.\n",
     "\n",
-    "We remember the following inequality for joint distributions entropy (we recall that $\\Sigma_d$ is a probability simplex in dimension d, and $U(r,c)$ is the set of transport matrices on this set) :\n",
+    "ie:\n",
     "\\begin{align*}\n",
-    "    \\forall r,c \\in \\Sigma_d, \\forall P \\in U(r,c), h(P) \\leq h(r) + h(c)\n",
+    "    H(r) &= -\\sum_{i} \\left( \\sum_{j} p_{i,j} \\right) log\\left( \\sum_{j} p_{i,j} \\right) \\\\\n",
+    "         &leq\n",
     "\\end{align*}\n",
     "\n",
-    "By the concavity of entropy, we can introduce the convex set\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By the concavity of entropy (we proved in the notebook called InformationTheoryOptimization) we can introduce the convex set\n",
     "\\begin{align*}\n",
     "    U_{\\alpha}(r,c) := \\{ P \\in U(r,c) | KL(P\\|rcˆT) \\leq \\alpha \\} = \\{ P \\in U(r,c)|h(P) \\geq h(r)+h(c)−\\alpha \\} \\subset U(r,c)\n",
     "\\end{align*}\n",
@@ -588,11 +611,6 @@
     "This quantity is also the mutual information $I(X\\|Y)$ of two random variables $(X, Y)$ should they follow the joint probability $P$ . Hence, the set of tables P whose Kullback-Leibler divergence to rcT is constrained to lie below a certain threshold can be interpreted as the set of joint probabilities P in U (r, c) which have sufficient entropy with respect to h(r) and h(c), or small enough mutual information. For reasons that will become clear in Section 4, we call the quantity below the Sinkhorn distance of r and c:"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": []
-  },
   {
    "cell_type": "markdown",
    "metadata": {},