03_summary_statistics.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "03-summary-statistics",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MIT-LCP/sccm-datathon/blob/master/03_summary_statistics.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1G_TVh1ybQkl"
      },
      "source": [
        "# eICU Collaborative Research Database\n",
        "\n",
        "# Notebook 3: Summary statistics\n",
        "\n",
        "This notebook shows how summary statistics can be computed for a patient cohort using the `tableone` package. Usage instructions for tableone are at: https://pypi.org/project/tableone/\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "L9XF77F2bnee"
      },
      "source": [
        "## Load libraries and connect to the database"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wXiSE558bn_w",
        "colab": {}
      },
      "source": [
        "# Import libraries\n",
        "import numpy as np\n",
        "import os\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Make pandas dataframes prettier\n",
        "from IPython.display import display, HTML\n",
        "\n",
        "# Access data using Google BigQuery.\n",
        "from google.colab import auth\n",
        "from google.cloud import bigquery"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "pLGnLAy-bsKb",
        "colab": {}
      },
      "source": [
        "# authenticate\n",
        "auth.authenticate_user()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PUjFDFdobszs",
        "colab": {}
      },
      "source": [
        "# Set up environment variables\n",
        "project_id='sccm-datathon'\n",
        "os.environ[\"GOOGLE_CLOUD_PROJECT\"]=project_id"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "iWDUCA5Nb5BK"
      },
      "source": [
        "## Install and load the `tableone` package\n",
        "\n",
        "The tableone package can be used to compute summary statistics for a patient cohort. Unlike the previous packages, it is not installed by default in Colab, so will need to install it first."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "F9doCgtscOJd",
        "colab": {}
      },
      "source": [
        "!pip install tableone"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "SDI_Q7W0b4Le",
        "colab": {}
      },
      "source": [
        "# Import the tableone class\n",
        "from tableone import TableOne"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "14TU4lcrdD7I"
      },
      "source": [
        "## Load the patient cohort\n",
        "\n",
        "In this example, we will load all data from the patient data, and link it to APACHE data to provide richer summary information."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "HF5WF5EObwfw",
        "colab": {}
      },
      "source": [
        "# Link the patient and apachepatientresult tables on patientunitstayid\n",
        "# using an inner join.\n",
        "%%bigquery cohort\n",
        "\n",
        "SELECT p.unitadmitsource, p.gender, p.age, p.ethnicity, p.admissionweight, \n",
        "    p.unittype, p.unitstaytype, a.acutephysiologyscore,\n",
        "    a.apachescore, a.actualiculos, a.actualhospitalmortality,\n",
        "    a.unabridgedunitlos, a.unabridgedhosplos\n",
        "FROM `physionet-data.eicu_crd_demo.patient` p\n",
        "INNER JOIN `physionet-data.eicu_crd_demo.apachepatientresult` a\n",
        "ON p.patientunitstayid = a.patientunitstayid\n",
        "WHERE apacheversion LIKE 'IVa'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "k3hURHFihHNA",
        "colab": {}
      },
      "source": [
        "cohort.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qnG8dVb2iHSn"
      },
      "source": [
        "## Calculate summary statistics\n",
        "\n",
        "Before summarizing the data, we will need to convert the ages to numerical values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oKHpqwAPkx6U",
        "colab": {}
      },
      "source": [
        "cohort['agenum'] = pd.to_numeric(cohort['age'], errors='coerce')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "FQT-u8EXhXRG",
        "colab": {}
      },
      "source": [
        "columns = ['unitadmitsource', 'gender', 'agenum', 'ethnicity',\n",
        "          'admissionweight','unittype','unitstaytype',\n",
        "          'acutephysiologyscore','apachescore']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3ETr3NCzielL",
        "colab": {}
      },
      "source": [
        "table = TableOne(cohort, columns=columns, rename={'agenum': 'age'}, \n",
        "                 groupby='actualhospitalmortality',\n",
        "                 label_suffix=True, limit=4, pval=False)\n",
        "\n",
        "print(table.tabulate(tablefmt = \"fancy_grid\"))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LCBcpJ9bZpDp"
      },
      "source": [
        "## Questions\n",
        "\n",
        "- Are the severity of illness measures higher in the survival or non-survival group?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2_8z1CIVahWg"
      },
      "source": [
        "## Visualizing the data\n",
        "\n",
        "Plotting the distribution of each variable by group level via histograms, kernel density estimates and boxplots is a crucial component to data analysis pipelines. Vizualisation is often is the only way to detect problematic variables in many real-life scenarios. We'll review a couple of the variables."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "81yp2bSUigzh",
        "colab": {}
      },
      "source": [
        "# Plot distributions\n",
        "cohort[['acutephysiologyscore','agenum']].dropna().plot.kde(figsize=[12,8])\n",
        "plt.legend(['APS Score', 'Age (years)'])\n",
        "plt.xlim([-30,250])"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}