python/2_prediction_outcome.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predictive Process Monitoring - Outcome Prediction\n",
    "\n",
    "This notebook is part of the [starter package](https://github.com/fmannhardt/starter-predictive-process-monitoring) for predictive process monitoring. It contains an example of a prediction model for the purpose of developing and applying predictive process monitoring techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "The following Python libraries are used, please refer to the installation instructions to prepare your environment:\n",
    "\n",
    "* [PM4Py](https://pm4py.fit.fraunhofer.de/)\n",
    "* [Pandas](https://pandas.pydata.org/)\n",
    "* [Numpy](https://numpy.org/)\n",
    "* [PyTorch](https://pytorch.org/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pm4py\n",
    "import numpy as np\n",
    "import sklearn\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from tqdm.auto import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Event Log & Data Loading\n",
    "\n",
    "We continue with bag-of-words or multiset of events model encoding generated in the [previous notebook](./1_feature_extraction.ipynb). Making use of the other encodings and adding others is left as exercise. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlretrieve\n",
    "import os\n",
    "\n",
    "# download from 4tu.nl\n",
    "urlretrieve('https://data.4tu.nl/file/33632f3c-5c48-40cf-8d8f-2db57f5a6ce7/643dccf2-985a-459e-835c-a82bce1c0339', 'sepsis2.xes.gz')\n",
    "sepsis_log = pm4py.read_xes('sepsis2.xes.gz')\n",
    "os.unlink('sepsis2.xes.gz') # clean up\n",
    "\n",
    "# Here we want the multiset (or rather list) of sequences view on logs\n",
    "from pm4py.objects.conversion.log import converter as log_conversion\n",
    "sepsis_log = log_conversion.apply(sepsis_log, {}, log_conversion.TO_EVENT_LOG)\n",
    "\n",
    "sepsis_returns = [len(list(filter(lambda e: e[\"concept:name\"] == \"Return ER\" ,trace))) > 0 for trace in sepsis_log]\n",
    "sepsis_log = pm4py.filter_event_attribute_values(sepsis_log, \"concept:name\", \"Return ER\", level = \"event\", retain=False)\n",
    "from pm4py.objects.log.obj import EventLog, Trace\n",
    "sepsis_prefixes = EventLog([Trace(trace[0:10], attributes = trace.attributes) for trace in sepsis_log])\n",
    "\n",
    "sepsis_df = pm4py.convert_to_dataframe(sepsis_prefixes)\n",
    "sepsis_act_count = sepsis_df.loc[:,[\"case:concept:name\", \"concept:name\"]].groupby([\"case:concept:name\", \"concept:name\"],sort=False).size()\n",
    "sepsis_bag = np.asarray(sepsis_act_count.unstack(fill_value=0))\n",
    "sepsis_bag.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "\n",
    "Let us try to build a basic prediction model based on this information. In this example, we aim to predict the binary outcome whether the event `Return ER` occurred or not. \n",
    "\n",
    "**Disclaimer: here *basic* means that the model and encoding is not expected to be of any quality. Also note that the prediction task, while useful, may not be feasible based on the prefix encoding that we chose. Treat the following code as an example and starting point only!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Target Variable\n",
    "\n",
    "Let us look at the distribution of the target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.unique(sepsis_returns, return_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For future processing we need 0 and 1 instead of True and False\n",
    "sepsis_returns = np.asarray(sepsis_returns).astype(int)\n",
    "sepsis_returns.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Scaling & Loading\n",
    "\n",
    "This often helps prediction models to perform better.\n",
    "\n",
    "**Important:** make sure to not compute the scaling with the test set included since there is a risk of data leakage otherwise. In other words, the test set should be separated before any pre-processing, which may use a property of the dataset, is applied. Of course, the test set is scaled as well but with the scaler *trained* only on the training set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import FunctionTransformer, MinMaxScaler\n",
    "\n",
    "scaler_x = MinMaxScaler()\n",
    "data_scaled = scaler_x.fit_transform(sepsis_bag)\n",
    "\n",
    "scaler_y = FunctionTransformer() # for binary values scaling does not make sense at all but we keep it for symetry and apply the \"NoOp\" scaler\n",
    "target_scaled = scaler_y.fit_transform(sepsis_returns.reshape(-1, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Definition\n",
    "\n",
    "Let's define a simple network and try to overfit. We make use of PyTorch to build a simple Neural Network. \n",
    "\n",
    "**Disclaimer: Again, this is just a simple example and not in anyway meant as a recommendation for a model.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NeuralNetworkBinaryOutcome(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(NeuralNetworkBinaryOutcome, self).__init__()\n",
    "        self.linear_relu_stack = nn.Sequential(            \n",
    "            torch.nn.Linear(x.shape[1], 64),\n",
    "            nn.BatchNorm1d(num_features=64),\n",
    "            nn.LeakyReLU(),            \n",
    "            torch.nn.Linear(64, 128),\n",
    "            nn.BatchNorm1d(num_features=128),          \n",
    "            torch.nn.Linear(128, 1),\n",
    "            nn.Sigmoid()\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        logits = self.linear_relu_stack(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use a standard training loop in PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(dataloader, model, \n",
    "          loss_fn, measure_fn, \n",
    "          optimizer, device, epochs): \n",
    "    \n",
    "    losses = []\n",
    "    size = len(dataloader.dataset)\n",
    "    \n",
    "    loop = tqdm(range(epochs))\n",
    "    \n",
    "    for epoch in loop:\n",
    "    \n",
    "        for batch, (X, y) in enumerate(dataloader):\n",
    "            X, y = X.to(device), y.to(device)\n",
    "\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            # Compute prediction error\n",
    "            pred = model(X)\n",
    "            \n",
    "            loss = loss_fn(pred, y)\n",
    "            measure = measure_fn(pred, y)\n",
    "\n",
    "            # Backpropagation\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            \n",
    "            losses.append([loss.item(), measure.item()])\n",
    "            \n",
    "        loop.set_description('Epoch {}/{}'.format(epoch + 1, epochs))\n",
    "        loop.set_postfix(loss=loss.item(), measure=measure.item())\n",
    "    \n",
    "    return losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And can use the following function to get all evaluation results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_all(dataloader, model, device):    \n",
    "    size = len(dataloader.dataset)\n",
    "    num_batches = len(dataloader)\n",
    "    \n",
    "    model.eval()\n",
    "    \n",
    "    result = []\n",
    "    original = []\n",
    "\n",
    "    with torch.no_grad(): \n",
    "        for X, y in tqdm(dataloader):  \n",
    "            X, y = X.to(device), y.to(device) \n",
    "            pred = model(X)          \n",
    "                        \n",
    "            result.extend(pred.flatten().numpy())\n",
    "            original.extend(y.flatten().numpy())\n",
    "                           \n",
    "    return np.asarray(result), np.asarray(original)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "Prepare the data for the PyTorch data loading mechanism."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "# We need float32 data\n",
    "x = torch.from_numpy(data_scaled.astype('float32'))\n",
    "y = torch.from_numpy(target_scaled.astype('float32'))\n",
    "\n",
    "# Always check the shapes\n",
    "print(x.shape)\n",
    "print(y.shape)\n",
    "\n",
    "ds = TensorDataset(x, y)\n",
    "train_dataloader = DataLoader(ds, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us check a random single sample from our data loader (always a good idea!) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs, classes = next(iter(train_dataloader))\n",
    "print(inputs[0])\n",
    "print(classes[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We train the model using cross entropy as loss function accuracy as easier to interpret measure to report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## if you want ot use a GPU you need to tweak the requirements.txt to include the GPU-enabled PyTorch\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "print('Using {} device'.format(device))\n",
    "\n",
    "# fix a seed to get reproducible results\n",
    "torch.manual_seed(42)\n",
    "\n",
    "model = NeuralNetworkBinaryOutcome().to(device)\n",
    "print(model)\n",
    "\n",
    "def get_accuracy(y_prob, y_true):    \n",
    "    y_true = y_true.flatten()\n",
    "    y_prob = y_prob.flatten()\n",
    "    assert y_true.ndim == 1 and y_true.size() == y_prob.size()\n",
    "    y_prob = y_prob > 0.5\n",
    "    return (y_true == y_prob).sum() / y_true.size(0)\n",
    "measure_fn = get_accuracy\n",
    "\n",
    "results = train(train_dataloader, model, \n",
    "                nn.BCELoss(), # crossentropy for binary target \n",
    "                get_accuracy, \n",
    "                torch.optim.Adam(model.parameters()), \n",
    "                device, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "results_data = pd.DataFrame(results)\n",
    "results_data.columns = ['loss', 'measure']\n",
    "ax = results_data.plot(subplots=True);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Accuracy: \" + str(results[len(results)-1][1]))\n",
    "\n",
    "true_returns = np.unique(sepsis_returns, return_counts= True)[1][0]\n",
    "true_not_returns = np.unique(sepsis_returns, return_counts= True)[1][1]\n",
    "\n",
    "print(\"Accuracy (never returns)\" + str(true_returns / len(sepsis_returns)))\n",
    "print(\"Accuracy (always returns)\" + str(true_not_returns / len(sepsis_returns)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Brief Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, that is a bit better compared to simply always saying that the patient does not return. But the accuracy on the training set (**not even considering a test set!**) is still varying a lot and the variation of the log and accuracy over the epochs trained does not look good either. So, let us still have a look at the individual predictions and their score depending on the ground truth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dataloader = DataLoader(ds, batch_size=256, shuffle=False)\n",
    "result, original = evaluate_all(test_dataloader, model, device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd_pos = pd.DataFrame({'Returns': result[original == 1]})\n",
    "pd_neg = pd.DataFrame({'Does not return': result[original == 0]})\n",
    "pd.concat([pd_pos, pd_neg],axis=1).boxplot().set_ylabel('Score')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There seems to be some separation but likely the prediction model will give us many false positives when used to identify returning patients in practice.\n",
    "\n",
    "**Of course, you should now compute the usual measures for classification tasks and the threshold for making a decision: recall, precision, confusion matrices, area under the curve and many other ways to deeply evaluate a prediction model. Always consider what would be the use case of your prediction.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why is this so bad? Let us have a look at the data distribution we put in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# count the unique vectors\n",
    "dist_bags = np.unique(sepsis_bag, return_counts=True, axis=0)\n",
    "\n",
    "# sort them with numpy\n",
    "unique_vectors = dist_bags[0][np.argsort(-dist_bags[1])]\n",
    "count_vectors = dist_bags[1][np.argsort(-dist_bags[1])]\n",
    "\n",
    "\n",
    "hist = pd.DataFrame({'Unique sample vectors': count_vectors}).hist(bins = 20)[0][0]\n",
    "hist.set_xlabel(\"Count\")\n",
    "hist.set_ylabel(\"Frequency\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many of the traces result in the exact same sample. Let us check what is the \"return status\" for the most common sample that represents more than 175 traces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# most frequently used vector\n",
    "unique_vectors[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find the sample indicies for this vector\n",
    "sample_indicies = np.where((sepsis_bag == unique_vectors[0]).all(axis=1)) \n",
    "sample_results = target_scaled[sample_indicies]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.unique(sample_results, return_counts=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is clear that, without additional information, there is nothing the prediction model can learn to represent this division for the exact same feature values. We can look at further examples, but it seems we simply cannot reliably predict whether a patient will return from the bag-of-words / multiset of events model in the Sepsis event log.\n",
    "\n",
    "This was just an example on how to use a predictive model with an event log to predict a binary process characteristic based on events contained in the event log."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "28aff1567d8aae5536826c1be921f2ff2e204808293d43dc67bdcb73bd29110e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}