intro_tidyverse.html

<!DOCTYPE html>
<html lang="" xml:lang="">
  <head>
    <title>MAT381E-Week 3: Data Importing, Manipulating, and Tidying</title>
    <meta charset="utf-8" />
    <meta name="author" content="Gül İnan" />
    <meta name="date" content="2021-10-30" />
    <script src="intro_tidyverse_files/header-attrs-2.11/header-attrs.js"></script>
    <link href="intro_tidyverse_files/remark-css-0.0.1/default.css" rel="stylesheet" />
    <script src="intro_tidyverse_files/fabric-4.3.1/fabric.min.js"></script>
    <link href="intro_tidyverse_files/xaringanExtra-scribble-0.0.1/scribble.css" rel="stylesheet" />
    <script src="intro_tidyverse_files/xaringanExtra-scribble-0.0.1/scribble.js"></script>
    <script>document.addEventListener('DOMContentLoaded', function() { window.xeScribble = new Scribble({"pen_color":["#FF0000"],"pen_size":3,"eraser_size":30,"palette":[]}) })</script>
    <link href="intro_tidyverse_files/tile-view-0.2.6/tile-view.css" rel="stylesheet" />
    <script src="intro_tidyverse_files/tile-view-0.2.6/tile-view.js"></script>
    <link href="intro_tidyverse_files/animate.css-3.7.2/animate.xaringan.css" rel="stylesheet" />
    <link href="intro_tidyverse_files/tachyons-4.12.0/tachyons.min.css" rel="stylesheet" />
    <link href="intro_tidyverse_files/panelset-0.2.6/panelset.css" rel="stylesheet" />
    <script src="intro_tidyverse_files/panelset-0.2.6/panelset.js"></script>
    <link rel="stylesheet" href="xaringan-themer.css" type="text/css" />
  </head>
  <body>
    <textarea id="source">
class: left, middle, my-title, title-slide

# MAT381E-Week 3: Data Importing, Manipulating, and Tidying
### Gül İnan
### Department of Mathematics<br/>Istanbul Technical University
### October 30, 2021

---


class: left

# Outline

* What is `Tidyverse Ecosystem`?
* `Pipe` operator.
* `Tibbles` alternative to `data frames`.
* Data importing via `readr` and `readxl` packages.
  * 01-importing.Rmd
* Data manipulating with `dplyr` package.
  * 02-manipulating.Rmd
* Data tidying with `tidyr` package.
  * 03-tidying.Rmd
  
---
class: middle, center

&lt;img src="images/time.png" width="90%" /&gt;

---
&lt;style type="text/css"&gt;
.pull-left {
  float: left;
  width: 50%;
}
.pull-right {
  float: right;
  width: 50%;
}
&lt;/style&gt;

.pull-left[
&lt;blockquote class="twitter-tweet"&gt;&lt;p lang="en" dir="ltr"&gt;Most data scientists spend almost 80% of their time inspecting and cleaning data rather than working on their machine learning models.&lt;br&gt;&lt;br&gt;But why?&lt;/p&gt;&amp;mdash; Ammar Yasser (@ammaryh92) &lt;a href="https://twitter.com/ammaryh92/status/1449261043138584578?ref_src=twsrc%5Etfw"&gt;October 16, 2021&lt;/a&gt;&lt;/blockquote&gt; &lt;script async src="https://platform.twitter.com/widgets.js" charset="utf-8"&gt;&lt;/script&gt;
]


.pull-right[
- "According to the "No Free Lunch theorem", most machine learning models have a set of built-in assumptions, and before you start training your model, you have to make sure that your data is in line with the underlying assumptions of your model."
- "Data analysis allows you to:
  1. Understand the patterns and trends in the data.
  2. Understand the data distribution.
  3. Make sure that your data is in line with the model's assumptions.
  4. Assess the quality of the data (noise, wrong values, missing values, outliers, etc).
  5. Assess the tidiness of the data (data structure, variable types, etc.)
  6. Clean your data to get it in the best possible form for your model.
  7. Feature engineering (extract features with predictive power from data)."

- "So make sure to spend some time analyzing and cleaning your model before moving on to your model. Always remember, "Garbage in, Garbage out".


]
---
class: center, middle

.pull-left[
&lt;img src="images/dataclean1.png" width="90%" /&gt;
]

.pull-right[
&lt;img src="images/dataclean2.png" width="90%" /&gt;
]

[Dezyre](https://www.dezyre.com/article/why-data-preparation-is-an-important-part-of-data-science/242)


---
class: center, middle

&lt;!-- First code block is setting options for theme of the slides  --&gt;  
&lt;!-- Second code block is setting global options R code blocks --&gt; 

# Introduction to Tidyverse Ecosystem

![Tidyverse](logo/tidyverse.png)                  

---

# What is the Tidyverse Ecosystem?

- [Tidyverse](https://www.tidyverse.org/) is **a collection of packages** built as a suite of "data science" tools with a focus on **importing, manipulating**, and **visualizing data**.


&lt;img src="images/data-science-workflow.png" width="80%" /&gt;

---
- There are about 30 packages in the `Tidyverse ecosystem`. 
- The **core** `Tidyverse ecosystem` includes the following packages:
  - `tibble`: alternative to data.frame class,
  - `readr`: open and organize the data,
  - `dplyr`: data manipulation,
  - `tidyr`: modeling and data management,
  - `ggplot2`: data visualization,
  - `forcats`: functions for working with factors, 
  - `stringr`: functions for working with string data,
  - `purrr`: code optimization and functional programming, and
  -  also, by default includes `magrittr` (source of the `pipe operator`). 
 

.pull-right[
&lt;img src="logo/logotogether.png" width="70%" /&gt;
]


---
- All these packages are designed to **work well together** and are based on: 
  - **tidy data** philosophy, where each variable has its own column, each observation has its own row,
  - use **pipe operator** `%&gt;%` to improve code development and readability, and
  - follow a **common functional structure** for code clarity and reproducibility. 
- Please visit [Tidyverse Style Guide](https://style.tidyverse.org/), especially the one on [Syntax](https://style.tidyverse.org/syntax.html).
    
 
---
# Install the Tidyverse Ecosystem

- We can install **everything at once** and access the `Tidyverse` as follows:


```r
# Install "tidyverse" package on your local machine.
# You only need to do this once per machine.
install.packages("tidyverse", dependencies = TRUE)
# By default, install.packages() will download packages from 
# https://cran.r-project.org, or one of its mirrors—so be sure you 
# are connected to the internet when you run it.
```

- This will install the following packages:


```r
[1] "broom"         "cli"           "crayon"        "dbplyr"        "dplyr"        
[6] "dtplyr"        "forcats"       "googledrive"   "googlesheets4" "ggplot2"      
[11] "haven"         "hms"           "httr"          "jsonlite"      "lubridate"    
[16] "magrittr"      "modelr"        "pillar"        "purrr"         "readr"        
[21] "readxl"        "reprex"        "rlang"         "rstudioapi"    "rvest"        
[26] "stringr"       "tibble"        "tidyr"         "xml2"          "tidyverse" 
```

---
- Now we can load package `tidyverse` into our `R` session.


```r
# You need to do this each time you start a new R session.
# You may need to update the packages (with update.packages() .
# function) from to time to time to get the latest improvements.
library(tidyverse)
```

```
#&gt; ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──
```

```
#&gt; ✓ ggplot2 3.3.5     ✓ purrr   0.3.4
#&gt; ✓ tibble  3.1.5     ✓ dplyr   1.0.7
#&gt; ✓ tidyr   1.1.4     ✓ stringr 1.4.0
#&gt; ✓ readr   2.0.2     ✓ forcats 0.5.1
```

```
#&gt; ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
#&gt; x dplyr::filter() masks stats::filter()
#&gt; x dplyr::lag()    masks stats::lag()
```

---
# What about those conflicts?
- When `R` loads packages, functions loaded in earlier packages are **overwritten** 
  by functions of the **same name** from later packages. 
- This means that **the order in which packages are loaded** matters. 
- We can see that there are two functions in the `dplyr` package which *mask* base-`R` functions of the **same name**.
- To make sure that a  function from  the correct package is used, we can use the following construction:


```r
package_name::function_name()
```

- where the double colon operator `::` is the **namespace operator**. 
- So if we want to access the base functions instead of the `tidy` ones, we should **explicitly** refer to specific functions with **namespace operator** such as :


```r
base::filter()
```
- As a general rule, it is better to load the `tidyverse` *after* all other packages; this will identify the conflicts.

---
# Example


```r
library(tidyverse)
library(MASS)
```

```
#&gt; 
#&gt; Attaching package: 'MASS'
```

```
#&gt; The following object is masked from 'package:dplyr':
#&gt; 
#&gt;     select
```

```r
#implies that you cannot use select function in dplyr package
```


```r
# but if you want to use select function in dplyr package, # use this
dplyr::select()
# or you can also use this
library(tidyverse)
library(MASS, exclude = 'select')
```

---

💡 Also notice that `library(tidyverse)` does not load every package installed by `install.packages("tidyverse")`. 

💡 We must use `library()` function to individually load the **non-core** `tidyverse` packages.


---
# Tidy Data

- **Tidy data** assumes that:
  - Each variable forms a column,
  - Each observation forms a row, and
  - Each type of observational unit forms a table.
  
&lt;img src="images/tidydata.png" width="70%" /&gt;

---

class: center, middle

&lt;img src="logo/pipe.jpeg" width="50%" /&gt;


---
# The pipe operator `%&gt;%`

- The pipe operator `%&gt;%` is a **special operator** introduced in the `magrittr` package:
  - allows us to “pipe” several functions into **one long chain** that matches the order in which we want to do stuff and, 
  - in this sense, takes the object/function call result on the left and "passes" it to the right; it does *not* make assignment by itself. 


```r
# in standard R
x &lt;- 1:3
sum(x)
```

```
#&gt; [1] 6
```


```r
# with the pipe, we can rewrite this as:
x &lt;- 1:3
# with the pipe  
x %&gt;% 
  sum()
```

```
#&gt; [1] 6
```

---

# Usage

- The pipe operator `%&gt;%` should always have a space before it, and should usually be followed by a **new line**.
- After the first step, **each line** should be **indented by two spaces**.
- If the arguments to a function do **not all fit on one line**, put each argument **on its own** line and indent.

---
- Root mean squared error (RMSE) is defined: 

$$
\text{RMSE} = \sqrt\frac{\sum_{i=1}^n(\hat{y}_i-y_i)^2}{n}
$$
where `\(\hat{y}_i\)` denotes the prediction and `\(y_i\)` the actually observed value.


```r
# in standard R
data &lt;- c(1, 3, 4, 2, 5)
prediction &lt;- c(1, 2, 2, 1, 4)
# calculate root mean squared error
rmse &lt;- sqrt(mean((prediction-data)^2))
print(rmse)
```

```
#&gt; [1] 1.183216
```

---
- Using the pipe operator `%&gt;%` makes the operation more intuitive: 


```r
data &lt;- c(1, 3, 4, 2, 5)
prediction &lt;- c(1, 2, 2, 1, 4)

# calculate root mean squared error the pipe way 
rmse &lt;- (prediction-data)^2 %&gt;% 
         mean() %&gt;% 
         sqrt() %&gt;% 
         print() 
```

```
#&gt; [1] 1.183216
```

---
class: center, middle

&lt;img src="logo/tibble.jpeg" width="30%" /&gt;
---
# What is a Tibble?

- The `Tidyverse ecosystem` uses **tibbles** as an **alternative** to the `R`'s base `data.frame` class. 
- [Tibbles](https://tibble.tidyverse.org/) are intended to represent the `tidy` data principles by design and provides opinionated data frames that make working in the `tidyverse` a little easier.
- `Tibbles` and `data frames` have many similar properties (rectangular data), but
`tibbles` returns more **readable output** in the console.
- We can create `tibbles` with `tibble()` function.


---
# Example


```r
# Create a tibble
# tibble() builds columns sequentially. 
# When defining a column, you can refer to columns created earlier in the call.
mytibb &lt;- tibble(
          "x" = 1:3, 
          "y" = 4:6, 
          "z" = x ^ 2 + y,
          "t" = c("math", "stat", "cs")
          )
mytibb
```

```
#&gt; # A tibble: 3 × 4
#&gt;       x     y     z t    
#&gt;   &lt;int&gt; &lt;int&gt; &lt;dbl&gt; &lt;chr&gt;
#&gt; 1     1     4     5 math 
#&gt; 2     2     5     9 stat 
#&gt; 3     3     6    15 cs
```

```r
class(mytibb)
```

```
#&gt; [1] "tbl_df"     "tbl"        "data.frame"
```

---
# Getting Help

💡 We can get **help on a package** by typing `package?packagename` such as:


```r
package?tibble
```

💡 We can get **help on a specific function** within that package by typing `?packagename::functionname` such as:


```r
?tibble::tibble
```

---
- We can **coerce** a `data.frame` to a `tibble` and **coerce** a `tibble` easily back to a `data.frame` as well.


```r
head(iris)
```

```
#&gt;   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#&gt; 1          5.1         3.5          1.4         0.2  setosa
#&gt; 2          4.9         3.0          1.4         0.2  setosa
#&gt; 3          4.7         3.2          1.3         0.2  setosa
#&gt; 4          4.6         3.1          1.5         0.2  setosa
#&gt; 5          5.0         3.6          1.4         0.2  setosa
#&gt; 6          5.4         3.9          1.7         0.4  setosa
```

```r
class(iris)
```

```
#&gt; [1] "data.frame"
```
---


```r
as_tibble(iris)
```

```
#&gt; # A tibble: 150 × 5
#&gt;    Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#&gt;           &lt;dbl&gt;       &lt;dbl&gt;        &lt;dbl&gt;       &lt;dbl&gt; &lt;fct&gt;  
#&gt;  1          5.1         3.5          1.4         0.2 setosa 
#&gt;  2          4.9         3            1.4         0.2 setosa 
#&gt;  3          4.7         3.2          1.3         0.2 setosa 
#&gt;  4          4.6         3.1          1.5         0.2 setosa 
#&gt;  5          5           3.6          1.4         0.2 setosa 
#&gt;  6          5.4         3.9          1.7         0.4 setosa 
#&gt;  7          4.6         3.4          1.4         0.3 setosa 
#&gt;  8          5           3.4          1.5         0.2 setosa 
#&gt;  9          4.4         2.9          1.4         0.2 setosa 
#&gt; 10          4.9         3.1          1.5         0.1 setosa 
#&gt; # … with 140 more rows
```
---
# Tibbles vs. data.frame
- There are **two main differences** in the usage of a tibble vs. a classic data.frame: **printing** and **subsetting**.
  - `Tibbles` have a refined print method that **shows only the first 10 rows**, and all the columns that fit on screen.
 -  In `Tibbles`, each column reports **its type**, a nice feature borrowed from `str()`.


---
- We can explicitly `print()` the `Tibbles` and control the number of rows (n) and the width of the display. 


```r
# coerce iris data set into a tibble
iris_tibble &lt;- as_tibble(iris)
```


```r
# `n` specifies the number of rows
# `width = Inf` will display all columns:
#  i do not have enough space, so i set n = 15
iris_tibble %&gt;% 
  print(n = 15, width = Inf)
```

```
#&gt; # A tibble: 150 × 5
#&gt;    Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#&gt;           &lt;dbl&gt;       &lt;dbl&gt;        &lt;dbl&gt;       &lt;dbl&gt; &lt;fct&gt;  
#&gt;  1          5.1         3.5          1.4         0.2 setosa 
#&gt;  2          4.9         3            1.4         0.2 setosa 
#&gt;  3          4.7         3.2          1.3         0.2 setosa 
#&gt;  4          4.6         3.1          1.5         0.2 setosa 
#&gt;  5          5           3.6          1.4         0.2 setosa 
#&gt;  6          5.4         3.9          1.7         0.4 setosa 
#&gt;  7          4.6         3.4          1.4         0.3 setosa 
#&gt;  8          5           3.4          1.5         0.2 setosa 
#&gt;  9          4.4         2.9          1.4         0.2 setosa 
#&gt; 10          4.9         3.1          1.5         0.1 setosa 
#&gt; 11          5.4         3.7          1.5         0.2 setosa 
#&gt; 12          4.8         3.4          1.6         0.2 setosa 
#&gt; 13          4.8         3            1.4         0.1 setosa 
#&gt; 14          4.3         3            1.1         0.1 setosa 
#&gt; 15          5.8         4            1.2         0.2 setosa 
#&gt; # … with 135 more rows
```

---
- Another option is to use `RStudio’s` **built-in data viewer** to get a scrollable view of the complete dataset. 


```r
# run this on R console
iris_tibble %&gt;% 
  View()
```
---
- We can pull out a single variable by `$` and `[[ ]]`.  
  - `$` only extracts by **name** and
  - `[[ ]]` can extract by **name** or **position**.


```r
# run this on R console
# Extract by column name
iris_tibble$Petal.Width
# Extract by column name
iris_tibble[["Petal.Width"]]
# Extract by position
iris_tibble[[4]]
```
---
 - To use these in a pipe, we need to use the **special placeholder** `.` :

```r
# run this on R console
iris_tibble %&gt;% .$Petal.Width
iris_tibble %&gt;% .[["Petal.Width"]]
```

- Use `as.data.frame()` to turn a tibble back to a data.frame:
 

```r
class(as.data.frame(iris_tibble))
```

```
#&gt; [1] "data.frame"
```

---
# Data Importing
- One of the most commons ways of **storing and sharing data** for analysis is through **electronic spreadsheets**. 
- A spreadsheet stores data in **rows** and **columns** and it is basically a **file version** of a **data frame**. 
- When saving such a table to a computer file one needs a way to define when a new row or column ends and the other begins. This in turn defines the cells in which single values are stored. 
- Sometimes the first row contains **column names** rather than data. 
- We call this a **header** and when reading data from a spreadsheet it is important to know if the file has a header or not. 
- **Most reading functions assume there is a header**. 
- To know if the file has a header, it helps to look at the file before trying to read it. 
- In `RStudio` we can do this by navigating to the file location, double clicking on the file and hitting _View File_.
- We start by demonstrating how to **read in a file** that is already **saved on your computer**.  

---
class: center, middle

&lt;img src="logo/readr.png" width="30%" /&gt;

---
# The readr package

- The [readr](https://readr.tidyverse.org/) is the `tidyverse` package that includes functions for **reading rectangular data** (spreadsheets like 'text', 'csv') into `R`. 
- The `readr` package can be loaded directly:


```r
library(readr)  
```

- The following functions are available to read-in spreadsheets:

| Function       | Format                       | Typical suffix          |
|----------------|------------------------------|-------------------------| 
| `read_table()` | white space separated values | txt |
| `read_delim()` | general text file format, must define delimiter | txt |
| `read_csv()`   | comma separated values|  csv |
| `read_csv2()`  | semicolon separated values | csv |
| `read_tsv()`   | tab delimited separated values | tsv |
| `read_lines()` | reads up to n_max lines from a file | - |


---

💡 We can get help on a package by typing `package?readr` and get help on a specific function
within that package by typing: `?readr::read_csv`.


```r
# let's get some help
?readr::read_csv
# focus on the arguments: file, col_names, skip, n_max
```

- More details on [Tidyverse readr](https://readr.tidyverse.org/).

---
class: center, middle

&lt;img src="logo/readxl.png" width="30%" /&gt;

---
# The readxl package

- The [readxl](https://readxl.tidyverse.org/) package provides functions to read in **Microsoft Excel** spreadsheets.
- The `readxl` package does not come with `tidyverse`. For that reason, we need to install `readxl` package and then load it.


```r
# install readxl
install.packages("readxl")
```

```r
# load readxl
library(readxl)
```


| Function        | Format                  | Typical suffix |
|-----------------|-------------------------|-----------------| 
| `read_excel()`  | auto detect the format  | xls, xlsx|
| `read_xls()`    | original format         | xls |
| `read_xlsx()`   | new format              | xlsx |

---

💡 We can get help on a package by typing `package?readxl` and get help on a specific function
within that package by typing `?readxl::read_xlsx`.


```r
# let's get some help
?readxl::read_excel
# focus on the arguments: file, sheet, col_names, skip
```


- More details on [Tidyverse readxl](https://readxl.tidyverse.org/).

---

&lt;style type="text/css"&gt;
.pull-left {
  float: left;
  width: 50%;
}
.pull-right {
  float: right;
  width: 50%;
}
&lt;/style&gt;

.pull-left[
&lt;img src="images/epi1.png" width="80%" height="50%" /&gt;
]
--
.pull-right[
&lt;img src="images/epi2.png" width="100%" height="100%" /&gt;
]

--
- Please walk around https://epi.yale.edu/.

---

&lt;img src="images/googlemob.png" width="90%" height="100%" /&gt;
--

- Please walk around hhttps://www.google.com/covid19/mobility/.

---

&lt;img src="images/transmonee.png" width="90%" height="100%" /&gt;
--

- Please walk around https://transmonee.org/.

---
- **Hands-on example:** Visit `01-importing.Rmd` file.

---
class: center, middle

&lt;img src="logo/dplyr.jpeg" width="30%" /&gt;

---
# The dplyr package
- The [dplyr](https://dplyr.tidyverse.org/) package is one part of a larger `tidyverse` ecosystem that enables us to work with data in tidy data formats. 
- Here are basic `dplyr` functions:


| Function       |  Description                                       | 
|----------------|----------------------------------------------------|
| `filter()`     | pick observations by their values (chooses rows)   | 
| `select()`     | pick variables by column names (selects columns)   | 
| `arrange()`    | sort rows by variables                             | 
| `mutate()`     | create new variables with functions of existing variables | 
| `summarise()`  | collapse many values down to a single summary |

---
# filter()
- The [filter()](https://dplyr.tidyverse.org/reference/filter.html) function is used to **subset the rows** of `data` based on the conditions applied to the column values to determine which rows should be retained.
- The first argument to this function is the data
frame and the subsequent arguments are **comparison operators** applied to the columns.


```r
#The filter() function is used to subset the rows of .data, 
#applying the expressions in ... to the column values to determine 
#which rows should be retained. 
filter(.data, ...)
```

- It supports **unquoting** in variable names.
- Multiple conditions can be applied via **Boolean operators**.
- More details on [filter()](https://dplyr.tidyverse.org/reference/filter.html).

---

- Remember the **comparison** and **Boolean** operators.

&lt;img src="images/comparison.png" width="90%" /&gt;&lt;img src="images/boolean.png" width="90%" /&gt;

---
# select()

- The [select()](https://dplyr.tidyverse.org/reference/select.html) (and optionally `rename()`) select variables in a data frame based on their **column name**.
- The first argument to this function is the data
frame and the subsequent arguments are the **columns to keep**.


```r
select(.data, ...)
```
- Variable names are quoted (or **unquoted**) expression separated by commas.
- The `select()` keeps only the variable you mention.
- The `rename()` keeps all the variables.
---
# Overview of selection features
- Tidyverse selections implement a dialect of R where operators make it easy to select variables:

&lt;br&gt;

```html
   : for selecting a range of consecutive variables.

   ! for taking the complement of a set of variables.

   &amp; and | for selecting the intersection or the union of two sets of variables.

   c() for combining selections.
```

---
- It is worth knowing that `dplyr` comes with a number of [select() helpers](https://www.rdocumentation.org/packages/dplyr/versions/0.7.2/topics/select_helpers), which are functions that allow us to select columns based on their names. 
- These functions allow you to select variables based on their names.

&lt;br&gt;

```html
   starts_with(): starts with a prefix.

   ends_with(): ends with a prefix.

   contains(): contains a literal string.

   matches(): matches a regular expression.

   num_range(): a numerical range like x01, x02, x03.

   one_of(): variables in character vector.

   everything(): all variables.
```

- More details on [select()](https://dplyr.tidyverse.org/reference/select.html).

---
# arrange()
- The [arrange()](https://dplyr.tidyverse.org/reference/arrange.html) **orders the rows** of a data frame by the values of selected columns. 
- The first argument to this function is the data
frame and the subsequent arguments are comma separated list of unquoted variable names, or expressions involving variable names.


```r
arrange(.data, ...)
```

- To sort from highest to lowest value or from Z to A, add `desc()` to `arrange()`.
- More details on [arrange()](https://dplyr.tidyverse.org/reference/arrange.html).

---
# mutate()
- The [mutate()](https://dplyr.tidyverse.org/reference/mutate.html) **creates new columns** based on the values in existing columns and **preserves** existing ones.


```r
mutate(.data, ...)
```

- The new and edited columns will not permanently be added to the existing data frame -- unless we explicitly save the output.
- **New variables** overwrite existing variables of the same name.
- The `transmute()` adds new variables and drops existing ones.
- More details on [mutate()](https://dplyr.tidyverse.org/reference/mutate.html).

---
# case_when()
- The [case_when()](https://dplyr.tidyverse.org/reference/case_when.html) allows us to vectorize **multiple** `if_else()` statements.
- It is `R` equivalent of the `SQL CASE WHEN` statement.
- A sequence of two-sided two sided formulas. LHS determines which values match this case. RHS provides the replacement values. 
- See the example in `02-manipulating.Rmd`.

---
# summarise()
- The [summarise()](https://dplyr.tidyverse.org/reference/summarise.html) summarizes one or more variables in data
frame.
- Summarizing should be done with an expression that returns a single value like `min()`,
`max()`, `n()`, `sum()`.
- To use the function we just add our new column name, 
 and after the equal sign the mathematics of what needs to happen, column_name = function(variable).  
 

```r
summarise(.data, column_name = function(variable))
```

- You can add multiple summary functions behind each other.  Here are the most commonly
used functions:

| Explanation  | Function              | 
|--------------|-----------------------|
| Center       | `mean()`, `median()`  | 
| Spread       | `var()`, `sd()`       | 
| Range        | `min()`, `max()`       | 
| Position     | `first()`, `last()`, `nth()`|
| Count        | `n()`, `n_distinct()`|
| the number of observations | `n()` |
| the numbers of unique values of the variable | `n_distinct()`|  

- More info [summarise()](https://dplyr.tidyverse.org/reference/summarise.html).


---
# group_by()
- In most cases, we want to get summaries by a group.
- The `group_by()` is often used together with `summarise()` which takes the column names as arguments that contain the **categorical** variables for which you want
to calculate the summary statistics.


```r
group_by(.data, ...)
```


- We can also group by multiple columns.
- More details on [group_by()](https://dplyr.tidyverse.org/reference/group_by.html).

---

- **Hands-on example:** Visit `02-manipulating.Rmd` file.

- **Study by yourself**: https://www.riinu.me/2020/02/r-filtering-with-na-values/.

---

class: center, middle

&lt;img src="logo/tidyr.png" width="30%" /&gt;

---
# The tidyr package

- The [tidyr](https://tidyr.tidyverse.org/) package describes a standard way of storing data that is used wherever possible throughout the `Tidyverse ecosystem`.
- Tidy data is data where: 
   - Every column is variable.
   - Every row is an observation.
   - Every cell is a single value.

- Here are basic `tidyr` functions:

| Function          |  Description                           | 
|-------------------|----------------------------------------|
| `pivot_longer()` |lengthens data, increasing the number of rows and decreasing the number of columns.|
| `pivot_wider()`  |The inverse transformation of pivot_longer(). |
| `drop_na()`      |drops rows containing missing values. | 
| `replace_na()`   |replaces NAs with specified values. |  

- More details on [tidyr](https://tidyr.tidyverse.org/).
---


&lt;img src="images/wilddata.jpeg" width="90%" /&gt;

- **Hands-on example:** Visit `03-tidying.Rmd` file.

---
# Exporting data 

- Similar to the `read_csv()` function used for reading CSV files into `R`, there is a `write_csv()` function that generates CSV files from data frames.


```r
write_csv(cth_data_filtered, "data/cth_data_reduced.csv")
```

---

# Attributions
- More on data types [Data types](https://r-coder.com/data-types-r/).
- All images used in this slide are taken from the web.
- This lecture note is mainly developed by following sources:
   - [McMasterl](https://socialsciences.mcmaster.ca/jfox/Courses/R/ICPSR/tidy_lecture-slides.pdf), 
   - [Stat Duke](http://www2.stat.duke.edu/~cr173/Sta112_Fa16/data_wrangling.html),
   - [Tidyverse](https://www.tidyverse.org/),
   - [Psych252](https://psych252.github.io/psych252book/data-wrangling-1.html),
   - [R4ds](https://r4ds.had.co.nz/tibbles.html),
   - [Data Science Labs](https://datasciencelabs.github.io/pages/lectures.html) and
   - [Cengel](https://github.com/cengel/R-data-wrangling).


    </textarea>
<style data-target="print-only">@media screen {.remark-slide-container{display:block;}.remark-slide-scaler{box-shadow:none;}}</style>
<script src="https://remarkjs.com/downloads/remark-latest.min.js"></script>
<script src="assets/remark-zoom.js"></script>
<script src="https://platform.twitter.com/widgets.js"></script>
<script>var slideshow = remark.create({
"highlightStyle": "github",
"highlightLines": true,
"countIncrementalSlides": false,
"ratio": "16:9",
"navigation": {
"scroll": false
}
});
if (window.HTMLWidgets) slideshow.on('afterShowSlide', function (slide) {
  window.dispatchEvent(new Event('resize'));
});
(function(d) {
  var s = d.createElement("style"), r = d.querySelector(".remark-slide-scaler");
  if (!r) return;
  s.type = "text/css"; s.innerHTML = "@page {size: " + r.style.width + " " + r.style.height +"; }";
  d.head.appendChild(s);
})(document);

(function(d) {
  var el = d.getElementsByClassName("remark-slides-area");
  if (!el) return;
  var slide, slides = slideshow.getSlides(), els = el[0].children;
  for (var i = 1; i < slides.length; i++) {
    slide = slides[i];
    if (slide.properties.continued === "true" || slide.properties.count === "false") {
      els[i - 1].className += ' has-continuation';
    }
  }
  var s = d.createElement("style");
  s.type = "text/css"; s.innerHTML = "@media print { .has-continuation { display: none; } }";
  d.head.appendChild(s);
})(document);
// delete the temporary CSS (for displaying all slides initially) when the user
// starts to view slides
(function() {
  var deleted = false;
  slideshow.on('beforeShowSlide', function(slide) {
    if (deleted) return;
    var sheets = document.styleSheets, node;
    for (var i = 0; i < sheets.length; i++) {
      node = sheets[i].ownerNode;
      if (node.dataset["target"] !== "print-only") continue;
      node.parentNode.removeChild(node);
    }
    deleted = true;
  });
})();
(function() {
  "use strict"
  // Replace <script> tags in slides area to make them executable
  var scripts = document.querySelectorAll(
    '.remark-slides-area .remark-slide-container script'
  );
  if (!scripts.length) return;
  for (var i = 0; i < scripts.length; i++) {
    var s = document.createElement('script');
    var code = document.createTextNode(scripts[i].textContent);
    s.appendChild(code);
    var scriptAttrs = scripts[i].attributes;
    for (var j = 0; j < scriptAttrs.length; j++) {
      s.setAttribute(scriptAttrs[j].name, scriptAttrs[j].value);
    }
    scripts[i].parentElement.replaceChild(s, scripts[i]);
  }
})();
(function() {
  var links = document.getElementsByTagName('a');
  for (var i = 0; i < links.length; i++) {
    if (/^(https?:)?\/\//.test(links[i].getAttribute('href'))) {
      links[i].target = '_blank';
    }
  }
})();
// adds .remark-code-has-line-highlighted class to <pre> parent elements
// of code chunks containing highlighted lines with class .remark-code-line-highlighted
(function(d) {
  const hlines = d.querySelectorAll('.remark-code-line-highlighted');
  const preParents = [];
  const findPreParent = function(line, p = 0) {
    if (p > 1) return null; // traverse up no further than grandparent
    const el = line.parentElement;
    return el.tagName === "PRE" ? el : findPreParent(el, ++p);
  };

  for (let line of hlines) {
    let pre = findPreParent(line);
    if (pre && !preParents.includes(pre)) preParents.push(pre);
  }
  preParents.forEach(p => p.classList.add("remark-code-has-line-highlighted"));
})(document);</script>

<script>
slideshow._releaseMath = function(el) {
  var i, text, code, codes = el.getElementsByTagName('code');
  for (i = 0; i < codes.length;) {
    code = codes[i];
    if (code.parentNode.tagName !== 'PRE' && code.childElementCount === 0) {
      text = code.textContent;
      if (/^\\\((.|\s)+\\\)$/.test(text) || /^\\\[(.|\s)+\\\]$/.test(text) ||
          /^\$\$(.|\s)+\$\$$/.test(text) ||
          /^\\begin\{([^}]+)\}(.|\s)+\\end\{[^}]+\}$/.test(text)) {
        code.outerHTML = code.innerHTML;  // remove <code></code>
        continue;
      }
    }
    i++;
  }
};
slideshow._releaseMath(document);
</script>
<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
  var script = document.createElement('script');
  script.type = 'text/javascript';
  script.src  = 'https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-MML-AM_CHTML';
  if (location.protocol !== 'file:' && /^https?:/.test(script.src))
    script.src  = script.src.replace(/^https?:/, '');
  document.getElementsByTagName('head')[0].appendChild(script);
})();
</script>
  </body>
</html>