In this section, we explore importing data into R, focusing on the readr package. readr is part of the tidyverse, providing an efficient way to read tabular data like CSV and TSV files. There are many packages designed for specific type of data import. For example, readxl package is designed to read excel files, haven package is designed to read SAS, SPSS and Stata files. We will explore only readr package in the next section.
We will focus on the following topics:
readr package.readr functions.Please install and load the readr package before proceeding. If you have not done so already please follow the instructions in the previous session’s Exercise 7.
readr packagereadr is part of the tidyverse suite in R and is designed to efficiently read and write tabular data. It is known for its simplicity and speed compared to base R functions. The key features of readr package are:
readrMethods of importing data
There are two primary methods for importing data into R:
1 - From local file on a computer 2 - From the Web directly
I have generated simulated data similar to iris data set and saved it in the data folder on my computer. You can use the iris data available in R, OR read in any data in CSV or TSV file format.
iris data set is a famous real world data set that is available in R. It is a data set that contains information about iris. It contains 150 observations and 5 variables. The variables are sepal length, sepal width, petal length, petal width, and species. The species variable has three levels: setosa, versicolor, and virginica. The data set is available in R and can be loaded using the data() function. The data set is also available in CSV format on the internet.
We will use the simulated data set to demonstrate how to import data into R from a local file.
1- Importing data from local file
Important things to know before importing data from a local file
In this example, we’ll use the variable name iris. Remember, you’ll need to provide the file path on your computer. By doing this, the data will be loaded into R and assigned to the variable iris. You also get a brief summary of the data set.
# Import data from local file
iris <- readr::read_csv("../data/iris.csv")Rows: 150 Columns: 5
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (1): species
dbl (4): Sepal.Length, Sepal.Width, Petal.Length, Petal.Width
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.While you will be working with your projects in R you will face different scenarios where you will need to import data from a local file.
Below are a few examples of different scenarios for importing data from a local file.
project/
├── data_file.csv
└── script.RImport using:
data <- read_csv("./data_file.csv")
project/
├── data/
│ └── data_file.csv
└── script.RImport using:
data <- read_csv("./data/data_file.csv")
project/
├── data_file.csv
└── session/
└── script.RImport using:
data <- read_csv("../data_file.csv")
project/
├── data/
│ └── data_file.csv
└── session/
└── script.RImport using:
data <- read_csv("../../data/data_file.csv")
project/
├── data/
│ └── data_file.csv
└── session/
└── subfolder/
└── script.RImport using:
data <- read_csv("../../../data/data_file.csv")
Have look at the iris data with function head()
We can use the iris data for further exploration load the iris data set using the data() function
data(iris)You will see the iris data set in the environment pane. You can also use the head() function to view the first few rows of the data set.
head(iris) Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1 5.1 3.5 1.4 0.2 setosa
2 4.9 3.0 1.4 0.2 setosa
3 4.7 3.2 1.3 0.2 setosa
4 4.6 3.1 1.5 0.2 setosa
5 5.0 3.6 1.4 0.2 setosa
6 5.4 3.9 1.7 0.4 setosa2- Importing data from web
In this section, we will learn how to import data from the web. We will use the read_csv() function to import data from the web. The read_csv() function takes the URL of the data as an argument. The URL is the location of the data on the web. The URL is a string, so it must be enclosed in quotes ““.
# URL of the dataset
url <- "https://raw.githubusercontent.com/allisonhorst/palmerpenguins/master/inst/extdata/penguins.csv"
# Read the data into R
penguins_data <- read_csv(url)Rows: 344 Columns: 8
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (3): species, island, sex
dbl (5): bill_length_mm, bill_depth_mm, flipper_length_mm, body_mass_g, year
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.In this example we first stored the web address into a variable url. Then we used the read_csv() function to read the data from url into R. We assigned the data to the variable penguins_data. We can now use the variable penguins_data to access the data.
head(penguins_data)# A tibble: 6 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<chr> <chr> <dbl> <dbl> <dbl> <dbl>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 39.5 17.4 186 3800
3 Adelie Torgersen 40.3 18 195 3250
4 Adelie Torgersen NA NA NA NA
5 Adelie Torgersen 36.7 19.3 193 3450
6 Adelie Torgersen 39.3 20.6 190 3650
# ℹ 2 more variables: sex <chr>, year <dbl>3- Saving and reloading your own data
So far we’ve imported data that already existed as a file. But you can also go the other way: take a data frame you’ve built in R and save it as a CSV file with write_csv(), ready to share or reopen later. Let’s do this with the fruit_data data frame from the Vectors and Factors session:
# Recreate the fruit_data data frame from the Vectors and Factors session
fruits <- c("Apple", "Banana", "Cherry", "Date", "Elderberry")
counts <- c(23, 15, 19, 10, 5)
fruit_data <- data.frame(Fruit = fruits, Count = counts)
# Save fruit_data to a CSV file in the data folder
write_csv(fruit_data, "../data/fruit_data.csv")# Now read it back in -- this is exactly what you'd do if a
# colleague sent you this CSV file
fruit_data_reloaded <- read_csv("../data/fruit_data.csv")Rows: 5 Columns: 2
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (1): Fruit
dbl (1): Count
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.fruit_data_reloaded# A tibble: 5 × 2
Fruit Count
<chr> <dbl>
1 Apple 23
2 Banana 15
3 Cherry 19
4 Date 10
5 Elderberry 5Add a new fruit and count to fruit_data (e.g. rbind(fruit_data, data.frame(Fruit = "Fig", Count = 8))), save it to CSV again, and read it back in. Does the new row appear when you reload it?
Understanding your dataset’s structure is vital in data analysis. R offers essential functions to examine and comprehend this structure. This guide will introduce these tools, crucial for familiarizing yourself with your data before proceeding with analysis or visualization. R’s functions enable effective data summarization and structural inspection.
Basic Functions for Data Exploration
Viewing Data
head() and tail(): These functions show the first and last parts of your data, respectively.
head(iris) Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1 5.1 3.5 1.4 0.2 setosa
2 4.9 3.0 1.4 0.2 setosa
3 4.7 3.2 1.3 0.2 setosa
4 4.6 3.1 1.5 0.2 setosa
5 5.0 3.6 1.4 0.2 setosa
6 5.4 3.9 1.7 0.4 setosatail(iris) Sepal.Length Sepal.Width Petal.Length Petal.Width Species
145 6.7 3.3 5.7 2.5 virginica
146 6.7 3.0 5.2 2.3 virginica
147 6.3 2.5 5.0 1.9 virginica
148 6.5 3.0 5.2 2.0 virginica
149 6.2 3.4 5.4 2.3 virginica
150 5.9 3.0 5.1 1.8 virginicaSummarizing Data
summary(): This function gives a quick summary of the data in each column, such as mean, median, min, max for numeric data, and frequency for categorical data.
summary(iris) Sepal.Length Sepal.Width Petal.Length Petal.Width
Min. :4.300 Min. :2.000 Min. :1.000 Min. :0.100
1st Qu.:5.100 1st Qu.:2.800 1st Qu.:1.600 1st Qu.:0.300
Median :5.800 Median :3.000 Median :4.350 Median :1.300
Mean :5.843 Mean :3.057 Mean :3.758 Mean :1.199
3rd Qu.:6.400 3rd Qu.:3.300 3rd Qu.:5.100 3rd Qu.:1.800
Max. :7.900 Max. :4.400 Max. :6.900 Max. :2.500
Species
setosa :50
versicolor:50
virginica :50
These are the numerical columns of the dataset.
Sepal.Length, Sepal.Width, Petal.Length, Petal.Width:
For each of these columns, the summary output shows:
Min.: The smallest value in the column. 1st Qu.: The first quartile (25th percentile), meaning 25% of the values in the column are below this number. Median: The middle value of the column when the values are sorted in ascending order. It divides the data into two halves. Mean: The average of all the values in the column. 3rd Qu.: The third quartile (75th percentile), indicating 75% of the values are below this number. Max.: The largest value in the column. For example, for Sepal.Length, the smallest value is 4.3, the median is 6.0, and the largest value is 7.9.
species: This is a categorical column, as indicated by the data types Class :character and Mode :character. The summary for this kind of column is different:
Length: The total number of entries in the column. In this case, there are 150 species entries. Class and Mode: These indicate the data type of the column, which is character in this case, suggesting that the species names are text data.
Understanding Data Structure
str(): This function displays the structure of your data, including the type of each column, the first few entries in each column, and the total number of observations.
str(iris)'data.frame': 150 obs. of 5 variables:
$ Sepal.Length: num 5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
$ Sepal.Width : num 3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
$ Petal.Length: num 1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
$ Petal.Width : num 0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
$ Species : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...dim(): Use this to find out the dimensions of your data (number of rows and columns).
dim(iris)[1] 150 5names(): This returns the names of the columns in your data.
names(iris)[1] "Sepal.Length" "Sepal.Width" "Petal.Length" "Petal.Width" "Species" class(): This function tells you the class of the data object (e.g., data.frame, matrix).
class(iris)[1] "data.frame"Practice Exercise Load a dataset into R (this can be any dataset of your choice, such as the built-in mtcars or iris datasets) and use the functions mentioned above to explore its structure. Write down your observations about the dataset’s size, structure, and types of data it contains.
More than the above described two methods you can also use the data from R. R has some built-in datasets. You can use these datasets to practice your data analysis skills. The mtcars dataset is a classic dataset available in R, containing data extracted from the 1974 Motor Trend US magazine. It comprises fuel consumption and 10 aspects of automobile design and performance for 32 automobiles.
Tasks
1- Load the Data
- Use the data() function to load the mtcars dataset into your R environment.2- View the Data
3- Summarize the Data
4- Explore the Data Structure
5- Determine the Data Dimensions
6- Retrieve the Column Names
7- Identify the Data Type
Questions for Reflection
Solution
Load the Data
data(mtcars)View the Data
To view the first few rows of the dataset, use the head() function.
head(mtcars) mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2
Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1Summarize the Data
For a statistical summary of the dataset, use the summary() function.
summary(mtcars) mpg cyl disp hp
Min. :10.40 Min. :4.000 Min. : 71.1 Min. : 52.0
1st Qu.:15.43 1st Qu.:4.000 1st Qu.:120.8 1st Qu.: 96.5
Median :19.20 Median :6.000 Median :196.3 Median :123.0
Mean :20.09 Mean :6.188 Mean :230.7 Mean :146.7
3rd Qu.:22.80 3rd Qu.:8.000 3rd Qu.:326.0 3rd Qu.:180.0
Max. :33.90 Max. :8.000 Max. :472.0 Max. :335.0
drat wt qsec vs
Min. :2.760 Min. :1.513 Min. :14.50 Min. :0.0000
1st Qu.:3.080 1st Qu.:2.581 1st Qu.:16.89 1st Qu.:0.0000
Median :3.695 Median :3.325 Median :17.71 Median :0.0000
Mean :3.597 Mean :3.217 Mean :17.85 Mean :0.4375
3rd Qu.:3.920 3rd Qu.:3.610 3rd Qu.:18.90 3rd Qu.:1.0000
Max. :4.930 Max. :5.424 Max. :22.90 Max. :1.0000
am gear carb
Min. :0.0000 Min. :3.000 Min. :1.000
1st Qu.:0.0000 1st Qu.:3.000 1st Qu.:2.000
Median :0.0000 Median :4.000 Median :2.000
Mean :0.4062 Mean :3.688 Mean :2.812
3rd Qu.:1.0000 3rd Qu.:4.000 3rd Qu.:4.000
Max. :1.0000 Max. :5.000 Max. :8.000 Explore the Data Structure
To understand the structure of the dataset, use the str() function.
str(mtcars)'data.frame': 32 obs. of 11 variables:
$ mpg : num 21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
$ cyl : num 6 6 4 6 8 6 8 4 4 6 ...
$ disp: num 160 160 108 258 360 ...
$ hp : num 110 110 93 110 175 105 245 62 95 123 ...
$ drat: num 3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
$ wt : num 2.62 2.88 2.32 3.21 3.44 ...
$ qsec: num 16.5 17 18.6 19.4 17 ...
$ vs : num 0 0 1 1 0 1 0 1 1 1 ...
$ am : num 1 1 1 0 0 0 0 0 0 0 ...
$ gear: num 4 4 4 3 3 3 3 4 4 4 ...
$ carb: num 4 4 1 1 2 1 4 2 2 4 ...Determine the Data Dimensions
The dim() function provides the dimensions of the dataset.
dim(mtcars)[1] 32 11Retrieve the Column Names
Use the names() function to get the column names.
names(mtcars) [1] "mpg" "cyl" "disp" "hp" "drat" "wt" "qsec" "vs" "am" "gear"
[11] "carb"Identify the Data Type
The class() function reveals the data type of the dataset.
class(mtcars)[1] "data.frame"Questions for Reflection
Dimensions of the mtcars Dataset:
Use dim(mtcars) to find the dimensions. The mtcars dataset has 32 rows (cars) and 11 columns (variables).
Categorical Variables:
By examining the dataset using str(mtcars) or summary(mtcars), you can identify that the mtcars dataset does not explicitly contain categorical variables as all columns are either integer or numeric. However, some variables like gear and cyl (number of gears and cylinders, respectively) can be considered as categorical in certain contexts.
Average Value of MPG:
To find the average value of the mpg column, use the mean() function.
mean(mtcars$mpg)[1] 20.09062Missing data is a prevalent issue in real-world datasets, arising from various sources like data entry errors, equipment malfunctions, or participant dropout in studies. It poses challenges for many machine learning algorithms, which often require complete datasets. Thus, identifying and handling missing data is a crucial step in data analysis.
We will demonstrate how to identify missing data in R using the airquality dataset, which records daily air quality measurements in New York from May to September 1973.
To begin, load the airquality dataset into your R environment using the data(airquality) function.
Using the is.na() Function
The is.na() function in R helps in checking for missing values. It returns a logical vector where each value is TRUE if it’s missing (NA), and FALSE otherwise.
# Check for missing values in the airquality dataset
missing_values <- is.na(airquality)head(missing_values) Ozone Solar.R Wind Temp Month Day
[1,] FALSE FALSE FALSE FALSE FALSE FALSE
[2,] FALSE FALSE FALSE FALSE FALSE FALSE
[3,] FALSE FALSE FALSE FALSE FALSE FALSE
[4,] FALSE FALSE FALSE FALSE FALSE FALSE
[5,] TRUE TRUE FALSE FALSE FALSE FALSE
[6,] FALSE TRUE FALSE FALSE FALSE FALSECounting missing values in a specific column
# Counting missing values in specific columns of the airquality dataset
num_missing_ozone <- sum(is.na(airquality$Ozone))
num_missing_ozone[1] 37function sum() is used to count the number of missing values in the Ozone column.
num_missing_solarR <- sum(is.na(airquality$Solar.R))
num_missing_solarR[1] 7Counting missing values in each column
# Counting missing values in each column of the dataset
col_missing_values <- colSums(is.na(airquality))
col_missing_values Ozone Solar.R Wind Temp Month Day
37 7 0 0 0 0 function colSums() is used to count the number of missing values in each column of the dataset.
# Counting the number of columns with missing values
num_cols_missing <- sum(col_missing_values > 0)Counting missing values in each row
# Counting missing values in each row of the dataset
row_missing_values <- rowSums(is.na(airquality))function rowSums() is used to count the number of missing values in each row of the dataset.
head(row_missing_values)[1] 0 0 0 0 2 1# Counting the number of rows with missing values
num_rows_missing <- sum(row_missing_values > 0)num_rows_missing[1] 42# Counting the number of rows with no missing values
num_rows_complete <- sum(row_missing_values == 0)num_rows_complete[1] 111After identifying missing data, you can decide how to handle it. Common strategies include:
1 Removing Rows or Columns: If a column or row has too many missing values, it might be best to exclude it from analysis.
2 Imputing Values: You can fill in missing values with estimates, such as the mean or median of the column.
When a column or row in your dataset has too many missing values, it might be impractical to impute them. In such cases, you might choose to remove these rows or columns from your analysis.
Removing Rows
To remove rows with any missing values:
# Removing rows with any missing values
airquality_clean <- na.omit(airquality)Removing Columns To remove columns with any missing values:
# Define a threshold for removal (e.g., 50% missing values)
threshold <- 0.5 * nrow(airquality)
# Removing columns with missing values above the threshold
airquality_clean <- airquality[, colSums(is.na(airquality)) < threshold]Imputing Values Instead of removing missing data, another approach is to fill in the missing values with estimates. This process is known as imputation. Common methods include using the mean, median, or a predictive model to estimate the missing values.
Imputing with Mean or Median
# Impute missing values in the Ozone column with the mean
airquality$Ozone[is.na(airquality$Ozone)] <- mean(airquality$Ozone, na.rm = TRUE)Alternatively, you can use the median, which is less sensitive to outliers:
# Impute missing values in the Ozone column with the median
airquality$Ozone[is.na(airquality$Ozone)] <- median(airquality$Ozone, na.rm = TRUE)Imputing with a Predictive Model
# Impute missing values in the Ozone column with a predictive model
airquality$Ozone[is.na(airquality$Ozone)] <- predict(lm(Ozone ~ ., data = airquality), airquality)[is.na(airquality$Ozone)]The above code uses a linear regression model to predict the missing Ozone values based on the other variables in the dataset. The lm() function is used to fit the linear regression model, and the predict() function is used to predict the missing values. The lm() function takes two arguments: the first is a formula specifying the model, and the second is the dataset. The predict() function takes two arguments: the first is the model, and the second is the dataset. The [is.na(airquality$Ozone)] argument is used to specify that only the missing values should be predicted. The predict() function returns a vector of predicted values, which is then used to replace the missing values in the Ozone column.
Key takeaways:
read_csv() (from readr) imports a CSV file from a local path or a web URL into a data frame; write_csv() saves a data frame back to a CSV file.C:/...) or relative (start from the working directory, e.g. ../data/file.csv) – check getwd() if R can’t find your file.head(), tail(), summary(), str(), dim(), names(), and class() all help you understand a newly imported dataset.is.na() flags missing values (NA); na.omit() removes rows with missing values, while imputation (mean, median, or a model) fills them in instead.Check your understanding:
Which readr function would you use to read a CSV file located at ../data/fruit_data.csv?
True or false: "../data/file.csv" is an example of an absolute file path.
Which function would you use to count the number of missing values in each column of a data frame?
What’s next: Continue to Basic Data Visualization, where you’ll plot the fruit_data you just saved and reloaded.