Working with CSV Files in R: Creating a Loop for Multiple Subfolders

R is an incredibly powerful programming language and environment for data analysis, and its flexibility makes it a popular choice among data scientists. One of the key tasks in working with R is handling CSV files, which can be found in various subfolders across different directories. In this article, we’ll explore how to create a loop that reads CSV files from multiple subfolders, stores their data in separate data frames, and combines them into a single list.

Understanding CSV Files

Before diving into the code, let’s quickly review what CSV files are. CSV stands for Comma Separated Values, which is a simple file format used to store tabular data, such as temperature readings from weather stations. Each row in the file represents a single observation, and each column represents a variable. For example, a CSV file might contain columns like date, temperature (°C), and station ID.

Looping through Subfolders

In this article, we’ll assume that you have a folder containing 10 subfolders, each corresponding to a different weather station. Within each subfolder, there are four CSV files: station134_2000_2005.csv, station134_2006_2011.csv, and so on.

The first step in our loop is to read the names of all subfolders using list.files(). This function returns a list of all files and subdirectories in the specified directory. We’ll use this output as input for our loop.

directory <- list.files()  # Get the names of all subfolders

Reading CSV Files within Each Subfolder

Once we have the names of all subfolders, we need to read their corresponding CSV files. For each subfolder i, we’ll use another list.files() call to get a list of all CSV files (periexomena) within that folder.

for (i in directory) {
  periexomena <- list.files(i, full.names = T, pattern = "\\.csv$")
  
  # Now we can read each CSV file and store its data in a separate data frame
  
  for (f in periexomena) {
    data_files <- read.csv(f, stringsAsFactors = F, sep = ";", dec = ",")
    
    # Add the data to our main dataframe
  }
}

Rbinding DataFrames

After reading all CSV files within each subfolder, we need to combine their data into a single data frame using rbind.fill(). This function takes multiple data frames as input and returns a new data frame with all rows from the original frames.

However, our current approach has some issues. When we use rbind() to add data from one dataframe to another, R creates a new row for each non-matching value in the columns. In our case, this means that when we combine multiple CSV files into a single data frame, it will have many more rows than necessary.

Using lapply() and do.call()

To avoid these issues, we can use the lapply() function to apply a function to each CSV file within a subfolder. The do.call() function then takes the results of this application and combines them into a single data frame using rbind.fill().

slotted <- lapply(setNames(nm = directory), function(D) {
  alldat <- lapply(list.files(D, pattern="\\.csv$", full.names=TRUE),
                   function(fn) {
                     message(fn)
                     read.csv2(fn, stringsAsFactors=FALSE)
                   })
  
  # Combine the data from all CSV files in a subfolder into one data frame
  do.call(rbind.fill, alldat)
})

In this code:

1. lapply() applies the function to each CSV file within a subfolder (list.files(D, pattern="\\.csv$", full.names=TRUE)).
2. The inner lapply() call reads each CSV file and stores its data in a separate dataframe.
3. do.call(rbind.fill, alldat) combines all data frames from the inner loop into one single data frame.

Creating a List of Data Frames

Finally, we can use lapply() to create a list of data frames, where each element corresponds to a subfolder in our original directory.

slotted <- lapply(setNames(nm = directory), function(D) {
  # ...
})

This way, we have a list (slotted) that contains separate data frames for each subfolder. Each data frame stores the combined CSV file data from its corresponding subfolder.

Conclusion

In this article, we explored how to create a loop that reads CSV files from multiple subfolders and combines their data into separate data frames using R. We used lapply() and do.call(rbind.fill()) to efficiently combine data from all CSV files within each subfolder, resulting in a list of data frames where each element corresponds to a subfolder.

We hope this explanation has provided you with a better understanding of how to work with CSV files in R and create efficient loops for combining multiple datasets.

Last modified on 2023-11-21