Missing values in big data research: some basic skills

Introduction

Missing value occurs when there is no data value for a variable in an observation. The phenomenon of missing value is universal in clinical researches involving big data. Nurses may forget to record urine output at a certain time point. Patients may have only one measurement of blood lactate, while the researcher is interested in exploring the impact of lactate trend on mortality outcome. Other reasons of missing values include but not limited to coding errors, faulty equipment and nonresponses (1). In statistical packages, some commands (e.g., logistic regression) may automatically deleted observations with missing values. There is no problem if there are a few incomplete observations. However, when there are a large number of observations with missing values, the default listwise deletion may result in significant loss of information. In such situation, analysts should take a close look at the missing patterns and find appropriate means to cope with it. The present article will introduce how missing values are handled in R, and provide some basic skills in dealing with missing values.

How missing value is handled in R

Missing value is represented by the symbol NA (not available) in R. When you read an Excel spreadsheet containing empty cells into R console, these empty cells will be replaced by NAs. This is different from STATA where empty cells are replaced with “.”. The same missing value symbol is used in R for both numeric and character variables. R provides several functions for handling missing value (Table 1).

Table 1 R functions to handle missing values
Full table

The function is.na() is the most commonly used method to indicate which element is NA. It returns logical values (FALSE or TRUE) and have the same length as its argument. Suppose we have six patients. Five lactate values are recoded and one is missing.

The returning vector of is.na() have the length of six. In the fourth place the value is TRUE, indicating lactate value is missing in the forth patient.

Someone may think of using logical test (e.g., lactate==NA) to examine the missing pattern. This can never be TRUE because missing values are considered non-comparable and you have to use missing value functions. The “==” operator returns NA when either argument is NA. By using which() function, you can locate which element of a vector contains NA. In the example, which() function returns 4, indicating the forth patient has missing lactate.

Next, you may want to describe lactate levels of the six patients. In statistical description, mean, variance and standard deviation are among the most commonly used statistics.

All of these functions return NA value because the vector lactate contains missing values. Fortunately, there is a function na.rm() to remove NAs when applying statistical functions.

The results are exactly what you want. Both mean and standard deviation are calculated based on the five patients with lactate values available.

Data frame with missing values

In real world setting, what you encounter is missing values in a data frame. Thus, this section focuses on how to handle data frames with missing values. First we create a data frame of three variables and five observations.

Note that the third patient have missing value on sex, and the forth patient have missing value on lactate.

The function na.fail() can be used to detect missing values in a dataset. If there is no missing value it returns the assigned object (see below). If there is missing value it returns an error message indicating there is one or more missing values in the dataset.

Although some good default settings in regression model can effectively ignore observations with missing values, it is useful to create a new data frame by omitting observations with missing values.

The function na.omit() returns the object with incomplete cases removed. As you can see, the third and forth patients with missing value in one variable are removed. Alternatively, the same purpose can be achieved by using the following code.

By applying na.fail() to the complete dataset obtained by na.omit(), the error message is replaced by a new complete dataset. The test passed when the new dataset contains no NAs.

Sometimes you may want to locate which variable of a patient contains missing value. Try functions is.na() and which() as described previously.

It returns 8 and 14 indicating the location of missing values. R counts the place in a column-wise fashion. There is no problem when the dataset is not large. However, if the dataset is large (always the case in big data exploration) you are overwhelmed by a long list of location numbers. The following code helps you to select observations with at least one missing value.

As expected, the returned results indicate the third and forth patients who have at least one missing variable. The function lapply() returns a list of the locations of missing values across variables ptid, sex and lactate. Unlist() simplifies list structure to produce a vector. When there is more than one missing value in an observation, unique() simplified it to list observations with missing values only once.

In occasions, you may be interested in counting missing values for a variable (number of missing values in a column). Then you can restrict your analysis to variables with less than one fifth missing variables.

Missing values in regression model

Regression model building is probably the most commonly used in statistical analysis. However, details on missing values are always omitted in regression model fitting. With small number of missing values, it is safe to fit a model by default argument. Problem may arise with substantial number of missing values and analysts have to understand how missing values are handled in model building. Here the difference between na.omit() and na.exclude() will be shown.

For the illustration purpose, I will regress lactate on patient id. Of course, this has no practical meaning.

As you can see, the na.exclude() function pads the residuals and fitted values with NAs where there are missing values. However, the na.omit() function simply exclude observations with missing values. In this regard, na.exclude() is a placeholder for missing values.

Some advanced functions for missing values

There are situations when you don’t want to simply delete observations with missing values. Or missing values may have special clinical relevance. In our example, missing lactate values may indicate that the patient have been fully recovered from shock (e.g., lactate is a biomarker of tissue hypoperfusion and hypoxia and clinicians typically do not order lactate for patients who are hemodynamically stable). Therefore, NAs in lactate indicates stable patients. Because NAs contain important information, it is wise to add a new category called “NA” to replace the missing values. You can try na.tree.replace() in the tree package for this purpose (2). However, this function is limited by the fact that it is only applicable for discrete variables with missing value. I create a new data frame for the illustration.

The data frame data.discrete contains three variables and the last two discrete variables have NAs. The error message returned by na.fail() indicates NAs in the data frame data.discrete. Then, a new data frame called newdata.discrete is created by using na.tree.replace(). The na.fail() function returns the new data frame without error. As you can see missing values are replaced by string value “NA”. The limitation of na.tree.replace() is that it stops if any continuous variable contains NAs.

Note that an error message is returned because the data contains continuous variable with NAs. The problem can be solved by using na.gam.replace() function in the gam package. Note that gam package requires simultaneous installation of foreach package (3).

Note that the data frame newdata pass the na.fail() test and returns the new dataset. For variable sex the missing value is replaced by string value “NA”, and the missing numeric value of lactate is replaced by the mean of the available lactate values.

Summary

The article reviews some basic skills in dealing with missing values in R. missing values in R cannot be compared by using logical operators and thus specific function is.na() is fundamental in judging whether an object contains missing value or not. There are varieties of tips to do with missing values in data frame. Some useful codes are to locate and count missing values by using lapply() function. Lapply() is very useful in that it applies functions across all variables while avoiding the time-consuming loop functions. Regression model performs casewise deletion of missing values by default. However, the performance of na.omit() and na.exclude() with respect to residuals and fitted values of NAs. There are also some advanced packages for handling missing values. These include na.tree.replace() in tree package and na.gam.replace() in gam package.

Acknowledgements

None.

Footnote

Conflicts of Interest: The author has no conflicts of interest to declare.

References

1. Montez-Rath ME, Winkelmayer WC, Desai M. Addressing missing data in clinical studies of kidney diseases. Clin J Am Soc Nephrol 2014;9:1328-35. [PubMed]
2. Ripley B. Tree: classification and regression trees (2007). (R package version 1.0-26).
3. Hastie T. Gam: generalized additive models (2011). (R package version 1.06.2).