3. Univariate Outlier Detection

Two things were emphasized in Chapters 1 and 2. First, outliers are an important practical complication in data analysis. Second, they come in several different varieties. This chapter is devoted to the problem of detecting one of these outlier classes—univariate outliers—for two main reasons. First, univariate outliers are the easiest to detect in a dataset even though, as the examples presented here illustrate, their detection is a nontrivial task. An important advantage of considering the univariate case first is that the results lay the foundation for consideration of the more complicated outlier detection problems discussed in Chapter 4. Second, the GSA framework converts the search for various types of data anomalies into a univariate outlier detection problem, a point discussed further in Chapter 6. Many different univariate outlier detection procedures have been proposed, but the main three considered here are of the following general form. First, a reference value x0 and a measure of variation ζ, are computed from the data sequence {xk}. Then, a threshold parameter t is chosen. Finally, every data value in the sequence {xk} is tested to determine whether it is an outlier according to the rule | xk − x0 |>tζ ⇒ xk is an outlier. 3.1 To convert this rule into a practical outlier detection procedure, it is necessary to answer the following three questions, and the main purpose of this chapter is to provide some practical guidance in answering them: 1. How do we define the nominal data reference value x0? 2. How do we define the scale of natural variation ζ? 3. How do we choose the threshold parameter t? Intuitively, (3.1) says that if observation xk lies too far from the reference value x0, it is declared an outlier. The variation measure ζ, characterizes the “natural spread” of the data values, giving us a useful basis for calibrating distances from x0, and the threshold parameter t determines the aggressiveness of the detection procedures. In particular, setting t = 0 declares any value different from x0 an outlier, no matter how small the difference between xk and x0, and taking t large enough usually guarantees that no outliers will ever be detected.