Outlier

Definition

An outlier is an observation far from the rest of the data. It may be an error, a rare but genuine event, or a sign that the data come from two kinds of things; deciding which requires looking at it, not just deleting it.

In plain terms

A price of 1,400 among prices near 14 is probably a typing slip (14.00 became 1400). A single huge sale may be real and the most important point in the dataset. Automatic rules, such as flagging points beyond 1.5 times the interquartile range from the quartiles, or with a large Z-score, only nominate candidates.

Good practice: investigate, record the decision and the reason, keep the original value, and report results with and without the point when it matters. Prefer summaries that are not thrown off by extremes, such as the median.

Example

Eight prices: 12, 15, 14, 13, 16, 15, 14 and 1400.

1400 lies far outside the fences and is flagged. A mean of 187 describes none of the items; the median of 14.5 describes all but one. A quick look at the source would show whether 1400 is a slip for 14.00.

Common misunderstandings

• “Delete outliers automatically.” Some are the most informative points; others are errors to fix at the source, not to hide.
• “An outlier is always an error.” Heavy-tailed quantities such as sales produce genuine extremes regularly.
• “The mean is robust enough.” One extreme value can move it arbitrarily far.