[FIXED] What is sigma clipping? How do you know when to apply it?

Issue

I'm reading a book on Data Science for Python and the author applies 'sigma-clipping operation' to remove outliers due to typos. However the process isn't explained at all.

What is sigma clipping? Is it only applicable for certain data (eg. in the book it's used towards birth rates in US)?

As per the text:

quartiles = np.percentile(births['births'], [25, 50, 75]) #so we find the 25th, 50th, and 75th percentiles
mu = quartiles[1] #we set mu = 50th percentile
sig = 0.74 * (quartiles[2] - quartiles[0]) #???

This final line is a robust estimate of the sample mean, where the 0.74 comes 
from the interquartile range of a Gaussian distribution.

Why 0.74? Is there a proof for this?

Solution

Suppose you have a set of data. Compute its median m and its standard deviation sigma. Keep only the data that falls in the range (m-a*sigma,m+a*sigma) for some value of a, and discard everything else. This is one iteration of sigma clipping. Continue to iterate a predetermined number of times, and/or stop when the relative reduction in the value of sigma is small.

Sigma clipping is geared toward removing outliers, to allow for a more robust (i.e. resistant to outliers) estimation of, say, the mean of the distribution. So it's applicable to data where you expect to find outliers.

As for the 0.74, it comes from the interquartile range of the Gaussian distribution, as per the text.

Answered By - Ken Wei

Answer Checked By - David Marino (PHPFixing Volunteer)