Outlier detection

Outlier detection is a fundamental issue in data mining and machine learning. Most methods calculate outlier score for each object and then threshold the scores to detect outliers. Most widely used thresholding techniques are based on statistics like standard deviation around mean, median absolute deviation and interquartile range. Unfortunately, these statistics can be significantly biased because of the presence of outliers when calculating these statistics. This makes their use inaccurate. To overcome this problem, we propose a two-stage thresholding method (2T). Most obvious outliers are first removed by using a more conservative threshold, and the same process is then repeated for the processed scores. Experiments show that this two-stage approach significantly improves the results of all the three existing thresholding techniques.