Syllabus →

Endsem pyqs →

Personal Notes →

Data Mining: Outlier Detection

In data mining, an outlier is an observation that deviates so significantly from other observations as to arouse suspicion that it was generated by a different mechanism. Detecting outliers is critical in many fields like fraud detection, network security, and fault detection.

Challenges of Outlier Detection

1. Definition of Normality: Establishing a clear definition of what constitutes normal behavior or data is challenging, as it varies from dataset to dataset.
2. High Dimensional Spaces: Outlier detection becomes increasingly difficult as the dimensionality of the data increases. This is due to the "curse of dimensionality" where the data becomes sparse, and traditional distance measures become less meaningful.
3. Type I and II Errors: Distinguishing between true outliers and noise is difficult and can result in false positives (Type I errors) or false negatives (Type II errors), where outliers are missed.
4. Scalability: With large datasets, efficiently finding outliers becomes computationally expensive. Scalable algorithms are necessary to handle big data scenarios.