Anomaly Detection

CORE DEFINITION

In data mining, anomaly detection is the identification of items, events, or observations that do not conform to an expected pattern or to other items in a dataset. Typically, anomalous items can translate into problems such as bank fraud, structural defects, medical issues, or text errors. Anomalies are also referred to as outliers, novelties, noise, deviations, and exceptions. Particularly in the context of abuse detection and network intrusion, the interesting objects are often not rare objects, but rather unexpected bursts of activity. This pattern does not follow the usual statistical definition of an anomaly as a rare object, so many anomaly detection methods (especially unsupervised ones) will fail on such data unless appropriate aggregation is performed. Instead, clustering analysis algorithms may be able to detect micro-clusters formed by these patterns. There are three broad categories of anomaly detection methods. Under the assumption that the majority of instances in the dataset are normal, unsupervised anomaly detection methods can detect anomalies in unlabeled test data by finding instances that are most dissimilar to the rest of the data.