How to decide whether to apply Supervised Learning or Anomaly Detection
Let’s quickly find out when to use Gaussian Distribution and when to use Supervised Learning –
· When there are very small number of outliers, or, positive examples and a large number of negative examples, this makes it a skewed data-set. In such scenario, it’s ideal to use Gaussian Distribution. When we have large number of both positive and negative examples, we can use Supervised Learning.
· When we have many different “types” of anomalies and it’s hard for any algorithm to learn from positive examples about what the future anomalies will look like, it’s better to use Anomaly Detection. Future anomalies may look nothing like any of the anomalous examples seen before. On the contrary, when we have enough positive examples and future positive examples are likely to be similar to the ones in the training set, use Logistic Regression.
· Fraud Detection, Error in manufacturing, Monitoring machine in Datacenter – are examples of Anomaly Detection
· Email Spam, Weather prediction, cancer classification – are examples of supervised learning.
· Remember, if there are too many fraudulent behavior, Fraud detection might as well move towards Supervised Learning.
Let’s say you have identified to apply Anomaly Detection. How do you decide whether to use normal Gaussian Distribution or Multivariate Gaussian Distribution ?
· If you want to manually create features to capture anomalies, use Gaussian distribution. Use Multivariate Gaussian distribution if you want to automatically capture correlation between features.
o For example, for monitoring machines in data center, let’s say you have x1=CPU load and x2=network traffic.
o Come up with extra features, like, x3=CPU Load/Network Traffic=x1/x2 and x4=x1^2/x2
· Gaussian distribution is computationally cheap, whereas Multivariate Gaussian distribution is computationally more expensive, because it has to compute the inverse of a matrix.
· Gaussian distribution works well even if training set size is less than number of features. Whereas, for Multivariate Gaussian distribution number of training set must be approx. 10 times more than the number of features.
· In case of Multivariate Gaussian distribution, make sure that there are no redundant features as well. A matrix becomes non-invertible (or, Singular) if it has redundant features.