Performance measures for classifiers _ Accuracy & Mis-classification Error
Binary classification
Binary classification is the task of classifying the elements of a given set into two groups on the basis of a classification rule. Different techniques can be used for binary classification such as logistic regression, naïve Bayes, support vector machine, neural network, decision tress, random forest etc. binary classification is also known as binomial classification or two-class classification or 0/1 classification
Performance measures for binary classification models
There are many model evaluation matrices to measure performance of a binary classifier or to compare different models. The very first measure is ‘accuracy’ of the classifier, which measures percentage of correctly classified cases. However, it is not the only metric to evaluate the model. Even sometimes, it is not a reliable measure of model performance because of ‘accuracy paradox’.
There are many other evaluation metrics and plots, which are useful (some of these work only for binary classifiers). We should not consider any of these in an isolated way as there is not a best way to evaluate any system, but different metrics give us different (and valuable) insights about the performance of a classifier. We will discuss different measures and their applicability.
To calculate performance measures, firstly we need to understand the confusion matrix (2 x 2 matrix for binary classification).
The confusion matrix:
A confusion matrix (also known as error matrix) is a table of dimension n x n associated with a classifier shows the predicted and actual classification, where n is the number of different classes. [Confusion matrix (Kohavi and Provost, 1998)]
For binary classifier, we’ve presented confusion matrix as below, where out of total (A+B) observations, number of observed positive classes are A and rest are negative classes.
Table_1
In the above matrix olive colored cells shows the number of classes which are correctly classified, while red colored cells presents incorrect classifications. Let’s understand some terminology of this matrix:
TP (True Positive): As the name implied, these are the cases, where we predicted positive and they do have positive outcome.
TP (True Negative): Similarly, these are the cases, where we predicted negative and they do have negative outcome.
FP (False Positive): These are the cases, where we predicted positive but they do have negative outcome. So, it’s a part of error.
FN (False Negative): Similarly, these are the cases, where we wrongly predicted negative but they do have positive outcome.
Prevalence:
The ratio of positive outcomes observed in the data is known as prevalence.
Hence, for truly balanced data, prevalence is 0.5 and for highly unbalanced data it is far from 0.5 [close to 0 or 1 according to class lebels].
Accuracy & Error:
The fraction of correctly classified cases is known as accuracy of the classifier.
For the above matrix, out of total (A+B) cases, number of correct classifications is (TP+TN).
Similarly, we can define mis-classification error as number of mis-classified (incorrectly classified) cases divided by total number of cases.
Accuracy Paradox:
Intuitively, for any model, higher accuracy (& lower error) is desirable. But there are cases where the model with lower accuracy has greater predictive power. It might seem counter intuitive but for unbalanced data [very less (20 % or less) cases in one class and rest in other class, for example, fraud detection, medical diagnosis, spam filters, titanic survival prediction], accuracy can be misleading and we need other performance measures.
Especially in case of predicting rare events, a model can make prediction for all cases to be in the majority class and achieve high classification accuracy; the problem is that this model is not useful in the problem domain. For example, in fraud detection, if there are 8% cases of fraudulent transactions (& 92 % genuine transactions) in the training data and our classification model predict all transactions to be genuine (majority class), model accuracy becomes 92 % (very high) but this is not a realistic model as it fails to detect any case of fraudulent transactions. Here, accuracy is not a reliable measure of model performance.
This is called ‘accuracy paradox’.
For other evaluation matrices, wait for my next post.
Happy Learning !