Performance measures for classifiers:F1-Score
In continuation of my previous posts on various Performance measures for classifiers, here, I’ve explained the concept of single score measure namely; ‘F - score’.
In my previous posts, I had discussed four fundamental numbers, namely, true positive, true negative, false positive and false negative and eight basic ratios, namely, sensitivity(or recall or true positive rate) & specificity (or true negative rate), false positive rate (or type-I error) & false negative rates (or type-II error), positive predicted value (or precision) & negative predicted value, and false discovery rate (or q-value) & false omission rate.
I had also discussed accuracy paradox, the relationship between various basic ratios and their trade-off to evaluate the performance of a classifier with examples.
I’ll be using the same confusion matrix for reference.
Precision & Recall: First let’s briefly revisit the understanding of ‘Precision (PPV) & Recall (sensitivity)’.
[You may refer https://learnerworld.tumblr.com/post/153292870245/enjoystatisticswithmeppvnpv & https://learnerworld.tumblr.com/post/152722455780/enjoystatisticswithmesensitivityspecificity for detailed understanding of these ratios]
Precision can be interpreted as ‘proportion of positive identifications was actually correct’.
Precision = TP/((TP+FP) )
If FP = 0, then Precision = 1
Recall can be interpreted as ‘proportion of actual positives was identified correctly’.
Recall = TP/((TP+FN) )
If FN = 0, then Recall = 1
Trade-Off: For evaluating the model performance, we must observe both precision and recall. It’s quite easy to understand that there is a trade-off between the two.
If we try to maximize recall (i.e., reducing false negatives); the classifiers boundary will minimize precision (i.e., increasing false positives) and vice-versa.
Therefore, we need a measure that relies on both precision and recall. One such measure is ‘F - Score’.
F1-Score: This is a weighted average of precision and recall. F-measure is calculated as a harmonic mean of precision and recall. [Harmonic mean is used in place of arithmetic mean as the arithmetic mean is more sensitive to outliers*.
F1-Score = (2*Precision*Recall)/((Precision +Recall) )
The F-Measure will always be nearer to the smaller value of Precision or Recall. For problems where both precision and recall are important, one can select a model which maximizes this F1 - score . For other problems, a trade-off is needed, and a decision has to be made whether to maximize precision or recall.
F1 score is a special case of the general Fβ measure (for non-negative real values of β):
Fβ = ((1+ β^2 )*Precision*Recall)/((β^2*Precision +Recall) )
Two other commonly used F measures are the F2 measure and the F0.5 measure.
As the F-measures do not take the true negatives into account, and that measures such as the Matthews correlation coefficient, Informedness or Cohen's kappa may be preferable to assess the performance of a binary classifier.
References:
*https://www.quora.com/When-is-it-most-appropriate-to-take-the-arithmetic-mean-vs-geometric-mean-vs-harmonic-mean
Sasaki, Y. (2007). "The truth of the F-measure" (PDF). Van Rijsbergen, C. J. (1979). Information Retrieval (2nd ed.). Butterworth-Heinemann. Powers, David M W (2011). "Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation" (PDF). Journal of Machine Learning Technologies. 2 (1): 37–63. Beitzel., Steven M. (2006). On Understanding and Classifying Web Queries (Ph.D. thesis). IIT. CiteSeerX 10.1.1.127.634. X. Li; Y.-Y. Wang; A. Acero (July 2008). Learning query intent from regularized click graphs (PDF). Proceedings of the 31st SIGIR Conference. See, e.g., the evaluation of the [1]. Hand, David. "A note on using the F-measure for evaluating record linkage algorithms - Dimensions". app.dimensions.ai. Retrieved 2018-12-08.