Evaluation Metrics for Classification Problems

1. Accuracy
   • Definition: The proportion of correct predictions (both true positives and true negatives) out of all predictions made.
   • Formula:Accuracy=True Positives+True NegativesTotal PredictionsAccuracy=Total PredictionsTrue Positives+True Negatives​
   • Use Case: Suitable when the classes are balanced. However, it may be misleading in imbalanced datasets.
2. Precision (Positive Predictive Value)
   • Definition: The proportion of true positive predictions out of all positive predictions made by the model.
   • Formula:Precision=True PositivesTrue Positives+False PositivesPrecision=True Positives+False PositivesTrue Positives​
   • Use Case: Precision is important when the cost of false positives is high (e.g., spam detection).
3. Recall (Sensitivity, True Positive Rate)
   • Definition: The proportion of true positive predictions out of all actual positives in the dataset.
   • Formula:Recall=True PositivesTrue Positives+False NegativesRecall=True Positives+False NegativesTrue Positives​
   • Use Case: Recall is crucial when the cost of false negatives is high (e.g., medical diagnosis).
4. F1-Score
   • Definition: The harmonic mean of precision and recall, providing a balance between the two metrics.
   • Formula:F1-Score=2×Precision×RecallPrecision+RecallF1-Score=2×Precision+RecallPrecision×Recall​
   • Use Case: Useful when you need a balance between precision and recall, especially in cases of class imbalance.
5. Area Under the ROC Curve (AUC-ROC)
   • Definition: AUC measures the ability of the model to distinguish between positive and negative classes, based on the ROC (Receiver Operating Characteristic) curve.
   • Use Case: The higher the AUC, the better the model is at distinguishing between the two classes. Useful for imbalanced datasets.
6. Area Under the Precision-Recall Curve (AUC-PR)
   • Definition: AUC-PR is similar to AUC-ROC but focuses on the precision-recall trade-off. It is especially informative when the classes are imbalanced.
   • Use Case: AUC-PR is preferred when the positive class is rare, as it focuses on the performance on the minority class.
7. Logarithmic Loss (Log Loss)
   • Definition: Log loss measures the uncertainty of the predictions based on the probability output. A lower log loss indicates better performance.
   • Formula:Log Loss=−1N∑i=1N[yi⋅log⁡(pi)+(1−yi)⋅log⁡(1−pi)]Log Loss=−N1​i=1∑N​[yi​⋅log(pi​)+(1−yi​)⋅log(1−pi​)]where yiyi​ is the actual label and pipi​ is the predicted probability.
   • Use Case: Suitable for models predicting probabilities rather than class labels. Common in logistic regression and neural networks.
8. Confusion Matrix
   • Definition: A table that summarizes the performance of a classification algorithm by showing the number of true positives, false positives, true negatives, and false negatives.
   • Use Case: Useful for a deeper understanding of the errors made by the model.
9. Matthews Correlation Coefficient (MCC)
   • Definition: MCC is a measure of the quality of binary classifications, providing a value between -1 (perfect inverse prediction) and +1 (perfect prediction). A value of 0 indicates random guessing.
   • Formula:MCC=TP×TN−FP×FN(TP+FP)(TP+FN)(TN+FP)(TN+FN)MCC=(TP+FP)(TP+FN)(TN+FP)(TN+FN)​TP×TN−FP×FN​
   • Use Case: A good metric for imbalanced datasets, as it takes all four components (TP, TN, FP, FN) into account.
10. Cohen’s Kappa
    • Definition: Cohen’s Kappa is a statistic that measures inter-rater agreement for categorical items, correcting for the agreement that occurs by chance.
    • Formula:κ=Po−Pe1−Peκ=1−Pe​Po​−Pe​​where PoPo​ is the observed agreement and PePe​ is the expected agreement by chance.
    • Use Case: Used to assess the reliability of classifiers, especially when multiple classifiers are involved.