Understanding Confusion Matrices (with Visuals!)

In the last video, we explored how logistic regression can be used for binary classification. In this video, we'll take a quick look at how to evaluate a model's performance on a testing data set using something called a confusion matrix. So, in our binary classification case, we have a 2x2 confusion matrix. On the left side, we have the actual true values and on the top, we have the predicted values from our model. If you look at the first cell, it represents true positives. Meaning the actual class was positive and the model also predicted it as positive. The next cell in the first row corresponds to false negatives. Here the actual class was positive but the model incorrectly predicted it as negative. Moving to the first cell of the second row, we have false positives. These are cases where the actual class was negative but the model predicted them as positive. Finally, the last cell represents true negatives. Meaning both the actual and predicted classes were negatives. Now, based on these four values in the confusion matrix, we can calculate several important evaluation matrix. The first one is accuracy, which measures how often the model is correct overall. The formula is simple. Take the sum of all the correctly predicted samples that is true positives plus true negatives and divide it by the total number of samples. Next we have precision. Precision tells us out of all the samples predicted as positive, how many were actually positive? In other words, it measures the model's exactness when it predicts a positive class. Then we have recall also known as sensitivity. Recall measures out of all the actual positive samples, how many the model correctly identified as positive. We also have specificity which measures the proportion of actual negatives that were correctly identified as negative. This is sometime called the true negative rate. Finally, there is the F1 score which is the harmonic mean of precision and recall. It gives us a single metric that balances both precision and recall. Especially useful when dealing with imbalanced data sets. Now imagine we have a model that predicts whether someone has cancer or not. Let's say we have the following confusion matrix as an example. I encourage you to pause the video here and try calculating the different metrics yourself. Once you are done, resume the video. You should get these answers. Now the confusion matrix is not limited to binary classification models. We can easily generalize it to multiclass problems. In this case, multiple actual categories listed on the left side and multiple predicted categories along the top. The diagonal elements represent the instances that were correctly classified by the model. For example, if you look at the first diagonal cell, it means that the actual class was one and model also predicted it as one. We can also calculate evaluation matrix in a similar way. For instance, to compute accuracy, we sum all the diagonal elements, the correctly predicted samples and divide by total number of samples. To calculate precision for each class, we first focus on one class at a time and treat it as the positive class. Precision tells us out of all the samples the model predicted as this class, how many were actually correct. Next, we have recall which also focuses on one class at a time. Recall measures out of all the actual samples belonging to this class, how many the model correctly identified. Finally, we calculate the F1 score for each class. The F1 score is just like before, the harmonic mean of precision and recall, giving us a single number that balances both. Now, when you are training models in practice, you don't need to manually calculate these metrics using the formulas as discussed. There are many libraries like scikitlearn in Python that provide built-in function to compute them easily. But truly understanding how these metrics work under the hood is crucial. It helps you interpret your model's result correctly and make better decisions about improving or comparing models.