Cross Validation Explained!

Key Takeaways

hello everyone and welcome to another episode of Code Emporium where we're going to talk about validation techniques and their importance so in typical regression what we want to do is when we have a cloud of data points we want to draw a curve through them it kind of looks like this now mathematically this would involve creating a linear regression perhaps with parameters Theta 1 and Theta 0 as the slope and The Intercept of the line and we would evaluate this or train this model using the mean squared error and the mean squared error is the difference between the actual value and the predicted value which is going to be essentially just the equation of this line now why you mean squared error is actually so good is because it can be decomposed into a bias term as well as a variance term now I've explained a lot about bias and variance in another video but essentially good machine learning models have both a low bias for the estimate of thetas and also a low variance for the estimations of the thetas and because mean squared error is proportional to both of these a lower mean squared error would involve creating a better model mean squared error is used during training as well as evaluation time of your model now with the with model evaluation why are validation techniques themselves required well the goal of validation and in general to evaluate your model you want to find the true model performance on unseen data and in the most Ideal World we could basically just keep getting data from The Real World and and just validate it probably with like mean squared error However unfortunately we in the real world don't really have access to just unlimited amounts of data and instead we are working with a data set or just a fixed amount of data that we use for both training and evaluating our models and so validation techniques become important working with this limited block of data now one type of validation technique is the hold out validation technique where here's our block of data which we split into a train set and a test set the train set is used to train the model and the test set is used to evaluate it and in this case we compute the holdout validation by taking the mean squared error of all of these test examples one of the fallacies of this method that you might hear is that it doesn't work that great if your data set is very small this is true for some part because this instance of the data set is actually just one instance from the broader real world data this could be for example just randomly a bad sampler or randomly a really good sample that might not be representative of how data is in the real world and what we want is the true model performance of unseen data so hold out validation may not be up to the par in that case however holdout validation is pretty reasonable with lots of data available and this is because of the law of large numbers this is a statistical principle that essentially states that as this test set goes on increasing the average or rather the mean squared error of this sample is going to be representative of the true unseen mean squared error of the population and based on our true goal that's exactly what we want and so hold out validation can actually work pretty well when you have a lot of data now another technique that's typically used in validation is cross validation where we essentially just break down our data into equal splits and we use one split for testing data whereas the remaining splits we use for training data and when we compute the mean squared error for our test data we can store that in a variable error one then we repeat this process with a different chunk where we train on the remaining chunks test on this chunk and get an error too and we repeat this process depending on the number of chunks that we decide in this case it's five so we have five errors and in the end the cross validation error is just going to be the mean of all of these errors now a neat thing that this provides over the holdout validation set is that in the case that the data set is small we now can use every single example for both training and evaluation however remember in the holdout validation set we have only some amount of data that's for training but this remaining data which we could have used for training is now used for evaluation and this training data when it's very small could have been very crucial to the model if we had more of it and so cross-validation tends to be a very preferred technique with lower limited amounts of data aside from this cross-validation also takes an average so if there are some like crazy error examples that you get here just off the fly all of those errors when you average them up will eventually come down to their truer value and so they're not affected by these outliers as much because of an average and so it's for this reason that cross-validation actually becomes very useful and versatile now when you want to use holdout versus cross validation is not a very specific rule there are General guidelines though like for example based on what we've discussed is just the amount of data where if you have a less amount of data you can probably use cross-validation however with more amounts of data the holdout group and cross-validation both perform pretty well but there are also other considerations to think about for example time dependence this means that the data that you've collected is actually determined at very specific points of time and if you have a time dependence in your data you can't really use cross-validation in the form that we discussed because of the concerns of data leakage to illustrate data leakage let's look at an example so let's say that this data block is arranged in ascending order of time so this is the earliest data this is the latest data in this particular iteration of cross-validation where we are testing on every single example in this set each of these examples are later than any of the samples that were used to train the data so this has no data leakage because this is exactly how we would use the model in production however in this case we are now testing on data where the training set could have included data from the future this is not representative of how we would actually use this model in the real world we can't use training data that we haven't seen yet and this is a good example of data leakage and it's because of this that you might see the error become low during evaluation but in the real world the model will actually not perform that well and will actually see a higher error and so with the time dependence cross validation in its current form is not usable that said there are also some time series Renditions of cross-validation that you may be able to use but for this case I've now just indicated them with hold out to use holdout validation now if there is no time dependence however you can use cross-validation and if you have lots of data you can use either cross validation or hold out as a recommendation like I mentioned before these are just recommendations but there are not strict hard and fast rules now another question that you might be thinking is how many partitions do we really need to use in cross-validation in fact we can just have two partitions one that's used for training and one that's used for evaluation or we can split this entire data set into partitions such that every sample is its own partition and this is the idea behind a specific case of cross-validation called leave one out cross validation now in order to determine the number of partitions you might want to consider how well does this actually perform and also what is the amount of compute that is required and one of the best ways to get an idea of performance is through simulation let's say that we simulated some data and we have now 40 data points for which we want to perform some form of cross validation now with 40 data points we can have the number number of partitions go as high as 40. this y-axis here is going to be the performance in terms of the complement of the mean squared error from this graph we kind of see that as long as we set the number of folds to be like around 10 or more the performance of our model actually doesn't change with even increase in the number of folds in Cross validation we also see something even similar with a larger number of data points with like 200. in this case we can now increase the number of folds all the way up to 200 but there isn't really still any kind of performance difference even if we had a very low number of partitions so here the performance itself is dependent on the amount of data because for smaller amounts of data you might have to increase the number of partitions but beyond a certain amount it really doesn't affect performance itself now for computation if you increase the number of partitions chances are it's going to take a lot more time to train but whether this is actually a huge concern depends on the kind of model you're using in the amount of data that you have because if the model itself only takes like maybe 20 to 30 Seconds To Train it's okay to increase the number of partitions without being too worried about computation power and resources and so how many partitions and cross validation really becomes less of an important question as you have more and more data and that's all I have for today I hope this gives you an intuition of what cross-validation is and how to use it very practically as well as its advantages and disadvantages of different types of techniques if you do like this video please do give it a like comment down below if there's anything that you agreed or disagreed with or if there's anything that you just learned today I'm trying to grow a community here and I want it to be about learning thank you all so much for watching and for your support and until next time