How to pick a machine learning model 4: Splitting the data

Key Takeaways

most often we are not handed separate training and testing datasets but rather one big grab bag of data when this is the case we are responsible for splitting it ourselves when we do this we have to keep in mind that there are two different broad categories of generalization interpolation and extrapolation depending on which we're interested in we will divide the data differently an example of interpolation is if we want to estimate the value of annual temperatures that we're missing from the middle of our data set in this case we know several values both before and after an example of extrapolation would be if we wanted to make estimates outside the range of our original data either for dates that come before or dates that come after another type of extrapolation would be applying the pattern to a different town nearby in both cases we would try to infer something about the world that extended beyond the reach of what we had measured if we wanted to divide our data set to test for interpolation performance it would be straightforward we could randomly sort every year's data into one of two bins testing or training this is what we did in part one however if we wanted to divide our data set to test for an extrapolation that would require a little more subtlety if we were interested in making predictions about the future for instance we would have to make sure to test the model on data from the future that it had never seen and never been trained on otherwise it would have an unfair advantage knowing what the temperature will be in two years helps to make a better prediction about what it will be next year knowing future year's temperatures would tip the model off to any upcoming trends or changes in temperature pattern most importantly this is an advantage that the model will not have when we put it into practice making predictions - honestly split the data into training and testing sets for extrapolation we would have to divide it by date training on all the data that came before and testing on all the data that came after that date this would give us a more realistic assessment more generally when dividing data into training and testing sets we want the data in the testing sets to be independent of all the data in the training set otherwise it's not a true test of the models ability to generalize there are subtle ways that data can be dependent on each other ways that knowing some bits can give you an unfair advantage when trying to predict other bits determining what constitutes independence often requires some domain knowledge for example if we wanted to test the generalizability of our model we could test it on the temperature data from another town but we would have to be careful our models ability to predict the pattern in annual temperatures for a town that was only one kilometer away wouldn't be sufficient that data would not be independent one kilometer is so close that the two towns would share not only underlying weather patterns but also the hyper local weather quirks using measurement from one to predict the other would not test the models generalizability however if we tested the model on temperatures from a town that was a hundred kilometers away that might be far enough away to be considered independent and would provide a better assessment of the models generalizability the ultimate measure of whether a train test split is appropriate is how the model will be used in practice is the training testing split representative of the conditions the model will experience when it's implemented if so you're good to go if not you're testing error might be artificially low giving you a false sense of security about how well your model is performing in high consequence applications being blind to your models weaknesses can lead to some very uncomfortable situations taking a close look at how you split your data into training and testing sets can save you this pain so now that we've covered the preparatory steps join me for part 5 where we talk about how to choose model candidates and how to handle hypothesis driven and Theory driven modeling