8.2 Intuition behind bias and variance (L08: Model Evaluation Part 1)

Key Takeaways

yeah so in this video we are now talking about the bias variance decomposition at least giving a brief introduction to the bias and variance decomposition so the last video ended with a question what does it mean if we say the model has high bias and high variance and if you read other research articles or blog posts or textbooks you will find that very often people say a model has high bias or a model has high variance so we are currently now in the process of dissecting it what it means yeah and we will do that by looking at the so-called bias variance decomposition and there's also the so-called bias variance trade-off and then later in a later video we will see how that relates to overfitting and under-fitting which are a little bit more intuitive to talk about yeah so what is the bias variance decomposition so it's a decomposition of the loss into two terms into the bias and variance actually there's a third term the noise term but we will not look at the noise term explicitly in this lecture for simplicity so here we are going to focus on decomposing a loss for example the squared error or the zero one loss into a bias in the variance term and that helps us understanding machine learning algorithms but also then how it relates to under-fitting and overfitting of a model and in the context of last lecture where we talked about ensembl methods it can also help us to better understand why they perform better than single models so for example bagging reduces the variance compared to a single model and boosting may decrease the bias in compared to a single model so both backing and boosting kind of manipulate how bias and variants behave compared to the single decision tree models used in the ensemble so when people say in practice a model has high bias this usually refers to the bias of the loss and when people say a model has high variance that usually refers to the variance of the loss so what are bias and variance again so this is the figure we already looked at when we talked about the ensemble methods briefly so here this is just summarizing the different scenarios where we can have a low variance and a low bias this means that we usually make a very accurate prediction so there's a very small error term so the outputs are not very scattered and they're also very much on target where the center here is ideal it's like the bull's eye when you're playing darts you can have a low bias oh so when you average it you are still correct but then you can have a high variance where all the estimates are very scattered and yeah a very bad case if you if you have a high bias and a high variance so not only are your estimates very scattered but also they are very far off from the center and then the fourth possibility is having a high bias and a low variance it's also not very ideal because the estimates will be very consistent but they will also be far off from the correct value so let me now to illustrate this redraw some of the figures i showed you in the ensembl method lecture so assume there is some true underlying data generating function f of x and this is uh think of it as a natural phenomenon this is just something that generates our data that's some true phenomenon or function but in practice um yeah there's usually some noise also so uh in blue you can think of the dots as the real data points that can be observed let's say that can be measured by some device so if the target the true underlying function is let's say the temperature at some point then the blue points would be like the actual measurements that we can make with thermometers for example so there's usually some noise term the noise in the measurement and in practice usually we also only have a subset of the population so we don't have all the data points the whole population we only have subsets so we have training sets that are usually of course smaller than the population so here on the right hand side i have three possible training sets so train set one train set two and train set three and they're just random samples from the population here the blue points so i just randomly sampled from them so we have these three possible training sets and you can see they are relatively different from each other even though they come from the same population or the data generating function so here i plotted linear regression models that will fit to either training set 1 training set 2 and training set 3. so the blue line is a linear regression model fit on training set one the orange line is a linear regression model to fit to training set two and the green line is a linear regression model fitted to training set three so this would be a case where we have a high bias model and the reason is that we are for all these three models we are very far off from the true target function the data generating function so if we would make a predictions with these linear regression models the predictions would be obviously very bad if based on the x value our task is to predict these y values on the y axis the true values here so given these models that would be a high bias scenario however the bias and some points is actually very low so or zero and this is where the model does make a correct prediction so let me use maybe the color right here so there are some points where the model would actually make correct predictions right so if we give it or every of these models if we give them x value of 5 it would actually make the correct prediction here so this one would actually be a good prediction so here the bias is zero but everywhere else i mean most of the data points in the whole range here is very far off from the true function so this would be a high bias model and also in some regions here there would be a high variance because as the training set varies we also see the models might be very different so let's maybe take a closer look at the high variance case so here i fit an unpruned decision tree so this has a high variance and that is because if we have different data sets it would significantly affect the model so um having different training sets even though they are from the same population if it causes the model to be very different then we would have a high variance so the predictions would be very scattered depending on how the training set looks like even though all the training sets are from the same population maybe also to just jump back by one slide so this would be a region where the models or the model predictions are very robust to the different training sets so here the predictions are very consistent so this would be a low variance case because the model predictions are very consistent here it would be more scattered if we have a more complex model a model that is more complex than a linear regression model here an unpruned decision tree for regression then this would fit the data very very closely right and if the training set just changes a little bit the model would look very different so here i have three unpruned decision trees and you can see all the three models are very different so this would be a high variance case so unpruned decision trees have high variance whereas linear regression models usually have high bias on non-linear data sets so what happens now if we take the average so if we take the model average that would be an ensemble method where we just average the model predictions so if we take the average here on the left hand side we can see actually that the average is a little bit closer to the true function than each of the individual models it's a bit hard to see on the left-hand side i mean it's a very very small data set and we only have three models where we are averaging so on the right hand side i'm overlaying the average with the other models and you can see roughly that the red line here the average of the three models is actually closer to the true function than any of the other models so we are here reducing the variance by averaging over these high variance models which is kind of like what begging does right yeah so that was um the intuition behind bias and variance now we are going to talk about some of the terminology that we will use when we are going to define bias and variance more formally and then also do the bias and variance decomposition so um we will use the term point estimator or theta hat which basically is a function that is used to find an approximate value of a population parameter using the data using samples or training examples so this is some true parameter that we want to estimate and this is our basically our approximation of that so then we can define the bias as follows so the bias is then here the expectation of theta hat minus the true parameter so this is on the expectation you can also think of it as averaging so if we are averaging over all the estimators the point estimators and subtract them from the true perimeter that is our bias term so yeah that was the definition of the bias term now we can also express the variance term as follows so here we have the point estimator squared which gets rid of the negative values and then the expectation of that minus the expectation of the point estimator same term is here but also note squared and then subtracting these from each other i find it personally easier to understand the variance if we write it as follows so we can rewrite it as follows where we have the expectation of the point estimator minus the point estimator let's say for for one model um so if you take a look at this there's a square term to get rid of the negative values so this can be done many times so for example for all training data sets so imagine you have changing training data sets and then if you would take the expectation of that that would be our variance term so in other words this term it kind of um yeah gives us how how far the average point estimator is away from a particular point estimator so it's basically giving us the idea of how spread these um point estimators are or here in a more visual way so i was drawing this little person here shooting um arrows using a little bow here and the variance would be basically how scattered these arrows are when they land so how far they are away from each other the higher the variance the more further scattered these are and let's say here's here's our target that we want to hit and the distance from the target this is our bias right so the target you can think of it as my theta here and the distance here is between the average so if we take a look at the average here minus the distance to the target that would be our our bias here and for the variance you can think of each of these lines as a theta hat and then we are subtracting it from the average so we are checking how far if um this is our average so how far they are scattered and then the average of how far each um point estimator is away from the average of the point estimators it's this term so if we take a look at the average of the scatter that is then our variance term in practice there's also some noise but for simplicity in this lecture we are ignoring the noise so the noise would be about the target so the target could be also a little bit um uncertain so we could have some noise so that is something we will ignore though in this following slides to keep things more simple so here i'm also recapping this so the bias is the difference between the average estimator from different training samples so training sample here means training set so if we have different training sets like i was showing you earlier the bias is the difference between the prediction for a certain data point on that data set averaged over multiple training sets and then minus the true value that we want to predict so if we think back of oops so if we think back of the case where we had this true function here and let's say we want to predict um this y value belonging to a particular x value here then um depending on what training set we use we may be um off by certain points if we have the green data set and we fit a model to that the prediction might be um let's say here maybe for a different data set if we put fit the model prediction may be here and for another data set it might be here so if we then consider the average of these so the average of these three points might be the center here how far is it away from the true value which is down here so that would be the bias term so the expectation is over the training sets and the variance provides an estimate of how much the estimates vary as we vary the training data so for example by resampling and again this is um when we have our case here we want to make a prediction we want to make the y prediction for a given x and the variance tells us if we have three model predictions that are like this it tells us how spread out they are from each other okay so we are now doing the bias variance decomposition of the squared error loss ignoring the noise term so for simplicity we will ignore the lowest term and just focus on bias and variance and that is what we are going to do in the next video for the squared error then we will take a look at how that relates to overfitting and under fitting and lastly we will then also do the bias variance acquisition of the zero wonders