Exponentially Weighted Averages (C2W2L03)

Key Takeaways

I want to show you a few optimization algorithms that are faster than gradient descent in order to understand those algorithms you need to be able to use something called exponentially weighted averages also called exponentially weighted moving averages in statistics let's first talk about that and it will use this to build out two most sophisticated optimization algorithms so even though I now live in the United States I was born in London so for this example I got the daily temperature from London from last year so on January 1st temperature was 40 degrees Fahrenheit and I know most the world uses a Celsius system but I guess Olivia United States which uses Fahrenheit so that's four degrees Celsius and on January second it was 90 degrees Celsius and so on and then about halfway through the year a year has 365 days so this would be sometimes day number 118 would be sometime in late May I guess 160 degrees Fahrenheit which is 15 degrees Celsius and so on so start to get near warmer to one summer and it was colder in January so you plot the data you end up with this with day one being sometime in January that you know being the beginning of summer and that's the end of the year kind of late December so this would be January of January first as you middle the Year approaching summer and this would be the data from the end of the year so this need to look a little bit noisy and if you want to compute the trends kind of the local average or a moving average of the temperature here's what you can do let's initialize V zero equals zero and then on every day we're going to average it with a weight of 0.9 times whatever is the previous value plus 0.1 times the that day's temperature so theta one here would be the temperature from this first day and on a second day we're again going to take a weighted average so 0.9 times of the previous value plus 0.1 times today's temperature so on v2 plus 0.1 times theta 3 and so on and the more general formula is V on a given day is 0.9 times V from the previous day plus 0.1 times the temperature of that day so if you compute this and plot it in red this is what you get you get a moving average or what's called an exponentially weighted average of the daily temperature so let's look at the equation we had from the previous slide it was VT equals previously we had 0.9 do not turn that into parameter beta beta times BT minus 1 plus and appreci with 0.1 then turn that into 1 minus beta times theta T so previously we had beta equals 0.9 it turns out that for reasons we'll go into later when you compute this you can think of VT as approximately averaging over something like 1 over 1 minus beta days temperature so for example when beta equals 0.9 you could think of this as averaging over the last 10 days temperature and that was the red line now let's try something else let's say it beta to be very close to 1 this is 0.98 then if you look at 1 over 1 minus 0.98 this is equal to 50 so this is zeal think of this is averaging over roughly the last 50 days temperature and if you plot that you get this green line so notice a couple things with this very high value of beta the plot you get is much smoother because you're now averaging over more days of temperature so the curve is just you know less Wiggly is now smoother but on the flip side the curve is now shifted further to the right because you're now averaging over a much larger window of temperature and by averaging over a larger window this formula is exponentially weighted average formula it adapts more slowly when the temperature changes right so there's just a little more latency and the reason for that is when beta 0 0.98 then it's giving a lot of weight to the previous value and a much smaller weight just point 0 to 2 whatever you're seeing right now so when the temperature changes or in term she goes up or down this exponentially weighted average just a decimal slowly when beta is so large now let's try another value that you set data to another extreme let's say it is 0.5 then this by the formula we have on the right this is something like averaging over just two days temperature and you plot that to get this yellow line and by averaging only over two days temperature you have a muchness as if you're averaging over much shorter window so you're much more noisy much more susceptible to outliers but this adapts much more quickly to the temperature changes so this formula is how you implement an exponentially weighted average again is called an exponentially weighted moving average industry 2/6 literature and I'm going to call it exponentially weighted average for short and by varying this parameter or later we'll see essentially hyper parameter of your learning algorithm you can get slightly different effects and it will usually be some value in between their workspace that gives you the red curve which you're maybe looks like a better average in temperature than either the green or the yellow curve you now know the basics of how to compute exponentially weighted averages in the next video let's gain a bit more intuition about what is doing