Practical Time Series Analysis

Key Takeaways

hi guys my name is Priyanka I'll be the moderator for now so uh Rahul if you want uh you can directly go ahead share your screen and we can start the webinar right away yeah sure I think that would be good yeah okay thank you sucks yeah so hi everybody uh this is Rahul so before I start on uh the time series analysis I'll actually introduce myself so I'm a working professional with seven years of experience um currently working with financials here in Bangalore so I've worked on multiple domains starting from Aviation to e-commerce and now into finance and dealt with a quite problem involving time series analysis so that's all about what has happened in the time series part throughout my career so here today since I've already lost a lot of time here so I'll just directly go to the presentation and the Very idea that I have is I want to give you an intuition of why are we dealing with time series problems separately okay and how can we delete on a real life okay I want to connect those charts okay so I just shared my screen uh let me know if you all can see it uh so the overall agenda of the session will be to start off like why I will be learning time series separately so uh if you know right we we have been dealing with a lot of problem like people who are from the background of ml or data science they know that we have regression problem right classification problem all the supervised unsupervised everything right why are we learning time series separately and why there is so much a buzz around learning so many algorithms and time series separately okay so let's come to that so if you could just in enlarge this so if you could take one example here where you have uh on the left hand side if you could see it I have a predictor here and I'm trying to see what is what is the sales of a sales of an ice cream based on one of the variable as temperature okay so in all these uh Recreation problems that you see we can actually find out the sales based on interpolating the information that we already have it okay let's say at today the temperature was this and I have this point that you see it here that will be my sales right so temperature is something that is going to repeat so if I am plotting something of let's say from um let's say from five degree Celsius to some 40 degree celsius right now any of the temperature that we see is might going to repeat it right so if I want to find out the sales of ice cream at any given point I'll have a lot of points in my nearby right who is going to actually support or Justify my predictions isn't it so we have a lot of points so this kind of problem is interpolation problem why interpolation because we are finding from within the range of values if you see this this is the range of values for temperature okay so I have a lot of points in this now what happens in case of Time series we one of the accesses becomes what we call as a panel data become time right right so now in case of this the time is not going to repeat isn't it because what has actually occurred at any instance p is not going to happen again and if you want to go and predict any other instances that's not going to be very beneficial right because we are not interested in doing something that has happened on the past isn't it so now what happens is as and when we go on predicting the future the unpredictability increases is it because let's say I'm predicting this point here right with some error okay and how am I predicting with whatever I have learned it from the past okay now this is this will have some error isn't it now I'm going to use this RNA and predict the next time instance okay so I'm going to predict the sales at next time instance now what happens is the previous point also had error right and I am using that also to predict the next point so what happens is this errors are carry forwarded okay so this is what going to this is what is going to happen in time series and this is this is what it makes it very difficult and we deal time series separately okay and the more or less all the all the problems that you see in Time series are extrapolation problems now okay why because we are predicting outside the range of the current data that is okay now so we'll talk about two things here one is autocorrelation and second thing is uh partial order correlation okay I'll take an example of sales of ice cream okay so when we are actually thinking of predicting this right as one variable like sales of an ice cream so one idea could be that whatever sales is there today right might be dependent on the sale price which was there the last one and so on maybe two months back isn't it so this is how I can actually uh depict it in a little ground field okay so we have this price at this month okay and price at the last month our price two months back okay now there are type of effect if I could show it here the one is maybe the price that you get today okay is dependent on the price that is from the last month so this effect that you can see and the another one is let me use The annotation here so this effect that you can see another one is that price two months back affects your last month price and that directly affects your current price so in order to get to price at two there are two parts this is one path and this is another part okay now how do we find the direct effect which is there okay in order to find out the directive of what we call as a partial autocorrelation that means and remove this indirect effect which is there okay so finding the indirect effect right which is there it's very easy so what do you do one is that uh just like you have a correlation right in mathematical you can find out the correlation between two Series so I can have two series as X and Y where you have prices for January March and pain right so this you can see you have two months price difference so you have price per Jam your price for March new Feb and April right so you can find the correlation between this the correlation which you find in this is nothing but your peace here which is a direct correlation okay which is very easy you can just find it now difficult part is finding the PAC which is the direct effect okay so how do we find the direct effect so easier that we can find the correlation between the lagged series which is there so this is nothing but your lag seeds so series one and series two which is two months okay now in order to find out the lag series what we'll do is we'll use vertical as a regression okay if you see this panel I have which is the price today is dependent on some coefficient into price what was there last month some coefficient which and price which was there two months back okay and when I'm actually predicting this price there might be some is nothing but your pacf value now you can very well argue in this that why it has that direct impact not the indirect impact that it can have this path right it will not have it because if you see this beta one coefficient right this is already taken care so price uh whatever is there from the last one the impact which is there is taken care by beta 1 okay so beta2 tells me what is the direct impact of the price today with whatever the price was there two months back okay and this is how we are going to calculate partial autocorrelation Factor okay and pacf is very important because this actually tells you removes all the indirect effects that could be there from the N time series back and takes care of direct impact let's say you want to predict the prices of uh sales price of ice creams from five months back so I can actually say what is the impact at T minus 5 so this coefficient is going to give me the pacm value okay and ACF is very easy that we saw we can directly find out the correlation between both of the lab intercities when I say lab series it's just the same series you take a lag okay okay now so why it is called Auto regression model AR model Auto regression is because we are regressing on the same variable okay regressing it with the same value just a language series okay so that is your Air Model that you have now now whatever model that we'll talk about right it assumes that the model that the series that I have is actually stationary okay what do you mean by stationary okay so in us in a for a Time series to be stationary we check for for three factors okay one is the series should have a constant beat okay the standard deviation should be constant Hindu should not be seasonality okay let's see how these things are right let's see it in new diagrams okay so if you see this diagram here right this is a Google stocks which is there if you see this series The the mean is constantly increasing is it changing right if you take any timestamp let's say from January 16 to uh Jan 2017. so in this case if you see this is the media isn't it and if you take any of the timestamp here the mean is Shifting so your mean is not constant okay so this is not a stationary time series okay we'll see how we can make this time series stationary okay uh in the coming uh slides here okay now this is one thing where your name is actually changing over time this is also not stationary why because if you see there is a seasonality which is there okay if seasonality you can see it uh like a cycle which repeats after certain period of time okay so this series is also not stationary okay this series because if you see the fluctuations is increasing or decreasing right so that means your standard deviation or the variance is not constant isn't it so the these were the conditions let me go back to the previous slide these are the condition to check whether this series is stationary or okay now we talked about one model which is auto regression okay when I say this this if you see it right let me go here okay so this you see it when I say that it is actually dependent on the previous time stamp only okay so when I say price at T is dependent only on price at T minus one this is nothing but your ar1 model okay because it is dependent only on previous timesheet okay previous lags and this you saw about stationarity how do we test it statistically like if the series is stationary or not we have a way to do it which is ADF test which is augmented ticket Fuller test okay if you put it in Python we have a language equals test model where we have this test you can use it and based on the p-value that you get you can say that this series is stationary or not okay now let me take one example in what in case where what happened when your series is about how to deal with that okay now if you talk about this series only right there is an increasing Trend which you can find it okay it's increasing linear Trend okay that means if I talk about this increasing linear Trend here that means in this there is a constant change that is happening on your y-axis okay and if I take a difference okay if I take a first order differencing that means I I subtract let's say YT the YT minus 1 then I'm well that change is going to be constant isn't it and if that change is constant that means whatever whatever fluctuation and whatever the pattern is there in the data is going to be preserved okay and if I change this to this if I do its first order differencing the series is going to look something like this okay this is going to the uh the pattern which is there is going to be preserved okay this is one of the way which you can use it to make the series stationary because any of the model that we are going to build it does not necessarily we we are not sure that the series is what we are getting is going to be stationary okay thank you so now let's move on to the next model so what we talked about till this point is that whatever price we are going to predict is dependent on the previous price values means the prices which has happened in the previous last time now Ma talks about the price that you are going to predict today it's not only depend on the previous fact let's gets dependent on the error that has occurred on the previous instances okay let's take this example you see a girl here is she she's asking that to get chocolates every day okay so at different time instances I'm I'm saying this like it time instance one the guy gets 20 chocolates okay and there is an error of minus one okay now the girl is telling that your actual chocolate should be actually 19 okay so the next day what the guy does is it actually brings the number of chocolates okay which is equal to the meal Value Plus some error that has happened on the previous day so what was the error that has happened on the previous day this doesn't minus one okay and there's a coefficient which is associated with this which is 0.5 okay that's how you actually predict the next day chocolates okay the next day the guy is going to take our pink chocolate is equal to this okay again he goes to that then again it's uh the lady is telling that okay you you got two chocolate strong okay so the number of chocolates we can match it less so the next thing what the guys does is since the error was two is going to get new number of checklets right with some coefficient 0.5 which is different okay so this is how it's going to be so if you just plot these right whatever number of chocolates the guy is bringing you see that this is fluctuating across a average value right this we call as a moving average that's why this is called as a heme model Right Moving average model okay how do I predict the price so price at any time p is you have some mean value in this case he was always bringing 20 chocolates hand plus minus he was doing based on the error that has happened in the previous stage okay so that's why if you see there's a coefficient that is associated with this hello what is the error that has happened previously so this equation that you see here right this is nothing but a maq model okay because we are taking care of error that has happened until two period of time okay now what have you covered we have covered that okay the price today can depend on the previous values now the price could depend on the error that has happened in the previous days okay now what happens when we accommodate both of these then it becomes my our number okay so how how what do we do in case of armor model we have let's say let's take an example of Arma 1 1A is nothing but your p and Q so p is a r p that means it is going to depend on the P previous values and Q which is there as a one that means it is going to depend on the two previous errors okay so this is how the equation is going to be okay now next thing is nothing but the integrated that means maybe your time series that we talked about might not be stationary okay now what since it's not stationary instrument which will not restrict ourselves for not building and doing the production right so what we do with that if you see this there's a linear Trend in this right just like we saw it in the Google stocks which was there if we transform this series right by doing this difference how do we do it we calculate a new series Z of T okay and how are we calculating we take x t plus 1 and subtract it with x t okay so we just do this differencing okay this is called first order difference okay we do this difference in the pattern which is there in this series is going to be preserved okay and then what we are going to do we are going to do prediction on this why because the sales because the series is stationary right we can see what are the conditions your mean is constant here right there are no seasonality which is then right your variance is the fluctuation that you see is okay so we are going to do prediction on this series okay let's say we find out uh uh the p and the Q value we'll come to the code when you see how we can find p and Q from the Pac and pacf and ACF plots so we'll we'll actually do the prediction for this series okay once this series we have done the prediction that means let's say we are trying to find what will be the value at Point K okay this we will get it okay then what we are going to do we are going to transform this back to the series how we are going to do it just like we this this transformation was subtracted right we are going to add this address so this is the z t with t plus 1 that is a one Gap you can accept K is z at K minus 1 plus X at K minus 1. okay and then again you can change this x of T minus 1 to again this change right which is will be Z of K minus 2 plus X of K minus 2 and so on okay and finally we are going to get this value which we already at this point okay fine now let's talk about uh segments okay finally before going into segments let's get into some uh uh code here in Python so that we have an understanding of how to actually deal with this okay so I have got one basic uh data here and then I'm going to just Sue it so this is one of the time series that you can see it right this is one of the time series that you can see how are you going to check what is the p and the Q value that means at what P it is a relate like the previous time series and the queue which is previous errors so we have ECF plot okay which is there inside stats model Graphics dot time series plots time series analysis plots so we have these two functions plot KCF and plot pacf okay so we're just taking one basic data set here okay and then we are just plotting this and how do we plot the ACF and pacf club we are going to call this function and pass on the series okay Lex is one of the parameter that means say we want to find out like till 100 lakhs oh this is related okay so if you see this if this has calculated the correlation Factor till 100 okay this blue band that you see this is nothing but the error pad okay that means anything that is below this inside this blue band that means we don't have enough statistical significance that this lag is related with this number of legs okay let's say at uh let's say this is 80 okay so this correlation factor is very less negative correlation it is very less so we don't consider this okay it might not be related with the uh current uh like current price which is 10 okay but if you see at these additions there are high correlations right uh you can ignore the first one because it is correlation with the same series okay the second that you see is with the next lad series which is p minus one okay and so on so in this case you can find out okay what will be the p and the Q value okay so in this case where my P value will be one two three right similarly you can do a pacf plot what pacf plot will tell you what is the direct correlation for the price today with the previous slides okay so this you can see it right for uh this lag one you have high correlation so I will let me discuss this suspect yeah so you have this stock prices that we showed for Google stocks data this series also is following a linear Trend upward okay and this is not stationary because your B is constantly changing over time okay so we can do a differencing on this to make the city stationary and are we going to do the difference we're just going to take a first order difference okay means subtracting the series one shift okay so once we do the shift we can plot the same stocks and see how this how this looks okay okay to this [Music] okay let me take an example of uh me model here and how we are going to do it okay so this is the series that we series of data that we that this is how it looks like we are going to calculate what is the p in the Q value by KCF and PAC IO this is the maximum number of lacks that then what we are doing is given a series I am just splitting it up into test string test and train data in order to see that how my prediction is going to be okay and then I'm actually fitting the arima model so this arima model is also inside the stacks model Library so we're fitting an arima model here so you can provide the order for this here which is my P D and Q values so there is two values you can see it is equal to 2 here okay and this looks like completely stationary so what we have done is we have provided D is equal to zero okay and then what we do is with the model we are fitting the data once the model effect is done we can see the model summary okay how this model has been built so if you see this these are the coefficients that we get so this is the constant now with M A as L1 the coefficient that I was showing through here beta 1 and beta2 here right L1 and Vector coefficient these are the coefficients so what is going to be my final equation my predicted value is going to be constant plus 0.37 into error that has occurred on the previous two instance uh two period back okay so this is nothing but my uh ma2 model because we are considering only the errors here not the previous steps okay yeah so that's how we are going to predict it and once we have the predictions done we can see it how it looks like so this is the actual data this is the test data that you see it from your blue line from here and this is your predicted data okay and with the testing that uh prediction you can actually go and check the metric of how good is my prediction so there are ways to do it this way you can calculate mean absolute percentage error and means come back here so till this time we have our AR which is auto regression on the previous values we have ma which is based on the errors that has occurred before we have Karma which is taking it together and Karima when we have series which is not stationary okay now the next thing that we're going to talk about is sarima okay till this time we have not taken into consideration the S part seasonal part okay now let's look at this data okay okay so in this case if you see this pattern over this 1996 to 1997 January this pattern is some kind of repeat in each of the year isn't it that means there is some kind of seasonality in the speed okay so in this case we go for what we call as okay so we had p d and q now similarly we'll have a capital P D and Q both are analogous is just that capital P D and Q is part of the Season okay so it's in the seasonality that we are seeing it how the seasonality today is dependent on the previous seasonalities okay or on the errors that has occurred on the previous season okay that's how we are going to build a salima model okay and then let's talk about this important factor which is aimed here okay small and just ignore this this is actually smaller okay so small m is the number of teeth it takes for the seasonality to repeat so if you take any of the time series data you actually plot it you can actually find it okay what is the number of periods it takes for the seasonality to repeat so if you in in this case if you see right it takes around 12 months isn't it so from this to this it takes 12 months to actually repeat the same seasonality okay so in this case my small M will be 12. okay now let's go back to the code here okay so this is one of the catfish sales data okay similarly the sales series looks like this it's very actually uh the way is very simple you take your data get the series out of it so one one of the time one of the accesses becomes your time one other one is any of that uh predictor that you are trying to make sales okay okay so in this case if you see there's a little bit of trend upward Trend so you can actually uh remove the trend by so this is not stationary by taking a differencing you can remove that right so this is how it looks like right then you can similarly you have to find your ACF and bacf values get your training and test data separately you divide your data into uh some part of training right like you take some 80 into training into twenty percent of your data point as testing data okay and then we have a library called study Max okay X X is nothing but the exercise okay so let's talk about this whenever you are actually building a sigma model right there might be one exogenous Factor also that could actually impact your salima okay let's let's take an example here uh let's say you are trying to predict One account balances okay over time let's say you're trying to predict One account balances now one of the exogenous Factor could be the interest at which the bank is giving you right the interest which bank is giving you one of the factors because let's say the bank is giving you a higher interest you will probably not take out your money right you'll you'll keep it in the bank only so this is one of the factors out apart from your previous whatever pattern ing so in in such scenarios when you know that you have one other feature as well which could actually impact your predictions to use Siri Max okay so you might have some seasonal component you there might be some AR or ma component from the previous uh patterns and there might be some other exog variables okay so in this function only Siri Max you can actually pass here exog is equal to your series which is there series column okay so settings is also inside this stats model Time series States based ceramics okay so this is how we are going to build a semi Max model so in this case we have to provide one is a small PDQ value which is there which is your ACF value is there a relation with the previous order of differencing for the previous errors okay other thing that you need to provide is a Capital PDQ and your M values okay m is nothing but your 12 in this case okay so you provide these things and you can actually build your Serie Max model here you just have to do a model dot fit once you have built your model you can actually summary and once you check the summary here right you will get your coefficients okay associated with each of the factors okay fine let me go back so till this point we have taken out seasonality and your exoc okay now we have not taken into consideration what happened to the residuals so whatever prediction that we do there might be certain residuals to it right so we take out those residuals and actually calculate the volatility okay so you calculate the volatility and the variance which is there and see so if you see this plot right here right this is your uh volatile volatility of the variance you can say maybe let's say this is time and you can see at the early time periods here right the volatility is less and then it increases and after some time there is there is a again similar pattern of decrease in the volatility and there is a increase here and again there is a decrease that means there is some pattern in your in your residuals right so what we are going to do we are going to capture this so we are going to model this volatility using okay let's talk about what Arch is actually Parts is first is auto regressive right because the error term that you are trying to model is actually based on the previous internal system right because you are taking the error that has happened today you are trying to model it in the previous values isn't it so this is auto regressive why this is conditional conditional because volatility today you're trying to check it based on the previous values so if you see this the volatility may be on the let's say this is Gen 2 Jan to March right and this is let's say April to June right and this is July so it's July volatility today is very conditional right it is going to repeat after certain period of time so that's why this is conditional this is very much position dependent okay now I'm not going to getting into maths because I think we have very less time available right now so I'll just uh give you an intuition of how we have moved from cast to culture okay so in this case is nothing but your generalized okay generalized Arch is why because when we are actually predicting using Arch we are saying that we are not only taking in concentration the previous values but also considering the volatility that is being there on the previous instances let's say for example what I am predicting today is based on the previous what has happened on the previous day right and it is also dependent like how jumpy it was on the previous day so we are also trying to capture the volatility okay so that's what it's going to happen in card so that is why we call this at centralized not only taking the previous days values but also the volatility that has happened previously okay fine so this I have taken a small note for this just to give you a parallelogram here on how we are making the uh jump from thought process jump from AR to Arma similarly we are doing a jump from Cars to cut let's understand this so when telling that we were going to model ar1 we are telling that 80 is dependent on 80 minus 1 which is on the previous day plus some error that has happened okay so if we have these two factors no we can actually predict the today's value okay so what happened okay second R mode we had dependence from E T minus 1 which is the previous day which is p you can see this is a one so that is the previous day and then you have 2 is equal to one so the error that has occurred from the previous day so this is e t minus 1 and the error that occurs today so you can see how we have jumped from ar1 to Arma 1. now let's go to this Arch so Arch uh mathematical derivation I'm not getting in but this is how it looks like okay so your a t which is the prediction today is dependent on error today but also the volatility today okay this is the volatility and this volatility is dependent on what is the value on the previous day now this you see on Arch right when you move from Cars to garch just like we were taking the errors also that has occurred on the previous day in this case also what we do is we take into account okay what has been the volatility yesterday but today apart from that I'm also going to check out what is the volatility that was there of the previous day okay so this was only the volatility today we this in this we are going to capture both the value which was there on the previous day and the volatility which was there okay this is this is kind AR to Arma that we have from Hearts to garch we this is also a similar jump considering the moving average okay so I think we have some minutes left so I'll just go to garch model here okay so I'll just show you where your cards model is there so you have this series these things are very same you have your KCF and ECM uh you have your variance calculated okay and then you have a medical Arch mark so if I see this there's a library called Arch in this you have Arch model okay and in this case you can actually pass your p and Q value and if you provide this policy equal to class this is going to actually consider uh building the garch model okay so what we can do is given your mean series which is there let's go to any of this given the main series which is there you can actually go about plotting the data then finding which is the best model starting by plotting the ACF and the PC of the plots on those plots you can get the p and Q value and if you see any seasonality in the data you can actually code for sarima okay and then if you see any factors which can actually apart from a pattern on the data which is there there might be another feature which is actually impacting it you can go for side effects okay once you have actually plotted predicted the data you can take the residuals okay by how do we find the residuals you take take what is the prediction you subtract what was actually so you have the residuals plot the recipients okay once you have plotted the residuals you can actually see okay is there any pattern on the residuals that I see can I still can I still actually uh find a pattern on it okay so once if you find any pattern on your residuals then you can actually model that also using Arch and garch model okay so if in this case you can very well ask like how on the volatility part how we find that uh how we have to model it like what is going to be my P and Q value this is very similar just we have uh the way we have actually calculated normal 10 series we are going to take the residuals and find the p and Q value okay for your courage and Arch model so we are going to provide that and do the prediction once we have this prediction you can actually accommodate this with the actual prediction that you have done it from any of your previous models okay so in this case you can see okay this is about the volatility in your predicted volatility is something here okay so in this case you can see that some part of volatility will still be there in your final prediction that means your prediction is going to be better okay yeah so that's all on this part so whatever is easily okay whatever is late we have something called as a white noise this is something that you cannot predict okay this is just a noise okay so this is where you actually stop it and say that okay this is something that we are not going to predict it and this is going to be a noise so why white noise will have following condition the mean is going to be zero so in this case you can see that this is fluctuating around zero so this is kind of uh White Noise right sorry yeah so then you have to have the standard deviation constant right and also there should not be any correlation between the lags right so the noise that you see today should not have a pattern right okay it is dependent on the previous uh residuals okay so in this case if you see it right there is a seasonality so this is not a wideness so what is the error what is the problem with this the problem is that we have not captured the seasonality which we could have done it using your sarima and sarima so we did some mistakes there so you can actually see okay this is the residual so it has some seasonality component which you are not able to capture it okay so this is how you can go back and fix that so this was a brief on how what's the thought process of uh building a 10 series model Okay so the way time series have evolved it I'll put a link to it so we have been modeling using a lot of neural network models deep learning okay and uh I'll put a link to it that the conferences which is there you're using Transformer models to predict the prediction right uh predict the future values uh so you can go through the conferences I know this is a little time it is difficult to actually accommodate everything I've actually maybe I've given you a little idea on how to go about 10 series data okay so that's all from my set uh have a good day yeah yeah thanks a lot Rahul for delivering Sachin insightful session on the behalf of analytics vidy I would like to uh thank you for your time and would like to apologize for the inconvenience cost in the beginning of the session yeah thanks thanks a lot and everyone yeah and everyone I've put up a feedback poll so please provide in your valuable feedbacks on the session and I would uh thanks everyone for joining this session and really appreciate that you guys waited till the starting of the session thank you if you wish to connect with me you can connect me you can have this question yeah thanks I hope yeah thank you everyone cool thanks everyone for joining in see you tomorrow in tomorrow's sessions we have two sessions