Model Guesstimation (MLOps)

Key Takeaways

so yeah hello everyone my name is Siddharth I am a along with ram goswami we are the two of the moderator and we are the part of a data science team at analytical video and we'll be moderating this session for the whole time so as I have already launched a one port please please uh do attend that whole and participate in it and uh for those who have joined this first time uh I'll just let them know why are we arranging these data session every day because these sessions are really really important uh to know about data science and data science domains so uh so basically data is is a one hour data one are dedicated to the data and it's a series webinar led by the top notch industry expert where they teach and and tell us about the data science knowledge so as we know that we have a session model and guest estimation ml Ops so let me tell you about how this session will work and what what is model guest estimation so as already uh Toshiba has told you about that these are very essential goals for machine Learning System to achieve by analyzing these reports you may determine whether or not the selected model is suitable for deployment or not so it is or if it is only designed use case or model that cannot be deployed or productionized before launching it to the cloud so yeah in this data session as I have already told that toshith will explain in details the process of model guesstimation and its need and its various applications in real life so before uh starting this session I would like to remind you few few points we are recording this sessions and we'll make the recording available in few days on our YouTube channel probably two or three days uh please please use the Q a question uh try to use q a q a section for asking any of the any sort of questions you might have uh you can ask it my during the sessions also or and probably two May answer these questions during the session all right by the end of the session it's up to him so and also we have already share a feedback hole to us and we will share the feedback poll at the end of the session and also it is ready mandatory to participate in it so yeah now it's up up to us people in this data session all the virtual stage is yours please hit it all right thank you sudharth thank you very much hello everyone and uh let me show you my screen so what I'm planning to cover today uh I already shared with you all right uh first 20 25 minute uh we will look into the slide and then we will jump into the code side and first we'll understand the concept of uh the topic which I am planning to share with you all and then the rest of the thing we will look into the code side all right sorry to interrupt I'm really sorry I really I haven't introduced you to that Indies I'm really sorry for that no problem actually okay so uh guys let me tell you about push it push it is currently working as a lead data scientist at HCL technology having 10 plus of years of experience in the field of uh artificial intelligence product development and research he also worked in various uh reputed companies including Emerson address plus and Hauser and in soft in soft Etc so yeah I guess we have very uh impactful keynote speaker and and he's also an author and a content creator also so hope you will learn a lot from authorship so yes now tashid you can continue the fun hour and you have all yours thank you sudar thank you very much thank you for the nice introduction anyway I was I was about to share all right no problem okay let me show you my screen guys everyone uh okay uh let me know when it is visible to everyone and then we will start all right so I hope my screen is visible to everyone yes no yeah perfect so yeah it's visible so let me you know keep it here somewhere so that I can see uh what people are responding it here right so today we are gonna talk about something called Model guesstimation so what what what you understand by looking into it right so guess the Guess the model uh which should be productionized right it uh is your model is production ready right that is the most important criteria which we are going to cover today okay yeah so this is me I am tushidave and I'm a lead data scientist in uh scl Technologies uh associated with various product and research team and uh I'm working on a lot of research given that type of product what we are developing it here and uh I handle the team inside my under me there are a lot of you know people who are working with me along uh for the research side and uh we do take care when whatever the launches and everything is going to happen in the future given this uh like in sele Technologies right so my first question to everyone this is the everything you have in front of you you have a model one you have a model two you have a model three you have got your accuracy of 90 and your model is 90 93 confident for each prediction okay this is there is another model two and model three by looking into these three particular models can you tell me which model is performing better any guess any one of you you can you can put it over the chart and then we can have it maybe three model three lot of people are saying three all right three three three three three three and so on and so forth perfect one okay okay all right all right thank you very much so intuitively I will say uh we can give a model one a priority oh there is no there's no relation with the person and recall currently my friend so just hold on your breath okay so I will say model three uh sorry model one I will consider as a as a ah for the productionization point of view why because what it is saying I am 90 confident that my model has given you the accuracy of 90 currently but that model is saying I am 93 confident whatever prediction I will give it is going to be 93 confident this is what I'm saying it here wherein in model 2 and model 3 see the difference what is coming up right model 3 saying I am 92 percent accurate but I am 97 confident I will give you class label as a zero or class table as a one right the difference between these two is close to three percent here and and four percent here right that is a challenge here right so you always have a two types of predictions one prediction is a hard prediction wherein you will say uh you predict the value of y uh which is your target variable which is nothing but your binary classes and predict the value of x oh sorry uh given the data set X right now another thing is something called probabilistic prediction when you you take the probability of Y and given the data set X right you will understand you will understand RT just just give me some time here you can understand it okay wherein Y is a binary class 1 or 0 may be true or false or fraud non-fraud or any of the other classes you take right so in this case what happened when you apply the probabilistic prediction it normally gives you the ranking so you need to rank the model that is being called as a ranking prediction and then you pass the dragging prediction into the hard prediction that is the flow normally you take when you proceed for the uh probabilistic estimates right when I say this when I say this the most important criteria what we we need to cover it up here is uh take the probabilistic prediction right I am saying the probabilistic prediction is going to be I am 93 confident for each observation or each observation is going to be 93 correct for the 93 percent that is what I meant for here given the accuracy of the model 90 I am for each data point or each observation it is saying in unseen data I am 93 confident you will get a class level 0 or 1. that is what it meant for here all right so let's move further imagine you have a h you have an education you have a total experience and occupation you fed into a model of random Forest imagine you you have chosen the random Forest as a model right now what you will do you will basically calculate the probabilistic estimate here right this is your predictive probability right so definitely uh these are some of them are categorical features as you might have noticed then what we will do given this categorical feature right we will convert into the numeric and we got a probabilistic value which is 0.2 right so what I am saying it here when I got this I am saying I am 20 your model is saying I am 20 confident people will get the get the given this information given this age education total experience occupation your uh you will achieve the um salary of maybe more than 50 percent uh sorry more than fifty thousand dollars or less than fifty thousand dollars right I just took an example of census income uh you might have heard of this particular data set it's a very popular data set uh so I just took that particular example and kept it here for your easy reference right so let's move further so this is the intuition basically you will achieve out of it I am saying 20 who obtain the Y heart or P hat which is the probability you have got is a 0.2 are likely to receive salary more than fifty thousand dollars given the features you fed into the model that is what the intuition you will take out of it right now if you look into this page so we will cover we will try to understand the overall scenario how this has been applied and why this is important why it is not uh it is very uh people don't cover it but let me tell you it needs to be covered in in a proper way so your first pointer is high risk use cases right now why why I'm saying high risk use cases right imagine imagine uh let's take an example of uh maybe Healthcare sector okay which is like highly uh critical sector we can we can consider right imagine patient is suffering from the specific disease right hence doctor definitely needs to give a some specific medicine to the patient yes or no right it is definitely required and imagine whatever the features that doctor has suggested to you you as a data scientist has has given you some of the important features you give me the accuracy of the model right you got the accuracy of 90 percent right but herein we also need to talk about the confidence of the model like how much confidence your model has so whatever the prediction you have got how correct it is right this is where the health sector problem comes into the picture because definitely either you will say you give a given the disease you suggest this particular medicine or you suggest whether the guy is having a heart disease or not but your model has to be very good uh you know in terms of probability I am 90 confident this person is definitely has a heart disease accuracy and uh probably probabilistic estimation both are very different that's the reason I clearly mentioned it here calibration or probability estimation is not at all equivalent to accuracy there is no correlation between them right now if I say given this information given for example you have run close to five models or six models because you you have no idea in in real world use cases what kind of model uh basically will work so there are multiple type of models there is an ad abuse there is a random Forest there is a logistic regression or bunch of classification models are here right so imagine in this way all the you imagine you have ran like close to seven models you you applied all the grids some model gave you 90 accuracy some model gave you 89 accuracy everything is looking pretty good to you like given the or you consider as a prisoner recall wherever the false positive false negative someone was someone was uh telling me about uh some imbalance thing and all right imbalance data set or XYZ but even though you you did everything here right still imagine you have got a accuracy of ninety percent for each model every model you might have looked into your present recall or you wanted to take care of your false positives as well you have taken care of everything but still are you confident out of this one of seven models which model will give me best possible result no one can say this or or any one of you can tell me then it's it's pretty good because you are like going to be a should be a lead data scientist or maybe a you know the head of the data science department in the in the Google right but overall the picture the my my my basic Point here is we need to figure it out the best optimized model given the data set okay now that the the method which I'm I'm trying to propose you know the method which I'm trying to share with you all will contain uh you can you can have you have an option to improve your models as well right so what are the mistakes you can you you will have to achieve the high probability value so there are two labels in in your one is positive level and one is negative level right so for the positive level or true label what is the probabilistic score you have or whether you should consider it in a uh given the given the important features which has been mentioned in your data set you are achieving a good probabilistic value a good probabilistic result based on that data point right so this is what the important comes in uh important aspects comes here right now on the right hand side if you see uh uh okay so before before I jump onto the right hand side which the picture which has been in three dimensional space right uh I will not go there so let's understand the calibration or probability estimate is not at all equal into accuracy right well calibrate model can also have a low accuracy why imagine you have worked on the uh on a on a on a one data set where um you have which is highly balanced mod highly balanced data set okay it is not at all imbalanced highly balanced that is 50 50 um you can consider as a iris data set right 50 50 data set right 50 50 data points you have it so are we saying that given this fifty percent you have got accuracy of your model is 95 percent given given that data set is highly balanced now that is to say whatever the sample data point you will achieve or sample information you will achieve in your in your from your population in the in the future right you're a 95 confident this should fall under the iris or it should fall under the verticular or virginica right but here I am saying it is not at all possible it is going to happen there is a 50 probability will be applied into it for the given model whatever model you choose right so even though if you have a low accuracy of your model so if now imagine you have an imbalance data set right when we talk about where you you might you might have applied a small technique or over sampling or under sampling all this all this technique you might have applied here imagine you applied imbalance data set for the one class right you will always achieve accuracy of 95 percent so what you will do you will try to do oversampling for the negative class and try to bring it the same bring to the same level but here still you will get this particular problem here you will definitely uh May achieve accuracy of your model 90 percent but your model is not confident given the data point they will achieve the uh confidence score of 95 percent that is the biggest challenge here okay so there are two terms model can have a higher high accuracy and can have a bad probabilistic estimated values right now when we talk about your all the action depends upon the precise probability what do I mean with it imagine you're working for one company for the issuance of the loan they wanted to issue the loan right and they say um you as a data scientist can you tell me uh if I will provide the loan to this guy whether he or she will be able to repay me that loan or not right this is where the precise probability or that particular probability estimated value comes into the picture because your model may have a good score altogether 90 95 percent but it is not confident enough to provide you that information which your organization is looking for where you where you can save the money for your own organization right so this is where the uh this uh probably probabilistic estimate comes into the picture okay this is where the calibration comes into the picture so probably probabilistic estimate is not a loan work work alone there is a calibration is required for everywhere right now now I am saying I am talking about the model quality I want to assess my model quality here okay so I'm saying it here I am 80 or 90 confident my model will provide me the correct result so you as a data scientist needs to have a clear-cut information whether your generated model or developed model when it will be deployed given the data set is confident enough to give you good and accurate result right so this is where again this particular method comes in okay now here the next thing is model impactness or maybe you can call this as a simulation right so in in this case ah let me let me let me take you an example here I think that will be um better so imagine you're working for one company like automobile company and they are saying uh given the model how much money I can save given the model how much money I can say oh maybe you're working for one power company one power industry wherein you know bunch of steam uh is being has been flowing into the term uh boilers and turbines and uh and generating electricity so they uh that company is asking you okay can you tell me what is uh what is the impact of impactness of your model you have it right how much how much given the given the model how much money we can save uh whatever the model you have deployed there so this is where the impactness of the model comes into the picture by using the calibration or by using the uh probably probabilistic estimate and you are you're already all are aware nowadays In fairness and biasedness is the most important criteria how fair your model is how much data points are biased towards your model you when you notice in terms of uh imbalance data set because your data set is biased towards one class right so this is again a very important implications which which factor which we should consider it okay when we talk about calibration now the next Point here is recalibrating for the model drift now the model drift normally comes into the picture when we when we applied any model maybe imagine you applied random Forest as a model right or you might have applied uh Knight base as a model right now you might have noticed you fed your data point into this model after after doing all the pre-processing all the feature transformation and everything right now somehow what happened the covet came in right now when when the covet came in uh what is the problem will come either there is a there is a uh there is going to be a very uh this problem is going to happen what what you will say um problem pertaining to your is it I'm not audible guys because people are saying is a poor communication something is happening hello from my side uh your voice is quite clear maybe some of the okay all right so okay so here is the here is the calibration what I'm talking about right so imagine there is a probabilistic value which is between 0 and 1 and this is your calibrated line which you can see here this is your calibrated line right this particular calendar always been a diagonal there is no changes going to happen in terms of this right so this is your ideal line which you can consider currently 0 to 1. now I am saying that I am 80 confident enough that whatever the result I have is going to be uh is going to be uh confident my model is saying this so when we talk about this particular criteria this is how the the entire things works right you take the you take the data point you take your unseen data point and you score the specific value out of it so that's how the entire entire uh you know data points will come into the picture and each for the Unseen data specifically when I'm talking about the Unseen data there's no I'm not talking about the test data or maybe train data you leave it behind there's no correlation between them right no so if you notice here I have a data point uh probabilistic value as you can see here this blue line so I have run one of the night base model in one of the use cases and you can see uh for the 0.6 probabilistic value which has been mentioned here right and I'm saying when you got your 0.6 probabilistic value here what is the score on the y axis the fraction of positive it is just close to point one right so you are saying for the point one the the binning score what you are what you are taking out of the given data point is very low you are not at all confident enough your model is noted on confident confident enough to provide you the accurate result so this is being called a reliability curve this is being called a reliability curve and when we talk about the reliability curve uh this there has always been binning on the probabilistic value which is being developed and plotted on your X between in two dimensional space right so if we achieve this particular data this uh blue line which has been noticed here if you drop it right if you increase the lead length from calibrate from 0 to 1 you can keep it here that is going to be a most probable best model which you can consider for your use case given the model given the data set and this is how this is what we need to achieve we need to bring our product our uh this probabilistic model Line near to this calibrated Line near to this diagonal line that's how the entire calibration works all right so this is just a small intuition guys uh there is no uh such thing here you you might have noticed there is a lot of formulas and all everything here so reliability I'm just explaining my reliability curve here you take the group of predictions into the bins as I said earlier in my previous slide okay and the fraction of positive per win right you take the binning you take the probabilistic value you bin it and you keep it here in two dimensional space then you develop a confidence on it and when you develop a confidence on it you will say my model is 90 confident 95 confident enough to show you the result to give you the proper result out of it right so uh this is the most important uh you know expects expect we need to consider when we talk about the machine learning operation side so because what happened in real world data normally ingested in in you know continuous basis so there's there's might be a threshold values which has been defined there might be a you know bunch of data point like 20 000 or maybe one lakh data point is floating into your into your model right into your pipeline given the model in in your pipeline right so what happened you need to have to have a proper prediction and proper probabilistic value because this is where the revenue has been involved for any other organization so we need to consider this particular criteria when we talk about any kind of reliability curve and when we talk about the calibration all right so what I'm going to do here this is my train data set you you take your features and you're feeding into the random Forest right then you take your so make a note The Strain data set won't be used further okay now what you're gonna do you take the test data set you check the label the label what you're looking for label one or level zero whatever the positive label you have you predict the probabilistic value and then feed into the logistic regression model as one of the calibration model I am considering currently okay then you test this particular logistic regression or calibrated model from the validation set and then check the value how it has been performed between your test result and the validated result and then you prepare you will say I am this particular calibrated model will work better compared to your other models which has been notified of which you already developed in your given the from your use case all right so uh this is the uh you know just a small small uh you uh this thing plot which has been developed in my use case okay so you have a original one as you can see the original one is a blue line right and you can you might have noticed you can notice here it is like going away from your calibrated line right this diagonal is your calibrated line or maybe you can consider this as ideal line all right now this orange data point which you have which you have developed is real close to this ideal line now now you can say okay this is working pretty fine now it is working my calibrated model is working better compared to my prediction name compared to my previous model right so just a moment so what I'm going to do here is I am just uh I've developed a one this particular use case I just you know bought this particular data point here this data set here which is nothing but the census data set right so you have a h you have a work class you have an education you have a marital status and what you want to predict here given this information whether the person will achieve the uh salary more than 50 000 or less than fifty thousand dollars right this is what you want to predict here right so this is just a small pre-processing I have applied it here and now I've run the I've run my models so as you can see here when I run my models just for the after the small pre-processing right you have got accuracy of 85 percent it is 83 percent for random Forest classifier is also you have it and write a boost classifier also you have it right all the now but by looking into this by looking into these models we cannot say which model will work best in terms of predictions in terms of probabilistic scores because it is saying I have got a pretty good accuracy 83 85 percent I have got a on test data set it is good it is also pretty good results as you can see here all the models having a very good results given the data set [Music] okay just a moment is it visible to everyone not yet yeah no it's visible okay so let me restart it right yeah you can just go through it yeah so this is the data set which I bought it from the from GitHub given this information given this the label given this features right I want to predict whether the salvi uh of the person is going to be less than 50 000 or more than fifty thousand dollars this census income data set uh I hope everybody is aware of right and just this this is just a small pre-processing I've uh I've done it here so you can look into it later and con and then splitting into the training test for now and run my uh model okay and this is the accuracy what you have got here currently as you can see here right everybody so all the models perform pretty good nothing nothing wrong in it right but now nothing wrong in it right so now what I'm trying to do here is I want to develop a calibrated result for the random forest model this is my random forest model which I have chosen right now have a look what is happening here it is very much close to your calibrated line or maybe you can consider the ideal line which is supposed to be there's a taggle line what you have so you basically calculate the Brier score here something called brighter score right Briar score lower the brightness score better the model right we will look in in upcoming thing what is going to happen I will tell you now what I'm what I'm trying to do here is same data set I am converting into the three form three is splitting into three ways train test and validation okay as you can see here everywhere I have a 10 000 close to 10 000 data point wherein uh in your validation is wrongly written please ignore it it's supposed to be validation okay it's supposed to a validation so it should be it is 11 000 data points So currently you have it now this is very important stuff uh which I'm showing it here currently the difference between your trained data set model feeding and test data set model feeding so there's a drift as you can see here currently which is happening the drift between train and test your if you notice here the test drift is skewed in this case everywhere right so when we talk about this test drift and when we compare this thing with the validation drift if you have figured it out there is a skewness is coming in your in each and every data point or each and every uh data you are feeding into it so either what you will do either you will retrain your model okay do the do all the pre-processing all the feature transformation again and then you refeed into the model and get the predictions you can do that this is what you can do right but it is just information I just wanted to check the the difference between these two so I just mentioned it here just for the information I have shared that another thought with you now what I'm going to do what I'm going to do here what I'm trying to do here is as you can see I have developed with Kelly this my ideal line and each model so I've applied three models here random Forest the support vector classifier and like base all right so if you look into it this is the ideal line the probabilistic value into the binning format you have it here right which is very close to your ideal line currently but when you look into your SVC line it is quite far the trend is very different compared to your random Forest right the same is applicable to night base as well so knife base is very poor performant even though you can you might have a accuracy of your Model 95 or 90 percent but it is performing very poor right so if we if we look deeply all the three lines all the three this blue line or probabilistic estimated line I can say that okay my random Forest is performing better but is it really performing better we yet to check this how we can check we will calibrate it we will create a one our own model calibrated model and check with this particular uh uh run the forest model all right now have a look so this is this is I am doing I have created a one calibrated model for myself which is being which is nothing but my LR model logistic regression model okay and I checked it with this right I still say the random Forest is performing better but compared to others the my calibrated line my calibrated model is performing better compared to other compared to SVC or nine base right that is the intuition basically we will take it out uh um from this given information from these plots right so if you look closely so later on I'm also testing it how this how this particular uh information is basically useful for you right just just give me a moment I will explain that part as well now if you look into it closely your random Forest is very close to your ideal line even though uh your this uh calibrated trend is is poor compared to random Forest now right now let's move further right now let me keep it here as a random Forest let me keep it as a random Forest here okay and now let's do the run and we will check Uncle calibrated values probabilistic values and calibrated probabilistic values so have you noticed the calibrated probabilistic values is good for random Forest wherein the uncalibrated probabilistic values yeah however if we talk about the uncalibrated probabilistic values which is poor that is to say we can say that we should consider random Forest is a better model we should be deployed for the prediction point of view in real world use cases however if you check with NF oh sorry NB Knight base let's check what is happening in in case of NB can you check what is happening the uncalibrated is 0.0 and calibrated is 0.02 that is to say the knife base is really performing very poor it cannot be deployed so if you notice here these are the major information which we should capture when we talk about deployment of the model so wherever the probabilities are there wherever the probability is uh or wherever the classification is there so definitely the probability is going to be part of it and it should be considered this probabilistic values or calibration should be considered before deploying any of the models right now uh this is a very important aspect uh which we should consider when we talk about the model drifting or any kind of drifting right the model drifting is a very important role uh because it will tell you whether you should go for the you should re uh train your model or you should stay back right so if you find find out if you figure it out in your in your unseen data in your unseen data set if you figure it out that there is a distribution in this distribution this drifting is happening right so in that case either you will go and retrain the model or you will stay back so this is a very important criteria which we should cover uh when the model has been deployed in the production all right so that's this is the end of the topic which I wanted to cover cover with you know cover with you if you have any doubt please do ask the questions I am happy to answer those [Music] each and everyone so yeah go ahead okay yes A2B testing calibration will differ definitely calibration is something that you want to see which model is best for your data set right so that is how it will differ how do we monitor it with object detection kind of use cases okay so in case of object detection what happened yeah in case of object detection what happened uh you can consider there are multiple classes are there right so with one you for example let me take one one of the use case for example um there is a tool okay and from that tool there are a bunch of vehicle or passing and you want to detect whether this particular tool in passing passed out vehicles from that particular tool is it is a truck or car or or bike or something right so in that case what will happen um you take the multi-class detection there so you calibrate first you develop your model any of the model whatever you have chosen okay and then you recalibrate it by using the same method which I have applied it and then you can say okay please uh this particular line or probabilistic values are this currently um and you can get this particular confidence level in you simple so the the method for the tabular and object detection will work in the same way no such differences will come how do we interpret calibration curve I mean when line is above ideal or below ideal line okay so line is above ideal or below line what happened if it is moving in the same trend if it is moving into the same Trend we can say that it is very much close to it your your calibration line if the calibration line is very close to your ideal line and it is moving towards in the same direction you can say that okay this calibrated model is pretty good performing good compared to other models the same is applicable with the other models so normally we we say okay this particular data point should fall between zero and one this probabilistic value should fall between 0 and 1 for sure so is there a package for the calibration and model Evolution as you present it no there is no such packages are there uh uh there is a one library is there but it is in in GitHub but uh it is not yet fully developed SN beta stage currently so we need to write our own code as I've showed here currently right so you need to develop your own probabilistic value likewise this can you automate calibrate calibration checks to detect data drift or you manually check periodically okay calibration check normally happen when you are at the development stage right and wherein the data drift normally normally takes place when your model has been deployed so this is a story of before feeding into the uh before before you feeding your model into the productionization stage it's it's before that right yeah any other question right we have some question in the Q a one can you answer them if it was we have a few of them there foreign okay yeah sure and get the definitely can we use the weight of evidence for model calibration okay so I would like to understand the when we talk about the weight of evidence for the model calibration so what is the evidence you will consider hit here the probabilistic values you will consider as evidence right so when we talk about the when we talk about any kind of problem probabilistic value which you want to calibrate those are basically you can consider as a weight for the given use case but the but in the end you need to Define how much your model is performing on the Unseen data how good your model is performing on the Unseen data that is a most important thing what we need to consider when we talk about the weight of evidence for the model calibration why we need calibration to figure it out the best model or you can consider which model has been performed better and how good it is for the pro on on the given probabilistic values how good it is and it is meeting your ideal line or not that's the reason you need a calibration how we can detect the drift the difference between mean and mean of your train and test will detect the drift okay so if you have a so let me share you uh here I think just a moment so as you can see here right the difference of your terrain data set enters data set if you will take out the mean of the strain and test if they are very close to each other you can say okay this is performing better if it is far you will say there is a drift is happening in in your training test no this is not applicable in case of regression guys because ah wherever the probability probabilistic value you can Define only in terms in case of classification problems it is not applicable in terms of regression okay how do you decide which model to use as a reference model I am not deciding it it has been defined based on this particular probabilistic value and these particular these plots if it is very close to your your model trend is very close to your ideal line you can say this particular model is can be used for the deployment or for the productionization you can apply any of the models it could be a it could be n number of models it could be Knight base it could be uh a random Forest it could be anything he has gaussian like this you're right yes when you have identified your model is performing best and it is very much very close to your ideal Trend then you can go for the deployment because you have already figured it out uh given the seven models given the five models your this this your that this particular model is performing best out of those seven models can you automate calibration checks to detect data drift or manually check periodically okay so make a note uh the data drift and calibration both are different okay uh data drift is like completely based on your train and test raw data set or maybe that data set you should consider which which you are feeding into the model for which the data point you're feeding into the models right this is where you will check your data data drift so there is a there is a you already created a pipeline and you already done everything and now your numerical values are ready to feed into the model this is where you check your data drift because uh when you feed into the model and you will see for each data point how your model is performing uh given the sharply value if when we talk about the sharply value you calculate the sharply value and you see the performance of your model for the each data point right also you can check there itself for the when your data points are going coming into the batch batch format like in 20 minutes or 15 minutes there are 50 000 or 20 000 90 000 data points are coming up how that distribution look like for for your test data set and how it is comparing comparable with the Unseen data set when we talk about uh it is moving into the batches from data ingestion point of view calibration is is validation for the model location how do you estimate the calibration and take a decision before deployment okay so as I said earlier if your model is very close to this ideal line you will consider this as a best model so in below if you notice here your calibration line is is poor compared to random Forest currently right so if you note if you put random Forest here and I have generated few random data points for myself as you as you notice here right and converted everything into the numpy array and and checked it so it will generate a value between 0 and 1 first thing and now you can see here your uncalibrated random forest model probabilistic is 58 however in case of calibration it is 50. thank you so that is to say we can say that okay my random forest model is performing better compared to my calibrated model which is nothing but a linear regression model oh sorry logistic regression model right so that's how you can uh you can particularly you can easily look into it and figure it out how I can improvise the calibration to reach the ideal line how I can improvise the calibration to reach the ideal line what does it mean actually I did not get the question foreign because this question is not clear to me some of the attendees are one one want you to set uh share this notebook if is it possible for you yeah please I will do that I'll be shared with with you and then you can yeah you can directly uh send the link in the chat box yeah all right yeah is there okay is there any other question I can help you with yeah so you might be thinking uh there is a there is a very important this someone has said right about the ROC curve so Roc and uh calibration both are very different right it is absolutely correct what you uh Samantha saying why because Roc Curve will definitely give you line between 0 and 1 but uh there is a threshold value which you you define right but calibration by looking into the ROC between the tpr and fpr you cannot say your this particular the area the highest area and under the Curve Model is the best model for you you cannot just blindly say this you your focus is going to be like this you want to figure it out your model the model what you have chosen is giving you how confident that particular model is and whether the it is able to achieve the 90 accuracy or 80 accuracy or not for each data point that is what you're looking for so your model has to be very confident enough you are confident but is it your model is confident this is this is the question so we as a data scientist need to look into it pretty closely all right is there any other question I can help you with guys or please do ask I am happy to answer those so always make a note um when you are whenever in your real world scenario uh whenever you encounter with the with bad data set or because data set in real world normally very skewed okay so and you are performing a kind of classification problem in your data set try to apply and figure it out your best possible model right and this is how you can figure out your best possible model you take any of the use cases it doesn't matter that use cases belong to accuracy or prison or recall or anything you can choose that best prison voila model but there is accuracy still are there right by by taking that information you calibrate uh start comparing with other models create a calibrated model for yourself compare with this ideal line ideal dotted line and see how your model is giving you the result and that model only I would I would suggest to deploy never go for it it is very much possible you may you might need to deploy your your calibrated model as well it all depends it all depends process of calibration to reach the ideal line I meant is there any standard way there is no standard way salima uh it is it is a comparison of the model it is a comparison of the model uh your calibrated model as well as the other seven models right it is a comparison of the all the models how it is closer closure to close to your ideal line as you can see in the plot how it is close to your ideal line yeah yeah is already shared on on the chat I think everybody is having it uh actually uh that link uh is working but then that link is requiring access so maybe you need to give them the access let me open it for all just a moment guys I'm attaching the link again do copy it and test it is yeah making it public yeah it is public now please do use it for your purpose all right so sudarth are we done or is there anything else yeah I guess we have answered uh most of the questions right so yeah I guess we are good enough so okay thanks a lot on the behalf of analytical Vidya so I would like to thank you for devoting your time and delivering such a great session I'm sure it will it it was really insightful and really comprehensive and appropriate for a lot of experience expertise or students so hopefully we can conduct more sessions with you in the future and request attendees to please participate in the feedback poll as it help us to conduct more such sessions if you face any kind of difficulties please connect with us at editor at the rate analytics with the platform and so tomorrow we have another session uh at the same time 7 PM IST which is on getting started with AWS ec2 some another speaker will be there obviously a top notch speaker so as I said recording of the session will be available to after two or three days on our YouTube channel do check our YouTube channel article with you can find another a very good data session which are being recorded before so yes uh I guess we are on the winner on our end and thank you thank you guys for attending the session and thank you question once again for delivering such a great and uh knowledgeable session thank you thank you very much so okay guys now I'm closing this webinar and hope you have a great day and uh have a good night thank you see you tomorrow okay so uh some of the links have been uh posted in the chat box uh with about the YouTube channel about the data session which are being will be delivered tomorrow please do join please do register and I hope uh this one hour session will help you a lot and progress in your career thank you thank you guys for attending