Role of Optimization in Machine Learning & Deep Learning | DataHour | Analytics Vidhya

Key Takeaways

foreign for this mathematically intensive session okay so not Everyone likes to you know attend these kind of you know uh a bit theoretical and this is a deep dive you know a lot of uh things which are expected to be known okay for machine learning deep learning okay especially the mathematical part so not many people are very keen to get into the mathematics part of it so first of all I must congratulate you guys to take such interest in knowing what is really going behind the scenes of this machine learning and deep learning algorithms okay so uh I think I welcome you all once again okay on behalf of the analytics with their team and thank you for inviting me for this data session I'll try to keep it within the time okay a part of this session is going to be theoretical and the second part is going to be Hands-On I'll actually show you the optimization in action okay so moving on uh so giving a quick overview of what we are going to discuss okay but before that so these are the four mathematical pillars on which your data science machine learning deep learning is standing okay so one is of course linear algebra probability statistics so anyone who wants to have an expertise in data analytics especially okay business intelligence data analytics and business analytics okay so for you guys probability statistics hypothesis testing confidence intervals those kind of topics are very very important first especially the guys who are interested in machine learning okay deep learning data science part of that okay for them the remaining three topics are very important number one is linear algebra multivariate calculus that is calculus of multiple variables okay also called as your Matrix calculus and the last topic is convex optimization so there are enough resources for linear algebra you might have already attended enough sessions on linear algebra talking about Matrix multiplications and you know eigenvalue eigenvectors Norms the use I mean so if you want if there is a popular demand I can take a session its applications as well uh calculus everyone knows used in Back propagation gradient descent so all of these are you know calculus based approaches to get the hyper parameters and all and the last topic that you see here is the convex optimization so there are two words one is convex one is optimization and I'll demystify both these words connect with our main topic of interest and see where all these things are being used okay and why should I bother it okay so quickly going towards the agenda so these are the 10 points I am going to cover looks too much but uh I'll try to be concise precise uh so yes first thing is why should I bothered about optimization okay and then where else is optimization used in machine learning deep learning and all what are convex functions because I use that word called as convex optimization so you need to have a basic understanding of what a convex function is okay and then yes then exactly we need to understand what do you mean by optimization what are you actually meaning to do what you are meaning to say and all that right and then yeah you want the slides yes absolutely all yours okay don't worry and then there are different types of optimization problems okay and then how do we solve these optimization problems what are the different approaches which approach is better to solve what kind of optimization problem that is the point number seven and finally I'll show you optimization in action using a python code okay and finally I'll give you a quick summary of where else there is optimization used in machine learning deep learning and finally I will also talk about some applications of optimization beyond your data science so that is the agenda for today's session yes warmed up let's get started any questions so I have a continuous watch on the uh you know chat window so if there are any questions please let me know promptly okay perfect so starting with why should I bother about machine learning so the optimization in machine learning so okay here I am expecting the audience to have some decent understanding about machine learning because I'm not going to lecture on what machine learning is all about supervised and supervised regression classification so if you are familiar with these terms can I can I assume that the audience is familiar slightly with these terms because the focus is more on discussing the optimization uh I cannot you know explain the domain of machine learning in this short span of time right okay so everyone understands what the features are attributes okay so features attributes different names essentially these will be your columns in our data set okay these are independent variables and then typically you have one column called as Target or the outcome or the labels different names again so that is the dependent variable so the overall task of machine learning is to find the relationship of the independent variables with the dependent variable okay so that is what your machine learning model tries to do it tries to find out the pattern or the relationship between these independent variables and the dependent variable that is what the model represents okay so that is what the model tries to understand and express fine so we have the features and the target so your machine learning model will Express the relationship between the features and determine so let me go ahead and take a simple example um okay uh is the last part visible slightly fine so let's say I've got a small dummy data I've got a small dummy data I am just taking this small W data throughout the discussion okay slightly hidden okay I'll take care of that not I mean any useful content will not be hidden okay uh I'll just try to make sure okay so anyway this equation is coming up in the next slide also so this data set uh it's not a big data set but essentially I'm going to take this particular problem throughout this session I mean okay so just remember this so what I've got is a toy data set and further part of the session will focus on this toy data set so that we can actually Implement something so what I have is a regression problem you can very well see I have an independent column teachers okay what we call so I've got three values one three and five and the target value to be predicted is 4.8 12.4 15.5 a small dummy data or toy data set fine so this is your actual value of y y will represent the target variable so please you know keep some of the variables in mind okay so you don't have to keep noting down everything but if you remember some of these terms this will help progress in this session fine okay so y represents your target variable why had if you see in the equation below that represents your model so I want to do a straight line fit of my data so your actual data points are represented by these three uh squares can you see three uh red squares okay so these are the three red squares I'll just try to highlight them once again okay so this for the x square is this Omega Red Square and the red squares okay so these are the actual data points which I have plotted between X and Y and then you have some predictions coming from the modeling okay so the predictions coming from the model are everything along the straight line and the original data points are given in the blue circles okay so if you very well see there is some difference in the predicted value so which are the predicted values the predicted values are coming from that red colored line and the blue circles that you see that is your original data that I've plotted so there is some difference that is your error term right so we wanted to predict this now let's say I wanted to predict this and what I've got this so there is a difference here so similarly here on the right side I wanted to predict this but what I got is this so that vertical difference is your error okay fine so I've just kept that plot on the right side once again for the reference so basically there is by intention in find the equation of the best fit line simple my intention is to find the equation of this best bit line which I am representing by a red colored uh you know line here and I have also given a very bad fit a poor fit line which is represented by a yellow line layer so just for a comparison say how a basket line would look and how a poor fit line would look okay so obviously the line is being represented by this equation Y hat equal to W naught plus W1 X or X1 where W naught will represent your intercept and W1 is the slope of this lens so till here everyone is fine everyone knows what is regression so we will not get into all the details so yes so in machine learning this best bit line is obtained by this algorithm called as a linear regression and these W naught and W1 is also called as model coefficients okay also called as weights you can call so-called them as weights or modeling coefficients correct uh epic pen out okay I'll try my best fine because sometimes I need to keep annotating okay I hope this is at one corner fine so as I just now mentioned we'll be getting some error from the model so error from the model is nothing but the predicted values from the model you can see the predicted values from the model yeah so these are my predicted values from the model and this is our actual value so predicted value minus the actual value is your error term fine now my claim is this best bit line will have the minimum error simple a simple claim is my best fit line will have the minimum error for the prediction do you guys agree with me yes no quick but here the question is what type of error you are talking about okay so then there can be different ways in which you can express this error okay that's another thing which we'll talk about in the next slide okay there could be multiple ways in which I can express this error okay so one of this way is the mean square error function one of these ways is mean square error function MSE so it is sum of the squares of the error terms and then divide by the total number of data points that's what this expression says so you take this sum of the squares of the error terms that is what this says sum of the squares of their returns and divide by the total number of data points that becomes the mean so mean of the square of the error terms which we call as the mean square error uh this is not the only choice so just for the sake of reference I'm telling you there could be multiple choices for your error functions or cost functions or loss functions multiple names are there for the same thing so multiple choices are there you can use the sum of the error terms you can use the absolute sum of the error terms you can use the sum of these squares of the area terms SSE I have chosen to go ahead with the mean square error function and you can also use the root mean square error function okay so multiple ways you can express this error but the main claim Still Remains the Same that the best bit line will have the minimum value of this whatsoever defined error function so you have to narrow down to one error function okay and by default or as a standard practice for regression problems most of the time we will consider the mean square error SD um error function for prediction right y mean square error and not the other functions I mean that we can discuss sometimes in some other data session okay there is a very interesting analysis why only mean square error and not any other error but that itself will take another half an hour for me to explain right Okay so how do I Define optimization name okay I'll just connect the previous slide with optimization problem so maximizing or minimizing any quantity is mathematically referred as optimization so if you remember in the last slide we wanted to minimize the error that word minimization is connecting us to the optimization Okay so after minimizing what do you get in hand the solution that solution is called as the optimal value okay so I'll I'll you know explain this a bit more but before I get to that let me just quickly give you a quick overview supervised learning is of two types one is regression one is classification these are the different algorithms used in regression problems and then you have again different algorithms used in classification problem what do you do in regression we want to find out the equation of the best weight line as you see here okay you want to find out the equation of the biscuit line now that can be obtained by a simple linear regression Ridge lasso and then elastic net these are regularized regressions you can use polynomial regression random four is multi-layer perceptron SATA goes gradient boost I got this picture somewhere from the internet I just copy paste it here but yes these are not the only algorithms there are many many more this is not an exhaustive list and in the case of classification what we are interested is to find the equation or mathematical representation of this separating hyperplane so how do I separate the two classes can you see there are two classes over here so this is one class and this is another class so I want to separate those two classes so I want to get the equation of that separating hyperplane right so that is the motivation that is the intention this is exactly what you are doing when you say that you are creating a ml model so what is our ml model doing the ml model is trying to express the equation of the best spit line in some format or your ml model is trying to express this decision boundary in some format right so that is what your ml model is trying to do or DL model is trying to do correct that Clarity has to be there okay so now we have different loss functions so what I'm trying to say is in each of those different algorithms for regression problems there are different loss functions being used okay so indirectly we have got an idea that all these algorithms are using some sort of a minimization they are trying to minimize some sort of a loss function okay just like I said that in linear regression we are going to minimize the mean square error loss function similarly in the case of polynomial regression you can again minimize the mean square error loss function in the case of region lasso you can have the SSC that is the sum of the squares of the error plus the penalty term what is the penalty term that's basically your Lambda times the L1 Norm or the L2 Norm the square of the L2 down depending on what kind of uh regularization you are introducing so for the ridge you will take the square of the L2 now and for the case of lasso you will take the L1 now so there is a penalty term basically and this overall term sum of the squares of the error plus the penalty term that thing gets minimized so you see how did I Define my minimization problem or optimization problem you want to minimize something and what are you minimizing you are minimizing these loss functions is that clear we are going to minimize these loss functions for different different algorithms we have different different loss functions to be minimized so the same loss function is not being minimized in all the machine learning algorithms is that clear okay I can slightly pause okay so how we are connecting optimization so what is optimization ultimately optimization means minimizing a quantity that was the clear definition if you want I can go back to that slide so minimizing or maximizing any quantity is mathematically defined as optimization so but what are we minimizing what are we minimizing so that's exactly what I'm trying to convey here in regression problems these are the error functions these are the different error functions which we are minimizing right these are the different error functions so if I can quickly go to I have a habit of going to the reference Pages yes key learn if you want to go for Ridge r i d g d so everyone knows this Rich function right so I'm just I'm just quickly showing you okay what it says you see the main documentation pay is the very second line it tells you there is some minimization going behind the scenes right that is what we are doing today that is what we are learning what is that minimization happening and why is it that needed correct so it clearly tells you this is your SSE term some of these points of the error and then there is some regularization parameter okay you can call it as Alpha you can call it as Lambda and this is a square of the L2 noun which is essentially the sum of the squares of the modular coefficients so we'll not get into all the details but anyone who knows what is Rich and lasso definitely knows what I'm trying to say okay so this entire thing is going to get minimized who is going to minimize how is it going to get minimized that is what we are going to discuss in the next slide right so this is what I just wanted to quickly tell you that if I just keep on showing you the documentation page for each of these functions you'll clearly realize that let me show you SK learn and lasso okay so if you go to lasso there also you will get so this is the objective function and again we are trying to minimize this loss function clearly tells you that we are trying to minimize this loss function right okay so there are some questions on the Q a uh is there any difference between cost function and loss function so this is the question I am answering is there any difference between cost function and loss function no they are all synonymous different names of the same thing like features attributes independent variables convey the same thing okay Target output label they convey the same thing similarly cross function loss function objective function all these three things are same Dr rajita your question um why CNN belonging to regression ccnn I think there was neural network written it was supposed to be neural network so just ignore okay so neural networks can be used to solve uh both regression and classification problems right so that is the thing that is the way we have to take that okay good so both the questions are answered done and done okay coming back I take it this way I hope that works fine so if you go to the support Vector regression page again you will get something okay some sort of a loss function is being used okay fine so in the case of decision trees again we are taking the sum of the squares of the error some some optimization is happening everywhere okay so if there are General comments uh I am I may not be able to see them if there are questions please try to post it in the Q a okay so that is where I can take it so that everyone sees what question I am answering okay so there is one request to everyone so if you have questions please try to use the Q a okay for classification problems again logistic regression uses binary cross entropy error I mean I am not going to explain what this is this is similar to your log loss okay so anyone who knows logistic regression definitely knows what is a binary cross entropy uh error that is one by n summation minus y log P minus 1 minus y log 1 minus P something where p is a probability that a data point belongs to the default class and that probability can be obtained from your sigmoid function okay so in a way that's a login simple support Vector classifier use the hinge loss I mean if you go to the documentation page know where it is mentioned unfortunately but if you go to the Wikipedia page you can easily see that your support Vector machines use okay s v m Wiki you can see that it uses okay so first I'm trying to you know taken away very clearly that what loss functions are being used because these are the loss functions or the cost functions which are going to be minimized okay so I'll just search for the word hinge h i n g e and you can see that word here loss okay so the support Vector machines use the Hindi loss okay that is the point we are trying to convey okay fine so coming back in deep learning architectures the good thing is you can create your own loss function uh and you can customize that loss function depending on the problem that you are solving okay and we have a you know big list of loss functions available okay you have a huge list of class functions available in deep learning so if you want I can show you for deep learning what all so I go to the Keras dot IO page okay and if you go to that page go to the API docs so you will see the list of all loss functions which are available okay so from here left side can you see on the left side uh losses are mentioned so I'm clicking on that losses and these are all the possible loss functions you can customize in your networks okay deep learning networks you want to use them solve probabilistic losses are there regression loss functions are there classification loss functions cosine similarity care Divergence binary cross entropy laws everything categorical cross entropy and everything right so these are all the possible loss functions popularly used in deep learning okay so I think by now everyone has a fair understanding that machine learning algorithms are going to use these loss functions to minimize the error so what is that error unless you minimize the error you will not get your model it is as simple as that you have to minimize some error and that error function is same as your loss function you put your error function is same as the loss function so there is no uh confusion on that right perfect okay so there are two more questions um is it same as meta heuristic I have a slide on heuristic well take that question later and uh there is one question by Anonymous attendee L1 loss cannot be minimized because it is non-differentiable right what you said is uh half correct they are not minimizable analytically but you can still minimize using iterative solvers okay so that also I'm going to discuss absolutely taken your point okay so obviously the next question is can any mathematical function be maximized or minimized what is your opinion I give a pause here what do you think I want your opinion in the chat box okay so what if what if my function is like this this is my function it is monotonously increasing so think about Y is equal to x square okay so sorry yeah so y equal to x square or think about Y is equal to e to the power of x do you think they will ever have any minimum these are monotonous functions so monotonous functions do not have any Maxima or minimum so not every function can be maximized or minimized okay so that should be very clear by now okay not every function that is the reason we are discussing about possible loss functions because those loss functions should have Maxima or minimum that is the important thing okay okay so let me further discuss about what a convex function is and then strictly convex and then we'll also talk about concave function so what is the convex function any function which has a Minima in a given range okay so you can have functions which can have multiple maximum Minima also so for example I'll just draw a function here maybe so this function has got one minimum and another Minima so in this range there is one minimum in the another range there is one more minimum so this is sort of a convex function but a strictly convex function will have exactly one minimum in the entire range of the X values right the one which I have drawn in this slide okay so that is your strictly convex function having exactly one point of minimum so in machine learning we prefer to have those kind of strictly convex functions because if you have multiple Maxima and Minima the problem is your algorithm might get might get stuck in these local maximum minimum which is what we do not want so this is the local Minima whereas this is a global minimum okay so these are the additional Concepts local minimum and Global minimum so such kind of loss functions are not advisable so we'd normally prefer to have strictly convex loss function which has exactly one Maxima or minimum right so in machine learning we prefer that the loss functions are not just convex but also differentiable so that was one of the questions raised by one of the attendees here so yes we prefer that the loss functions uh that we are discussing should be differentiable so that that derivative uh leads to the point of Minima so what happens to the derivative the at the point of Maximum Minima the derivative will have a what is the derivative derivative is nothing but a slope right so the slope of the intercept okay at the point of Maxima or Minima will be zero the slope of The Intercept at the point of Maxima or Minima will be 0 and that slope can be obtained by differentiating that function so that is why I talked about differentiability but as someone pointed that for example this is how the plot of the L1 loss function comes from Simply that's online okay and that is not strictly differentiable but still we can minimize that because we have some other ways of solving right so we'll see that so what about non-continuous functions those non-cardenous functions cannot be even discussed because they are completely logical I don't want uh my you know model to be built in a way that it is applicable only in certain scenarios so what is the point in making a model which is only applicable in certain scenarios because uh you know if the function is so the model can be built only where the loss function is continuous so you don't want to give you know conditions to your model so ideally we will never prefer a loss function which is not even continuous if it is not differentiable that problem can be solved but if it is not even continuous I would straight away discard that cross function so such a loss function is most useless is that clear everyone fine okay so I'll give you some more examples of the convex functions being applicable to machine learning okay so I have got a lot of questions uh okay I'll take them one by one okay so it seems people are super excited okay okay so uh some convex functions used in machine learning so the loss function of your lasso as we discussed some time back this loss function this is how the plot of that loss function mostly looks okay that is here sum of the squares of the error terms and this is a penalty parameter or the penalty term okay that is a convex function because it does have a Minima but it is not strictly differentiable but we can still minimize whereas the last loss in the case of Ridge is differentiable is differentiable and it is strictly convex so how does that look that looks like very simple curve I'll just clear the drawing and the loss function in the case of Ridge will look like this very simple thing okay so okay good and then we have the mean square error loss function already we have discussed in the case of linear regression is also strictly convex why is that MSC loss function strictly convex can anyone tell me why is that convex function a very simple question to everyone why do you think that MSC is a convex function why do you think MSC is a convex function and I'll give you a hint so let's see the understanding here I'll give you a hint okay so look at the formula for MSC loss function my question is why do you think MSC is a is a convex function first of all so can you see that it is a two degree polynomial first of all it's a two degree polynomial and any two degree polynomial is a parabola and the beauty of this Parabola is it does have a Minima okay right once again I repeat your mean square error function is a two degree polynomial first of all you have to understand it's a two degree polynomial you are taking the sum of the squares of the error terms so it's a two degree polynomial any two degree polynomial in general ax Square formula okay will give you a parabola a two degree polynomial will give you a parabola I can I will demonstrate this in the passing code just wait okay it is there in the next couple of slides so I'll take a generic equation of a two degree polynomial and I'll show you the plot it comes out as a parabola so since MSE is a two degree polynomial you can very fairly assume that it is a parabola with a minimum so this is clear so what is optimization finally and how is it different from any standard mathematical problem okay so any problem that finally deals with maximizing or minimizing any quantity that is your optimization okay and uh okay so can you see here I have drawn a two degree polynomial it's a two degree polynomial 2x Square minus 3x plus 1 and this is the exact plot and this is a snapshot coming from a Jupiter notebook okay so this is a two degree polynomial and you can very well see there is there is of course now if I want if I am asking you find the roots of this polynomial so what do you mean by The Roots Roots means at which point the polynomial is having so this f of x is having a value of zero so those are called as roots so you can clearly see that this polynomial will have two roots one is at 0.5 and the other is either one so this polynomial has got two roots and we know that formula okay x equal to minus B plus minus under root P Square minus four AC by 2 with that formula we know right by which we can calculate the roots of any two degree polynomial okay now finding the roots of any two degree polynomial is not an optimization problem why because you are not minimizing or maximizing anything you are trying to find out or you are trying to solve this equation so solving this equation is not equivalent to finding out the Minima so when I say that I am solving an optimization problem my solution is this and not this or this is that clear everyone so finding the roots or finding out the points where this function is becoming 0 is not an optimization problem that is how it is different from any standard mathematical problem okay if I ask you the roots of this polynomial after 1 million years the answer will still be 0.51 but if I ask you where is the Minima of this function under certain certain set of conditions okay then the value can I mean your final answer can keep changing every other day depending on what conditions I am imposing okay so that is how you know an optimization problem is very different from a standard mathematical problem where you are equating a function a function of x to 0 you are equating a function of H to 0 and then you solve it for the values of X where the f of x is becoming zero that's a standard mathematical problem mostly you deal with such kind of problems but in optimization we are talking about maximizing or minimizing that function okay perfect okay so this is how the mathematical version of that problem looks actually fine so here we are trying to find the minimum value of the f of x so you can call it this way so how do I how do I uh you know read this equation okay so listen to me carefully how do I read this so find the minimum value of f of x where f of x is equal to x square minus three X plus one so here your solution would be listen very carefully here this solution would be the minimum value of the so what is the minimum value of the function and you see on the y-axis the minimum value of the function would be somewhere here that is probably negative 0.125 that is the you know center point of point one and point one five so you will get somewhere here so that is the solution to this problem there is a solution to this problem now another type of immigration problem talks about the argument so how do I rate the second problem so the second problem is read as this find the argument X listen to me carefully and this is how your machine learning problems are so your machine learning problems are not of this type so machine learning problems your deep learning problems are of this type which is why you have to listen to me once okay so find the argument X for which f of x whatever it might be find the argument X for which f of x has a minimum value find the argument X for which f of x has a minimum value where f of x will be your actual loss function in the machine learning case or in the declining case that will be the actual loss function whatever that means so this f of x is your last function or cross function or objective function and this x becomes a decision variable that is the generic term but in machine learning this x is nothing but your model coefficients right so most of your machine learning problems are of the second time where you are not interested to know the minimum value of the mean square error rather my solution is to find out my problem is to find out where is this function having a minimum so that is the solution to this problem is that clear number so the solution to the second problem is the point of Minima I am interested to know where is that Minima occurring the solution to the first problem was this so these are the two types of uh you know mathematical sorry optimization problems the way you formulate them okay perfect so someone has already posted that x equal to three point three by four zero point seven five very good so what you did was to differentiate that function equate it to zero derivative of that function is 4X minus 3 and once you equate it to zero means you are saying that the slope at the point of Minima will be zero so the derivative represents the slope so 4X minus 3 is the derivative equated to zero solve you will get x equal to three by four absolutely thank you thank you okay I don't want to get into too much mathematical stuff but the bare minimum things should be still there so finally what are the different types of optimization problems okay we have constraint optimization we have unconstrained optimization okay so unconstrained optimization one example is the case of linear regression that is exactly what we have been discussing till now okay so in the case of linear regression we want to minimize this loss function what is this loss function this loss function is your main square error loss function so this loss function will be a function of your model coefficients W so we want to minimize this mean square error loss function with respect to which variables your variables are not X your variables are the model coefficients those slopes and intercepts right so how do I read this how do I read this so find the model coefficients these are the arguments so find the arguments you can say or find the model coefficients for which your mean square error loss function is minus given what is that vertical line representing given and X Y is your data so for a given data give me the optimal values of the model coefficients where the minimum of the mean square error function occurs clear where the minimum of the mean square error function occurs so that is exactly what you are trying to Define in the case of a linear regression okay perfect the other case is a constrained optimization so what you are saying is I've just given a simple uh optimization problem or W1 okay right now so how do I read this so minimize X1 minus X2 whole square plus x 2 minus X1 whole square whatever so there are two variables here subject to some constraints so what you are saying is uh these constraints have to be followed so there can be hard constraints there can be soft constraints and there can also be bounds so depending on the constraints and the bounds you define something called as feasible region to define something called a squeezable region and your solution which is the optimal value will lie in this region okay what do I mean by that I'll give you a simple example so if I go back to my previous slide this one if I want I can introduce one constraint so here I say find the minimum value of this function subject to a constraint here that your value of x must be greater than or equal to 0.0 and 0.5 so so this is your constraint you are saying that subject to this constraint so now whatever constraint you have introduced as per this constraint your optimal value of x cannot exceed 0.5 you see in this case your solution the feasible region is this so this is a feasible region this is the physical dream right this is a feasible Vision that's it so this is no longer Your solution exactly this is no longer Your solution Your solution will lie in the feasible region that is exactly what I was trying to tell you so every other day you keep changing the constraint although the main problem is same but the solution can keep changing depending on what constraints you are imposing okay so now the next question is is there a scenario where we get constrained optimization in machine learning anyone anyone is there any machine learning algorithm where we encounter constrained optimization anyone anyone constraint optimization lasso rich is actually not a case of constraint optimization Okay so answer is your support Vector machines svmo okay okay support Vector machines okay so I think I had that page open I quickly go to that so you have the constraints uh I can just show you okay can you see this subject to these constraints so these are the constraints I'm not going to explain what this okay but you want to find the maximum separating hyperplane okay so you want to find out this hyperplane which has the maximum margin you want to find the hyperplane which has the maximum margin but subject to these two constraints so these are the constraints that you have to satisfy so this is what you want to minimize can you see you want to minimize this but subject to this that is exactly the price clearly written in your spectral machines okay so support Vector machine is a case of constrained optimization clip so wherever you get subject to means you have imposed a constraint over there clear now there can be hard constraints there can be soft constraints okay so Hardware strain means that that constraint has to be followed whatever happens and if you and there can be conflicting constraints also and if those conflicting constraints are hard constraints you may not end up with any solution so that is why you can you know impose a soft constraint also okay or if the data is not linearly separable then you have to use something called external trick which is or you can go with a soft margin variant so what is the soft margin variant that is where you are removing the heart constraint not exactly removing what but so not get into all the details but that only take away from this slide is yes support vectoral machine is the case of constraint optimization if you understand that much understood right that is sufficient so yes now uh what are the different approaches to solve optimization problems there are two big approaches one is our analytical solution what we also called as ordinary least quiz and the other is iterative solution so someone had us gradient descent somewhere okay so your iterative Solutions are your gradient okay how to choose which one so see all the optimization problems cannot be solved on pen and paper so theoretical solution is someone solved that problem you remember someone solved in the chat box by differentiating that function 4X minus 3 equal to 0 okay someone solve that problem and you got that optimal value that is exactly analytical solution so one actually solved that right three by four perfect that is exactly your analytical solution which I am referring to which will give you the exact solution but the only problem here is not all types of optimization problems can be really solved most of the real world problems the actual business problem the actual data problems all the problems that you get on kaggle on your different companies everywhere they cannot be solved no no analytical solution can exist okay so for that you need a different approach and that is where we have the iterative solution okay so this is going to give you an approximate solution but it can solve all types of optimization problems that is what I have clearly written overlays right okay before I move ahead okay there are already short of time why the loss function should be continuous ah okay maybe I'll take that at the end uh I try to explain that can we have concave function to be used the concave function C uh the only problem with the concave function is you can have a concave function no problem suppose you want to maximize the profit of a company so that's the maximization problem right but the solvers which are available in Python and Commercial solvers they are hard-coded they are made in a way to solve minimization problem only that is the only problem so you have to convert your maximization problem into a minimization one and how do you do that that's already done here if you see you wanted to maximize this margin this margin was supposed to be maximized can you see somewhere you want to maximize this margin right this margin is given by 2 over Norm of w so where it is mentioned this margin is given by two more so this is the total margin you want to maximize but instead of maximizing this you take the inverse of that function and minimize so some trick has to be applied either you take the negative of that function and minimize or you take the inverse of that function and the device but the point I'm trying to make is all these solvers available on the internet on all the softwares commercial solvers every one each and every one can solve only minimization problem okay so you have to convert a maximization problem into a minimization and that's what is done here how do you realize why are we minimizing Norm of w instead of maximizing 2 over Norm of w we just took the inverse so instead of maximizing that quantity we minimized the inverse of that function right perfect so that question is also answered and what else would we able to download I'll give you a link where most of this information is available okay so don't worry and please can we get the slides or record a session is already recorded So that is already fine so let me go ahead okay we have another 10 slides to cover and a small python code okay so how do we finally solve okay as I just now mentioned we have the analytical method we have the iterative method okay so analytical method is covered in your linear regression class everyone knows the linear regression class right and the numpy qualified function I'll demonstrate this right away okay and the iterative solutions what do you do in a iterative solution you start with some guess value okay and keep going downhill that's what exactly does right so I'm not going to explain what is gradient descent but everyone knows what gradient design is all about right so you pick up some you pick up some so this is you pick up some uh you know guest value and slightly keep on iterating till you reach downhill right that's what the gradient present tries to do you are descending downhill so you start with some guess value and you know find out the value of the loss function and then update the weights okay and using those updated weight again you get the new predictions so that is your pretty decent and then you have variance also for that gradient like stochastic gradient is in many bad ingredient is an advanced gradient design techniques are also there which are used in deep learning let me show you that so if I go to this link called as optimizers so what is an Optimizer quickly what is an optimizer common sense now everyone should be able to answer this what is an optimizer so Optimizer is the algorithm which is solving the minimization problem Optimizer is the algorithm which is finding the Minima that is an Optimizer right so the scikit-learn API refers to this by the name as solver okay remember so we have different names again so circuit learn refers to this as solver the Sci-Fi API refers to this as a solver but the Keras in deep learning we refer this by the name as Optimizer okay so Optimizer so again I can just quickly show you these are all the available optimizes but what are these functions trying to do they are again some variant of your gradient descent sdd is your stochastic gradient descent again atom is adaptive momentum adaptive Delta at a grad adaptive gradient that is atomax medium that is nestero adaptive gradient descent again and then many more are there right so these are all variance of your same gradient descent but a bit more advanced because they have the momentum terms and they are adaptive okay so these are all different algorithms which of course are not going to get into all the details but one of the story is while you are training your neural networks you are trying to minimize some MSE loss function or binary cross entropy loss function for categorical cross entropy loss function some loss function you would have taken while defining the neural network and compiling you remember anyone remember pollution step you are defining this Optimizer right so exactly okay coming back okay uh you have some heuristic solvers and meta heuristic solvers again so like genetic algorithms which are evolutionary algorithms based particle swarm optimization and Colony simulated anything so anyway maybe we can have a separate session on these heuristics logos which are actually very very interesting and they really solve a lot of Industry problems okay uh in sci-fi we have the nelder made Power lb of GSS limited memory Broad and Fletcher bullfrog Channel algorithm sag is a stochastic average gradient descent and all that so I'll quickly do a quick demo and then we are good to go uh okay fine so I'll take another five seven minutes will that be okay the moderator 5 to 10 minutes not more than just a fine fine yeah yes because anyway we started 10 minutes late so I'm counting my time okay thank you yeah okay because I don't want to skip the last seven inch slides which is the most important okay because there I am discussing the real application part they uh you know walk around okay good thank you so now that we know that we have so many solvers and these are the two approaches okay now the question here is I have okay let me show you one more page one more page before I go ahead let me show you one more page sci-fi dot optimize dot meaning minus let me show you this page very interesting page so all the optimization routines whatever I mentioned in my last slide are present here okay and most concrete machine learning algorithms are calling this function behind the scenes you may not realize this even the logistic regression let me show you the logistic regression okay even the logistic regression or the scikit learn does this so it does have the solver I'll just show you so can you see this solver option lb of GS where is this lb of GS coming from I'll show you this LPF GSS coming from here lbfgs limited memory Broad and pleasure algorithm okay so now in this page you will get all the list of solvers available for machine learning so these are all the followers and if you go down here in the logistic regression page you have the choice of all these followers the Newton's conjugate gradient lvfgs live linear sag Saga and all that okay so you are only specifying but where are these actually hard coded where are these coming from they're coming from this function so this function is being used behind the scenes so again what do you see logistic regulation the simplest mlr statistical algorithm that you have been knowing for classification is also using optimization behind the scenes you see now further the documentation page is very good actually you should actually read the documentation however habit of reading documentation this clearly tells you that these are the algorithms available for unconstrained optimization so you can use the conjugate gradient you can use the pfgs method you can use the Newton's conjugate gradient dog leg press regions Newtons conjugate gradient crylock method and so many and then you have the bounded and constrained minimization so that is where you can use the nether mid which uses the Simplex algorithm you can use lb of GS you can use the powers method and so on and for other kind of constraint optimization you can use your slsqp also that is a sequential and all that so this solves majority of your problem which algorithm to use or what kind of problems is partly solved because it is very clearly mentioned what to use when and if you're still struggling for the case of logistic regression is very clearly mentioned that all different followers don't support penalty so if you are deciding that I have to go for L1 penalty then your choice is limited annual penalty is uh supported only by lib linear and Saga if you are going for L2 penalty which is the default then you can select any of these so the default penalty is L2 if you see and there is very clearly mentioned here regularization has been applied by default what regularization L2 regularization is already applied by default here okay so which means we have to use LPF GS so if you want to apply L1 penalty then you have to change this because ldfgs does not support L1 penalty so the choice of the solver depends on many things number one it depends on what kind of constraint you have okay whether do you have a constraint altogether or not whether it is a linear constrained non-linear constraint whether you have an equality constrained inequality constraint okay and then whether what type of regularization you want to use that also determines what kind of solver you want to use yeah does that give you a fair enough idea now is the optimization a part of tuning yes I do have one uh comment in one of the slides about hyper parameter tuning which also involves um optimization absolutely absolutely so coming back to this problem okay uh I'll just uh go back to so what I did was this is my MSC loss function this is my MSC loss function and Y hat I had defined it to be a w naught plus W One X so if I substitute this will be carefully if I substitute this y hat in this expression then the total loss function looks like this the net loss function looks like this this is a net loss function okay after a substitute can you see yeah can you see this is your y hat now clearly what see what I want to conclude from here listen to me carefully what do I want to conclude your net loss function is not a function of your features because features is your data data is constant the loss function is not a function of your y the loss function is a function of your model coefficient these are the actual variables and that is why I said your loss function is a function of your model coefficients which are the actual variables X and Y is your data my dear X and Y is your data and data is constant you can't change the data to fit a line a hypothetical line right you have to change the line to fit the data please understand X and Y is your data data is always a constant so what is the actual variable in this the actual variable is your model coefficients that is why your loss function is a function of model coefficients now that is clear so what did I do I plotted my last function with respect to the model coefficients okay I plotted my last concept with respect to the model coefficients right and then when you solve all that this is the final thing that I get Okay so how do I use this equation I have one slide where I am actually going to use this equation this was my dummy data and uh I just plug in the values I just plug in the values so what is X Y Bar X Y Bar is basically the mean of X Y values X bar into y bar that is the product of the mean of X values and the Y values then the denominator term here is the mean of the square values of X and the other term is the square of the mean so if I just plug in the values I'll get some optimal values we can see 2.67 into 0.88 so let me switch to the python code to the python code okay and demonstrate this okay so first I have just taken the same thing same three data points as an Empire array can you see I have taken my data points X and Y powerful session thank you thank you John Okay so this is my data points I have taken is an Empire array and what I am doing is a simple polynomial fitting so this qualified function can fit a one-dimensional data only one dimensional data please remember and after that it will return you the polynomial coefficients I am converting those polynomial coefficients into a polynomial object which I can evaluate later fine so can you see two point six seven five two point eight seven five can I go back to the code sorry slide 2.67 and 2.88 perfect it matches exactly it matches what does that mean this is the exact equation being used by this function this is the exact equation I'm telling you the exact equation being used most people don't know what is being used behind what equation okay let me ask you so most of the times people think that the linear regression gradient descent is used I can tell you that is absolute nonsense linear regression does not use ptn basic mind here it uses ordinary least squares and if you're still wondering why am I learning all this I'll just show you quickly do I have the linear regression page no okay I'll just show you the linear regression page and unfortunately there is no details there is no equation it only tells you I am using ordinarily squares and people have literally no idea literally people have no idea what is OLS nowhere in this entire page any expression any equation for OLS is there there is no expression for OLS in the entire circuit learn API because the creators of this API assume that you guys know what is OLS okay this could be a bit critically I mean fortunately they assume that you know what is the wireless okay whereas most people don't even know what is optimization forget about knowing even all this okay and still many many people I encounter even they think that linear regression uses gradient descent absolute nonsense it cannot be more wrong that than that okay so if you're still wondering why am I giving these equation is the equation used behind the scenes right perfect okay uh before I go to the last slide I'll go back to my python code once again so you see that this is the polynomial coefficients I used and then I am using the linear regression class from your scikit-learn this is our X and Y once again okay I am doing the model fit everyone knows this right and see The Intercept is 2.875 and 2.6 perfect the same value is what I got from here 2.675 2.875 so what is the model of the story here more of the story is my linear regression actually uses ordinary least squares and what is the expression for the ordinary displays my dear this is the expression convinced everyone okay fine so we are almost nearing uh completion but before I do that I'll give you some more uh applications so let me go with that function so I want to minimize that function which function 2x Square minus 3x plus 1 the same function which I used in my slides remember someone had already solved 0.75 now I am going to solve you solve for you that same expression so what I defined as a Lambda expression the same function which I want to minimize then from PSI Pi dot optimize import minimize do you see I mean using the same function so it's not just Theory I am doing the Practical now so this was the same web page I'm using that same function now so you are minimizing this function with some guess values what is X naught here you have to provide some rough guess where the minimal lines okay and this can be obtained from the plot we had seen from the plot we got a graph value at 0.7 something right see the solution here absolutely 0.74999999 and what did I tell you sometime but that these are iterative solvers and they will only give you approximate solution did you notice and that is the reason you do not have the answer as 0.75 is that clear that is the reason if you are still wondering and pondering why is my answer not 0.75 because I have already declared that these are iterative solvers how many iterations did it take by dear two iterations how many times the functions was evaluated six times the function was evaluated okay number of jacobians jacobians are nothing but derivatives so three times the function was differentiated okay and this is the minimum value of the function what is Hessian this is the double derivative of the function you need the double derivative to prove that the value that you have got is indeed a point of midi Mark so let me go back to high school mathematics so the first derivative will tell you the point of maximum or minimum that's what everyone knows but the second derivative of the function will tell you whether that point is indeed a Maxima or minimum right so if you evaluate the second derivative at the point of maximum or Minima and if you get a positive number it relates to a Minima and if you get a negative number you get a Maxima so going back to that function if you double differentiate what do you get 4 right if we double differentiate this function you get 4 what is the first derivative 4X minus 3 now the first derivative is 4 4X minus 3 so the double derivative is only 4 and what is the inverse of that Port what is the inverse of that Port 0.25 exactly and that is positive so what is HCl ACN is nothing but Hessian is nothing but your double derivative and what what psychic learn is printing is the inverse of that so one is one upon 4 0.25 and that is positive so indeed we know that we are talking about a Minima indeed we are talking about okay finally there is one more quantity called as Jacobian so this is your value of the function uh sorry value of your derivative this is the value of your derivative at the point of proposed Minima or hypothesized Minima at this Minima what is the value of the derivative and this is the value of that derivative and you see the value of the derivative represents the slope and the slope is fairly close to the not exactly zero but fairly close to zero is that clear so Jacobian represents the slope basically Jacobian represents the slope at the point of maximum or minimum so the slope the slope is nothing but a derivative so Jacobian is the derivative the final Jacobian is your derivative of your function at the point of Max more minimum right and that is supposed to be close to zero within some tolerance there is some tolerance yeah I don't want to go into all that but there is some tolerance if you see there is some tolerance also there is some tolerance okay there is some default value of that tolerance so depending on what tolerance you are imposing okay this value can increase or decrease okay so moral of the story this tells you that our derivative is purely close to zero which means this solution is reliable okay any question so what is the function value telling a function value is telling you what is the value of the function this is if you are talking about the mean square error function this will be the minimum value of the function what is the minimum value of the function so this is the minimum value of the function zero point negative 0.125 on the y axis and on the x axis it is 0.75 is that clear now on the x axis it is 0.75 that is the location of the minimum that is the location of the Minima and the value of the function at the point of Minima is negative 0.125 perfect okay so uh there is if you want you can change the method if you want you can change the method okay and in this case what happened if we change the method uh I did get 0.75 in this case very interesting okay I'll not get into all the explanations but this time the function took 24 iterations so it's up to you in just two iterations you want very very close or do you want to go to 24 iterations and actually reach the point of minimum so depending on all these different techniques you can just try them okay so you can just find each of these techniques and see what values you get okay okay so let me just take up the last two slides and we are good to go okay so I already discussed about this these are the other small words discussed about this okay so where else I think I have fairly given you a good idea okay I don't want to repeat this support Vector machines deep learning hyper parameter so someone had this question so yes indeed the hyper parameters can be optimized by the concept of um optimization again now whether you are talking the upper parameters of a machine learning algorithm line suppose for support Vector machine you want to go with what type of terminal you have different choice of Kernel linear RBF polynomial you can optimize the kernel or if you want to go with a polynomial then you want to optimize the polynomial degree you want to optimize the value of C what is your regularization parameter or you want to optimize the value of gamma that influences the amount of uh influence each data point will have on the decision boundary right that is what the gamma can be using so each of these hyper parameters for every machine learning algorithm can be optimized by using some techniques so normally we use grid search right uh so what are the other applications now this is where I am falling short of time so what I'll give you is uh because of the interest of time I give you a link I give you a link okay and in that link I have given at least 30 applications of optimization which are not machine learning based will that be okay instead of me explaining each of them guys yes can I do that perfect so if that works then um I think I can just a minute I'll give you a link of that um and then we can wind up I'm almost done that's the last slide anyway okay so if you go to the analytics video blog web page this is the main web page let me share this link with you okay [Music] with everyone and if you see this article this summarizes everything including the code everything whatever I have discussed today is all there so you don't need my slides at all all these slides have been created from here everything in detail is very neatly nicely explained written by me and published today for you guys especially just because I wanted to take this session I wrote this article for you people okay thank you for your time and as I said all the codes are there you can copy paste the code and try it by yourself so whatever the code that I've been running is also there where else is optimization used I have given you at least 24 applications right so I'll give you the link for this article please go through it it's all yours download this article and please make use of this so everything you don't need my slide anymore it's all there okay so I'll just paste the link for this article on the chat box please mark this bookmark this okay so it's all yours it's all yours finally um these are some commercial solvers which are also mentioned in that article and this is my final slide thank you so much for your time let me take a couple of questions okay and yes you can scan that code that will directly take you to my LinkedIn page you can use your Mobile uh scanner I mean the LinkedIn app has got a scanner you can use that LinkedIn app scanner connect with me okay so hello Prashant yes before we proceed to answer your questions I would like to request the attendees to please fill in the poll about feedback as it help us to conduct more successions so we have people 10 percent from Africa another 10 person from Europe seventy percent are from Asia but we do have participants from North America and South America as well so none from Australia so that is the result for report thank you and uh the other question was on the feedback [Music] um okay so most of the people have zero to three years of experience 65 percent good which is good and another feedback has sorry poll has come up please please I request everyone all the remaining 81 participants to give a feedback okay I'm waiting and then I'll take some three questions are still painting in the Q a so if you guys are still there I am ready to answer those three more questions okay so most of these questions are actually answered somewhere in this article but definitely if you think still there is a need for a discussion I'm still there yes we can continue yeah thank you thank you so one or two questions I'll take and any book recommendations absolutely a very good question and um please allow me uh this is one super book I would recommend everyone to follow this book by Professor Singh ratio s Rao on engineering optimization this you can easily get on Amazon okay a valley publication uh Indian audience can easily get this book okay Indian audience especially okay most of the optimization courses conducted in top tier one institutes like iit's entities use this book as a textbook fine for the international audience I would like to suggest another book ml ml p l e a u Edgar an email below that is one of the standard textbook across most of the universities across this planet Earth I'm not aware of the optimization books being followed on the planet Mars but at least on this planet any course on optimization cannot be complete without following this book on Edgar and human okay uh where is this book not coming up I'm sorry let me do a Google search [Music] so the name of the book is optimization of chemical processes okay although the name says it is specifically for chemical processes but the first 20 chapters are absolutely awesome okay they are pretty generic okay so yes you can easily get the PDF this is the name of the book okay so optimization of chemical processes by Thomas Edgar and David human blog okay I probably misspelled so this is one where you will easily get this book in your libraries if you're coming from academic background okay otherwise you can also catch hold of the other book by saying the ratio as raw access now yes yes now just search SS now in optimization you will easily get that book okay so these are absolute Beauty okay fantastic books to follow so that answers the question um kindly share the book name uh yes this is the book I have just answered that Shashi and uh should I try to optimize coefficients to get optimization see anyways your machine learning algorithm doing that for you for you okay okay that is what the machine learning algorithm is already doing for you so you do not have to explicitly break your head with these minimizers unless you have a physical problem that you are trying to solve so some of these physical problems I have already you know conveyed in that article if you go to My article some of these physical problems I have very clearly discussed okay so if you are facing some of these physical problems okay let's say you want to minimize the cost of the uh manufacturing of product so you can express that cost in the form of a Manpower cost raw material cost inventory cost Transportation cost logic so once you have all the numbers you can make a mathematical expression and want to minimize that for that you will have to use the minimizer so I have already shown you sci-fi dot minimize function right so that is what people are doing otherwise if your problem is just two complex you can make use of some of these commercial solvers and especially I should mention about or tools this is a Google's product which is open source okay so fantastic optimization tool is this so that is also thank you Lena thank you so much Lena had already our husband question why should the loss function break the bacterias I have already answered I think and uh there is one last question is it the same as meta see optimizations have many you know formats heuristic but are heuristic and these are you know straightforward optimization so I mean there are multiple application area so everywhere you will not get the same standard optimizations okay so I have tried to summarize what are the cases of straight forward optimizations but there are cases where you would indeed use genetic algorithms simulated let's say some part of my PhD thesis also I have used genetic algorithms and stimulated anything in particle swarm optimization perfect so follow this article it has all the things that we have done for the day and thank you so much [Music]