The Jacobian : Data Science Basics

Key Takeaways

[Music] hey everyone welcome back so today we're going to be talking about a cool mathematical object called the jacobian so it has one of those scary sounding names but i promise it's not too difficult let's start really easy with a function of one variable so we have f of x is equal to x squared if i asked you to take the derivative of this guy it should be pretty simple we just do df dx is equal to 2x pretty simple let's go up one level let's visit multi-variable calculus where you learn that we can put not just one thing into a function but we can put several variables into a function so here's one example we have f of not just x but now it's x1 and x2 and that function is given by this guy over here not too difficult to extend this now we can call these partial derivatives instead of just derivatives so they have this delta symbol here and we ask about what's the derivative of that function with respect to each of its inputs so we call it partial of f with respect to x1 is this guy and partial of f with respect to x2 is this guy so still nothing too crazy going on now the jacobian all it does is just extends this one level further because what we saw here we still had one function total it's just that that function was now inputting several variables now the jacobian asks what if we have several functions each of which is a function of different variables so here's an explicit example of that so we have f of x1 and x2 so this notation here by the way is the exact same as this notation here you can really pick either one i just chose this notation to say that hey this is a function of a vector or in other words a function of multiple variables so that's just a notational thing so we have this function called f which takes in several variables here just two variables but the the difference here between this and this case is that this function f is really two functions so it's a function that takes in a vector x1 and x2 and it also outputs a vector which is f1 of x1 x2 and f2 of x1 x2 and explicitly it's given here that's the mathematical formulation so let me just backtrack and make sure everyone's on the same page here we have two variables x1 and x2 the first thing we do is put them into this top function f1 and we get this result here the next thing we do is take those two variables put them in the bottom function and we get this result here so that this function f its job or its objective is to take in two variables x1 and x2 and also output two variables namely this guy and this guy or f1 and f2 so now that we're all on the same page about what's going on here what does it mean to take derivatives in this case well we can just extend directly from this example because now we have two variables and two functions so there should be two times two or four possible derivatives to get in this case and that's exactly what's collected in this table here or matrix that's probably the better way to think about it so explicitly we have the partial of the first function this guy with respect to the first variable the partial of the first function with respect to the second variable the partial of the second function with respect to the first variable and the partial of the second function with respect to the second variable and we can just work them all out and it looks like this so hopefully i remember my calculus well enough to have that correct but we have this two by two matrix which collects all the possible derivatives and this two by two matrix is what's called the jacobian so our notation is j for jacobian we subscript the name of the function which is f and we explicitly put in all the variables that were used here x1 and x2 so you'll see different notations for this guy but the jacobian in a nutshell is just taking all possible derivative combinations based on how many functions you have and how many variables you have and collects it into this nice neat matrix now that we hopefully understand what the jacobian is we are going to talk about the most important part of this video which is why is the jacobian useful in data science or stats of course it's cool to look at these things but i personally don't think that there is any value if it's purely theoretical i think that it needs to be used for something in data science so what is it used for in data science the best example and the most buzz wordy one i could think of is in the context of neural networks where it's pretty important so let's take the rest of this video and talk about that so here's a neural network i have a whole video on neural networks it would be good to watch that but it's not necessary i can walk you through what's basically going on here so there's one input x so to be clear here x is just a single number it's not a vector so we have the single number x and that's called our input and some stuff is going to happen and i'll describe that stuff to you but at the end of the day all that stuff leads to another single number at the end of this neural network and that's called p so i call it p because a lot of times you'll be using a neural network to try and figure out probabilities of some event happening that's just one application so we have some input x which is one number and we have some output p which is another number so our goal and everything we're about to talk about is going to be try to figure out what is dp dx in other words if i change x by a little bit what is the effect that has on the final output p to do that we're going to need to better understand what goes on in the middle because obviously we're not going directly from x to p there's some machinery in the middle going on so let's talk about it step by step that's just a three step process the first step is that this x goes to the first hidden layer which consists of two numbers a1 and a2 so by the way these plus ones are not variables they're just there to represent the bias term which won't really even be that important to the model i just wanted to do this justice so the first layer is a1 and a2 which is given here so this vector a1 and a2 a1 and a2 are both functions of a single variable x so if you look back to the previous layer before this the only input coming in is x and so they're explicitly given by this which are just linear functions of x so to be explicit a1 is a function of x which looks like some coefficient times x plus this c is just a bias or coefficient term a constant number a2 is also just a function of that same variable x and it's given by this coefficient term times x plus different bias term now if i ask you what's the jacobian of this guy it's actually even easier than the example we did here because these are all linear functions but let's go through it explicitly just to be sure so if i ask you what's the jacobian of this function here it's a function of one variable and it has two functions in it so the jacobian's dimensions are going to be two by one it's going to be important to talk about the dimensions of the jacobian in this process so explicitly how we get this first number is we take the derivative of the first function with respect to the first and only variable so that's the derivative of this guy with respect to x that's simply just this coefficient term here exact same thing for the second coefficient term here so that's the jacobian of the first part of the process now let's go to the next part of the process so we go from the first hidden layer to the next hidden layer a little bit more complicated so now we have this new vector b1 b2 which are these two numbers here and b1 and b2 are both now functions of two variables so you know that by looking at the diagram if you look at b1 there's two variables coming into it if you look at b2 there's two variables coming into that so this is the same dimensional case as we are looking at here two functions two variables which means that the size of our jacobian is going to be two by two and we can go through all the math you can do the same derivative calculations but you'll find the jacobian in this case looks like this so very similar to up here where it's just collecting all these coefficient terms so we have a two by two jacobian here and let's go to the last part of the process the last part of the process is going from the second hidden layer to the final output that's even easier than the one we just looked at because that's just a function of two variables and there's one function one output which is p so explicitly that looks like this we can take the jacobian of this step two in this case there is one function of two variables so the jacobian is going to be one by two and it looks like this so take a minute pause rewind write some stuff on a piece of paper make sure you are convinced that these jacobians are correct make sure that this architecture roughly makes sense and now let's revisit our goal which is trying to figure out how does a small change in x which is our input affect the output the final output which is p now if we weren't using jacobians if we were trying to do this by hand it could get a little bit cumbersome and a little bit messy because we have to think about every possible path there is to get from x to our final output p and in this case there's four paths which isn't a lot so explicitly those four paths are going to a1 then b1 then p a1 to b2 then p a2 to b1 then p or a2 to b2 to p so there's four paths but you can imagine as your network gets longer and as you get more nodes in each layer there's going to be many many paths it can be really hard to keep track of all that and then we have to add up the results of those derivatives together in order to get our final result now here is the magical part the shortcut if we choose to use jacobians we can get that same exact number dp dx simply by just multiplying together the three jacobians we've collected along the way so if we do j3 times j2 times j1 let's make sure the dimensionality is correct so it's a one by two times a two by two times the two by one so the final result is a one by one scalar which is exactly what we were expecting and the awesome thing is that this scalar is exactly the result we're looking for which is dp dx an intuitive way to think about it is that these jacobians were collecting the derivatives at each part of the process so it should make some intuitive sense that if we multiply these derivatives together which is kind of like tracing through all the possible paths through this network that we're going to get exactly the result that we're looking for so this is one awesome application of jacobians and so they're not just a cool notational thing they are something that gets used a lot in order for us to calculate these gradients a lot more efficiently and a lot more reliably so hopefully you learned about jacobians in this video if you did please like and subscribe any questions in the comments below are welcome and i will see you next time