Singular Values vs. Eigenvalues : Data Science Basics

Key Takeaways

[Music] hey guys welcome back this is gonna be a really brief follow-up video from our singular value decomposition video where I'm going to tie together the notion of singular values and eigen values singular vectors and eigen vectors because I think that it's really helpful to know that the thing you just learned is not some completely new invention of math or stats but it's basically just a renaming or reformulations for something you already know so let's start from where we left off which is that the SVD of any M by n matrix M is given by u Sigma V transpose and we learned that each of these components has special properties U and V are orthonormal matrices which means that they contain vectors which are linearly independent to each other and Sigma is a diagonal matrix P by P containing the singular values and also U and V contains singular vectors both left and right let's next consider some transformations of m so let's consider M transpose m and m m transpose so these are both valid multiplications and the resulting dimensionality is n by N in this case and M by M in this case so both of these things are square matrices that's going to help us get into the terrain of eigen values now let's think about what would happen if I took M transpose m that matrix and multiplied it by V so what I have exactly done here so M transpose M which is the matrix we were talking about earlier multiplied by V V again being the matrix here containing the right singular vectors of our original matrix M so let's just go through this calculation so we can actually open up this M transpose and M since we know it's literal form right here so if I were to take the transpose of this guy I'm gonna get V Sigma transpose u transpose and then writing em just as you see it here we get u Sigma V transpose and of course all that which is the expression of m transpose M gets multiplied by V now we can do lots of nice cancellations because of the properties we talked about earlier namely this guy V transpose V + u transpose u both are equal to the identity matrix because of the fact that U and V are orthonormal so that's how I get to this step which is this first V Sigma transpose which is all that's left we can do another simplification because Sigma is a diagonal matrix which means its transpose is equal to itself so really we're just gonna have V Sigma transpose it looks like we didn't do anything very crucial but let's look at this in a different form opening up V so we can really see what happened so we see we have M transpose M times V so this was the original thing that we had written up here just with the V matrix now written in its full form with all of its P vectors and what we have on this side basically if we were to take this Sigma squared which is basically a diagonal matrix where you square each of the singular values on the diagonal then we can write it in this form so Sigma 1 squared V 1 o and all the way to Sigma P squared V P so what this means if you look at this this is saying that take this matrix times V 1 and you get back Sigma 1 squared V 1 take this matrix times V P and you get back Sigma P squared V P which means that V 1 through VPN vectors of M transpose m and their corresponding eigenvalues are Sigma 1 squared all the way to Sigma P squared so just to say that in a different way v1 through v4 we were just calling the right singular vectors can also be thought of as I invectives not of M itself but of M transpose m and Sigma 1 through Sigma P which we were calling these singular values up until now also we can call the square of them the eigenvalues of m transpose m so that's this powerful link between singular values and eigen values they're not really two separate entities but they're linked in this way and we can do a very similar thing for you I won't go through the calculation because it's the same but we get the same conclusion which is that u 1 through u P which again are the component vectors that make up matrix u are the eigenvectors of mm transpose which is this other guy and the eigenvalues are also the square of all the singular values and to finish this video let's just look at this in a matrix form so we know that M transpose M times V is equal to V times Sigma squared that's literally what we derived right here so now if we just give these guys separate names we can call em transpose M just a V we'll just keep calling V and let's call Sigma squared as lambda now get back at the same exact form as the eigen decomposition that we looked at before so the crucial takeaway in this video is basically that the singular value decomposition and the much simpler eigen decomposition are linked in a very very intimate way so if you have any questions at all please leave them in the comments below like and subscribe for more videos just like this and I'll see you next time