Variational Autoencoder [VAE] from scratch | Intuition + Coding

Key Takeaways

Welcome back to a new lecture. Today we'll be discussing about variational autoenccoders or VAE. So the context of taking up this topic at this point of this course is twofold. One is um as a standalone architecture VA is very important. So even as somebody who has nothing to do with vision or multimodel large language models, VA as a separate topic in itself it's it's very significant for you to understand. It teaches you a lot of things. Second thing is if you want to understand deep generative models like stable diffusion or even the diffusion model which was originally proposed before stable diffusion it's very important that you understand variational autoenccoders in the first place. So today's lecture I have divided into two parts. First we will try to understand the architecture of variational encoder autoenccoder and what it does and in the second part we will be coding a VA from scratch. Coding wise in my opinion today's lecture will be relatively easy for you because you have already coded much more complex things as part of the transformers for vision series. So today's coding will be relatively straightforward but we can do a little bit of experimentation with the model itself to see what are some different outputs that results from our different uh let's say hyperparameters or latent space dimension etc. So we'll discuss all of that. So first thing is I had posted a video about autoenccoder. So I recommend you to have a look at it. But in any case I will give you a quick recap on autoenccoder which is easy to understand. So autoenccoder has this following architecture. It's fairly straightforward. It has an encoder. It has a decoder and what the encoder does is if you have a data like an input image encoder will compress it into a space called as the latent space. So latent space has number of dimensions which is much less than the original number of dimensions in your input data. So if you had a 28x 28 image you will have around 784 pixels right and each of these pixels you can consider as one dimension. So overall here the data is residing in a 784 dimensional space. But the latent space will have a much smaller number of dimensions. It can be 10, it can be 64, whatever. But the number will be much smaller than your input embedding dimension. And the purpose of this latent space is to sort of represent the same information which is contained in your input data in a much more compressed format. And how do we know if this compression is valid? To know if this is valid, the only thing you need to do is try to decompress it. So using the 64 or 10-dimensional representation of your data in the latent space, use a decoder to reconstruct your original image. If your reconstruction looks very close to your input data, it means your latent space is able to represent your input in a much more compressed fashion. So this is the basic idea behind autoenccoder. Now how does the encoder and decoder look like in terms of architecture? It will be bunch of neurons in a fully connected neural network. So usually if you have an input image of 28x 28 pixels, you will convert this into a layer of flattened uh pixels. So this will be 784 and then the flattened layer will be projected into let's say 256 dimensions and you can project this 256 dimension after some rail activation. Um so let me draw activation like this. After this you can project it to let's say 64 dimensions and this will be your latent space and then after each layer you will have radio activation then project it back to 256. Then project it again back into your original dimensionality space. So this part you will call as encoder. This middle part you will call as latent space. The second half you will call as decoder. This same architecture you have seen in many many uh instances. If you have if you remember or if you are familiar with unit architecture you will see such an encoder decoder architecture. If you remember how spatially our images were getting shrunk in a convolution operation. It has lot of similarities with what the encoder does. So the latent representation is nothing but a vector. basically using a vector to represent your entire image. So if this vector has 16 dimensions, you will say your latent space is a 16 dimensional space. All right? So you're you're hoping that with this encoder you are able to compress whatever information is there in 28x 28 pixels into just 16 numbers. So you're going from 784 numbers to 16 numbers. This autoenccoder however has some problems. We will discuss that uh in today's lecture before going into variational autoenccoder. So first thing is latent space is very compact and lower dimensional which means a higher dimensional bigger data you can compress into a smaller data right that is the advantage of latent space but this is what is happening in a autoenccoder. Let's say I pick a point from my latent space. So now this is the smaller dimensional space. I pick a point from the latent space and let's say this point resulted from my um encoder. So I encoded some image into this point. Then I'm picking that point and I'm trying to decode it using a trained decoder. Okay. So here I'm only considering the decoder part of my uh autoenccoder. So if this point resulted from a trained encoder then I can reconstruct an image with fairly good um clarity or accuracy and this reconstructed image will look very similar to a input image. So if my input images are from emnest the reconstructed image will also look like an emnest image. But the problem is that latent space in an autoenccoder is very disorganized and irregular. So there is a very specific reason why this term is used. So the the the reason is if you are sampling from a random point in the latent space. So let's say this latent space you you pick a random point either this point or this point. So it's a random vector and now you are trying to decode from this vector the original image meaning you are trying to reconstruct the original image using this vector which is a random vector in your latent space. Then what typically results is a messy reconstruction. You will almost never get the original image back or you will almost never get an image which resembles the emnest image. It will be it will look like a meaningless image. I will show you an example in the code itself. Now one might think okay I can resolve this issue by making sure that I am sampling from nearby a reference point. Right? So what I mean by that is I take my trained encoder. This is trained encoder. This is my input data. My input data using the encoder is converted into an encoded reference point. So let's say this red point is my reference point. So this red point is basically is the encoded form of this image of the five letter five. Now I'll say now the sample point which I'm picking from my latent space is this green point which is very close to my uh encoded reference point. In your opinion, do you think this will result in a a very good, you know, high quality reconstruction or do you think this will still result in a messy reconstruction? So here what we are doing is we take an image which we know is a good image. we are passing it through an encoder and through the encoder we are producing um the encoded version of the this input image and then very close to that we are taking a reference point that reference point do you think will it result in a good um you know good meaningful image and here is one more question okay let's say this red point is the embedding of uh digit let's say three or something like that or digit five itself and let's say this point is the embedding of um maybe digit 8. Okay. Now if I if I pick a point between these two points let's say I pick this point. So this point is exactly in the middle of the reference point for five and reference point for 8. By just normal human logic, we may expect this point to be something that is a mix of five and eight, right? Or it may be closer to five or closer to eight or a mix of those two. A mix of five and eight can be let's say a six because uh you know if you add one extra thing here it is six. If you add one more thing here it resembles eight. So maybe you might think okay maybe it might look like a six. But what actually happens is it doesn't matter in the latent space if you are having an autoenccoder in the latent space even if you pick a point that is very close to a reference point you will still get an image that is meaningless you will get some gibberish the reason for that is in the autoenccoder you are doing some hard training so by hard training I mean this data data point you are saying that this data point which is this image of five which actually can have a lot of variance right five I can write five like this a bit slanted a bit like this all of these are maybe fives so if ideally I were to construct a latent space I would say this point represents five and I have a distribution around this point such that irrespective of whether I pick something from here or from here or from here in the latent space all of them should be equivalent to different variants of five right so ideally in according to human logic any version of five slanted or thick or anybody's handwrite handwritten digit of five is a variant of a a mean tendency a mean five can be something with no slant and average thickness then variance can be something with higher thickness lower thickness more slant planted or whatever. Those are all variance of the five around a mean. But here we are not even considering the fact that there can be a variance. What we are simply doing is we are hard coding this uh digit five into a latent space and we are trying to reconstruct and we are trying trying to train this model. Let's also discuss about how exactly would you train if you want to train a reconstruction like this. What kind of loss would you provide? How would you penalize the autoenccoder in this case for making poor reconstructions? How do you penalize the autoenccoder? What is the loss function? The loss function can simply be mean squed error, right? You compare the reconstruction with the original image pixel to pixel and mean square error would simply be 1 by n sigma i = 1 to n yi cap where yap is the reconstructed pixel value for i pixel minus yi which is the original pixel value for i pixel squared. And when do you think this mean squared error will be minimum? mean squareed error will be minimum when the reconstruction is as close to the data as possible. Right? So you are forcing the this data point to be represented with a very specific latent point such that that very specific latent point when you reconstruct results in an image that is very close to your input image. You are not allowing the latent space to kind of explore different possibilities. You are making through mean square error you are enforcing your latent space to pixate one latent variable corresponding to one input image. And because of that reason the the the term irregular is used is because if the latent space was regular and organized then it would look something like this or it may look something like this. It would have been let's say it's a 2D latent space. I might I might be able to say something like all the points in this region of the space belong to digit five. All the points in this region of the space belong to in general digit three something in between will look like neither digit five nor digit 3 something in between. So this is called as organized and regular latent space. Um but uh an you know disorganized or irregular latent space would not have these properties. For that there is no guarantee that if this is cor if this is the latent variable corresponding to digit five there is no guarantee that if you pick something just close to that there's no guarantee that this will also resemble the you know vector corresponding to digit five. It may be completely random. Um the concept and problem okay understood the concept and problem with normal autoenccoder but why would I pick a random point from the latent space great question. So originally when autoenccoder was introduced people also had the thought that oh can I use a decoder now to produce in generative AI you want to produce or generate images right so if you want to generate an image would you pick can you you know randomly generate a latent space point and can you use that point to generate a completely new image so that can you consider autoenccoder to be a deep generative ative model and what is deep generative model? A deep learning model which has the capabilities to generate in this case generating image. That's the reason why you want to pick a you might want to pick a random latent space point and hope that that random latent space point may correspond to a completely new image of a handwritten digit of five or handwritten digit of seven. But in autoenccoder it is not strictly a deep uh generative model because when you are trying to reconstruct an image you will never be able to reconstruct a completely new looking image because the moment you pick a random point from the latent space you will end up with a messy image reconstruction. You have to be very very close to or extremely similar to the reference point to generate a meaningful image. So the autoenccoder is a really good model which can be used for compression. If you want to compress the data and store it, it's it's really good for that. But if you look at the decoder individually, even if it's a trained decoder, this is useless because it can only be used for reconstructing the actual reference point. It can never be used for image reconstruction for a completely new or completely random latent space point. But this is where the idea of variational autoenccoder was introduced. Can we somehow organize our latent space such that different you know regions in the latent space may mean something. That was the idea. I'll show you one code um for autoenccoder first and then I will show you um the the you know construction of variational autoenccoder. So this is the code for autoenccoder. It's very simple. I'll go through variational autoenccoder and this is just a subset of that. So once you understand variational autoenccoder autoenccoder will be very easy. So this is for fashion emnest data set. I'll simply run the code. But what happens here is you are embedding your fashion emnist images which are basically images of I think 10 classes of different uh fashion wares into a 32dimensional or whatever the latent dimension is into that space and then you will try to reconstruct those same input images using autoenccoder. So let me show you an example uh once this code is executed. So don't worry about not understanding the code right now. I am going to show you how to construct this exact not this code for a slightly more complex uh version because variational autoenccoder has few more complexities. Okay. So for my first question is in autoenccoder this 0.03 03 is mean squared error. Okay. And can you just try to think about what does 0.03 mean? Mean of square of error between pixel values. So if pixel values are in the range 0 to one, I mean 0 and one. Zero means uh black, one means white. Yes, I will also share this notebook. Yeah. Okay. My execution is done. But this top row, you see two rows here. The top row are the original images of different uh you know uh upper body wares. Bottom row are the reconstructed versions of the same image. So I encoded it and then decoded it. How does the encoder look like? It's a neural network. I'll show you that when we are constructing I when we are discussing the variational autoenccoder architecture. I'll show you that it's fairly easy. But look at the difference between top row and bottom row. Top row is the original images. Bottom row is the reconstructed images. Reconstructed images are a little bit blurry. But still we can make it less blurry by maybe running for more epochs uh and and maybe increasing the latent space dimension to slightly higher. Right now it's 32. Maybe we can increase it to 128 etc. But this is not what I want to show you. What I want to show you is this. So here at the end of the code I am picking a random latent vector. Okay. So random latent vector uh of dimension equal to latent dimension. So latent dimension is 64. I'm constructing a random 64dimensional vector and I'm trying to use the trained decoder to produce an an image and I'm just plotting the image. Look at the image. This looks completely random, right? This this resembles nothing like a a fashion um you know a wear wearable thing. This difference will be even more pronounced if I change the fashionist data set into emnest data set. So here let me change it to mnest and let me show you the originals the reconstruction and the reconstruction using land random latent vector. Okay here it's in this notebook is the decoder taking the same point from the latent space encoded by the encoder. Yes. for reconstructions. Yes. But for the last block which is the random reconstruction, the vector that is being selected is a random vector. But the image reconstruction the two rows which I showed you in that uh we are picking the same vectors. So here again mean square error is 0.01. So what this means is on an average each pixel has only variation of 0.0. 03 and pixel values can be maximum of one right so percentage wise it's in my opinion it's like around 3 to less than 5 percentage roughly per pixel the variation is only 3% so that is what mean squareed error of uh actually the mean of squared of errors so percentage would be even square root so the error between pixel value is square root of 0.03 03 uh that would be point um square root of this will be 0.5ish.55 or something like this because.5 multiplied by.5 is um 055 perhaps 0.03 03 to 0.5 uh.17. Okay. All right. My bad. 07. So on an average pixel value of each pixel deviates by around.17 where the actual pixel scale is somewhere between 0 to 1. So if a pixel value was one the value can be up to you know uh it could be like 0.83. 83. So it's like a 17%age variation something like that. So just just to you know give you an idea of what does the mean square error of 0.01 mean. But look at this. Look at the originals. So top row are original images. Bottom row are reconstructions. Reconstructions make sense, right? Reconstructions are looking very similar to input images. However, look at this now. Look at the reconstruction using random vector. This is using a random vector. It resembles no emnest image. This is nothing like an emn mnest image. And I can run this multiple times. Each time it will produce a random vector reconstruction and each time it is a meaningless image. So this is the problem with um autoenccoder. Now let's discuss what exactly is variational autoenccoder and how does it solve this problem. So in variational autoenccoder the main thing is the reason why I wanted to introduce variational autoenccoder is because I eventually also want to teach you diffusion model and variational autoenccoder understanding lays a lot of groundwork behind uh diffusion models. So if you understand variational autoenccoder that's a great step towards uh deep generative models like diffusion model. The main advantage of autoenccoder is in in its compression. So if you want to compress your data in a certain uh lower dimensional variable or a lower dimensional space then autoenccoder is used. In variational autoenccoder uh we introduce something called um uh latent space but with a probability distribution. So this is the architecture of variational autoenccoder but I will explain what this means. So this is autoenccoder bottom one top one is variational autoenccoder and I will show you uh what this mu what this sigma means. All right. The paper that introduced variational autoenccoder is super super famous. It has 50k or 55k something like that that many number of citations. This first author Dri Pingma he's extremely famous. He's also the main author behind the model uh the optimization uh Adam. So and that has revolutionized deep learning as we know today and his Google scholar if you visit he has almost like 300,000 plus citations which is very very big. So this is the Adam paper in which he was the first author and that paper alone has 240ish thousand citations. His second most famous paper. So in these two years he was ultra productive and uh so just to you know make sure that you know who worked behind this. The paper is extremely mathematical. It has a lot of ideas from probability base theorem etc discussed. But today we will try to understand the variational autoenccoder from more of an intuitive point. uh if we were to construct variational autoenccoder how will we define the loss function and within that lot of understanding of the math behind uh variational autoenccoder will automatically come so these are the differences between VAE and autoenccoder so in autoenccoder as I showed you it's directly running a deterministic mapping meaning one input is always mapped to one single latent vector and similarly one latent vector is directly mapped into one reconstruction ction in variational autoenccoder we are performing a probabilistic mapping into latent space meaning a given image can be mapped into within a region of a space it can be mapped into anywhere so so technically the input image is mapped into a um something like a normal distribution from that normal distribution we will sample a point and that point is what we use for reconstruction I'll show you this in a moment in a in a diagram you will understand it much better then the main purpose of autoenccoder is compression dn noising etc but latent space itself as I showed you it has it's irregular and it's not very organized there is no such thing that this entire region is dedicated for handwritten digits of three this for digits of four etc there is no such thing in VA on the other hand uh the latent space is regularized in autoenccoder one thing I mentioned is a little bit wrong. I should not say that it's completely disorganized. That is not correct. If you in general plot in autoenccoder, you know, uh if you make a plot of um let's say all the handwritten digits of one, all the handwritten digits of three, you will of course see clusters. So I will show you one image of such clusters. So see this. So these are created using a latent dimension two for different handwritten digits. And you can definitely see that there are clusters. Okay. But it does not mean that if I pick a point from here. So there is no guarantee that that point will map to digit one. It can be completely different. There is no guarantee that something which I pick from here will map to a point uh like digit zero. There is no such guarantee. So although when you convert your input images into latent space you may seem like okay there is there is good uh clustering which means the cluster the space which where the clusters belong may have some meaning that is not the case. Now in autoenccoder what we are trying to optimize is reconstruction loss which is the mean squed error loss where mean squareed error is forcing the decoder to um directly copy whatever the input is. So if if input image has certain pixel distribution the output should be almost as close to that as possible for you to minimize the mean squed error and autoenccoder is not a reliable generative model. It's a good compression model but not a generative model. But on the other hand, variational autoenccoder can generate new data. I will show you that. And um the optimization which is the reconstruction actually has two terms. One and I'll explain what those two terms are but it's not directly penalizing the model for not performing a perfect reconstruction. That is what variational autoenccoder does. And I will also explain what does this latent space regularization mean. So latent space is very uh disorganized in autoenccoder in variational autoenccoder there is a regularization happening so that latent space different regions in latent space actually mean something. So we'll all we'll all discuss we'll discuss all of this when we are looking at the architecture. So just look at this diagram in autoenccoder a digit like this is directly mapped into this region this point in the space that point in the space if you are reconstructing you will get the image back uh if the the model is very well trained. So if the encoder and decoder is trained for many epochs and if the latent space dimensionality is not too small the reconstruction will look very close to the original input image. In variational autoenccoder what we do is just please pay close attention here. What we do is we do not map this input image. Uh now this x-axis y axis in green color is the image space. And of course as I mentioned image space in the case of amnest has 28x 28 which is 786 dimensions. This image from the image space is not directly mapped into one single point from the latent space. What we do instead is this point is mapped into a gshian distribution of certain mu and certain sigma. So gshian distribution as you know will look like this graph in uh one dimension. So this is what is gshian an inverted bell-like curve. It means uh what this means is it has a mean value. It has a standard deviation, two standard deviation, three standard deviation. So you know you know how the goshian distribution looks like. So here you are mapping the image from the input to a gshian distribution of mean value mu and standard deviation sigma. Now from that gshian distribution you will randomly pick a point. If you randomly pick a point from a gshian distribution of mu mean value mu and standard deviation sigma, you are very likely close to a pick uh you know probabilistically you are likely to pick a point closer to mu. Right? So if your probability distribution is something like this and if you pick a random point from this distribution 10 times where are the points likely to be concentrated? Do you think those points will be likely concentrated far away from the mean or do you think those points which we sample will be closer to the mean? the points will be obviously closer to the mean because in a normal distribution um the peak of the probability distribution is at the mean which means uh not at the mean it's closer to the mean which means if you sample from that distribution n number of times more than often you will end up sampling points from closer to the mean some points may be sampled from away from the mean similarly when you are constructing A probability distribution like this using your point by constructing mu and sigma and from that distribution when you are sampling a point you have no guarantee that the sample point is always the same point. If you sample now you may you may end up sampling this point. If you sample later you may end up sampling this point. If you sample later sometimes the point may be far away. Right? So you have no guarantee that the point which you sample from the latent space is always always the same corresponding to a given image. There is no such guarantee. Then what do we do with that sample point? That sample point is now passed through a decoder to perform the reconstruction. So just pay good attention. Each image is mapped such that a gshian distribution is constructed. From that gshian distribution, you pick a sample point and that sample point will be the point in the latent space to be mapped or to be used for reconstructing the image. Now if you are doing this right for the same input image you will end up with a uh different type of reconstruction because you are sampling a different point but when you are doing this over and over for large number of input images the output will not be such that the output will not be such that the input images of letter n is almost almost always something like this. There is no such guarantee rather the input images of nans might might be represented by a gshian which lives somewhere here in the space input image or somewhere here in the space let's say input image of the letter three might be somewhere here in the space input image for four might be somewhere here in the space we can plot and see this so basically what we are doing is we are forcing the input image to be represented using a gshian in the latent So input image is not one to mapped to a point but it's rather mapped to a distribution. Now what we are doing is from that distribution we will sample a point using that sample point we will construct the reconstructed image which means now we have two things to think about. Uh okay there is one more question for each digit a separate distribution is created. In fact for every image a separate uh distribution is created. Not even for each digit for every single image you predict a mu and sigma and from that uh so mu and sigma means you are predicting a goshian and from that g goshian you will sample a point there is one more important thing to be noted oh okay great question how do we pick a random point great question I'll come to that so one more thing to be noted is that in the latent space when we are constructing this normal distribution. So n of mu comma sigma means or n of mu sigma square means variance is sigma square and mean is mu. We say that the distribution of my input points right. So input points are the actual images of my emnest images. But input points can be some random images also because in by 28x 28 which is 784 dimensional space emnest images are not the only images. There are lot of random meaningless images living in the image space. But here what we say is that the latent space corresponding to my emnest image space will have a normal distribution such that mean is equal to one sorry uh zero and standard deviation is equal to 1. So this is like a standard normal gshian distribution which is centered at zero and standard deviation is equal to one. So what we are essentially trying to do is whenever we are trying to predict mu and sigma we want to predict ideally such that mu is closer to zero and sigma is closer to one. Uh there are a few questions um this flowity now latent space would be a collection of gshian distributions. Yes, latent space will be a collection of gshian distributions. But but because we want those gshian distributions to be a standard gshian distribution or a normal gshian distribution, the mu will be forced to be closer to zero and sigma will be forced to be closer to one. So now there are two things we want to enforce here, right? One is the reconstructed image should be closer to my original image. That is one thing. If this is not looking like a nine, then there is no point in even doing this. So of course the reconstructed image has to be looking similar to uh my original image. Second thing is we want to penalize the model, right? We want to penalize the model for predicting a gshian distribution with mu and sigma as the mean and standard deviation if it diverges too much from the standard normal gshian distribution. Meaning if the model predicts something where mu= 5 and sigma equal to 25 model has to be penalized because this gshian whose mu is 5 and sigma is 25 is very different from so n of mua sigma square we want to be making it as close to n of 0a 1 as much as possible. So we penalize the model if the model is not able to do this. Okay. So mu comma sigma penalty. So think about this. We are penalizing variational autoenccoder for two things. One for making poor reconstructions and one for making u you know poor predictions meaning one for making predictions of gshian distribution which are far away from standard normal gshian distribution. Now let me ask you how do you penalize the reconstructed image? So now we are discussing the loss function. How do you penalize the reconstructed image? We already discussed this obviously mean square error um you compare pixel by pixel take the square of difference of pixels add them up divide by number of pixels for sure. How do you penalize n of mu comma sigma square being different from n of 0 comma 1? How do you penalize this? Could you penalize something like you know you want mu to be closer to 0 and sigma to be closer to one. So can you do something like mu minus 0 the whole square plus sigma - 1 the whole square. How about this? What about this loss function? So my total loss function will be L = um you know 1 by N sigma I = 1 to N YAP minus Y I the whole square plus predicted mu minus supposed to be whatever is the mu which is zero plus predicted sigma minus sigma that is supposed to be the whole square how about this can I can I do something like this I mean logically you can there is no problem but the only issue is when you are doing this mu minus 0 and sigma minus one you are not comparing two distributions here you are comparing two distribution you are not just comparing two numbers you're not comparing mu to 0 and sigma to 1 what you want to compare is this distribution you want to compare against this distribution This is where the term KL divergence comes. Can you please try to recolct where did we discuss about KL divergence before in this course? We had discussed about KL divergence once. So KL divergence is the way to compare how close or how far away are two distribution. KL stands for Kbach lieler divergence. It is a way to compare two distributions. If the two distributions are exactly same as each other, if these two overlap and they are completely same as each other, kale divergence will be zero. Otherwise, they it will keep increasing. Kale divergence also has a you know very nice formula. So if you are comparing the reference distribution P against the distribution to be compared against Q. So in our case the reference distribution is n of 0 comma 1 and the distribution you are comparing that uh you know distribution that you are comparing with the reference is n of mu comma sigma or mua sigma square if you want to mention in terms of variance then p is this 0 comma 1 the one with mu equal to 0 and sigma equal to 1. Q is the one with predicted mu and predicted sigma. Then if you have discrete number of points meaning if in your probability distribution it's not a formula but you have bunch of actual numbers. So you have a probability distribution. Let's say P equal to 0 uh 0.1 0.2 0.7 and let's say Q equal to 0.7 0 um 3 0 0 last term is zero. Then this you can compare using this sigma formula. But if it's a continuous distribution you can simply integrate it. So instead of sigma P of X log P of X by Q of X, you do sigma P of X log P of X by Q of X DX. And now P is the reference distribution. Right? The good thing is in our case since it's a Gshian distribution, Gossian distribution has a formula. So this is the formula. Gossian distribution has this formula 1x square sigma square roo<unk> of 2 pi exponent of -/ x - mu by sigma square. This is a very standard formula. The good thing is since both of these are gshians, we can construct the KL divergence formula in this way. So KL divergence between P uh and Q actually P is the reference. So actually this should be over here and this should be over here. This is a mistake. So p is my reference and q is my uh distribution that I'm comparing against my reference is equal to half * mu²/ * mu² + sigma square minus log of sigma square minus 1. This is the formula for comparing the two gshians the kl divergence loss. Now compare this against the formula which we just discussed the the naive formula which we discussed. Where was it? Um this one this will be mu square this part will be mu square 2 + sigma square + 1 - 2 sigma right just compare this against so mu² + sigma square then + 1 - 2 sigma right that was the formula which which we will we would have ended up with if we were not comparing the two distributions but rather we are just comparing mu and sigma Now look at this. They are very close to each other. Actually technically you could use this formula also. But then you will not be strictly comparing the two distributions. You'll be directly comparing mu and sigma against 0 and one. But when you are actually comparing two distributions the formula that you will end up will be something like this. The derivation of this is a little bit uh not involved. We don't have to do it right now. Now the total loss will be the mean squared error loss which is also there in autoenccoder. So this is the MSE plus the KL divergence loss and this KL divergence loss basically regularizes the latent space meaning it gives the latent space provision to produce a probability distribution corresponding to an input rather than one deterministic point. So it makes the space more smoother. We will see what this smoother look like when we produce an output image. Right? So this is the final variational autoenccoder loss. Now I want to also discuss one more thing which is the encoder decoder architecture. Uh before discussing the um you know before doing the coding. So let's look at how the architecture would be. Let's say my latent space is this and let's say I want my latent space to be 16 dimensional. So what would this be in a neural network? What would this 16 dimensional thing look like? A layer with 16 nodes, right? Something like this 16 nodes. Okay. Now this is my latent uh space. So I'm denoting it by zed. So before the latent space I will have a hidden layer. This hidden layer let's say it has 256 dimensions. Before the hidden layer I will have a flattened layer of my uh inputs which are of 784 dimensions. Correct? So it's basically like a multi-layer perceptron and this will be fully connected. This will be fully connected. Only difference is here there will be rail activation. Okay. So after the weights are being applied there will be rail activation and here there will be my input image of 28x 28 pixels. So you are flattening the 28x 28 pixels. Then there are weights here W11 W12 etc. So there will be a matrix which multiplies uh whose dimension is of 720 84 by 256 in Python code. Um okay why are we grounding? There's a question uh why are we grounding the standard why are we grounding to standard gshian this will collapse all distributions in the latent space correct so what we are deliberately trying what we that's that's an assumption made actually in reality what would happen is your image space would be something like this let's say image space is this I'm sampling an image from here okay from this space my latent space might look like this some some random distribution So if I were to pick and let's say this is the space where all the emnest images are residing. If I were to map all these emnest images into my latent space my latent space will have some random shape. Now now what the assumption that we are making is this latent space actually is assumed for simplicity that it looks like a gshian which means it has a mu and a standard deviation. So um so that we can have a easy way to map from my images to this distribution and then from back from this distribution to image in in case of regeneration. So what the encoder does is is it creates a mapping from this P of X which is a probability distribution to P of Z which is another probability distribution where X is the space where you have input images Z is the space where you have latent. This is what encoder does. What the decoder does is it learns to map something back from this latent space distribution back into your image space and we don't know how this space looks like because we don't have all the handwritten digits in the universe available to us. There are many many many different ways in which handwritten digits can be made. Which means P of Z also there is no way for us to know what this you know random shaped P of Z may look like because we cannot obviously construct you know corresponding Z for every single X in existence. We can't do that. So for simplicity we assume that the latent space distribution is like a gshian and that to standard goshian. So it's an assumption to made to uh make the calculation of this simpler. Now one more thing is uh yeah so this is the encoder and in encoder you have ru now this is latent space after latent space you will have the decoder. So decoder again this layer will have 256 nodes and then the last layer will have 784 nodes and this will all be fully connected. This will be fully connected. There will be Railu activation here. There will be RAU activation here. Um and one more question I have is what would be the activation at the last layer over here right after this 784 dimensions. U what will be the activation function used? Will it be radio? So think about it if we if we use softmax what would happen many of you are saying softmax by based on kind of impulse but think about it if I have soft max see I don't this is a 784 distribution every pixel matters here right we are not trying to find the one single pixel with the best possible value just think about it we want all pixels matters matter because All pixels play a role in creating the reconstruction. So it's not like we want to find the one pixel with the best probability. It's not what I what we want. What we want instead is the values of every single pixel should be within a range of 0 to one. Correct? So every pixel should be 0 to 1. The problem with relu is that ru is unbounded. Re's value can be less than zero. Sorry, it can't be less than zero. is from 0 to plus infinity. So the better better activation function to use at the last is sigmoid because sigmoid looks like this. Sigmoid's value is bounded between 0 and 1 irrespective of what the input is. Input can be from minus infinity to plus infinity. Tanh we cannot use because tanh will be from minus1 to +1. So uh the problem with relu is that in the previous layers we used relu but then uh that is because unbounded activation functions have expressive power. However the final output that we want is not unbounded. It's a bounded output. So we use sigmoid in the last layer. Okay. So just note that. So if you now look at the overall neural network architecture for this it's very simple. The only additional thing we need to know is where exactly is mu and sigma coming into picture. So here we mentioned about latent space Z but we did not talk about mu and sigma. So what is actually done is there are two layers. So from encoder so this is the first layer of encoder. This is the hidden layer of encoder. Then in the latent space actually there are two parallel layers. This layer has 784 nodes. Sorry not 784. Uh let's say latent space is 64 dimensions. 64 nodes. This has 64 nodes. This is fully connected to this. This is fully connected to this. So this will be 256 nodes. This will be 784 nodes. And these 64 nodes will represent mu. These 64 will represent sigma. Okay. So these are two disconnected uh nodes. So uh I mean two disconnected layers within the latent space. So whatever is there in these 64 nodes will represent mu. Whatever is there in this will represent sigma. And then we will construct the KL loss based on what this mu and what this sigma look like. That is the first thing. Second thing is now to predict my reconstruction. So my reconstructed image will be let's say six or whatever. I need to start with some random point in the latent space, right? Some some random Z. How do I randomly sample from a distribution? My distribution looks like N of mu comma sigma. Correct? It's a normal distribution whose mean is mu, standard deviation is sigma. I need to sample from this. Correct? Yeah, I'll explain once again. We need to first predict mu and sigma. And then from that distribution, I need to pick a random v random latent vector z. And this random latent vector will be finally used for reconstructing my original input image. Now my question is how do you sample? How do you get this Z? One way to sample is you use a some random number generator and or or you know whatever you know uh numbers you have in your mu comma sigma space you randomly pick a number. The other option is any number which you have in a normal distribution. You can express it as so this number you can express it as mu plus one standard deviation or mu + 2 standard deviation or mu + epsilon standard deviation where epsilon is a variable. So any number in a distribution I can represent by saying this number is equal to mu plus some variable multiplied by standard deviation. So if the if the variable is exactly over here then it will be mu plus 0 standard deviation. And if mu is also zero this will be zero. If the variable is if the randomly picked number is far away then it will be mu plus let's say th00and standard deviation. It is it is very far from the mean. So essentially all we need to do is define a random variable epsilon which is sampled from a distribution like this. So we sample a random number called epsilon and then we say my number that I'm picking which is zed is equal to mu plus epsilon* time standard uh distribution. The reason why this is done is um when we are constructing the loss right as you know the loss is dependent on it's a function of mu and sigma right which means I should be able to and it is also dependent on other things but it is also a function of mu and sigma. So for constructing gradient descent I should be able to do D do L by do mu or I should be able to do D do L by do sigma. I want to be able to do this. But if I'm randomly picking a variable and saying my zed is equal to some random number epsilon then my zed is not dependent on mu and sigma. But here if I say zed is equal to mu + epsilon * sigma. Now my zed is dependent on mu and sigma which means I can define eventually some parameters which will be like so let's say do l by do mu is equal to do l by d z multiplied by do z by do mu using chain rule or I can say do l by do sigma is equal to d l by d z multiplied by doz by do sigma so I can have a differentiable function which means I can perform back propagation. So just to reiterate, we construct the neural network the decoder such that in the first layer I have 784 nodes, second layer is a hidden layer in which I can have variable number of nodes and third layer is also variable which is the latent space. But in the latent space I am predicting two things mu and standard deviation. So mu will be again 64 dimensional standard deviation will also be 64 dimensional and these are fully connected. And these are also fully connected. Now from this mu and uh from this mu and sigma I will sample something such that my zed is equal to mu + epsilon * sigma. The dimensionality of this zed will also be 64. So all of these are 64 dimensional. Then I will project the 64dimensional thing uh vector into 256dimensional hidden layer. Then from that I will project it into 784 dimensional layer. From this basically I have my image reconstruction. Then I have two errors. One error is by comparing the input image to the output reconstruction that is the mean squared error. Another error is by comparing this predicted mu and predicted sigma which is n of mu comma sigma. we are comparing it against n of 0a 1 and constructing the KL divergence loss and then the total total loss is MSE plus KL divergence loss. So this is the um this is the total loss which is used for defining the um architecture of variational autoenccoder mu and sigma. How are they calculated? They are just calculated using the neural network. they are the parameters in this these neurons basically. So if if latent space is 64 dimensional, mu is 64 dimensional it's calculated here. Same for sigma. Um so they are basically learned through the neural network. So initial mu and sigma will be random because the parameters will be random. Eventually it will become closer and closer and closer to 0 and one. Uh but not extremely close but closer and closer because there are two losses. Right? If you try to minimize KL divergence loss, maybe your mean squared loss will increase. So you cannot minimize both. You are trying to minimize the sum of both. So there is no guarantee that KL divergence will be such that mu and sigma are close to 0 and one. There is no such guarantee because maybe if you make mu and sigma extremely close to 0 and one, your mean square error might increase a little bit. So during gradient descent the network will figure out what is the best way in which the total loss can be uh minimized by minimizing MSE as much as possible and minimizing KL as much as possible. All right. So now I think we can code and during the coding all the rest of your questions will be answered. Okay. There is a question since here uh image size is small and dense representation will work. However for higher resolution we can use convolution encoder and decoder. uh in that case latent space can be of higher dimension um or it has to be single dimension vector. How will it be? So latent latent space is already higher dimensional. If you if you think about later dimen later lat latent dimen uh latent representation space is already 64 dimensional or 128 dimensional right but it's a space of vector even if you think about um a three-dimensional like an RGB image that also you can think of something that is so let's say this emnest was RGB okay 28 by 28x3 you can say that this is represented in a base whose dimension is 784 multiplied by 3 whatever that number is 2,100 plus whatever that is right so just by having additional channels no complexity is arising other than your computational cost increasing so here we are essentially performing operations similar to CNN except convolution operation is not there so it is not a convolutional neural network it is like a multi-layer perceptron Um then there are so finally if you think about it the latent space is at the end of the day a space multi-dimensional space where vectors can live. That's all. It doesn't matter whether the input image is color image or black and white uh grayscale image. Uh okay let's now start coding and we can take up some more questions in the middle of today. So please open uh an empty you know Google collab and just name it as anything via from scratch. I will share this with you in case you want to follow my code as such. But there is no no need for that. I mean this time we are not using any pre-trained weights or anything. So variables you can use any variable you want. All right. So let let us start with imports. The first is um import torch for pytorch. Then import torch.n NN as NN which we'll be used using during coding the class for variational autoenccoder as we have always been doing torch.optim Optim as uh optim for defining optimizer then from torch vision import data sets which will give us access to mnest fashion mnist etc and transforms for um performing transformations like that you can convert images into tensors then from torch doutils uh dot data Import data loader. This is for creating batches of data. And then finally for plotting images, import mapplot lib piplot as pl. Numpy is not needed. Fairly straightforward. These are the only imports we need. And then we can just define the device uh device equal to same thing device equal to coda. Five codas available LCPU. Uh here I want to pick something like maybe V6 60 minus one uh TPU. You can pick anything. This code is going to be very simple. So no need to you know no need to have any GPU. Um oh yeah tot I was having an extra space here. My bad. All right. So these are the imports and device. Now we can have our uh code block for creating batches. We can say batch size is equal to something like 128. Then transform equal to transforms dot compose and inside this the only transform we are going to pass is transforms do.2 two times. So, so earlier we sometimes used to do transformations like normalizing using mean, standard deviation etc. Here we are not doing that. Just converting the images into tensor format. Then we say train data set equal to data sets dot fashion or let's start with mnest actually not fashion mnest mnest of the folder to download is root equal to dot / data. So this is where our data set will be downloaded to and train equal to true. This is training data download equal to true uh and then transforms for transformation we'll use this transform. So transform equal to transform maybe to reduce confusion here I can call it as data transform and this is I'm this is what I'm passing here. Then this I'll keep as a third argument. That is usually what is done. Doesn't matter. Then here we don't really need validation data because we are just looking at the reconstructions directly. So we don't necessarily need the validation data. We can just um um use only training data to just see how the reconstructions look like in the training. Uh train loader is the data set loader for train. So data loader of and here we have to pass data set and data set is nothing but this train data set. Then we have to pass batch size which is defined by this. So batch size is equal to batch size. Then shuffle equal to true which you may or may not define not needed but we keep it since we have been doing this in every code. That is the train loader and that's it. So this is how you create batches. Now comes the main part coding the VAE class. So first we can define a latent uh dimension latent dim. Let's say this is 64 dimensional space latent dimensions. Okay. And you can say class v ae of nn dot module so that we can inherit properties from nnn define init and self we pass here rest of the things whatever we want to pass we'll figure out and then for inheriting properties from my um uh parent super then uh self dot_init actually we don't have to pass this is enough then we can write first we can write the encoder then we can write the decoder then we can write the um forward pass the forward pass will be a function like this define forward uh self comma uh x right inside this bunch of things we will write. So first let's write the encoder. In encoder we start with the image. So first is a fully connected layers going from uh 784 dimensions into um 256 dimensions which is the hidden dimension. So here I can say input dim equal to 784. This is the mnest 28 x 28 hidden dim let's keep it as maybe 256 and latent dim equal to this latent dimension. So I'm just passing this variable here. Um so in encoder self dot fully connected neurons one in layer 1 is equal to nn.linear linear and this is from what to what are the two arguments that I should pass here. I am going from input dimension to hidden dimension. So please note anything after n dot is uh trainable parameters. Then self dot fully connected to actually there is a fully connected we don't have to say fully connected to instead it's better to say fully connected for predicting mu and fully connected for predicting u standard variance. So what we are actually going to predict is this not standard deviation but this term um so this you see this log of sigma square basically this is log of variance. So what we are going to predict is log of variance. I'll tell you the reason why we are doing this. The reason why we are doing this is the output from a a layer of neurons can be um you know if it is if is there the output will be definitely positive but otherwise the output can be positive or negative. So let's say we were trying to predict sigma square directly then we know that sigma square is always positive which means we have to enforce the neurons to always output positive values but logarithm of sigma square can be positive or negative because if sigma square is less than one logarithm of a number between 0 to 1 is spanning from minus infinity to zero correct and if the number is greater than one logarithm can be greater than zero So by predicting logarithm of sigma square we are we don't have to force the neural network to necessarily predict positive values. So that's the only reason why we are trying to predict this term logarithm of variance. So from if we if we say we are predicting log of sigma square how do we get to sigma and let's say I'm calling this as log where logarithm of variance exponent of log is equal to what sigma square or um yeah this is equal to variance. So to go from to go from log to variance all you need to do do is take an exponent. So if you predict this logarithm of variance you can get sigma square or sigma or any quantity you want. So what we are going to do here is we will predict mu as such but here we will we will say we are predicting logarithm of variance. So now mu is nn dot linear from what to what? What would this be? Hidden dimension to latent dimension. So hidden dimension, uh, latent dimension. And what about this? What about self dot fc logar? Same thing because it's another uh set of fully connected neurons where the number of nodes are going from hidden dimension to latent dimension as well. Right? So these two but look at this these two are now in parallel. Uh mu is predicted from hidden hidden layer. Logarithm of variance is also predicted from hidden layer. All right. So that is the encoder. Decoder is simple. Decoder is self dot fully connected to equal to nn.linear. And here we are first projecting from latent dimension to hidden dimension. And in the next layer self fc3 is we are going from nn dot linear hidden dimension to output dimension. Output is simply same as input dimension. So I can just say because the final predicted image we are directly comparing with the input. So final predicted image will also have 784 by 784. So here I can say input dimension which is 760 7 uh 84 that is decoder. Now we need to say self dot relu because we'll have to use relu multiple times. This nn.relu and at the last layer we'll be using sigmoid. So nn self dot sigmoid equal to nn dot sigmoid. Okay there is a question if images are of higher resolution then does it mean I would need more hidden layers? Um not necessarily. I can like to preserve over I mean to to make sure that there is no over compression right yeah probably maybe you can add more hidden layers and um using that you can you can ensure that otherwise what if you are go jumping from 3,00 input dimension into 256 there is a lot of it's a lot of uh you know compression happening in just one layer but I don't think it's needed you can just have one hidden layer also but in general neural networks the more layers there are the more the you know expressive expressive power. Okay. Uh that is encoder decoder and now before forward pass we need to do one thing which is define encode. So this is the forward pass through the encoder. Then we can have a forward pass through the decoder which is decode self comma x. And we will also need before this the reparameterization. And what is reparameterization? Nothing but picking zed. So you're sampling zed from your standard uh from your mu comma sigma distribution. So reparameterize that's the official term used in the paper. It's not a term that I invented. here self comma mu comma uh log bar logarithm of variance this we'll be passing and in decoder we are technically passing zed in forward pass this is correct now we need to write these functions so first encoder so encoder we'll say h is equal to which is written equal to self dot radio of self dot fully connected in layer 1 of input which is x. Now this hidden is projected into mu and logarithm of variance. So mu equal to something logarithm of variance is equal to something. And what is mu? Mu is um self dot fully connected layer for mu of h. Similarly logarithm of variance is self dot fc log of same h. Then we can say from this we return mu and logarithm of variance. These two things we return. Then in reparameterization we say standard deviation is equal to torch dot exponent of 0.5 into uh log y. So this is essentially finally taking square root uh of exponent. Um then we are sampling from the same distribution. So epsilon equal to torch dot random number like standard distribution. Basically this rand n like means if this standard deviation is coming from a distribution we are picking another epsilon which is supposed to come from the same same distribution. And so random like is to sample another random number from a distribution which is very similar. Then we say zed is equal to mu + epsilon multiplied by standard deviation and then return zed. So now this zed is the random zed which we sample from our distribution resulted from the input image return zed. And then what will decoder do? Decoder will say the hidden layer output is self dot relu of self dot uh FC 2 of uh Z and then final output is equal to self dot sigmoid of self dot FC FC3 I think yeah FC3 of H which is this this H and then we return the output. So this output is now the final reconstructed image. Now we have to do a full forward pass through the whole thing. Then we say mu comma logar equal to this one. So self.enccode Then xz equal to self dot reparameterize of uh mu comma logarithm of variance. And the last thing is the x reconstructed constructed which is the reconstructed image is equal to self dot decode of zed and then I can simply return the reconstructed x which is the final image that is reconstructed. So fairly straightforward. Uh now let's run this to see if syntax wise everything is okay. Looks like it's fine. Oh, one more thing we might have to uh return is mu and logarithm of variance also we should return. You know why? For calculating loss. If you don't return this loss cannot be calculated. So just make sure this is also returned because outside the this class we need access to the mu and logarithm of variance for the k divergence loss. So speaking of loss, now it is time to code the loss. coding the loss. So define VAE loss. Here we are passing reconstructed X. Then we are passing uh X itself so that we can so these two are compared for mean square error these two guys. Then we are passing mu and log logarithm of variance. Then we say reconstruction reconclosal um then dot binary cross entropy and we and s we could simply let me ask if there is a direct MSE uh function or not. is the MSE function like this torch dot n dot okay so let's then try this uh nn.functional mse loss then reconstruction, x. So those are compared and then kl loss is equal to um this was a formula. The formula was half * I think mu square + sigma square minus log logarithm of variance minus 1. So 0.5 * mu² um plus sigma square which is um logarithm log dot exponent that is that itself is sigma square. So, mu square + sigma square minus logarithm of variance which is log bar and minus 1. This was the formula. Then actually I should do torch dot sum. So 0.5 * torch dot sum of all these guys. Okay. and then I can return the total loss is simply this plus KL loss. So this is the final loss function. There is a question is the formula for KL in this case of normal distribution comparison predefined as you shown from the internet. So KL divergence loss formula is derived. So this is the actual integral. Okay to this integral if you substitute the fact that P of X is normal distribution like this and Q of X is normal distribution like this and if you derive you will end up with this formula. So we can sit and derive it but then we'll be digressing too much from our core idea here. But essentially this formula is resulting from the fact that our distributions are two different normal distributions. Ones with mu equal to 0 and one the other one with mu equal to mu and sigma equal to sigma. So uh but this formula does not apply if the two you know distributions are not um gshian distribution. Just note that. But yeah this formula I directly got from the internet. I did not derive it here. But it's not difficult to derive as well. It's it's because it's all deterministic. It's a it's a well- definfined formula which you can substitute in the integral. Integral of course will give you some pain. But if you sit if you sit down and do the integration you will end up with this formula. Okay. Um so this you can just run the variational autoenccoder loss function and now we can define the model. So model equal to VAE we had defined this class. Uh we can also say two device just to make sure that model is also using the same device CPU or GPU as the earlier ones. Then optimizer equal to optim dot adom with a capital of and what we are optimizing is model dot parameters and learning rate also we can define to be let's say 1 e minus 3. So this is just the optimizer and model and now we can do the training loops. So first we say epoch equal to something like five just to make it fast. Then model train to keep the model in the training mode instead of in the eval mode. Then say for epoch for epoch in um range of epochs. So first we define a total loss function total loss variable whose value is initially zero. Then for x comma dash in train loader train loader is where my batches uh are there x = x dot view of -1 comma 784 dot to device. So please note the reason why this line is written is because my batches are like this. My batch size is uh 128 or something right? So my batches are like 128 comma um channel comma 28 by 28. This is how this will look like. My data will look like batch dimension, channel dimension, width and height dimension. What I actually want is in in my fully connected sorry in my flattened layer I just want 784 and then I'll have a batch dimension or whatever this dimension whatever this number should be. One thing to note here is if my input images are let's an exact multiple of 128. So let's say I had 1280 images in the input. If I divide it into a batch size of 128, I'll have 10 batches whose number of images are exactly 128. But what if I had only 1279 images then my batches will be such that last batch will not necessarily have 128 images. It will have only 127 images. So I cannot hardcode 128 here. But if I say it is minus1, 784 with used along with view function automatically will Python will calculate what the number here should be whether it is 128 or 127 or whatever that number is. So that is why we use this u view function. It's it basically automatically calculates whatever should be the replacement for this minus one. then optimizer.0 grad. This is to make sure that we are um starting we are not storing the gradients from the past batches. Um optimizer there should be a I here I'm missing an I. Then X reconstructed is equal to and yeah this comma mua log variable. Both these three things are output from the model. So equal to model of X. So from the model these three things are coming out as output. Then we say loss is equal to VA AE loss and into that I pass these things along with X. So X reconstructed comma X comma mu comma logarithm of variance because these four out inputs are being expected. Then we say um loss dot backward. So loss. So this optimizer for Z.0 is done for every batch separately. So if you put it outside here, it will be only applied for the first batch. It won't be applied for the rest of the batches. Um loss dot backward then optimizer.step step. So this is the backward propagation. These are standard. We usually don't write this but today since the model part was anyway easy. Total loss equal to total loss plus uh this loss loss dot item and then we come out of the loop and then we want average loss. Average loss is total loss divided by total number of batches. So equal to total loss divided by and how do you get the batch size length of train loader and then we can print it and we can say print um epoch and epoch number is epoch + one because epoch this epoch variable will start from zero. So epoch number is epoch epoch + one. So one out of um all the number of epochs. So epoch value is five. So this will be 1 2 3 4 5 sequentially it'll be displayed and loss is average loss um with four floating point precision. I hope the syntax is correct. Now we can hopefully run this. Okay, let's run this and see what happens. If there is any error, we'll see at that time. Okay, it seems to be running fine. Um, just some bracket is missing. That's all. That's okay. Okay, so the loss is coming pretty much down and now we can just plot the image. So for plotting the images, I'm just going to copy the code because I don't have to uh write this piece of code. I don't want to write this piece of code. This is just plotting the sample images. So here what we are doing is we are picking eight images from our sample uh from our training eight random images and we are plotting those eight images and comparing it with the reconstruction. Okay. So just look at this. So these are now the images and the these are the reconstruction very bad right? I think what we are missing is probably um number of epochs we can increase. What else can we do? I think everything looks good. Let me just run run this once more cuz the reconstruction is really bad. Okay, this is really poor reconstruction. I was in general getting reconstructions like this for um for lowdimensional latent space. But here my latent space is not so small. Maybe what I can do is increase the epoch and increase the latent dimension. Uh please try to run this yourself and let me know what output you are getting um in terms of reconstruction. Let me try actually binary cross entropy loss. Maybe that will converge better. Let's see this one cuz here it does not seem to converge at all. Slowly. Yeah, loss. Uh, so my loss here is okay. Here there is again clearly something is wrong. Let me check once more. So I was able to get losses like 0.01 or something. Our loss was like 0.06. So just wanted to see if reducing it further will improve the accuracy. But let's see what else am I missing here. Uh nn.functional dobbinary cross entropy that looks fine. K scale divergence. So mu mu ra to two with this we are doing. Okay. So perhaps instead of doing this should not make any difference. I hope um and logarithm of variance logarithm of variance exponent. Let's run this once more. Do you guys see any obvious mistake? Uh I'm not seeing any obvious mistake but I'm sure there is something because loss this high of a loss is not at all sensible. um log I have not missed I think let's do one thing let's run this code and come back uh the scale divergence multiply by minus one this one right uh I don't think so because the formula is the formula is this um let me see I don't think so this is the exact formula that I'm using half times uh you can write it as minus half times but then all the things inside will be flipped by minus one so I think the formula is correct where else can be the mistake let me check this training loop part. I'm not sure if I'm making any uh silly mistake. Uh let me just copy paste this. Okay, now the loss has decreased. So clearly there was some issue with my previous training loop. Let me just go back and see. This is my current training loop and this is from my actual working code. Okay. Okay. This is the mistake. Um the average loss when I'm dividing uh I should do train dot train loader dot data set obviously. Yes. So this is the this is the mistake. So I'm not dividing the length by the length of train loader the the actual data. VA loss pre check. Okay. I think if I change this now if I run hopefully it should work. I think this is the only mistake that I could spot. Uh yeah. Okay. Now it's working. All right. So I'm I was just missing this train loader data set. Uh okay. So let's just run the code once more. I'll use my now loss seems to be a bit high. I just want to try running this once on my original working code and see what kind of losses I was getting. Um code is pretty much the same but so here the number of epochs are 15. Uh the latent space dimension is 64. All right. Okay, losses seem to be on the similar order of magnitude. So maybe it's fine. Okay, so this is my convergence now around 100 and these are my reconstructions. All right, this is good. So these are now the reconstructed images and it's pretty good and if you run this each time it will pick another eight set of images and it will provide the the reconstruction. Okay now one thing I want to show you is reconstruction using a random uh Z that is the important thing so it will look a little bit random of course it won't be as good. So this is a random input. So here we are saying I want to pick random 16 random vectors each of whose dimension is same as latent dimension and let's see so it's better it's not the best but it's much better than autoenccoder autoenccoder was giving us things like this if you remember this was the auto I think this is disconnected okay but this was the output of autoenccoder but variational autoenccoder is giving much better output some of them are you know not making any sense but at least it's it's It's still looking like closer to uh emnest. Now this can look even better if we do work on fashion mnest. So let me just change this data set to fashion mnest and let me run this whole thing once again. So here there are a few experiments which we can try. Okay. Um one is changing the latent space dimension from 64 to just one dimension. What if you just did one dimension? Could you get a good reconstruction? So we can actually try that with autoenccoder itself. So autoenccoder let's say my embedding. So the question is can you compress your latent space so small and get good reconstruction. Okay. So here I'm picking latent dimension as two. All right. In autoenccoder and I'm just running the full code. I'll show you the reconstruction when the once the code is done and yeah this is the fashion mis data set. So here these are the images these are the reconstructions. So one thing about variational autoenccoder is that it always it has a sort of blurring effect. The reason why this blurring effect is coming is because you know uh you might have heard about gshian filter itself right goshian filter is it basically averages out the sharpness in the image. Here also something very similar is happening in the latent space. The data is distributed like a gshian and u depending because of the fact that there is a um distribution of the data there are no sharp edges that are created in the output. But in variational autoenccoder you can have sharp edges but the thing is it can only produce images which are exceptionally close to the the reference point. Even even if you're slightly far away from the reference point, you will have very uh you know useless reconstructions. So here these are the first row is the actual fashion image data. Second row is the reconstruction and these are for random vectors. So for random vectors from the embedding space see these are these are still meaningful. These are all reconstructed images from random input latent vectors. You see it's not senseless. It's sensible. Let me look at the two-dimensional reconstruction. So with two-dimensional uh latent space. So please note this is autoenccoder which I'm showing right now not variational autoenccoder. I just want to show you the reconstruction with autoenccoder. Look at the reconstruction. Um it's you know it's not that different. And how do you make sense of this? I have a code here which I want to show you for autoenccoder. So here what I'm doing is I'm plotting the u these latent this latent space itself directly on a scatter plot. So just look at this 3 8 they are looking very similar even after reconstruction five is also so 3 8 and five are looking very similar right 7 9 they are all looking very similar. So now if I plot the reconstruction, if I plot the latent space, look at the latent space in 2D, the ones with least intersection is um orange, which is one, digit one and digit 0. So digit one and digit 0 will never be confused. But rest of the digits have very high overlap. So let's look at digit one and digit 0. If I run this once again, hoping that digit one and digit zero come see one is getting reconstructed really well. Zero is getting reconstructed somewhat well but rest of the ones are you know six and two are not getting reconstructed properly. Six is getting reconstructed as something like eight. Two is getting reconstructed as something like eight but 0 and one is very good. Now variational autoenccoder in that also we could uh do the same. I think this one I'll reserve for an assignment. I wanted to give an assignment of plotting the scatter plot for uh latent space. So here this is the direct latent space plotted in a 2D scatter plot. My question is what if the latent space dimension was four or five or six or something like that. What can we do to plot? If I wanted to have a visualization of the latent space, what would I do in a higher dimensional space? I can simply pick two principal components, the top two principal components and plot the two principal components. Right? Here I have done 3D also. So uh if I keep my latent space dimension as three um it plots a 3D plot but beyond three uh it will plot using principal component analysis the two principal components. So this is one way to plot the latent space and see the reconstruction. So in general you you should see that the reconstruction becomes better if the latent space is having more dimensions. Why is that? Because then the loss of information is less. The smaller the latent dimension, the greater the loss that happens. Obviously, right? Uh if the latent space dimension is one, what does that mean? It means your entire image you are hoping to capture with just one number. If the dimension is two, you are using just two numbers to capture the information in an image. So obviously reconstruction will suffer because of that. Um so this also I just want to show you this 3D uh three-dimensional reconstruction. Obviously it will look better than two dimensions and I will show you the latent space how it looks like. So this is in 3D again not the greatest many uh one and zero are not getting confused but remaining digits are completely getting confused. It's very hard to say what is overlapping in 3D. Okay, this is a really bad 3D latent space. Let me run once more. Maybe I have to run for a few more epochs. I don't think we have reached too much convergence. Um and this is still autoenccoder in variational autoenccoder. Let's look at fashion emnest. We were about to see that. Okay. So this is the fashion emnest for the variation autoenccoder we just coded from scratch. I I showed you this is with latent dimension 128. previous one was 64 and reconstructions are fairly good but you see these patterns on the t-shirt on the jacket it's gone it's not there then uh these are the reconstructions from random latent vectors so truly variational autoenccoders are actual deep generative models because these are now generative AI images although it is the most primitive the kind of generative AI images you can construct these days are amazing but you know technically from a fundamental standpoint these images are generative AI images which which we generate from uh pure numbers numbers are converted into these images. We can do the same thing for color and higher dimensional images but the encoder decoder will become much more complex if you really want more expressivity and latent space will be more having more dimensionality plus uh you will have to run for more epoch. computational cost will increase. But this is something that is truly demonstrative and and you can see the power of variational autoenccoder compared to autoenccoder themselves. Autoenccoder could not have such a high reconstruction power but variational autoenccoders because it captures the uh latent variables with as a probability distribution. It has much higher power to learn and generalize. So that's the difference. So uh I think this is a good time to stop the lecture. I will post an assignment where I will ask you to plot the latent dimension in 2D, 3D and more dimensions using principal component analysis for emnist as well as for passion amnest that is one thing. Uh if there are any other data set like 10 or some some grayscale data set that you want to use feel free to use that and see how the reconstruction looks. Uh why autoenccoder is used instead of CNN? It's not instead of CNN, right? CNN is basically a convolutional neural network at the end of which you can do image segmentation, you can do classification, you can do detection. It's not instead autoenccoder is a deep generative model. CNN is not typically used for generation. That's the main difference. Um CNN is used within generative models but the usual CNN's like restnet is not used for generation. It's used for image understanding and then for making some predictions on top of it. Here it's neither for classification nor for detection nor for segmentation. This is a degenerative model. So this understanding will help you a lot in understanding diffusion model. That that is where we want to go next. So in the next lecture we'll be discussing about diffusion and you have you heard of how many of you have heard of LLMs diffusion based LLM? Have you heard of that? You might have seen videos uh uh JIF images in uh LinkedIn and all which were the the entire output of LLM is not coming in token coming token after token but the entire context window whatever is the context window like that much is displayed all at once text usually in LLM's text is displayed token token after token but uh in diffusion model you know you you produce the entire output at once so in LLM's also in text based uh generation also idea is possible. So um just just have a look at that. Uh I think Dr. Raj is releasing a series also about diffusion based LLM. So just check it out. But the same idea of latent space and embedding items in a latent space and then producing it not using variational autoenccoder it's using diffusion models which you are we are yet to discuss but the same idea will apply there. So I think if you understand v VA va in depth it will help you a long way in understanding many many other things uh especially in generative AI space. So uh thank you so much. I'll be posting one assignment based on VAE for autoenccoder there is no assignment but just for VA u so just try to solve it. So next lecture will most likely be the last lecture for diffusion model. Uh so we'll try to coding a diffusion model from scratch. I have to figure out how to make it simplified but we can do something about diffusion models and um about Dino V1 I will record a video a bit later and I'll post it so that you can go through the video and notes and code uh in a self-paced mode. So this boot camp is coming pretty much to an end. So next week will be the last uh lecture for the series. Then one model which I deliberately omitted is the mass mask to former. So mass to former I had discussed be in the beginning but the reason why I omitted is because we had discussed SAM and mass to former is basically completely obsolete in my opinion if SAM is around. So since we discussed SAM I thought there is no point in discussing mass to former. Uh and the paper SAM paper is anyway much much more popular. Um so the so far the topics that we have discussed are we have discussed generative model briefly u we are just starting it we have to discuss diffusion also before that we discussed about VLM um in the one of the earlier lectures we had discussed about um coding nanovm in between we discussed transformers for detection transformer for segmentation and transformer for classification and before all of that we started with the transformer architecture itself self attention um then modified self simplified self attention self attention with uh no masking self attention with masking then multi head attention etc. So the the path that we have taken to reach till this point is very long and it's very you know fascinating. So diffusion model technically will require two classes but what I'm planning to do is so there are two things to discuss one is the original diffusion model itself which was proposed in 2020 I think the uh the DDPM uh paper the second one is uh stable diffusion so stable diffusion is of course much complex so for stable diffusion most likely I will do a paper dissection video and I'll post it separately but in the next lecture I don't plan to cover stable diffusion in detail I will however cover DDPM So uh with that we will wind up the series but this has been too long. I expected it to wind up somewhere in the beginning of January or something but the number of lectures went longer than I expected. Um so GAN I have I have actually made a entire video on GAN um on I think it was in the I had an explainable AI series. I think in that I have made one entire video on coding GAN. Um so GAN produces more you know uh there is no there is no probabilistic space or anything. GAN produces very sharp images. It's really really good if you want to truly generate images belonging to one specific class. Um let me show you one website actually you must have seen this already. U this person does notexist.com. So these are images produced using GAN. I don't know which GAN model one of them. And each time you refresh this image, these are all this is not even you know the latest uh generative AI model. This model was proposed many many years back I think. And look at these images. These are all human faces but these are all fake people. These are all AI generated images. And look at the kind of you know uh how sharp it is. It's not blurred. It's it's very very sharp. It's impossible to say it's AI generated but it's completely AI generated. And um this model this GAN the original vanilla GAN model came much before. So uh I I'll find a video link and I'll post it in the um discord channel. But all right everyone thank you so much. Uh please attempt the assignment. Uh I want to do one kind of a big project. I want to propose a big project but for that I have to make sure that mo most of you have submitted your assignment. Right now I think most of you have not submitted the assignments. I can only see some names, common names who constantly submit assignments. So if you do submit all the assignments, I can post a like a a difficult project for you to attempt. All right, thank you so much. See you again next week. Uh it's been great. I hope you are enjoying the series. I know that many of you are yet to catch up on some of the last videos, last few lectures videos, but uh you know, do catch up. I think you will learn a lot. Thank you so much. See you once more next next week.