Build an Autoencoder in 5 Min - Fresh Machine Learning #5

Key Takeaways

is generating novel [Music] data hello world it's SJ and today we're going to learn about a special type of neural network called an autoencoder then we're going to implement our own autoencoder using tensorflow to generate handwritten digits an autoencoder is a simple type of neural network with only three layers it's got an input layer a hidden layer and an output layer just like your mom just kidding what makes an autoencoder special is that the output neurons are directly connected to the input neurons and the goal is to get the output values to match the input values so if I added an image of a dog to the input layer the output layer would then output that same image when we input that image it's a vector in N dimensional space which is sent to the hidden layer after some activation function is applied to it this reduces it to an M dimensional space so there are less Dimensions this process is called dimensionality reduction and it happens in every neural network so we can think of the first layer of the network as the encoder since it's compressing data and the last part as a decoder since it tries to reconstruct the original input from the smaller representations in the hidden layer so when do we use autoencoders all day all day well one use case is data compression kind of like creating a zip file for some data set that you can later unzip another is image search where you search Google using an image as your search query instead of text it's very likely that Google is using autoencoders to help fuel this use case whenever one of Google's Bots is crawling the web and finds an image it'll compress it using an autoencoder that would mean Google would have all of its indexed images compressed in the form of an easily searchable array whenever you search for a new image it'll compress it and find all the points nearest to it in the compression space since there is less noise then rank them according to their similarity there's also the one class classification use case that's when we only have a data set with a single class which is called the target class when we feed it to an autoencoder it'll learn to detect objects of that class when it receives an object that doesn't fit that class category it'll detect it as an anomaly whether it's a deadly virus or fraud attempt system downtime or someone who actually doesn't play Pokémon go that's right autoencoders are for you and if you want to train a deep Network you could use what's called a stacked autoencoder that is a set of autoencoders where the outputs of each layer are wire to the inputs of the successive layer once you train it on some data set you can use those weights to initialize your deep net instead of randomly initialized weights one of the more recent applications of Auto encoders is generating novel yet similar outputs to our puts like faces that look really similar but are different than the input faces for this we use a newer type of autoencoder called a variational auto encoder which learns a distribution around data so it can generate similar but different outputs there are lots of techniques that are used to prevent autoencoders from successfully reconstructing the input image like Den noising where the input is partially corrupted on purpose the idea is that if it can reconstruct an image despite it being corrupted it'll be a more robust decoder so let's build our own simple Auto encoder to learn to detect handwriting digits using tensorflow shall we first we'll import tensorflow the awesome machine learning library then nump the pythonic scientific Computing Library we'll also import our input data which is the collection of handwritten character images then we'll Define our autoencoder hyperparameters we know that each character image is 28x 28 pixels so we'll set the width to 28 then we'll initialize a variable that represents the number of input nodes we also want to have 500 nodes in the hidden layer I generally like to have 2/3 the number of nodes in my hidden layer as my input layer as a starting point one to purposfully corrupt our input data later so that our decoder gets even more robust in its reconstruction over time so let's set our corruption level to. 3 which isn't nearly as high as the US governments once we have these variables we'll use them to help us build nodes we'll start by creating a node for the input data then we'll create a node for the corruption mask which will help us reconstruct the original image we'll then want to create nodes for our hidden variables after we initialize our weights we'll initialize our hidden layer as well as the prime values for each that tie the weights between the encoder decoder and result in an output value after that we Define our model via a function that takes in our variable parameters we'll make sure to get a corrupted version of our input data then create our neural net and compute the sigmoid function to create our hidden State then we'll create our reconstructed input and return that so our model will accept an input and return a reconstructed version of it despite the self-imposed data corruption so we can utilize this function by building a model graph with it which we'll call Z so now that we have our model we need to create a cost function which is what we want to minimize over time the more we minimize it the more accurate our output results will be then we want to create a training algorithm which we'll call train op and use the classic gradient descent algorithm to help train our model which takes the cost function as a parameter so it can continuously minimize when we train it before we start training our model we need to load our data so let's read from our local directory we'll set one hot to True which is a bit widest operation that will make computation faster then initialize the variables for images and labels for both our training and testing data finally we'll begin our training process by initializing a tensorflow session we'll initialize all of our variables first then begin our for Loop which will iterate 100 times for every image and label we will retrieve an input then create a mask using the binomial distribution of our input data and use that as a parameter to run our tensorflow session using the training algorithm we defined earlier we'll calculate a mask for the outer loop as well and print out the results as we go along let's take a look at the results we can see our score gets better and better over time with training and eventually our neural net is able to reconstruct and classify handwritten characters for more information check out the links down below and please subscribe because I'm just getting started for now I've got to go boost some gradients so thanks for watching