Python Tutorial: Encoder decoder architecture

Key Takeaways

now you will learn about the encoder/decoder architecture in more detail a machine translation model works by first consuming the words of the source language sequentially and then sequentially predicting the corresponding words in the target language however under the hood is actually two different models an encoder and a decoder let's first look at the encoder in our example the encoder takes in 100s of english words as inputs and produces a compressed representation of the inputs known as a context vector then the decoder consumes the context vector as an input and produces probabilistic predictions for each time step the word for a given time step is selected as the word with the highest probability note that though the inputs to the encoder are ones and zeros the decoder produces continuous probabilistic outputs these models are also called sequence to sequence models because they map a sequence that is an English sentence to another sequence that is a French sentence let's understand this - an analogy a teacher is explaining what an elephant is and you never seen one the teacher explains it as it's big has large ears and trunk then you create a mental image of an elephant this would be the encoding process then your friend asks what does an elephant look like then you decode that mental image of the elephant by explaining the features of an elephant or perhaps by drawing one this is the decoding process to understand the encoder/decoder architecture better let's implement a simple model that reverses a sentence first the encoder receives an one hot representation of the sentence and converts it to word ids next the decoder takes in the word ids reverses them and converts the reverse ids back to the 100% ation resulting in the reverse sentence in order to implement the encoder you will first implement a function called words to one hunt which will convert a given list of words to one not vectors the resulting one hunter will have the shape number of words by Nam classes Nam classes is three in our example the encoder function is a simple function that takes in an array of one hot wetters as the argument and returns the word ids corresponding to the one hot vectors to obtain the word ids from the one hot vectors you can use the NP dark max function and Peter Arg max computes the index of the maximum element along a given axis since one hot vectors are laid out along axis one you can use axis equals one after defining the two functions you can encode a given sentence to do that first call the words two one horn honks as the first argument and word to index as the second argument to get the one hot vectors next call the encoder function with the one hunters as the argument to obtain the context vector finally print the context vector the context contains the corresponding word IDs of the words the decoder will take in the word IDs reverse the IDS and then return the 100's of the reversed words next you write the decoder as a function which takes in the context vector produced by the encoder and produces one hunters of the reversed words the decoder function first reverses the word IDs in the context vector in numpy and 1d array can be reversed by adding Colin Colin minus one within square brackets after reversing the word IDs the one hot vectors are obtained by calling the two categorical function you also need a helper function one hot - words which will convert a set of one hot vectors to human readable words to do that the one on two words function takes in an array of one hot vectors and a dictionary index to word which Maps a word ID to a word consequentially indexed word is the reverse dictionary of word - in this used in the encoder finally you can compute the decoder output by calling the decoder function with the context vector as an argument and obtain the reversed words by calling d1 hot words function with the correct arguments now let's implement the sentence