Data-efficient Image Transformers EXPLAINED! Facebook AI's DeiT paper

Key Takeaways

welcome we are here to discuss the best kind of paper you know those papers that are published on archive just before the end of the year because that is when everybody wants to finish stuff but does not directly have a conference to submit to so great christmas reading papers or not well it does not matter how much you like christmas and new year's eve papers since this one here comes with breaking news and the transformers are here to take over computer vision [Music] um no this is actually old news we had a previous video about the vit vision transformer by google that achieved state of the art on image recognition with a pure transformer architecture but by also using 300 million more images of their own not publicly released data find the link to the video in the description below then the actual news of this video is that facebook now managed to train a pure transformer too but unlike google they trained only on imagenet and achieved competitive results in this video we will discuss why is this surprising or even an achievement from facebook research how did they do it and what does it mean for transformers training on academic data sets as mentioned in the previous video about google's vision transformer the transformer architecture is more general than cnns or lstms for example cnns already know how to look at images patch by patch with convolutions containing the same parameters for every patch lstms on the other hand know how to handle sequences from left to right this inductive bias is intrinsically bound to their architecture but the transformer does not have these inductive biases for it the data does not have to be a 2d matrix an image or a sequence like text as long as it is a set the order if there is one is encoded in the positional embeddings hence the transformer can do computer vision and text processing or both combined by learning how to handle the visual or textual domains from the data itself but for learning this it has to see more data than the cnn or lstm counterparts then of course it is not surprising that google's vit had to look at 300 million more images to beat state of the art on imagenet on image recognition the facebook's deit's number of 84.4 percent on imagenet means a lot with around 300 times less data crazy how did they do it facebook's deit model processes images very similarly to google's vit model 16 by 16 image patches are transformed to image vectors because vectors are the representations that transformers naturally work with the authors of the deity paper call these image vectors patch tokens call them whatever you like these patch tokens are processed by transformer layers built by the usual multi-head self-attention and feed-forward layers that in the end have to predict the correct label of the image prediction which is enforced by the cross entropy loss but here one can see something that google's vid model did not have a distillation token this special token has to be predicted by a transformer to match the output of a cnn teacher model why a cnn because it has more priors about images therefore the teacher cnn model can learn from less data more easily than a transformer which without the inductive bias has more degrees of freedom and needs more data to train distillation is a cool idea to apply for the transformer here which simultaneously but with two different heads has to both maximize the cross-entropy loss to make correct predictions and also make the teacher's prediction this is distillation in the way of the transformer this is the way but unlike vit of the google researchers deit from facebook does not work with a lot of data and to alleviate this the authors apply three common techniques for cnns on transformers data augmentation optimization and regularization for data augmentation the authors do not provide that many details except that almost every data augmentation technique is useful if you are not familiar with data augmentation for images here are some examples flipping the image on both axes when it makes sense or slightly changing the contrast or brightness or saturation are useful ways to change the image in the eyes of the neural network but without changing its meaning now about the optimizers because transformers are very sensitive to different optimizers the authors choose the best one and search for the best hyper parameters like the learning rates and weight decay by cross validation well i mean for this you need the facebook computing power right furthermore after training they fine tune the transformer on images of a greater resolution than during training they train on 224 by 224 pixel images and fine tune on 384 by 384 pixel images by this even though the number of instances in the training data is the same for the network the effective training data is increased because the scaled higher resolution images are new for the network do not forget that these images are then converted to patch tokens or image vectors as ms coffee bean likes to call them so the transformer can work with these vectors but in order to make this work properly the authors need to upscale the images with a special bicubic interpolation with a specific regularization that preserves the norm of the image vectors if they would not do so the norm of the original image vectors would be too different to the norm of the upscaled image and the trained transformer would not recognize the relatedness between the data seen during training and what it gets during fine tuning okay so to generalize and to conclude d-e-i-t is like vit but i side trained with a procedure more adapted to a data starving regime and this is the cool thing about it leaving vit state of the art numbers on imagenet a little in the shadow especially now that it seems like transformers are here to stay it is great that a step towards data efficiency was made by deit making it possible to train a transformer on image data sets of millions of images instead of hundreds of millions of images and this is great perhaps you remember what miss coffee bean said in the vit video about transformers taking over cnns perhaps not but i can tell you that she is very surprised by this deit paper coming so quickly after vit and addressing the data efficiency aspect of visual transformers right away she really thought this might take longer but it's machine learning what did you expect you turn your head for a second and next an awesome thing happens how can you keep up with this but rather how to not be overly excited by this miss coffee bean are you done yes i am okay bye you