PyTorch ViT: The Ultimate Guide to Fine-Tuning for Object Identification (COLAB)

Key Takeaways

hi Community today we code Vision Transformers in my last videos I showed you how to do an image classification I showed you where the self-attention in Vision Transformer is and the status about cancer screening in 2023 independent if you have a sentence Transformer or a vision Transformer to Transformer architecture is almost identical so this means we have a pre-training model and then we find you on this model for a specific task and today I show you how to find your Envision Transformer on image classification you could do it in an industrial application or for a biological we take now the idea we have a drone that takes pictures and we have now to identify the type and the degree of the infestation and to fine-tune a vision transformer for this task and as I know you want to follow along in a notebook I have chosen here a very particular implementation from hugging face from Nate raw and it is called fine tune Vision Transformers for image classification with hugging face Transformers and here you have your color notebook I'll leave you the link in the description of this video I just adapted the notebook a little bit so here we go you know what a vision Transformer is and that has an encoder stack we went through this already so what we do we say pip install our data sets from hugging face enter Transformers from hugging face and then load the data set that we will use for fine tuning Our pre-trained Vision Transformers so this means we need now a label data set for our supervised learning and there we have now if you go here to hacking face and a specific data set it's called the beans so easy we have three parts here we have here a string and this string describes here a specific file path we do not need then we have here an image this is an important part and we have a label a class label describing the condition of this image for image classification and we have the label 0 1 and 2 2 means healthy and the other one indicate some past manifestation if you want to have more information on this we have here the data fields and everything and notice that the data set is already split in a training data set with 1000 images a validation data set with about 100 and a test data set also with about 100 images beautiful so here we go now we load these beans data set and then we have our images and our label describing the images in three classify classification tasks so yeah we have a look at the keys that you can see it data is downloading beautiful and I want to show you the type that we are working here because when we're working with tensors you have to be careful to transfer your dictionaries to tensors but at first here our keys are very clearly in our dictionary now that we have we have the training data set the validation and the test data set and if you look here this is exactly the number of pictures under the number of images that I just showed you so beautiful and then we just yeah we can have a look at the format yes of course but the interesting thing here is we use here the python Imaging library for the python interpreter with some image editing capabilities so we use the pill format beautiful let's have a look at one of those pictures so here we go this is here a typical picture that we're gonna analyze and you know that here this is not a healthy pic a healthy uh plant but we will have one of the first two labels now you can try to augment it with some luminance Channel or whatsoever but this is not the point in this video I'll show you in a later video but I would like to show you what we can do now with the classification so at first we have here healthy this has the index two we have then the angular leaf spot I think this is the index one and the bean rust really heavy past the infestation is I think zero so this is it let's print it out and we know that this picture this picture here is brown spot here on the leaf is classified as angular leaf spot Ted with angular leaf spot is here is the scenes in bean plants beautiful now you can say show me more examples so this is just that you have a three times three image visualization where you have the classes now in each row there's nothing to this code you have it in a link in the description of this video and it is just to show you angular leaf spot has irregular brown patches the bean rust has circle around spots surrounded by the white yellow ring structures and healthy well healthy just looks healthy a green leaf as you would expect so here we are the first row you have angular leaf spot so this is stock patches on our Leaf the bean rust this is the real serious part with here the little white spots around and then example three examples of a healthy Leaf beautiful so now we have to prepare our images for our vision Transformer and we have some beautiful thing here and yes I know we have a feature extractor what a feature extractor does is just prepares the pictures for you okay yes use the wrong transformation on your image and the model won't be able to understand what is seeing yes so more or less it is a normalization so now to normalize our images for our vision Transformer model you have some parameters you have here the sequence of means for each Channel you can determine to be used when you normalize your image the sequence of the standard deviation for your RGB red green and blue Channel you can say here Boolean operator if you want to normalize it yes or not you can choose here with a boon operator to resize the image and if you choose yes you have here the size normally default is 224 pixels time 224 pixels so beautiful so here we go and now of course we have here a vision Transformer feature extractor from a specific pre-trained model and our pre-trained model no wonder is what we used in the last two videos from Google The Vision Transformer base model with a patch 16 times 16 image size 224 times 224 with millions of pixels of images I'm sorry so and if you are not sure here we go hugging face you go here Google Vision Transformer base patch 16224 and you see here in the description this is the pre-trained model on the imagenet which has 14 million images as 21 843 different classes and the images at a resolution of 224 times 224 exactly what we need only a pre-trained model this is it and we fine tune it now on our specific data set so here we go yeah this feature extractor if you think this is something miraculous it is rather easy it just tells you yes I did normalize I did rescale yes this is it the wave is 224 times 224 pixel beautiful nothing else now if you want of course here you have to go from the image pixels now to a tensor and nothing easier than this you put in your image you say you return tensor please a pie torch tensor and your pixel values is now a tensor for each three channels remember we have an RGB red green and blue channel so for each of the three channels we have here the pixel values normalized so we do not have I don't know 115 000.2 and minus 6 million 800 but we have normalized pixel values beautiful okay now that we know how to read in the images and transform them let's write a function that will put those two things together to process a single example so what we do we have our feature extractor which will return the pixel values as a pi torch tensor and we want to have attached our labels remember the three classes Alfie rust and this pattern 0 1 and 3. let's do this we Define here this process example function and if we apply it here for example here on a single picture you have here your pixel values this is the tensor for RGB and then you have now attached a specific label describing the condition of the leaf in these images this is it great next Point yeah there's in pandas if you remember we had a panda data frame and then we used here to transform command for example to find a square root and the result of Euler's number raised to each element of the data frame and we just used here append the data frame dot transform and the function that you want to have and now more or less we do yambert we can do the same and now if we want to augment our data set we will use something similar we will use here the hugging face data set dot map command in batch modes and this will let you control the size of the generated data set freely it is nothing in particular it is just what we what I just showed you it is just here the most efficient way to encode it so we have here our transform function for the batch so we take a list of images and turn them into the pixel values with our feature extractor I just showed you in regard a pytorch tensorback and we include the label and we do this then we have defined this function yes yes yes and now we yes we already have it in cash beautiful we can now directly apply this to our complete our whole data set using the command data set with transform and we do this and now everything is now transformed yeah I know I know I know I'm coming to the training part yeah and if one have it look at this very easily what we achieved we have here the pixel value the tensor values and you're not going to believe it here is our label attached to it and now it is done for the complete training data set on what we will fine-tune our vision Transformers finally training so what we need for the training pipeline we have to define a collate function I assure you we have to Define an evolution Matrix it will be accuracy we have to load the pre-trained checkpoints yes and finally the training configuration with the learning rate the hyper parameter and everything we need to fine-tune now our pre-trained model now collate function is exactly what we had last time I don't have to repeat this the evaluation metric as I told you is simply accuracy you are familiar with this particular part and then of course we need a pre-trained model and as I told you we will use this from Google vat base patch 16 224 with our how many million pixel 14 million images beautiful so here we go we have now the head for image classification and this is now for the fine tuning part you remember we have a model Transformer with an image classification head on top exactly like we did with bird where we had them for the fine tuning you put a specific head on top of your bird model and here we do exactly the same we take now our vision Transformer model and we put here a classification head on top this is already pre-defined for us of course in hugging phase so all we have to do for constructing for Designing our model is we take here now in the fine tuning task this additional header we say we want to have apply this to the pre-trained model pre-trained model is of course our big model here and we want to have included the ID to label and the label to ID class this is of course clear that we have a forward and backward compatibility and we are ready to train now the only yes there's a special command but what we need now is for when we use the hugging face train and we do not code it in pure Pi torch but we're gonna use this trainer we have arguments you know these arguments where's the output directory what is the batch size I tried to use 64 by the way for our GPU the number of training epochs I set it to 8. you can choose here to go with half Precision but I say no I want to go with floating point 32. what steps you save it the learning rate you can Define if you want to push it to the hugging phase Hub as I know because I have just a demonstration if you want to load the best model at the end yes and and end and after defining everything we are ready that we say okay let's build now the trainer to fine tune our pre-trained vision Transformer model so we Trend we load from the Transformer from hugging face our trainer a trainer is simple but a feature complete training and evaluation Loop for pi torch and it is optimized for hugging face Transformers and this is exactly what we use here so this is the most simple way to build a training Loop so we say trainer the model is our model we defined the training arguments we just defined here this is here the data collator we defined the compute Matrix is accuracy the training data set is prepared the evaluation data set is prepared and for our tokenizer I showed you how we do this with our feature extractor beautiful yeah maybe I should execute this I didn't execute this two oh my goodness I'm sorry I'm sorry I forgot to execute everything the trainer is here and then we are now running trainer dot train this is the main command for the training and as you can see here oh gee how long this will take I have no idea number of epoxies eight my batch size is 64. oh yeah let's see if our TPU Ram uh would say at eighty percent but I hope it will be stable I hope we will not Crush but for the time being for eight epos it tells us about five six minutes I will be back with you in a second and we are back and you can see we are close finishing our 8 Epoch we have our 85 million free trainable parameters eight epos GPO Rambo stable system run was stable beautiful you can see here our evaluation our running evaluation has an accuracy of 99 so really nice yes yes yes yes yes of course if you do not want to use the hugging face trainer class you can go with your native pie torch remember this for epochs in range of number epochs is eight for each batch you have here the model with the patch defines the output you have the last backward propagation you have your Optimizer your learning rate scheduler everything as you know what we use here to train a functionality and also if we do not evaluate our newly fine-tuned Vision Transformer model here are our validation data set and our test data set and as you can see here the evaluation accuracy is also 99 beautiful so we it's set out to it to train to fine-tune this on our pre-trained uh Vision Transformer model and we achieve this now here you have in the notebook also if you want to push this model with your name with your credential to the hugging face hub for the community this is the way to do it you see it's really easy this is all that it takes and you have published your model this was it you see it is very easy to fine tune it but if you want to go a further step I want to give you two literature insights at first please have a look at the GitHub from Google research division Transformer here you have Vision Transformer and a mixer architecture here you have everything in a repository where to explain things beautifully of course also as you see here with imagenet 21k data set this is how we did it of course here you have then for the TPU cluster in the cloud you have here fine tuner model but as you can see here we are operating down in Jack's on nickel Cloud infrastructure if you are interested if you want to improve your training or your fine tuning there's a very interesting article a preprint from archive from June 2022 so version two and they examined how the best way to train and fine-tune vision Transformers and I said I found if you prepare your visual data your augmentation your regularization of your data you can get models with the same performance with compared to models that are an order of magnitude more have more training data so the better you prepare your visual data the more efficient your system your vision Transformer system will be well of course and just to give you a short preview here for example if you look here at this visualization you can see here your vision Transformer base and large and you see here normally if you have here the pre-training data size with 1 million this is about I don't know let's take the blue one the base model 32 you have here about 63 percent let's say here 63 percent and if you take the same pre-training data site but you augment it in a beautiful way and you have a regularizer then you can achieve suddenly I don't know 78 so you go for 163 to 78 here if you're in your accuracy after the fine tuning what we have just done so the better you prepare the input data the clearer all the pictures the easier it is the model to achieve a higher accuracy well this is exactly more or less what we expected so here you have your further literature I think it's really interesting but otherwise this concludes here our training our fine tuning a vision Transformer on a pre-trained visual Transformer I hope you enjoyed it this task is finished you see GPU Ram system Ram everything stayed within the normal boundaries so give it a try experience yourself maybe you do not take here a data set of I don't know leaves you can do it whatever you are interested in maybe it's course or maybe it's some technical content maybe some helicopter whatever you like if you have a training data set available a labeled data set where you have an image and a particular description that characterizes this image and you want to have a classification task it is as easy as I showed you almost the same as in there NLP Bird model so give it a try with vision Transformers I hope you enjoyed it and I see you in my next video