Image Classification Computer Vision with Hugging Face Transformers -Google ViT - Python ML Tutorial

Key Takeaways

dear fellow coders welcome to one little coder if you ask any nlp developer who uses hugging face a lot and you ask them like what do you want from hugging face or what do you like about hugging face and that you know you don't have somewhere else that is usually hugging face transformers and computer vision so an intersection of hugging phase transformers and computer vision is something a lot of nlp developers actually are or i should say machine learning developers who use hugging face a lot is something that they dream about and that dream is true today or probably i should say yesterday because hugging face now has a computer vision model it's called vision transformer and then it's also integrated as part of your existing transformer the advantage is you use the same api you use the same syntax and then now you can actually build a very good computer vision model using hugging face so hugging phase the company that is known for nlp has got a vision transformer which is for computer vision and this video is going to be about that in this video we're going to see how to use hugging fish hosted inference api for computer vision problem which in this case is image classification it's going to basically find out what the image is and then we're going to move into google collab notebook to see how to write a python code for the same thing so first let's get started so this is the page where the hugging face model is listed so this to be honest this is actually the model from google and it's on hugging face model hub so this model which is uh vit it stands for vision transformer it's a pre-trained model on you know image net 21k like 14 million images um with eight hundred 21 forty three classes at a resolution of two hundred and twenty four by two hundred and twenty four and it's also fine tuned on uh another one million images with a thousand classes at a resolution of two to four by two to four so it was introduced in this paper that uh that is listed here and this model uh this this pre-trained model you can actually use it within transformers just uh like you know how you would use your nlp libraries like gpd2 um gpt neo or any other nlp library so we're going to use it in the same way and same steps so and we would actually can now use hugging phase transformer and image classification free trained model and actually predict what image is this so first let's start with the hosted inference api so i'm a big fan of hugging fish hosted inference api primarily for the reason because this works on phone and then you can test how something works before even you know you spend um uh gbs uh downloading a model like um if you're doing it on google cola maybe it's a different topic altogether but if you're going to do it on your local machine it takes a lot of time um so if you if you if you want to test a model so this is an amazing option and if you are somebody who uploads a lot of model to hugging face model hub my request to you is make sure that your model works with hosted inference api so let's get started so what i'm going to show you is i have an image like a car and then i'm going to upload that car here so let me drag and drop it here and then after i drag and drop you can see that it is computing and after it computes it says it's a sports car with 0.845 uh probability um or score i should say and it also calls it convertible and then probably like 0.03 car wheel but what what stands out is that it has got an 84 percent probability or score to say that this is a sports car which means it actually works very nice one thing that you should probably understand is um we don't know whether this image was actually part of the training image highly unlikely it was not what if maybe this model was not even available when uh this image was uh this pre-trained model was created and that is that is the beauty of the entire transfer learning um world right so you like even if this image was not available there so still it works fine if you want to fine tune it for a different problem still it works fine so yeah that is that is all the greatest thing about it now that we actually see that this model works fine and um you know we we have tested it with the hosted inference api from hugging face model hub let's quickly move on to the google collab and then see how to actually do it uh on your python environment so the first step is you need to install the latest version of uh hugging face model latest version of transformers from let me put actually hugging face model hub okay so that's the first step and your next step is uh import import required libraries so you need transformers for um actually creating uh the model download sorry downloading the model uh doing a training doing classification and then finally predicting so you need a transformers va feature extractor va v80 feature extractor and v80 for uh for image classification these two classes from there and then you need image from pll this is primarily to process image pll if you are not familiar with pl is the python 3 fork or i should say like pillow is the python 3 fork of pil and once you install pillow you would get pl and then finally you need a request because we are actually trying to download the image from internet which we are going to use for classification so because of that like if you only want somebody to upload an image and then do it you may not require a request but because we are going to take a url from a user and then download that image from internet and then figure out what image it is like using classification that's why we need request after you've got all these three libraries next is define um in download image from the given url so we given you we give a url and from that url we actually can you know download the image so this is like the first step is you don't give the url the second step is you actually download the image and store it in image and the next one is i'm sorry the next one is display the image display the image the i should say downloaded image i mean it doesn't really matter a lot because what i'm saying you're actually listening to me but in case if you or just reading if somebody you've shared this google color notebook please please feel free to share this google collab notebook which i would link in the youtube description with anybody you like the whole idea is uh everybody should enjoy the way hugging face uh you know it's developing uh their open source libraries amazing contribution from the team once again you know heartfelt thanks from a fan of hugging face for whatever they are doing whatever they have been doing with nlp and then what they're going to do for computer vision i'm truly amazed by the team so let's display the image okay it's a cat or cats cute cats okay so i don't know what type of cat it is probably like maybe machine learning will teach me but i don't have that much knowledge about cat so i'll just simply call it cat or cute cats once you have that thing now it's time for us to download pre-trained pre-trained models or model so we have a feature extractor uh it's very similar to any hugging face uh or transformer model uh that you have used so if you have seen my recent videos about gpt 2 gpt neo you would have seen a very similar fashion so you would have a tokenizer in nlp world and then here we have a feature extractor and then we have uh the actual model so feature extractor instead of tokenizer and then you have uh the model so once you download uh or assign these two into the respective objects then the next step that you're going to do is you're going to extract features from the input model that you have extract features which is what we're going to pass on to the model for prediction so now model i should say maybe like inference oops and from that you are going to actually calculate the logits or just take the logits outside and then pass on that logit uh that your model has given you to predict so like we saw initially that uh there are this this model is uh fine-tuned on thousand classes and we are going to pass it to one of those like we are going to we are trying to find out which of these thousand classes come very closer to the one that has been predicted so that's what we're going to do so uh yeah i'm not going to say it says like model predicts so text model predicts one of the thousand image net classes so i think i've ran this okay i didn't run this run it run it run it run it so let's see so it says it's an egyptian cat so i don't know if it is an egyptian cat but uh but it says it's an egyptian cat so let's try with something that i understand or probably like you also understand which is a beautiful laptop so what is in this url is instead of this url i'm going to give this url downloading the image successfully done now display the image and this is quite the high resolution image um i don't know why did i pick up a very high resolution image but yeah this is a laptop that all of us almost are familiar with um a macbook but again um it's a laptop right so the model download we don't have to do it again because we are going to just do inference and we have to do feature extraction again model inference get the logits and then try to find the class and it says it's a notebook or a notebook computer and that is exactly right because it's a notebook probably like uh i think in the future we would probably would love to have a class called a macbook because we are a different class altogether but uh this does its job so if you want to try something something else as well uh i don't know what what object we can try like maybe like white house let's see take white house white house white house okay um and then which image like i don't want a very high resolution image but all the images that i've got are very high resolution probably this looks okay copy image address and then go to the notebook and then paste it probably like this okay let's see if it works does it actually display the image i think it didn't download it properly maybe i might have to display the image oh it's uh i think that's something okay i did a mistake because this this image if you see the resolution of this image it is 4206 by 2200 so it's a huge image i don't know what's going to happen let's see let's extract some features from it model inference get the logit and then do the prediction and it says it's a palace okay probably american president lives in a palace so and that palace is called white house and we know that it is white house and that desert so in a very few lines of code that almost all nlp developers are already familiar with which is a very popular hugging face in tags now you can actually do a very good computer vision so either you can either either you can um create a streamled web application using this or you can have a cli tool or you can just you know like do hand like you you can use it for any any number of purpose um that that you would like to you know use it for so this video was supposed to tell you how to do computer vision using hugging face with google's vision transformer and the task that we did is image classification but what hacking face is saying is that you can actually use uh this for if if you have uh like this model fine tuned for some other task other than image classification you can use it for that purpose as well but if you want to try out with any image please try to use hugging face hosted inference api but um please do not kill their server so if you want to use it for experiment purpose try to download the model and test it out yourself so that you are not overloading the server and once again a huge respect and thanks to the hugging face team for uh for actively developing and then releasing open source product this means a lot for the open source community machine learning community and especially for somebody like me who's an applied uh i would say data scientist i'm not doing research i'm not you know going um going and you know creating state of the art model but if somebody creates a state-of-the-art model people like hugging fish get it to me which i can actually put it in good use in an industry setup and thanks to hacking face for that and if you have any questions in this video please let me know in the comment section otherwise please give us a shout out to hugging face team for the amazing work that they're doing and also please give me a thumbs up i would really appreciate that thank you so much for watching this video see you on the next one take care