CODE to Fine-Tune NEW SmolVLM on Consumer GPU w QLoRA

Key Takeaways

hello Community today today we're going to f tune a brand new vision language mall and this is a real small tiny little small Vision language mall and you know what it's completely open source we know everything about it and there's a beautiful new code just 15 hours ago it was updated here by hugging face some stable code implementation they give you here the complete notebook and you know what you can run it here on collab and me maybe you can even run it here on your local machine so you see the code is rather easy just have hugging face then you have your normal Laur implementation and then you just load here a very specific data set visual question and answer version two data set you do a little bit of pre-processing for the text and for the image and then you have a trainer and we use here the hugging pH train implementation and you see this is all the code is then you train it that's it and ready so it couldn't be easier yeah you can publish of course your model so this is all the code this is a ready notebook for you to F tune a vision language mod but you know what this is such a beautiful Vision language mall that we should understand what it is so let's go here we go to hugging face we go to image text to text and we go to trending and here you see it hugging face here s small Vision language Mall instruct and you see updated just 14 hours ago by hugging face and it's so free that it's beautiful we can really use this and if you never had a vision language model this is the model to start this is the op mod where you have every knowledge and you can do it and you have to code to really fine tune it and even to evaluate it and pre-train it so let's have a look if we go there and we say here for example small Vision language Mall you see we have now four New Vision language mall we have the base model we have the instruction tuned model we have the instruction tuned and DPO aligned model plus we have here a synthetic data set tun model so let's have a look it starts really easy because if you go there in July of this year this was introduced in in face blazingly fast and remarkable powerful language model and of course we all started with language model and they had small models with 135 million 360 million and then 1.7 billion parameters and you can run this on almost every machine now they already said hey small language model that are really open that we can operate on local devices this could be the future and they give us even the pre- training data set so we know exactly on what they have been trained so the first version had a mix here version and then here for the fine tuning they give you exactly the data they give you the performance and then the data corelation so we know exactly on what data so beautifully we can with the knowledge of the pre-trained data now optimize our fine tuning all the performance data which is great and of course you have here if you go to hugging phase small language mod you have here everything for language model to but we today we are especially interested here in Vision language mall so we go to hugging phas textbook research here you have all the information that you need here for the data set the Corpus and then the language mall then the language Mall version two and then the vision language model and here we are now so let's have a closer look now you have to understand that the base model that they built was of course hugging face own eix now eix now inv version three was built of course on a llama model on an 8 billion free trainable parameter llama model if you want to know more about this version here of course perfectly compatible with all the hugging fist Transformer installation eix was a beautiful open multimodal model except aitr sequences of images and text input to prod used here whatever you need developed be hugging phase multimodal AET 2 version isn't this beautiful and it is so open that we know everything about how it was pre-trained how it was fine tuned everything that we are looking for you have to complete code if you want to further tune it everything is available for you because it is not a proprietary model now if you are looking here of her code this is easy just hold on yep so here we have it here under fine tuning here you have exactly here our vision language mod for the fine tuning as I told you just 15 hours ago now you understand why we have your edix 3 and this is simply the code for your data for your personal application that you find here the model according to your wishes if you want to have more code base you just go to hugging phas here for the Transformer library and you go here for other models okay you need a little bit more of an infrastructure but if your model name you can have here different models and if you want here the Lama 3.21 B Vision instruct that you further supervis the fine tune this is the simple code for you you see just some lines of code and you can find in your own Vision language model now let's see this in action and I simply go here to spaces and we have it here our small Vision language mod and I uploaded here an image and I just said describe the content and tell me the mood of this image now and you can have what's called now an assistant prefix okay and I say hey if you discover here a kind of a sensitive topic to humans like you know hospital or healthc care so some some sens topics hey be kind and gentle as an are in your reply so thisp my system prompt instruction if you want and then you have here the output and this is so nice because sometimes you need here a textual description of the visual content so that this textual description can be further analyzed by your xdi system and in most cases somehow your AI agent will be text based so beautiful transfer from image to text content but let's have a look what in my case here of my thumbnail of my last video what is now here the performance that you get a feeling for this so I get here the image depicts here a futuristic scene in what appears to be an advanced Medical Facility in this setting there are several elements that contribute to a surreal atmosphere in the foreground is dominated by two main objects a robot and a human lying on bat robot stands prominently at the left side of the frame facing towards the viewers right it has a Sleek white body with metallic detailing suggesting it might be some form of an AI or robotic assistance system its hat features large eyes and mouth openings which suest it has been programmed for communication purposes okay this is an interpretation the robot holds a tablet and has displaying here various digital information such as graph and charts okay this implies that a robot could serve multiple function like data analysis or providing support during treatment session for such a tiny tiny tiny little vision language mod that is completely open and you can tune to your likings to your task I think not bad background in the contrast to the immediate foreground the background shows part of a cityscape viewed from high above their tall buildings illuminated against night sky filled with stars indicating either daytime hours before sunset or nighttime when artificial lighting would suffice structures appear modern well-maintained hinting and technological advancement within the depicted environment now to the text element here as also is a text to text and they say okay at the top Center portion of the image EI and Healthcare is written in both capital letters followed by smaller text reading and hospitals below this title another section contains the subtitle cancer and EI further emphasing teams related to healthcare technology integration into cancer treatment except that I think to remember that this EI and Healthcare hospital is all in the same font size I think real nice and then we also I didn't even ask you know I just describe the content and the mood this is all I ask and I get even back here and interpretative Analysis this Vivid depiction likely represents how e technology can enhance traditional Healthcare practices significantly here in lies potential implications where machine take care take over certain task traditionally performed by humans thereby improving efficiency while reducing errors associated with manual human processes such scenarios highlight not only a technical aspect but also etical consideration around automation versus human intervention in clinical settings to summarize the image repres presents here scenario involving both robots likely used for assisting patients and discussion about integrating artificial intelligence into Healthcare System specifically focus on cancer treatment management this blend underscores the transformative power of combining Cutting Edge technology with medical expertise to improve the patient outcomes do you know I couldn't have formulated it any better so take your image go there this is free you can experience this yourself upload your image and see if this m is really great for your specific task of course as always I have here a recommendation and I would say here this is such a beautiful informative blog you know from hugging face and they tell you we have a Blog here explaining everything you can try the demo we just did you have a fine-tuning recipe and you also have here a Transformer integration for a DPO alignment but of course you can go with po reinforcement learning whatever you like so this is a beautiful open model Apache 2.0 small Vision language mod instruct synthatic base and now also instruct and DPO aligned for particular jobs where you need here safeguards into place and I just think this is amazing now if you want to read yet a complete documentation published today November 26 2024 those are the beautiful ERS so thank you to hugging face to providing this to us to the community open source free of charge that we can really use this as a building block to build up our own Vision language M small but yet Mighty they give you here a real nice idea in the ecosystem where they position here their system so you see here you know from Google p Jama I have here different videos on how to find you also this and then we talked about q1 q1 2.5 already and of course there was this proper this open source eix 3 8 billion but based on the Llama but now they moved away from the llama and they think this is something that makes me excited if you move away from a Lama mall and the explanation will come in tomorrow's video so they show you here what they want they go for here better performance and it is cheaper and faster and it is open source yeah pamas we be Moon dream 2 q1 2 Vision language M and they explain here everything from the capability the architecture The Benchmark the memory the throughput if you want here all the details beautiful I just want to show you here where is it here the memory and I think if you work locally and you are kind of this GPU po you see here the GPU RAM and yeah of course smaller is better if you just have here a very limited GPU you see for those modes you need quite some Ram here in in your memory 23 gab but you see here with those Ms you are here in the best case scenario of course at about five or polyam at 6 to 7 here so this gives you an idea where to position this new hugging face small Vision language mod in its four versions great throughput Beautiful video processing pipeline code everything is available to you but I want to show you here it is and they say it here finally today we replaced llama with here their own small language model to 1.7 b as the language backbone and this is what I'm excited about because I think this is an intelligent step yeah they tell you here aggressively compressed your the patched visual information by reducing the information nine times using pixel Shuffle strategy now they say we use patches here of a little bit higher dimensionality yeah for whatever reason and to change the vision backbone to use here the shape optimized sck lip with patches here and the inner patches of 14 * 14 read the documentation if you were into this I think it's a real interesting model it is so beautiful and they follow here the architecture of eix 3 we have some beautiful argumentation and documentation why eix 3 has has this particular structure on the performance data of eix 3 but if you look here I think that's great here you have all the different Benchmark and as you can see here in the first row this is our new M cuan yeah poly Jama Moon dream so you can compare for your particular task which M you are going to choose next anything this is it anything left to do no because I will give you here the link here to this python notebook in the description of this video and we already had a look at the code tested it works just import here your specific fine-tuning training data have a look let it run we have version that it runs here on a free callop so there's nothing holding you back to experience your first fine tuning here of a complete open-source Vision language mod that is transparent that we understand that is provided not by global corporation that is interesting your data but by hugging face it runs on consumer gpus and yes of course we have to accept it has some quanti uh low rank adaptation happening to reduce to compress the model a little bit further that it fits on a consumer GPU but otherwise if you want to do this the first time I think this is the perfect example and this is the perfect time to enter the area of fine-tuning your vision language model I hope you enjoyed it please follow Hugging Facebook please give to ORS who invented this 4 thei Community alike maybe you follow them on the different platforms and it simply would be great to see you in my next video