Why wait for KOSMOS-1? Code a VISION - LLM w/ ViT, Flan-T5 LLM and BLIP-2: Multimodal LLMs (MLLM)

Key Takeaways

hello Community Microsoft released a new product Cosmos one so let's have a look at this from March 1st 2023 here a beautiful publication from Microsoft about multimodal large language models and they call their system Cosmos one and if you read the paper you think hey there are some nice ideas inside but then you read Microsoft says it plans to make Cosmos one available to the developers and current GitHub page has no cosmos-specific code available and then I remember that I asked you about a week ago if you were interested that we code together some visual language system and I said hey I'm going with a vision Transformer the latest Transformer technology and now not from Microsoft the Bing chat but a flying T5 model from Google so you voted 62 words 2 3 said about yeah show us the code let's code together and one third also said hey yeah we would like to understand a complete Theory how to do this and how we can pre-train or fine-tune those monster systems like Vision Transformer or flying T5 systems and so I started to code and the videos already now so this is the cover of my first video I combined here Vision Transformers with a flying T5 system and the missing part to combine those system is blip2 so beautiful and the second video that's already uploaded to Google not released yet is here a complete code example I code here division Transformer the flying T5 llm and I have here complete code example where we build an app and I'll show you how it works now if you combine this functionality with the result that Microsoft showed here and its Cosmos one paper it is almost identical now if you look at the solution here you see that we use three Transformer we have here a vision Transformer we have here an llm and I use here Transformer architecture and in the middle here the combining element the Q format is also a Transformer so we have apple apple and apple and this is the easy thing because we're gonna freeze now division Transformer and the language Transformer and we have the ability to fine tune each element individually and maybe we do not even need a Microsoft supercomputer Center who does everything for everybody in one step on a supercomputer and I remember when I was looking for a technical solution this was a research paper number 17 and you're not going to believe it it's from Salesforce research and they described blip of course it is from end of January so it's more than one month ago in research this is a year but a very nice idea never thought I would have any Salesforce paper so have a look at this now this here is my YouTube Studio my timeline you see those last two videos are released today for me is March 2nd and you see those videos are already uploaded but they are triggered with a timer and you see that these two videos that I talked to you about let's have a little bit close up you you they would start they would be released on March 8th and March 10. and now reading that Microsoft is not providing the code of Cosmos and not open source decoder anything at all I thought hey maybe you would be interested that I release my two open source video with the code that you can go and you can build it this weekend of course there's another Beauty this system here has a vision Transformer as an individual part that you can go and fine-tune the vision Transformer let's say you are interested on a specific app and this app is only about I don't know military helicopter military weapon system and you have I don't know 1000 pictures of military helicopters and you have about I don't know thousand books about the content of military helicopter so you can train a vision Transformer individually then you go and you train or fine tune your plan T5 architecture and then just combine it with blip 2. so you have the advantage of going step by step we have a modular approach and we do not need this Microsoft Azure super computing we have the ability to fine-tune each system according to your needs of course what comes out is not like a Microsoft everything for everybody this is a highly specialized app that I'm gonna build here with you and it will only know then about military helicopter but for this you have a visual Q and A ability that is second to none so of course T5 and flan T5 there were a lot of questions for my viewers the difference and I have here six videos how to fine tune T5 and flan T5 system so they will start to be released March 16 March 18 March 20 March 24. the first three is about fine tuning P5 and flying T5 llms I show you here the complete code and give you a tutorial in pi torch on a free Call of notebook and then I have a little bit more professional code if you want I show you how you can use this with hugging face accelerate on a multi-gpu or multi-tpu environment and then I have a video where I show you exactly if you're interested in the cost how much does it cost just to fine-tune F9 T5 model with either we go with the three billion parameter the 11 billion parameter model I show you what it costs you so this is my future timeline for the next two three weeks I want to release those videos that you know what is coming up but I just wanted to tell you you know you don't have to wait for Microsoft Cosmos 1 and then everything is only on a Microsoft supercomputer no there are alternatives Technologies available today you can find them you can combine them you can code them you can make them open source available I share my code whatever I learned with you have a look code will be ready when the videos are released will be in a jupyter notebook and you can go and if you're interested if you want in the community tab you can vote now for an early release of my original llam videos so you can build your own Cosmos one in open source this weekend you do not have to wait for some proprietary non-open source application that Microsoft might release to some Developers I hope you enjoyed it and I see you in my next video