Aya Vision - The Challenges & Breakthroughs

Key Takeaways

[Music] hi my name is ahed I'm a research scientist at our lab we have just released I Vision which is a multimodel language model that can understand images and respond in 23 languages so here we gather with the the team that built I vision and we want to talk about it so first I would like to actually set the stage before going into details right so I know our lab is Works based on long-term research agenda and very ambitious research bet and the multimodel is one of these bets why it is well multimodel if we kind of go back to what our field is built around this idea of building intelligent machines the first convening for AI in the 1950s always thought of it as separate subjects so there was computer vision there was audio uh there was text and that's largely because each task was so difficult it was kind of unimaginable that you could tackle all of them at the same time this moment is very profound because as you're describing we now have this moment where the amount of compute power and the ability to model General representations means we can finally address these together but it matters because our own intelligence is so multimodal so if you think about how we navigate the world or even the fact that we manage to get up on these stools like that is very much because of our ability to orientate to use a combination of vision and language and uh auditory coordination to be able to decide and have a level of certainty about our environment so that's what makes I Vision but also this era of AI really special and profan for our ability to model the world as it is and not a fractured kind of partial vision of the world yeah that makes a lot of sense also like if you think about this AI models I mean uh the level of intelligence is really how comprehensive it understand our world right so I think it's unimaginable thinking just with the text domain that we need to incorporate all the domains that we have and that's super interesting research question but now I want to come to you basa as a senior researcher in our team so we always care about of course super interesting research questions but also now we are in Era that whatever we build gets adapted extremely quick why this is important to build a multimodal model for for the community to me the most important aspect of this uh this project is to make the multimodal accessible in anywhere on Earth so because uh one of the biggest vision of our lab is to make AI uh accessible anyware and it can't be only make with the language models because images also captures lots of cultural nuances we for example travel around the world and go somewhere and we see something and don't have any clue about it and there's no text even there's text we don't understand what's written there and we can just use our camera on the phone and just check what it is and ask to our model and it can give us some information about what it sees there I think that's pretty exciting as a vision uh for our lab and as a research project there are so many challenges that probably we will dive dive deep in the rest of the conversation and it's also like as a researcher it's also very interesting as a project I think that's also fun because that hits on why we focused on releasing such an accessible seven billion parameter model this is much more Compact and hopefully usable as well so really nice yeah exactly actually you gave me the perfect pass because I mean we want to build such a i me model that is very accessible which is also required like I was talking with my mom after we released I expense they use in WhatsApp but like she was asking can I send a photo like I was bit shy to say like not yet but now she can so but this this comes with the challenges right because it's really challenging machine learning problem yes so I would like to come to you SAR so you are a a research engineer that worked on this from the beginning so how different it is from building a text on language model to multimodal language model was harder than I anticipated to be honest um some of the systems uh when you when you work with text because text is such like light weight you can get away with like um like not optimizing it and like writing some like taking the easy way out but with multimodal uh what happens is because like images and like and videos if you move to other modalities there are they are so heavy in in terms of processing that you really write need to write like very good systems to kind of train and infer these so I think that was like has been the most challenging part for me yeah so the the infrastructure that is ready for training a text on language models is not enough to train a multi model is that correct yeah not even remotely I see yeah I mean we know as a machine learning people without infrastructure we cannot do anything that's super super important so Oliver you you are a research scholar in our lab so you are relatively Junior yeah and uh I mean and this is really ambitious research P right so when you started and what were you thinking how it would be working in multimodel project and how do you feel now yeah it's a very exciting opportunity especially like I before I only have a roughly idea how like how like motor train going to look like mostly like the idea read from paper on my school project in the the my studies uh but I for this project I feel like especially privilege and lucky to have the chance to participate in an end to endend project or model training project like uh not everyone has that kind of opportunity um so another thing that is really nice about I vision is it is multilingual it can respond any question in 23 languages so this is also really nice because it's connected to Aya movement right this is something that our lab I think proudly kind of established so why multilinguality is such an important and such also challenging kind of research Direction so I have vision is part of a wider commitment that we're doing which is how do we expand Frontier technology to serve the world we're covering 23 languages which is half the world's population and this already is incredibly technically challenging because a few components one is the sheer curse of multilinguality so when we add languages we typically have to do bigger is better scaling uh which to sar's point of the infrastructure required and just the dimensionality of what we're processing as data this adds a lot of complexity but the second is images more than anything else differ around the world so we are actually I think all from different places so um I think Ahmed Baza you're from Turkey Oliver is joining us he's from China so I um from Ireland I live in the US and then sarb is in India so these are all very different worlds and like how do we represent the complexity of languages in those worlds for me this is actually core to machine learning and I think am you and I have talked about this a lot but this is core to how do you adapt large models to new distributions and how do you make sure that that we do this in a way that's efficient that's performant and that we are representing the worlds that we we serve when we share these models I think I mean M this is such a nice also research Battlefield because so I do like you call a battlefield because I mean there are M more than objective that we need to reach right so I want to come to you basa so multilinguality itself as just sah mentioned as a optimization problem because there is lot of languages and there is limited capacity but it is required for accessibility but now we have also another modality vision and text and with for the I Vision one of the the target from the beginning for for our lab keeping having really good performance for vision visual understanding but also Main maintain the textual performance so now we have three different objectives that we want to combine so what do you say about this challenge yeah when you asked the question to Startup about how it's different than building a language model I was thinking it's like multiobjective optimization problem and it's like multi multi objective optimization problem and it's really hard uh the actually difficulty starts with the data because even for the language model it's really difficult to uh Source data for all these 23 languages some languages are really under represented and regarding that multimodal data uh collection it's like maybe not two times but five times more difficult because there is almost no data available that captures all these cultural nuances and aligns with the text comes with it so that was the most difficult part for our uh project I can say also like uh keeping the text performance high and keeping the English performance high and Chinese performance high that was another challenge you shouldn't forget what your mother learned before when you are doing this multiobjective training um and maybe other thing you need to maximize the crosslingual uh transfer for this case because for some languages you won't have maybe very limited amount of data only and you need to uh teach your model uh to like answer um properly for these languages I think one thing that's super nice that you touch upon and I want to continue a bit more the data so it is very challenging machine learning problem in terms of optimization but today all Frontier models starts with the high quality data I know Oliver like one of maybe every daytoday job uh thing that you do look at the data the PRS completions images try to understand try to estimate the quality yeah so when you start how was the quality that is available in public domain for this project yeah for the uh most of the image text here we saw today like those are all academic Benchmark those are very short and concise answer but that's not what we want from an AI model human usually prefer longer answer that explains your answer not just one short like Lether options from the questions so so that's the main challenge we have so for this project we need to like a caption the data set to make it more human prefer more natural more like like more human preferred M it might I add uh also one of the major sources of multimodal data on the internet are like image and ALT caption Pairs and those are very very noisy and if you use that to kind of train your model it it won't understand all the nuances and like the correlations really well so a lot of effort from our end was put on actually like cleaning the data and making sure it is representative of what what the model would actually see in in real in the real world and real world use cases so I think that has been one of the fundamental what do you say innovations that we've kind of focused on yeah I mean that is pretty interesting because U almost like two years ago we released iio 101 a text on the language model and we we had exact the same discussion about the data there were data that is very Benchmark oriented uh very short form completions and now we are in 2 years later later we are building a multi multimodel model and we have the same problem which is sort of interesting but yeah solving this is one of the gist uh to get the performance I totally agree with that so uh S I want to continue about this uh so throughout the development phase of I Vision did you have did you see or did you observe any surprising findings or sort of innovation that you did not expect actually in the beginning I think the most surprising part for me would was from used using our approach getting a decent model was surprisingly easy getting a very good model was surprisingly hard so how do you actually bridge that Gap from going from like a decent model to a very good model which actually makes the makes the real impact me like when people use use it and it actually gives them the Delight that they're looking for um in in that aspect I would say the one of the harder problems that we are trying to solve is keep the extremely good text performance that I expans had and one of the things that we did not want to compromise on is was our text performance and uh I mean I I expense is is a really good model I mean I I I realized that when I was working on preserve it adapting it to like new modalities uh it was it was hard to beat it so um we we experimented with a bunch of ideas we tried adding a lot of Text data to kind of preserve the distribution and it we all saw at gradation and then we uh with and I we were discussing when fine day like you know what what if we try to merge model Trin with on different modalities was kind of a wild idea but it worked so and then we kind of ran with it and kind of ran experiments with it to kind of improve it even further so that that was a very fun moment in like the research Journey for I Vision I would that was also very surprising for me I'm sorry like also how we get like tested out okay we had this half day that like we can SP for some implementation okay let's test it out yeah very yeah was super interesting go oh I was going to say so I actually think that was fantastic cuz I remember the update the next day and you're like oh this is incredible we ran the Benchmark multiple times yeah double check yeah um but you know it's interesting I think there's another aspect to this which was really hard with this project which was what is on Northstar because there's there's a lot of academic benchmarks for multimodal but to model is actually there's not as much work on how do we actually measure where we want to go and so Oliver spent a huge amount of time preparing an evaluation set which we're releasing along with I uh vision and I think this is worth talking about because it's measuring more open-ended problems yeah most of like Benchmark ACC measures the accuracy but when people use the model they don't people don't care about not say don't care about accuracy they more care about the response they got they want like really honest they want the AO to really understand their questions and give more like a fluent human perer natur natural response so we build a win rate Benchmark to to make out to evaluate our model if the model is like really giving response that's more is better and preferred by humans MH yeah yeah a major problem with like the multimodal benchmarks that we have recently is even if you have an extra like comma or full stop and it does like this exact string matching and it'll taret as zero and if it asks for the time if if if if you say the answer is like 4:20 p.m. over like 1620 which is like like you say in like military time that's also marked wrong it's correct but this is a wider crisis with our evaluation I think more and more we want to exactly what you were saying Oliver just capture the essence of where we actually want to go with artificial intelligence because our whole goal here is to create true intelligence and so I think part of that is creating evaluations that capture that actually that was so I'm going to ask this to the basa but my AA moment in the project was exactly this like we were checking the benchmarks I mean I was surprised like how limited it can be and it is extremely limited and I mean your your work is really really important in this building this evaluation I can okay what is your a moment okay I mine um yeah after creating this Benchmark uh with with Oliver um we were almost checking all the answers in also our language not only in English but Oliver was checking in Chinese I was checking in Turkish and I was so shocked that the answers are so good so I couldn't believe because we are generally used to like use models and which is uh good in English right so it's not so surprising a model can perform really well in English uh but when it performs really well in in your language which is just used in your own country it's really both I mean you you really feel proud at that moment and you get happy that what you worked on and what you achieved that's great so it was it's a super nice discussion I would like to close slowly now but just a closing and closing question and like I would like to be really brief so what do you think now we have I Vision what is next that we should go forward mhm I think um modalities are really important we started with image but I I think we should expand the modalities we cover so probably video should be the next I would say as as like a youngest member of our team what do you say I would think like more fine gr understanding of images and like like let's the multimodel model can really help your daily task for like more deeper understanding of those documents those like daily task they can really be like an aid to your daily life s do you have an answer I think it is it has to be definitely speech if if the next modality we would be adding uh because I I think the one of the core ideas is to make this model more accessible right and if we are are we if you are targeting people who uh who know language which are in like the tail end of the distribution uh it's most people like like a significant chunk of that population does not know how actually does not know to like read or WR write the language they know how to speak the language so if you are able to like point and ask questions and then get a response back that is like a magical experience I would say and that I think that is the kind of intelligence systems we should be like moving uh towards what does sar think I think he said it all that was amazing that was that was perfect I mean I really enjoyed discussing with you with you all guys so thanks thanks everyone thank you thank you ahed [Music]