Meta's Daniel Bolya on Perception Encoder and Improving Visual Understanding

Key Takeaways

So if I say a sign that says apple market clip will give you some things that loosely associated with that text. So here's a sign. There's apple somewhere on this sign. This has the big apple. So it's sort of related, but the first result from P are all exactly what you asked for. So like a sign that says apple market. And you could even do compositionality. So maybe like a red sign that says apple market. And P's first result will be that red sign. Sometimes you're looking for an image or a video with uh really specific content like packages loaded in a delivery vehicle. You hit search and some of the results that you see might be exactly what you're looking for. Uh some of the other results might be a little bit different. Well, recently a team at Meta released a new model called perception encoder which may uh drastically improve some of the results that you see not only from image search but many other tasks that require visual understanding. And today we're going to be learning more about perception encoder and comparing some of its capabilities to other models like clip. And to help us learn more about perception encoder, we are joined by uh Daniel Boa, who is a research scientist at Meta and was a major contributor to the project. Daniel, would you like to say hi to everybody and introduce yourself? >> Yeah. Hi, and thanks Patrick for introducing me. Uh, nice to be here and I'm happy to explain anything about perception encoder and how it works. >> Nice. Okay. Welcome. Thank you so much for joining us. Um, we are also joined by Peter Robisho who is the machine learning lead here at Rooflow. Peter, would you also like to say hi? >> Yeah. Hi. Uh, I'm Peter Robisho. Um, I've worked a little bit with Perception Encoder building some fun demos using the model and just overall a big fan. Uh, I have a little bit of experience with it. >> Nice. To get started, um, Daniel, I have a lot of questions for you. Um would you please just let us know what is perception encoder? What's an embedding? What are we looking at here? >> Yeah. So perception encoder is a family of vision encoders. Basically vision encoder is what takes an image or a video and maps it into an embedding or you know a vector of numbers that the a model can understand uh down the line. And so vision encoders are basically a core component that sees the world and translates it to some like sort of semantic understanding that our models can recognize. And PE comes in three flavors. The one we'll be focusing on mostly today is P core which is sort of the uh backbone of PE and that is trained to align images to text. So as you can see in like the top left example uh p core out of the box can align an image for instance this uh cat sitting on some grass to a line of text that sort of describes the image like a judgmental cat sitting on the grass. So P is actually very good at sort of aligning the representations of the image to uh a corresponding text. But there's also two additional models that we won't be sort of covering today. But though that can be used as components in sort of downstream um machine learning architectures. For instance, P language works very very well on um language modeling tasks. So if you're building a language model and need a vision encoder uh for that language model, uh P language is a very good choice for that. So that would be tasks like OCR Q&A like trying to figure out what these um bikers are holding up uh what what that spells. So like in this case it would be Indianapolis. And then we also have the spatial family of models uh for directly spatial tasks that don't really rely on text. So this would be things like detection or in this case you can see this depth map that was generated by the model. So with those three family of models, we sort of cover a lot of use cases that you would need for vision encoding. And I'll I'll hand it back to Patrick. you know, in my uh history here with uh doing live sessions at Rooflow, we've had we've had a lot of dog examples in uh in our sessions, I think this might be the first time we had a a cat example, which is a little bit surprising to me. So, you know, thank you for that. Um Daniel, I had another question for you, which is um could you tell me a little bit about I guess like the background of perception encoder? You know, what was the motivation? what was perhaps the problem that you saw out in in the world of models that you wanted to solve here? >> Yeah, so the there's kind of like multiple classes of vision encoder. Um, so the main class that perception encoder is supposed to fall under is the uh clip class of encoders which is trying to match images and text. That's what sort of p core is. But a big problem with those encoders is that um they only work for that specific task. like if you wanted to use a clip encoder for like let's say detection tasks um it's actually like not so good compared to something like let's say a dynino encoder or something specifically trained um for detection. So one of the things we wanted to set out to solve is try to find one sort of base encoder that could be used downstream for a lot of different applications. So like language and also spatial in addition to the like the original clip tasks. So one of the things we tried very hard to do is making sure that you know our PE core can still be used um downstream for these other tests and we weren't you know 100% successful in having one model that does that but we were able to come up with some techniques to make sure that the um so the base core pre-training step is actually as general as possible so that it can be um later easily tuned for language and spatial down the line. I have one more sort of basic question just about like what is perception encoder and that is um could you tell me a little bit about sort of I guess like the the different sizes that are available sort of the scale of the model um like how huge is this thing how small can we get things like that >> yeah so the base perception like the biggest perception encoder model is quite large so almost two billion of parameters um but we have uh releases down to all the way to tiny so tiny can you run on your uh laptop and stuff and all of them are sort of distilled from this big model. So they're all actually fairly good compared to other you know tiny clip models in the past. One thing I didn't mention before is like uh P core is also trained on videos as well. So previous like clip models and stuff are trained on only images but that's actually jointly the same model works on both image and video. Well, so speaking of um using PE on different kinds of images, videos, and kind of getting to know the capabilities, uh I want to go to our next slide here. And uh Peter, I was hoping that you could let us know a little bit about what's going on in this image. I see I see of course there's an image of um a guy with a beard. There's some text. There's like a similarity um percentage. Could could you explain what we're seeing here? >> Yeah. So in general, the class of clip models that Daniel referred to is at the main sort of task that the model is trying to achieve is aligning text and images. Um, and so what that means specifically is you take the image and you map it into the same sort of space that you map text into. And in that space you can compare the two vectors and see how similar they are to each other. And this demo um I built shows running PE base. Uh and the caption at the bottom is sort of a predefined caption that I wrote which is an image of a guy with a beer or a guy with a beard holding a can of sparkling water. And then we embed that text and embed the image in real time and sort of looping over the image see how similar that image is to that uh text embedding. And so the cool thing that you can see here is that the model understands this text and the higher you know the more that the image conforms to that text the higher the similarity is. So you can sort of imagine using this in a lot of ways to like detect when whatever image you're looking at is similar to whatever description you're looking for. I I have one more question about this which is um for the similarity 59.8% 8%. Like, is that is that a good score? Is, you know, because for me, my human eyeballs, I look at it and I think that's 100% an image of a guy with a beard holding a can of sparkling. Like, I mean, well, I don't know, maybe I shouldn't make assumptions. Maybe like 90 99% sure. But yeah, could you tell me a little bit about like the similarity score and is is that good? And is actually even good a term that I should be using here? >> Yeah, that's a really good question. So I would encourage whenever you're building an application trying to compare text and images to sort of calibrate the sort of what the maximum similarity you'll see and what the minimum is. Um, so an interesting sort of research problem that all clip models face is that because you're embedding the text with one neural network and you're embedding the image with sort of another joint neural network, um, you never achieve sort of perfect alignment of features. Um, basically although the loss is encouraging uh images that that are aligned with a caption to be as close as possible, usually there's sort of a gap between the the space between where images occupy and images and text embeddings occupy. So basically I would say that this similarity score is sort of like a it's usually the better a model is the the better it preserves the sort of ordering of similarity but it doesn't necessarily range from 0 to 100 all the time. >> Nice. Um Daniel did you have anything that you wanted to add to that? >> Yeah I think uh Peter's explanation was pretty spot on. Um, basically during during training, there's actually a huge set of uh text in that the model has to compare to a huge set of images and it has to sort of map them one to one in the training loss. And this this is actually a very illdefined problem because like one caption can apply to multiple images. So in it a lot of flip models will be actually fairly conservative with the similarity statistic because it can't like really jump the gun and say 100% when there might be another image in that batch that um sort of describes that or has that same caption or could have that same caption. >> Cool. I wanted to bring us back to this um topic and comparison of image search capabilities in different models. Um here we have like a video playing of some tests that I did on a tool that um Peter and Daniel built to help compare um image search results from perception encoder versus clip. And I think at this time we're going to switch over to Daniel's screen and he is going to actually be able to go through some of these um queries with us and we're going to take a look at how the results uh vary. So Daniel, if you want to go ahead and take over the screen sharing, I will stop sharing my screen. >> And then for our viewers, if you are um interested in a specific query and you want Daniel to try it out in this demo, um feel free to leave a suggestion for something that you want to check in the chat here um in Zoom. So with that, Daniel, I'll hand it over to you. >> Okay, great. So um a little explanation of this demo. So I'll search here like orange berries. Uh and on the left will be results retrieved with clip. So this is like based on the highest uh so clip similarity between the this text string and a set of like a couple hundred million uh images. And one and on the right will be perception encoder for the same uh model size. And immediately when you actually start playing around with this, you'll notice that perception encoder has a couple of capabilities that um if you use clip before um clip does not. So the first thing is compositionality. Um clip is very very notorious for ignoring compositionality in text. So what I mean by that is like if you search orange berries, the first result from clip is not orange berries, it's oranges and berries. So it just kind of ignores that the orange is modifying the berries and gives you an image of um both of them. While on the other hand, perception encoder is much better at like respecting compositionality. So uh in like all of the images that perception encoder returns, you see these orange berries, which like before I did this demo, I didn't even know existed. Uh but perception encoder is able to find them while clip just gives you a bunch of oranges and sometimes oranges and and berries in the same image. So in general um perception encoder represents compositionality very well. For instance, if I say a blue street sign with white text like the first result that clip gives is a street sign with a green black background. Um not the blue that we expected. It gives this sort of one one street sign here. But most of these are just like very um like photogenic images of street signs uh that don't like respect the actual query that we asked for. And first uh the first one that a PE responds is is is kind of a little bit wrong but the rest are like basically all spot on of um like what I asked for. And so in general um perception and code is a lot more robust to these sorts of things. And this includes robustness to like more hard images. So if I search like a raccoon at night, clip will actually spot on it will understand what you mean. So it will understand like a raccoon at night um and give you that image. But the second anything is is bad and like um is hard in that image. So for instance like there's a bunch of night vision images in this uh data set. um clip will completely fail. Like so it doesn't recognize the raccoon here. This is actually a possum. It's not a raccoon. And you can see clip returns a bunch of images of possums. Like this one's also a possum. So like clip actually just fails in identifying what this uh object is because of the adverse settings of the image. Whereas perception encoder is able to correctly identify uh raccoons even in night vision for instance. So these are all uh images of raccoons. And another thing uh perception encoder is very good at is text. So if I say a sign that says apple market clip will give you some things loosely assoc associated with that text. So here's a sign. There's apple somewhere on this sign. This has the big apple. So, it's sort of related, but the first result from PE are all exactly what you asked for. So, like a sign that says apple market. And you could even do compositionality. So maybe like a red sign that says apple market. And P's first result will be that red sign. whereas like you know clip is just you give you a red a red flag. So in general P is a lot better at compos compositionality. It's a lot better at text. Um and it's sort of a lot better at understanding more difficult uh images. And that's uh I can that's like a good point to pause to maybe Patrick if there's any um queries from the audience. >> Yeah, we do have a few. Um, I was going to Should we start from like simplest to hardest or hardest to simplest? Um, so there's one from John. Smoke detector. Just two words. Smoke detector with side. Interesting. Oh, this is the cliff gives you smoke. And then P gives you I don't know what this is. I'm not sure if there's Oh, is this a smoke detector? Sure. It might not be in the data set by the way. So this uh you know this is this demo is sort of constrained in sort of what images are actually in the data set. >> Yeah, I was about to say the same thing. Um that sometimes we find images that are just completely missing from the data set. How about um there's one another one from Daniel. 12 men in the factory. Oh, we can't do numbers in this one. So I have to do 12. >> Yeah. Well, I don't know if this is 12. I I So what I will say is P can count up to like three or four. I think this is 13 if I counted correctly. >> I see one guy. >> No, this is 12 actually. Yeah, this is Hold on. >> Okay. I did not expect this to work actually. >> Oh yeah, there's a couple guys in the background, but I think like the the main main people are 12. But uh in general PE is can count up to um like a couple. So like if you say one cat and two dogs, you can't do and um this is two two cats and one dog. Yeah, but basically let's say actually I did this query before two dogs in a car. So P is able to generally if the image exists in the in the corpus um sort of count up to a few and actually respect the compositionality of the fact that it's that's in the car. Um whereas clip can sort of count the two but in most cases these are these are actually pretty good too. Yeah. All right. We have another one which is um three boxes stacked on top of each other. Um, this is from James and I'm I'm wondering why he needs this image. Oh, got three boxes. These are I guess these are I don't know if these would be considered debatable, but yeah, general seems to work. Yeah, I I would be confident um P can kind of count up to up to like three or four. And then clip is Yeah, >> I had one that I tried out earlier which was um I think I wrote like an athlete kicking a soccer ball. >> Oh, how do you spell soccer? >> Now you Now you've broken my ability to spell. >> It's okay. We got spell check for this. Yeah, we got spell check for this. It's fine. >> We'll just edit out that part out later. Yeah. Yeah. >> And you know something if you scroll down a little bit um >> you know like some of the results are different like with clip like sometimes it's like not a soccer ball sometimes it's a it looks kind of like a ball. Um and then if you go over perception encoder you know you click like somebody kind of like winding up about to kick the ball. I don't know how you explain this or I'd be actually curious from your side like >> I find the perception encoder results like more aesthetically pleasing >> um as a as a researcher like how do you try to explain that or have you had any observation related to that? Like it's kind of like >> they're just more entertaining or more interesting to look at like how do we talk about that? >> That's actually probably because perception and code restraint on videos as well. So, of course, in this demo, we're only doing images, but you can do the same mapping from videos to text. Uh, and perception coder does have like a step where we train on very well annotated videos. And, uh, in videos are all about sort of the action shots like that's how we we actually um like in our video data set, we filter by um videos with specific like actual actions. So P will understand actions probably a lot better than Clip because Clip has only seen actions through the lens of like images where P has seen it through videos. There's another one that I kind of started off with here which was like um I think I was searching for like packages loaded into a delivery vehicle or maybe even just packages in a delivery vehicle. And um this one I thought was a little bit interesting because I see there's definitely like on the clip side there there are delivery vehicles um and then there's packages, >> you know, and maybe this is also just like a weakness of my query, right? Like yeah, okay, yeah, technically that is packages in a delivery vehicle. Um in my mind, I was kind of hoping to get the perception encoder uh results over there of like it's a van or a truck. >> Yeah. >> Yeah. Right. Um, but I I think my takeaway here is it's like the perception encoder was a little bit better at kind of like guessing what I needed or yeah like making up for my poor query. I think that's a in general because of the composition compositionality aspect. So a lot of time clip will treat your query as like a bag of words. So um it will just consider each word independently. So when you say pack a package in a delivery vehicle, it doesn't really care about ina. it only cares about packages delivery vehicle and then so it gives you um results with delivery vehicles and packages whereas perception encoder a lot can a lot better represent sort of that the fact that it's inside of a delivery vehicle which is why I think you see um much more images inside it but I'm wondering if like yeah so even in this setting if you give if you already spoon feed the the query to say like a van with packages inside of it this is very much not like like I mean this van could have packages inside it but we don't know right so the clip is just ignoring the packages part of it and just just focusing on the van whereas P is able to like you know union the two together >> all right beautiful very cool um let me see here do we have any other queries um I didn't see any other suggested queries from the audience did you want to demo anything else yeah >> yeah um one of one thing that actually is very surprising like after um like We didn't try for this during training uh but it ended up just working uh in that PE is very very good at like recognizing like geoloccation so like recognizing what country something is in just by the image. So if you do like a Canadian food truck which is something I really I really like to do. Um uh this isn't a food truck, but um so Clip Clip will just like sometimes give you Canadian food trucks uh and mostly just not give you food trucks. Um yeah, this but I I don't this doesn't seem like a Canadian specialty to me. Um but P will give you like explicitly find some way to connect this to Canada in the image. Like whether it be a Canadian flag, whether it be uh well another Canadian flag or I'm trying to look for a poutine truck. Yeah, a poutine truck. Somehow it'll always find it. And we I I've I've looked at like a running this on like a bigger set of images, like billions of images, and it will find like the smallest little details to map that to the fact that it's in Canada, like a a CA URL on the side of the food truck somewhere in like very small font or something like that. Um, so it's always like very surprising to me that it can it can find these things. Yeah, I see one of the one of our viewers suggested searching for something like a car with a Canada number plate or a Canada license plate. >> Canadian. Let's see. I don't actually know what Canadian license plates look. This is not I don't think this is legal. Yeah. >> Yeah. Well, it might also change depending on I guess the province. Maybe we should search for like a British Columbia license plate. Oh, but these are Vermont is not Canadian, but it's close. This is uh Ontario. So, this is actually correct from the clip side. This is also against that Vermont image. >> What is that? >> I Yeah, I can't read the actual license plate. So, I don't actually know. >> This is Ontario. >> Might be a limitation of our data set that we're looking at here. Oh, that looks like British Columbia. Yeah. >> Yeah. Like a Swiss mountain side. I think I did this before. I don't know this. Yeah, Alps. Um, yeah, I think Clip is good at that, too. It It's just in general like actually PE is um sort of the state-of-the-art by quite a bit on like geoger type questions. Um, like even compared to the other state-of-the-art clip levels out there. >> All right. Well, um I think for now we could always return to this later if we get a some other suggestions, but um we could sort of continue on to our next topic. >> Um I'll go back to sharing my screen. >> Um thank you very much for that demo. Let me rearrange a couple things. Um yeah, so we can move on here to um if you are a user of Rooflow and you're interested in how perception encoder might be applied um you know within the Rooflow ecosystem. I was hoping that Peter you might be able to tell us a little bit about the tools that are available for people at this moment. Yeah. So, we added perception encoder to Roboflow's inference package, which is a package that allows you to run a bunch of different model types um either sort of on your own hardware on your computer. It can also it has a client to Rooflow's web services. So, you can make requests to um our hosted inference services and run sort of larger models there. Um, so that demo that we showed earlier in the presentation was built using the code that's displayed on the screen. So not a ton of code to write like sort of a streaming um, webcam version of the PE demo. Um, and so if you're interested in trying out PE for yourself, you can just pip install and print and you know copy and paste this code. Um, and you could also use sort of P in other contexts where you'd use clip that way. >> Nice. Now, with that said, um, I was hoping to have, uh, sort of like one more kind of big topic before we wrap things up. And this relates to the possibilities, applications that are either unlocked or improved by having perception encoder. And I guess to give a little bit of context, you know, I work in marketing. I make web pages. I make slide decks. I make uh webinar content, video content. And um I often find myself on stock imagery websites and you know I'm searching for really specific things like you know I want to do a webinar about finding damage in uh shipping cartons or shipping containers and you know I'm sure that the stock website they must have videos of these shipping cartons that are damaged But when I search on their website, uh I can't really find them and then I end up spending a whole lot of time sifting through results myself. So if anybody from some stock websites uh they see this this call um this video, please get in touch with us and perhaps we can help improve the uh search results for you. Um, but you know, that's that's my example of how I could see something like Perception Encoder um improving like a real world use case. Um, Daniel, I was wondering have you seen Perception Encoder used out in the wild? Um, or may this might even just be something you built yourself, you're using yourself, or some other projects that you've seen people people do. >> Yeah, I think actually there are a lot of applications very similar to that. So not just like you know I want to find a specific example of a picture but as like a tool in the pipeline for training your own ML models. One of the big things is like which data do I annotate? How do I find um like images or videos to annotate uh that are sort of relevant to what I'm trying to solve? And a lot of times you'll have a big set of of images that and you want to find like specific um maybe failure cases of your existing model or uh some specific concepts that like new concepts that you want to annotate in in your data. And I've seen a lot of success using perception encoder to be able to do that search and filtering. So I mean like in the demo you could just do a uh like a search with a text query but you can also take you know an existing corpus of images that um you've already hand selected and like sort of average the embeddings together and search with uh the image embedding instead of the text embedding and just get a bunch of similar results. So this is not only can PE you know do text based search but you can also do the reverse right sort of image based um search and that is just really really useful to finding more examples of failure cases that you want to annotate or more examples of a rare concept for instance that you want to annotate and that's been I think used for very uh big success in some of our other projects. >> Cool. All right Peter I was actually hoping to ask you a similar question. And I think you have a lot of experience kind of interviewing people, chatting with people who aren't using computer vision kind like out in the world at work or for their own personal pro projects. Um what are some of the cool ways that you've seen people use models like this? So I think that you know sort of retrieval is like the obvious thing. Um, but I think in general sort of like any use case where you want some sort of semantic understanding to be built into sort of a downstream flow makes a lot of sense. So you can imagine sort like similar to the way this demo was was built. You like upload a bunch of images um on the fly and you sort of process them into just like their perception encoder vector and then you sort of have this representation that you can use to both sort of like look for similar images in a data set or like search over text. Um other instances uh that happen in the research world is I mean so PLM perception language model I think is the uh language model that was built using perception encoder text as a base. Um but because it has sort of more robust vision and language features it's is a good base to start building other models off of. Let's um keep moving ahead because I saw we have a lot of questions and I'd like to leave a little bit of time for them. Um before we start to sort of wrap things up and move on to questions, Daniel, do you have like any other announcements or things that um you hope the audience would be aware of? >> Yeah. Um keep a lookout. There's going to be some cool new models uh dropping pretty soon. Um some using perception encoder, some maybe adding some new capabilities of perception encoder. Um, so look forward to that. Um, and yeah, Perception Encoder I just uh recently learned was accepted uh to NurS as as an like top 1% oral at uh in in December. So I'll be both at ICTV and Nurops if you want to chat. >> Beautiful. Congratulations on on that. And uh I won't be there myself, but I'm sure there will be some people. So please, if you're at those events, stop by and say hi to Daniel. Um, so with that, I was going to take a quick break um, and say if you're watching a recording of this session somewhere and you want to join a session live in the future, please head on over to roofflow.com/webinar and you'll see a calendar of some upcoming events that you you can join in the future. Um, and also I want to move over to some questions and I'll do my best to to sort of rephrase things and uh make it easy for everybody to listen in and understand. But um really early on there was one question um that we had from Eric and he said clip is often uh used as a feature provider in object tracking applications. Are you aware of the use of perception encoder for this kind of problem especially for pedestrian tracking? So um do you kind of have anything that you any guidance that you could provide there? >> Yeah, I think for any task where you're already using clip vectors P is just a drop in replacement that improves performance. There have been I've seen works that like use clip clip-like models for like zeroot tasks that have replaced their uh model that they were using with P and instantly getting like a 7 8% um accuracy improvement. So basically anywhere you use clip right now you can just slot in u PE and immediately get a get a boost in performance uh for for most things. Um, and as for pedestrian detection specifically, if you wanted to sort of swap out like maybe the uh backbone encoder uh that you use for your pedestrian tracking instead of just the um sort of the clip embeddings, uh you can also maybe try out PE spatial as well. >> Cool. Um I'm going to jump around a little bit, but I saw one question that intrigues me. It was about the size um of PE and compared to clip and um they were wondering about you know how does it work if I wanted to say deploy on a smaller device like a mobile phone um it looks this person said uh it was John looks super interesting and uh for our use case they're using clip at the moment could you talk about that >> yeah so I think the smallest clip model that has been released has been like clip bas case uh we actually released two additional smaller uh models that are even smaller. So uh we have PE tiny and PE uh small and so those ones are actually like fairly lightweight like uh I think Tiny uh I not exactly sure on the parameters count but around 10 million parameters and for clip models like there's no like it's RP tiny model is like 10% better on most metrics than any other tiny clip model to date. So definitely give that a try if you want to sort of deploy on like very small use cases and I think that should work plenty well for you if you if you already using um clipbase. >> We should also think about how we get that uh tiny model working in Rooflow as well. It might be useful. Um there's another question here um from uh Mongal um and this one's uh kind of cool. He said, uh, can the, uh, could we talk a little bit about like the intentbased outcome of the search? Um, so this probably came in when we were doing the demo. Um, and he said, for example, if I search for I am hungry, uh, then provide pictures of food or food restaurants and not the text itself, you know, to sort of understand the intent behind the search. Um, are there any capabilities like that? Have you tested that out in the past? >> Yeah. So, what I will say is that PE and most clip models are trained on like web alt text. So, that is like when you upload an image and um you want something to like some text that describes the image to display um when the image can't load or if you can't like someone can't see the image. So, basically what what the model is trained on is what a human would write about an image. So, in a lot of cases, that actually might be intent based, right? So, if you upload an image, you you might like caption it, um, you know, I'm hungry and and show a picture of a nice burger or something. Um, so those sorts of things actually P and clip actually both do I think fairly well. Um, so yeah, I think it should be able to especially anything to deal with food like anything people would like generally upload to the internet a lot. >> Nice. Yeah. Um >> yeah, actually one of one of the demos, yeah, one of the demos I have is like um you know, a cat that's like mad at you or a cat that really wants food. And you can see that the P is able to sort of understand that, you know, with a cat that wants food. Um sort of it's it's in a kitchen looking at you really longingly. Um is that what P returns? Um so it sort of understands underlying semantics, but it's basically, you know, modulo what people actually upload online. >> Cool. All right, I'm going to merge a few different questions together. Um, but when when we were showing off that sort of proof of concept demo website that was designed for this presentation, um, there were sort of like a lot of questions about um, you know, is it possible to see the confidence results? Um, what's the database that we're using to to get the images? Can we see cosign scores in real time in the demo? Uh, I don't think we're going to update the demo right at this minute, but I guess when you're when you're creating your own app or when you're deploying PE, I'm I'm I'm assuming that there's probably ways for us to see some of these scores. Could you talk about that? >> Yeah. So, basically, in order to actually show these results, you need to calculate like the cosine similarity score, right? So um all of these results on the demo are basically sorted from highest similarity to to lowest similarity. So the the first result will be sort of the highest similarity. One thing I will caution is what like basically what pet Peter said earlier um these models never will give you 100% similarity. In fact like most of the time if the query is very short it will give you like 23%. So you'd have one thing that will be like 23% and one thing that will be like 10%. And then it will get it correct like the ordering is correct but the actual calibration um isn't the same. So if you are deploying this on your own test and you do want similarity scores, you should run the model beforehand on some test data and figure out how to calibrate the similarity scores from what the model does on your test data or your training data. >> Beautiful. Yeah. I remember a colleague was working on a website where what was it? It was the Timothy Shalamé comparison where you upload a picture of yourself and it tells you how similar you look to Timothy Shaé. That's his name, right? Yeah. And I don't remember he was tangling with this issue of sort of deciding how to interpret the score and and improve that. Um and sometimes it's the beauty is in the eye of the beholder or the Timothy Shalomé lookalike is in the eye of the beholder. Um, let me see if there's any other um questions that we should be getting through here. Uh, >> I saw a question about is it possible to fine-tune and >> Oh, yeah. Oh, yeah, please. Thank you. >> I imagine Rooflow will also have like a a way to do this, but um right now PE is actually integrated into Open Clip. So, like this the standard way people sort of train clip models today. You can actually go and use it as a like right out of the box from Open Clip and start fine-tuning right now. >> Yep. Cool. Yep. That was definitely on my list. I'm gonna take one more out of here, which is um from Daniel. Um this is actually something that I was thinking about the other day as well, but um so he said, "Have you ever seen singularity breakthrough performance when training PE with large scale sizes? Any specific differences between the weight size uh and performance? Is there kind of anything that you you could share here about like the the process of um training a model like this? >> Yeah, I wouldn't I probably wouldn't say singularity. I think that may be a little bit too strong, but uh definitely there are things that work at larger um both larger model sizes and larger resolutions that don't work in in smaller resolutions. So some things like the compositionality only we only start started seeing that when we you know train on a large model and with a specific uh way to train it that we describe in our paper. Um and we we don't you don't see that on the smaller clip models but it turns out that you can distill it back into the smaller models after the fact. It was very interesting the dichotomy there. Um so like the bigger model sort of discovered how to work well on OCR and how to work well on um like the the more compositional structure of text uh which you can distill back into the smaller models but the smaller models don't find that on on their own. Like in fact one of the very interesting things is our text encoder is architecturally identical um in both size and uh like number of parameters to clip text encoder. So in that demo the text encoder sizes and architecture were identical yet the text encoder in PE uh that PE model can understand compositionality while the clip one cannot. So that is like just some thing that just like a switch flipped because we you know train a big model in with a specific training recipe and um distilled that back into the smaller model and then now can just do that. So I think that's that's pretty cool um we found. >> All right. Well, um I think that we are out of time for today. So, um thanks to our our viewers and audience members for submitting all of these uh questions and coming to uh watch this session. And thank you very much to uh Peter and Daniel um for agreeing to to join and uh teach us a little bit more about perception encoder visual understanding and uh what you have going on there. >> Thanks for having me. It was great.