Computer Vision Study Group Session on SAM

Key Takeaways

yes okay cool so once again for the recording uh welcome to the computer vision study group in August today we will talk about segment anything so that's the paper probably a lot of people have heard about because it just made some waves when it came out and it only came out in April for me it felt like oh it's been there for so long but oh it's four months which maybe machine learning time is long um but yeah came out in April from MAA AI um and yeah as always I'd like to put a little theme to my presentations and start with an intro story so I I like to come up with some stories um that are more or less related to the topic and yeah today's theme is oh I think it h kind of neon neon Punk uh um ninja style I would say and yeah we start with the story and then we dive into the paper so it is a warm summer night in sex city where the buildings grow High and the streets never sleep uh for the story I created some images as you see uh with generative AI of course I used the stable or the sdxl model so the new one from stability AI is pretty good um yeah and we go on with the story so for months now Clan Wars have been shaking the city every day three big families are competing for Supremacy the micro Clan having found new Strength new allies and new ideas they want to rule the city the elok clan having been ahead of the other four long time the things are changing recently The Meta Clan it has seen better days with more money nevertheless they keep coming up with Innovations a key Concept in sex City's Urban Warfare is height and attack for which all clans have their own ninja forces to avoid these sneak attacks the glans introduced Special Forces to Monitor and segment the environment the segmentation measures were good but rather limited until recently when the meta Clan introduced Sam she's a result of many months of training in meta's special program to segment anything her abilities allowed her to do the work of many different segmentors by herself but even more important ly she was able to pass on her skills and enable others to achieve great things the streets of sex city still are a battlefield Sam won't be able to change that but her power enter disciples are to have a big impact in this city and that's it that's the end of the story it was a rather short story this time um because I focused bit more on like this this paper part because I feel like there's so much stuff in the paper actually it could have been two papers um yeah because in this paper you not only have this model like the segment anything model but also they describe the whole data set and how they made the data set and everything um I think a lot of researchers would have just really made it into two papers because you could easily do that um but yeah they decided not to so they have a long appendix and yeah a lot of information in their paper but here in the presentation today I will try to focus on the model a bit more than on like the data set yeah because I mean the data set is interesting very interesting um but the model is harder to understand I would say and and yeah I think that's what what we're trying to do here in the CV study group is mainly understanding stuff um and if you feel like it you can always have a look at most data set statistics yourself like where are all the images from or in which countries or whatever um so yeah okay I've talked a bit about the paper already um so let's first have a look actually at image segmentation right so what what is like image segmentation um so for me when I hear image segmentation there's always stuff like this it comes to my mind so we have like an an original image over here uh and then we have a segmentation map and in the segmentation map you can see different colors right in this case it's like a semantic segmentation map so you have um things that are the same class basically in the same color for example the cars are this greenish and the crown somewhat I don't know what you call it Tois Tois maybe the buildings are pink and the sky is red right so it's like a semantic segmentation map um so for me that's yeah like the most natural thing when I think about image segmentation then you often see this is overlays as well right like here and you can see aha okay here we have our cars and this Sky um yeah but actually there are like different kinds of image segmentation right so what I talked about now already is the semantic segmentation where we have like yeah these um Parts segmented by the classes they belong to and the important thing about semantic segmentation is that like the whole image is covered so every pixel has some value assigned right so there's no pixel that is like yeah well nothing so everything is something because in contrast to that there is instant segmentation where you don't need to have the whole image assigned somehow and we actually focus more on instances of objects right for example here again the cars right so every car has a different color and we we don't care about buildings about crown or about the sky um in instance segmentation it's really about finding single instances of things and then there's a pen optic segmentation um actually for a long time I thought this was called instant semantic segmentation but I learned it's panoptic segmentation which is just really a combination of of both um very we try to cover or where we do cover the whole image uh but we try to detect or segment as many instances as possible of these different classes we have right so panoptic segmentation is just really the combination of the form and two and yeah may be the most helpful but can also be the most difficult to uh yeah to get to actually so yeah one big problem now is like what if you want to detect new classes for segmentation right segment new classes it should actually be for example like signs or Billboards or or something like that so what we will have to do is annotate it right so we have to go to our data sets and then have to get up our labeling tool probably uh sit in front of our computer or whatever and then start labeling these pixels cuz it's like the hardest part I think for segmentation data set is really just labeling everything because it's it's like this dense task where we really yeah care about each pixel right like for object detection often you just have bounding boxes so you draw a bonding box here bonding box there bonding box there and it's all good but for this pixel stuff you even want yeah the best accuracy actually right because then you feel like OKAY model can learn best when it's best accuracy so it's a lot of work to actually create a data set for semantic segmentation um so so the question may arise okay there's so much work so much money that goes into it in the end isn't there like a better way so are there Solutions um actually this slide mainly exists because of the image um the image kind of happened when I played around with SXL model and was okay it's so cool it just has to be in the presentation so yeah Solutions question mark and one solution that comes to mind um is the socalled zero shot image segmentation right so what we do here is like we have an image and then we just say billboard signs and the model be give this prompt to basically so this is like our text prompt right billboard signs and the model can give us the mask with the Billboards and signs without any further additional training data so that's what zero shot means right basically we think it has never seen Billboards or signs um and then yeah it can segment it any way because it has learned so much about like the nature of it um yeah that it can just do it that's a zero short image segmentation and that what makes um Sam so outstanding because it is a way to yeah leverage zero shot image segmentation with prompts and what these prompts can be and the whole architecture is the next slide oh was yeah so let's have a look at like how can Sam do this whole thing how do they get to these zero shot capabilities and how can they do that so at the start of the whole architecture there's an image encoder right there you give your image in there and it gives you a representation of the image so tokens embeddings whatever you want to call it and this is only done once so for example when you're in their official two or whatever on their project page and you upload an image then they do this once this image encoder thing right they just get your image send it to the encoder and then they save the embeddings um for further stuff so not every time you you're trying to segment something in the image say do the image encoding step again that is only done once right in the beginning and then the the embeddings are there because your image does not change unless you upload a new one right U so the next step would be that we can add a mask as a prompt actually so the mask goes through some convolutional layers uh to get it in the same dimensions as the image embeddings and then it just gets added basically but there are other proms we can use as well so we can just put in points right you can say Okay want here here here or we can just use a bounding box as a prompt or a text and then those go through a prompt encoder actually and then the prompt encoder embeddings or the prompt embeddings are then fed to a mask decoder where also the image embeddings and The Mask if it exists um go in okay and in the end we get a mask out right so that's like the architecture now you might wonder like me okay that's good but what what is in the prompt encoder and The Mask decoder and maybe even the image encoder right um and after say the paper in the main part is a bit short on that so they put most of the explanation in the appendix probably do have more space in the main part for data set stuff but yeah if if you want to read about that you have to go to the appendix and I did that right so um let's have a look and start with the image encoder because it is actually at the start and the symbols one of the encoder decoder things here so yeah for the image coder of course we have an image right and we give that to a pre-trained um Vision Transformer they say they use one uh that is strained with the um mask Auto encoder stuff so the the my which basically just helps um getting a better quality even with fewer images and then they get the output of that and feed into some convolutional networks to in the end get to the embeddings they want so yeah it's not much magic in there so the vid that's actually the the compute heaviest part I would even say in the architecture is this this image thing here because what they also say like in their architecture they focus on uh on speed mainly so that this whole Sam segment anything model can even run in in a browser and that's they say okay this this first part yeah that's a bit compute heavy but we only do it once and then um you can prompt whatever so yeah here we get our image embeddings and the next step is a prompt encoder which is a bit more complicated um and I have to had to read it a few times actually to really like understand it um and I'm still not sure if I completely understood it even after looking at code and everything because it was it is a bit strange at points so here it say it's sparse prompt encoder because they say okay these are the sparse prompts and when you put it in a mask it's a dense prompt um so this is The Spar prompt encoder basically um and we start with like these points up here and the the bounding boxes and they get fed into they get fed to but um we first create positional embeddings from them right so we get our points which are somewhere in our coordinate frame in the original image and from the bounding boxes we take the upper left and lower right corner and we create the positional embeddings from them it's actually the absolute cidal positional embeddings if you know some stuff about position edings I don't know too much about it but um yeah they are absolute cidal positional embeddings here they are creating and then we take these embeddings like here are our position embeddings these ones like the four ones from the four points and then depending on where the point is um we add to it a background or a for Crown token like learnable embeddings as well here um yeah so background tokens get added to the point that are not part of the ground rules but are part of the prediction right so basically false positives and foreground ones are the ones where the ground RS and the prediction overlap so true positives right false positive true positive okay for the bounding box it's bit simpler um just get there one for top left and and one for lower right that you add as well like these are separate position embeddings you add them and for the text encoder they just take the clip text encoder actually so clip is an other model a multimod model that exists that has a text and an image encoder and they just say oh yeah we just use basically clip text encoder and send our text in there and in the end what we get are our prompt tokens or prompt embeddings out here right so maybe to say it now this this whole part with the the text is a bit strange because they talk about it a lot in the paper but in their released version there's no no hint of it right so they say yeah yeah okay for the text you can do it like this and that and um it even needs a bit of a different training strategy um but yeah they never really released something something that can use text out of the box so there are there are some more projects that can do that but from meta themselves there's to this date no model that can just take text in so basically the prompts are mostly um points or boxes yep okay so that was the prompt encoder now we get to the mask decoder which is I feel like the most complicated thing and you can see here we have our image edings right the green ones if you remember green is image embeddings then we have our token embeddings down here the blue ones and what we do is weate another token which is this purple one and that is a class token so if you're familiar with Transformers and stuff like bird or like in general class tokens are often used to yeah to be the output in the end right so when you want to classify something with a WID or whatever usually in the end you get the class token out and then you can say okay this is like a duck or a cat um and it's similar here because the glass token in the end is used uh to really create the mask and yeah so basically it's it's the mask output token you could say okay and all these well you five tokens you could say the number can be different right depending on how many prompts you have like if you use six uh points as promts you have like six of these and yeah so you first feed these prompt tokens here into the decoder block and the first step is a self attention so basically uh yeah it just helps in in getting these tokens more context aware about each other and the next step is a cross attention where the goal is to actually help the these tokens and the prom tokens become more aware of the image embeddings right so basically yeah in the paper they they write it as a cross attention from prompt tokens to image embeddings that's why I put this Arrow here um but maybe I I will think about it the other way because actually the the image embeddings teach the prompt embeddings about their information right oh next step is that we feed our prom tokens into an MTI layer perceptron thingy just for some dimensionality things and then in the last step of the decoder block we actually go the other way around with the cross attention so that um the image tokens get some information from the prom tokens right so this way we have like a yeah two-way cross attention here well two times one way you could say um yeah once in this direction once in this direction so in the end these tokens are somewhat connected and know about the information from the other tokens and we do this decoder block just two times so it just have like two layers quite a small decoder but as I already mentioned they try to focus on speed and that yeah that helps just having two layers okay then so after these two layers they take the image embeddings and feed them through some convolutions to upscale them a bit again because they will downscale in the image encoder and now they want to up scale it again they used some convolutions for it um and then again they have a cross detention here where for the last time basically the prompt tokens learn a bit about the image tokens and then the class token which I talked about before right gets fed to a multilayer percep Ron again and then the output from here goes here and with a DOT product you combine the image embeddings and the output mask embeddings basically and then you get your mask out that that's the mask decoder it's a bit of a long way in the end um and you might have some questions uh I understand that's something I had to look at a long time and always go to the paper then try to try to figure it out myself and yeah it's just well not that easy but also not that highly complicated I would say but now there's one problem actually with masks where they say in the paper Okay um yeah that's what happens when you use Point prompts and that's mask ambiguity I hope I pronounced it correctly um that means that when I put a prompt Point here right on on this cap um what do I mean what do I want to get out as a mask as a human right what what was my intention of getting do I want like only this this cap here or do I want to get the whole tire or wheel or do I want to get the whole car right and that's a problem um that they were facing apparently especially during validation I guess because I mean I could mean all three of these things right so most likely for me would be not the whole car maybe the whole wheel probably the whole wheel um but yeah this mask ambiguity thing is something they tackled by going to the m decoder again right you see here our class token and they actually use three class tokens here okay so they they not just have like one output mask but they say okay we just take three output masks and so three was like the The Sweet Spot for them they tried different numbers and yeah they say Okay three mostly is the best thing um because they have like subp part part and hole or something as you can also see here right like this would be like subpart part and Hole uh yeah and that's covered with these three output tokens so in the end um these things here are three tokens right and then you get masks out and not just a mask um and what they also do is that they have token in here that is just used in the end to give us the intersection over Union um scores of these masks that we then get out in the end here um yeah and why they need these scores I will tell you later it's some especially for validation and testing stuff uh okay but that is like the image of the Mask decoder what you see here now is like the the whole mask decoder and yeah let's jump to the next one which is the training procedure right because you might ask yourself okay how the hell do they train this now um I mean the task they are tackling they call it prompt based SE segmentation so of course the first thing is you need a ground Ru ground Ru mask for example for this car here then you need a prompt prompt is this point and then you get a prediction right ground truth prediction okay so now you have these two things Crown truth and prediction and you can take both of them and create the intersection over Union right that is like the uh yeah the base uh metric for most segmentation tasks so you have like the overlap of both of these masks and the union of both of these masks right so like the whole thing they cover if you just yeah put them on top of each other you get like the union and the overlap is the area that is present in in both of these masks and resist over section over Union they then use two losses actually the focal loss and a dice loss which is also F1 loss Dice and F1 I learned are the same I hope I learned it correctly um yeah and oh it is really similar to in this simple intersection over a union but oh a bit different um and the focal loss mainly helps us focus on uh hard examples Right Stuff where where it fails uh bigger or more often and then they combine these two losses uh linear linearly in the end um and they say they have it in a ratio of 20 to1 so they put some some weights on it yeah and it's just so the focal loss is way more important than the dice loss in the end for their training okay so that is what they do in in one iteration basically when they prompted it but there's more because they prompted it a bit more often right so for each mask they say like okay uh we did the first one the initial one and then we do eight more of these prompts in the areas where it is quite uncertain right or where is wrong or where is uncertain so we we put some more prompts there so that that's eight more so in total you can say there's there's nine prompts now and then they also saying we find out it helps if we just run it two times without adding any any prompt so they just keep it at like seven points prompt it again and then they add another Point um yeah and they say yeah this keeping the the same amount of proms is um mainly helping when you do it in the end once and somewhere in between somewhere in between doesn't really matter apparently when you do it but the one in the end does matter for some reason there's always these experiments I find quite interesting like how did they find that out how many times did they experiment with that right but yeah so that's um kind of what they say is helping so in the end you have um 11 times that you go through the whole loss thing and every time you actually get out three masks right as I showed before for this ambiguity problem and they only back propagate the loss for the like the the smallest intersection over Union right so they don't do it for all three mask but they just take like the the smallest insection of a union so the masket does best and they use that for back propagation then yeah okay that's how they train it basically and here are some results of it so they did validate it on 23 data sets and they mainly compared it here to this other model this ritm model that is also doing like prompt or like Point based uh segmenting and here they say okay they used one Center Point here that means they just took like the center of the ground Cruise mask and then they used that for prompting their model and they are doing really well on a lot of um these data sets so mostly better than the itm model but there are also some where they do worse and they also introduce this Oracle thing here um so they say yeah we also have our Oracle prompt here some Oracle and Oracle here also is like these dots right these these dashed lines and the Oracle is when they're actually looking at their intersection over Union scores and take the best mask um yeah the best mask according to this intersection of a union scores because apparently sometimes when you prompt the model it does not automatically give you like the best mask that would fit the crown TRS right again this ambiguity problem that you can get any of these three where the model things are is actually the best one right now but then when you go back and compare to the crown truth it was not the best of the three masks so for this Oracle they say okay we actually take what would be the best of the three masks so it's a bit of well I don't know if you can call it cheating but it's um yeah like they say okay if we would take the best of these masks then we would be here here here and here right then they would always be better but um in reality they are not so yeah they just wanted to prove I think that they can be better than the other thing um what's interesting is here that they say okay that's like the the machine the automatic validation but we also want to let humans rate it right so let humans have a look at it and there they say okay when humans look at our mask we are always better than the ritm um for these data sets here uh yeah and on here what we have here here there are more models actually the simple click and focer click which get better when they have more points so the simple click and focal click are not designed to be like one point segmenters but they're more yeah focus on having more points to help them segment and yeah here they also say okay Sam is way better when we do it like with just a few points um in the end when there are more points than the other ones are pretty good as well all right and here yeah here's the center points so always pointed are in the center and random points similar okay so that's like the segment anything model with results and now we have a short look on the data set stuff because as I said they also release the data set right with the whole segment anything model and they have this data set with like 11 million images with 1.1 billion masks it's really a big data set and they also talk about how they collected that actually so they used three stages the first one is the assisted manual stage second one the semi-automatic stage and the third one the fully automatic stage so at the first stage we just go through the stages now because I think it's pretty interesting um because labeling stuff or even using Sam to label your own data is like one of the main yeah main things I would see use for Sam Actually in labeling my data or my company's data right just making it easier for me not having to sit in front of the computer for like weeks just to get some segmentation data so how they did it how they created this huge data set for them is um in the first stage they have this model assisted thing so they have an image uh they give that to a human the human says okay here I I put some prompt in I put here here here here here um there should be some some object right uh and then give that to the the first version basically of of the Sam model and Sam gives us some some masks you give that back to the human the human can say Okay um I refined this mask here and this mask here and this mask there uh and then add that to annotate data set and then from time to time they retrain Sam on this data set even still in in this first stage of model assisted manual thingy they retrain it somewhere in between I think they don't say how long the stages lasted but I can imagine this stage was like the longest stage because it has a lot of human interaction um and yeah it probably takes some time so they just yeah retrain them to get better results in between so the second stage uh it's a semi-automatic stage so we still have the human have an image and we give the image again to a s and then what they did is that they used like the data from the last stage and trained an object detection model on this data already and then they use this object detection model to create prompts for Sam in this stage so to get better initial results basically without a human having to prompt anything and so then yeah you get this mask again give this to a human human says uh okay here you can do better there and there and there and then again you have the annotate data set and then from time to time they retrain um the same model right so yeah pretty similar actually just less human interaction and in the last stage the fully automatic stage um there's no human anymore because they said like yeah at this point we had a lot of stuff figured out like the mask and bigy thing and then we got way better actually and we didn't need a human anymore so they just take um the image give it to give it to the Sam model again then they create a git of 32 * 32 um dots to prompt it right and then they get a lot of masks out of it and then they have like these these three key concepts for masks to keep them or to dismiss them and it's like confidence stability and non- maximum suppression and with this these three concepts in the end they get their fully annotated fully but yeah they annotated data set uh yeah which in the end has 11 million images and 1.1 billion masks which is a lot um yeah right but one thing maybe about the whole data set creation and everything what bothers me about the paper is a bit so they do their best to talk a bit about ethics and like responsible AI in terms of hey so we created our data set and we checked the biases so they tried to include a lot of images from a lot of different settings and countries and so forth um but I really wonder about actually the humans that were involved in this whole data creation thing right because they never talk about like the humans that are uh used here um yeah I'm also a bit concerned about this like low wage uh yeah data annotating people so that's something I would have wished some information on from MAA but maybe well they didn't want to put it in there I don't know yeah that's just my my little critic about this data set stuff so right in the beginning in the story I already mentioned that um like one big advantage of Sam is that it actually can help other projects right to grow and it was amazing to see like right after it came out there was so many ideas from people and and many things that happened and I just put in um two GitHub resources here basically they are pretty similar so when you open it I open it somewhere or not um you will just get to get up page oh yeah now now I opened it four times of course and there's just a long list of projects that are leveraging the s model in some way right so if my computer is not dying now I can show you ah here we go so it's called awesome segment anything actually there a lot of reos that are called something with awesome segment anything or or aome segment I don't know um but yeah this is one of the ones with the most stars so I just chose that one and yeah you can see there's a lot of stuff in here like base papers base model papers so that's basically papers that are good to understand as well for the segment anything part um derivative papers and image meting reporting project yeah that's where I wanted to go right so I think one of the best projects actually is crowned segment anything here because that actually yeah lets you use text proms um so when you yeah want to use really like these text proms that they prom is in the paper but they don't deliver in the code then crowned segment anything is a good thing to go to because it can do that and yeah it's basically using this crowning dyo thing together with segment anything and then it yeah is pretty good I have to say but there are other projects um there's stuff like this magic copy which is a Chrome extension that lets you just do the foreground background segmentation pretty well and yeah there's also mobile Sam somewhere and Fast Sam which try to shrink the whole thing even more so you can use it on uh less powerful devices but yeah you can just scroll through that yourself if you want to um but yeah if you want to use it in code yourself right segment anything it's also available in the Transformers library from huging face right so as we on hugging face Discord here I suppose everyone knows hugging face um yeah and I just link the official docs here and there's Al some example notebooks from NE over here where you can see how to use it so here's two one is like fine-tune Sam on a custom data set and one is run the inference with met Sam which is a derivative of the original Sam model and you can have a short look at that just to give you a feeling of how to maybe use the S model with Transformers Library so we yeah import everything torch and Transformers so you can import the Sam model and the S processor and then we instantiate both of them here um we give like the huging face up for the model uh to the the address this is rang laab mm with base okay so it's a med Sam not the original Sam and then you can load a data set for example the breast conso data set here from from needs and yeah you can take any image out of the data set so here he takes this image he loads the image loads the the CR through mask basically so you can see it here and yeah in the next steps um he creates a box prompt hello can you hear me oh yeah I can hear now yeah uh sorry I can see your screen oh wait a second here it is right can you see it now uh yeah thank you okay cool thank you thanks for telling me um yeah so you get actually the the crown Tru thing here and from this Crown Tru mask he creates a a box prompt so he just adds some some noise basically to it to not have like really close but have like a bit of space around use this rent in right so yeah just creates the Box around it and then us it this uses this one as input box for the Sam model so you can see the box right it's a bit bigger than the original thing actually and then yeah you can use this uh you can call the processor put the image in and the input boxes and yeah in the end you can get it out again and visualize so it's actually I I feel like the the actual part of the model is is quite small but there's a lot of things to do around like yeah creating the box and I mean up here Neil just did a lot of code for uh making it visible like the image and the ground TOS mask so um in total the code is quite small that you really really need when you don't want to visualize everything thing right just when you want to make it as visual as need here oh it gets a bit longer but yeah I would say say absolutely feel free to check out his notebooks they are pretty good also the one for like the custom data set training um which might also be really interesting for a lot of people but it's even longer so I don't want to go through it all here yeah so that is it for now I would say wait I have some thank you slide here thank you uh yeah do you have any questions right now uh I have one question uh so uh it's about fine tuning Sam the the Sam model was trained with prompts or boxes and um and the losses but we need the prompt so in in case of fine tuning do we need to like for example metsa manually label the the pieces we want in the image um what do you exactly mean by the pieces do you mean the segments or yes for example uh you need for example to to point the foreground background or boxes and do we need for example if we have an x-ray uh like kidney we have to box the kidney to and then do the the training Loop okay something else so now we do actually have a look at this fine tuning data set because I have to say that one thing I I didn't look into myself very deeply but maybe we can do it together now because I mean we have the thing here anyway um so we have our Crown Tru mask some data set blah blah blah bounding box oh yeah okay he uses bounding box prompt then prepares everything okay so here he takes svit base model right so the original one from Facebook basically and uses this some data set here so it looks like he is using yeah so he's using box prompts right as before I guess he created these box prompts um from the crru masks with some some noise to it so when you have your your data set and you do have some Crown to mask already you can just create this bounding box around this mask like he did before it's the same here right some some noise to it and then use these boxes um yeah as a prompt basically for Sam to find unit as far as I can see here that's what's happening okay thanks yeah no problem okay any more questions if not that is it and thank you for attending and have a good day or night or whatever time it is at your place