Getting Started With Transformers for Computer Vision - Divya Swaminathan & Tony Reina

Key Takeaways

All right, I think we can start. Um, welcome everybody. So, today I've got I'm Tony and this is Diva and we're going to be talking about getting started with transformers uh for computer vision. So, at the end of this um you should have a basic understanding of what transformers are, how they work, and then Divia is going to show you how to actually use it like in a Jupyter notebook. Um, it actually is super easy now with with hugging face. That's what we'll be talking about. So the latest um like decade basically has all been deep learning driving kind of the AI uh revol this this latest wave of AI computer vision and natural language processing NLP have been kind of the two big uh areas uh of this. Um it can be argued that 2012 with uh AlexNet and Alexki's paper uh that really started this latest resurgence of CNN's for computer vision that was on image classification but then object segmentation semantic segmentation. Uh the one Divi is going to show you is object uh segmentation or actually object classific uh object detection. So yeah uh and then natural processing natural language processing uh 2017 is really when the Vaswani Google brain paper came out with attention is all you need talking about this transformer model. Um it was a brand new type of neural network uh or I'll show you it's it's almost brand new. There's there's really just one change uh to things that we you are used to uh and that literally transformed the NLP community. Um, this is really what people use now. Uh, so transformers, the T the big T like chat GBT, uh, if you use any models with a T in it, it's probably a transformer model under under the hood. What's so great about them? The whole reason that they were created was because initially for NLP, it was a lot of recurrent neural networks. those were very hard to train on GPUs because there was the the recurrence the sequential part of it actually limited its ability to be parallelized. Um so the sequential part was hard for that task. Uh transformer models end up scaling. They don't have to be there's no recurrence in it. You can actually scale it across lots of GPUs. Uh and then there's the the key takeaway was instead of recurrence they had this attention module uh that I'll go through but the idea here is this attention module that was the thing that would learn relationships between words. Um and so in in language word position in the sentence is actually important in defining the meaning of that of that word. Uh so this combination of parallelism with the attention mechanism that really opened up uh the doorway to this and so I I talked about two different fields CNN's being for uh computer vision and transformers being for NLP. Transformers actually be can be used in computer vision as well. The idea here is this is uh an image is worth 16 by 16 words. They literally just take a picture and divide it into a 16x6 grid and each uh tile of the image is a quote unquote word. And now you can feed it in and pretend like an image is a is a language by just breaking an image apart into tiles and considering each word to be a or each uh tile to be a token of the of the language that you're dealing with. If you look up here with the transformer encoder, this is the the basis of the network uh for the transformer the the norm layer and the ML MLP the multi-layered perceptron those were there before. It's really just this multi-head attention that is the new thing and so that's what I want to describe and then we'll go into an actual implementation. Okay, so it's harder to describe with the pictures being divided into tiles. It's a little bit easier to describe in the NLP domain because we have words and we all understand words. But the idea is there's some like atomic, you know, structure. There's some like smallest part of the thing. Uh a word, a word or a word piece or a token is the smallest part of the thing. But something like the word park uh with these three sentences means slightly different things. Um and it depends on the other word. It depends on where it is in the sentence and it depends on the other words that are in the sentence. Um, and so this is what they were trying to solve with the uh with this attentional layer is being able to change the meaning of the word park based off of the other words that were in the sentence. With everything on a computer, it's got to go into a number form at some point. So it knows nothing about what these it knows nothing about the word park, the letters P A R K. Um, it just needs to be put into a number form. And so if you in fancy terms we call this an embedding space but just think about it as this room right if each person is a word in the language um so you know you're the word park um you know you're the word soccer uh you're the word uh you know something like that so everybody's a word in this space uh if I just say I'm 000 I can define in a two-dimensional grid like two numbers that define where you are in the face. And it turns out that you can train the model, you can train models to to create this for all words in a language. And the way it ends up usually end up being training is that words that are similar. So all the animals are back there and all the colors are in one cluster and all of the the uh meals and food are in another cluster and all the transportation is in another cluster. So words that are similar in meaning seem to be in similar clusters in the language. And so they're going to play on that um with this. So imagine if you had this is the space um so I'm just showing as two dimensions. It's actually 768 dimensions. I'm doing the BERT model here. Um this is uh the embedding space for this sentence. uh initially in that in this room the word park would have been here in this open circle here the other words here so your park uh the word soccer and uh and play uh or like over here they're actually pulling on you and moving you slightly in the room slightly within the embedding space. So now you're slightly offset to the original meaning of the word play into this new uh version of the word play. So they call this attention like how much you you're in your same space, but how hard you pull park. Uh that's the thing that it's learning is how hard it should it should pull. So if I'm doing this sentence, it pulls it this way. If I'm doing this sentence, well, now maybe there's another word garage or something like that that's out here. That's moving you in a slightly different uh place. And so for the three different sentences, park is about in the right place, but it's in slightly different places. And that gives it the different context, the different meanings. Um so that's how the intentional layer is working is it's just saying that in a if if we consider this room to be an embedding space words are living there words that are close together mean certain things we can just perturb the space a little bit we can morph the space a little bit uh and that's what you see in these these three I'm going to move it over because it this is uh this is pretty complex but Diva is going to show you how easy this actually is and how most of this has been done for you in hugging face. >> Thank you Tony. Hi. So basically I'm going to talk about if you are curious about transformers how you can use them and how you can use them for image based tasks. So the hugging face platform is a platform where a lot of uh pre-trained MLAI models are available and you can just plug and play. Essentially the big success hugging face has had is with the transformers library. Uh and hugging face has a lot of different annotated data sets. So in my daily life I work a lot with images. But if I am curious about any other type of data set I can log in and I can you know look at audio, video, text and start playing with it. If you were just go to go to the landing page, you will see they have about 2 million models, half a million data sets, something called tasks and spaces. Tasks is basically uh the different uh machine learning tasks they have and the models they have to accomplish these tasks which could be image uh object segmentation, image classification, textto image generation, etc. and space is the part of the platform where you can deploy your code. If I do a search with the word transformer on hugging face, I end up with thousand models, 250 data sets. Um so yeah now if I was I want to start with image classification and so all you need to do is uh go to perhaps Google collab open up a notebook or you can even set up an environment open up a notebook locally install the transformers library and here I took an image from the hugging face portal this is an image of a bird and it's essentially just four lines lines of code and I get a classification result that this image is a mech with a probability score of one. So you basically from transformers will import this abstraction which is called a pipeline. It is a userfriendly abstraction uh that allows you to use a pre-trained model on an image that you have just taken from the web. The main thing is you have to create this uh pipeline. I mean here I can choose whatever name I want a classifier a pip uh put it in this abstraction. The first argument that goes in is the task you want to perform. In this case I want to perform the classification task and then I choose the model that will go in. So you have lot of options on hugging face that you can pick for image classification. This happens to be the default model. So the main thing is specifying the task is mandatory. Specifying the model is optional. If you do not specify it, you will get the same result. So once you pipe your image through, you end up with a result. result is a list of all the labels that have been generated and the first uh uh the first element is the highest probability score. So most likely your classification uh now if I wanted to segment these birds all I need to do is change the task object detection now but because it's a different task the model is different the same rule applies specifying the task is essential model if you do not specify a default model will be pulled in same thing uh the image is passed through this uh segment pipeline and I have the result like it shows up in segmentation models. You have a probability score, you have a label and you have the bounding box. So two birds were identified with a probability score of one. So the main point here is that the barrier to entry uh for a classification and segmentation task for a general image is just it's very low. This is essentially four lines of code and you're getting your result out. Um I am showing here it's these are four different images from the cocoa data set and they have been passed again in the same this four lines of code and it does a pretty good job. You have all your detections. So yeah, it's easy. But Tony and I live in the world of biological images. uh uh so what happens if we just take some of the images we are used to and put it through this pipeline. So these are examples of GFP tagged cells that I just picked from the internet and clearly the model is failing. So that is the first result or the first sort of uh thing I want to uh pass on here that default models will definitely fail on custom image data sets. So depending on your uh problem, depending on your imaging data set, um uh you will have to tune these models. So here's a case study. I wanted to look at breast cancer classification. So the first thing I did is I went to the hugging face portal and I looked for uh a data set for cancer images. I came across this uh multi-scale u multicancer data set. So this had about 130,000 images of several cancer types. So I took a subset of these images. So I identified all breast cancer images. So that was about 10,000 of those two different categories benign malignant and you push it to the hub. So this way I generated a practice data set. So now this is available for anybody to use. If you want to play with breast cancer data uh classification, you can go and simply load this data set in the notebook and you will have the uh it's again a transformers abstraction called data set dictionary. So now I have 7500 images in the train data set 1250 validation and 1250 in test. So uh you can this is an example of what those images look like. These are histopathology images different from the GFP tagged images but there is just in in the biology space uh there is a lot of variability uh within the kind of image data uh one comes across the modality which is being used for imaging the task you have on hand. So yeah, it it can be very varied. Uh if I were to take it and put it into these my four simple lines and do an image classification task, this is what I get. Image was classified as a neatode. Highest probability of 0028. So no, it won't work. You have to So what can you do? You have to fine-tune a model. And the finetuning part is that here I'm uh choosing to work with the goo Google vision transformer model. So I can just call it in. And the main thing is you freeze all the model layers but train the final classification layer and that also is achievable. So it's very it's it's easy to familiarize yourself with the transformers library and use it. So again in three lines of code I have I have instructed the code that okay freeze all the layers only train the classification layer uh and then you run the training. This is my result. I trained the model only for three epochs in the interest of time. uh it took about 40 minutes and on the validation data set it gave me 96% accuracy. Uh recall if you plot the confusion metrics uh the recall is pretty bad 48% uh recall for malignant cancer. So two lessons here uh for breast cancer classification or cancer classification accuracy should not be the metric number one and one has to improve recall. Recall is the one that is important. The other thing I want to say I did not show this model all of the 7500 images. I actually only showed thousand images. So now I have run uh an exercise end to end but I have to go back and now figure out how I can improve this um performance but overall it's a nice little project to familiarize oneself uh with the hugging face platform uh and to work with transformers library and if you have never worked with image data to get started with computer vision as well. So the point is if you were to do this 10 years ago you would be sitting and writing the model but right now things are different and um yeah I will repeat myself barrier to entry is low. Uh what are the pain points in uh just from experience in any image based problem? Uh the pain point is always the annotated data set depending on what problem you are wanting to solve. Uh you have to make sure the data is available. uh it always takes a lot of time, effort and money to generate and annotate data particularly uh biology based uh data sets. The other thing I will say is it's important to get clarity on how the model is ingesting data. These are all tensorbased uh models, right? You need to be very sure what dimension is going in, what dimension is coming out. That's a slippery slope also uh with I only showed uh segmentation uh sorry classification today. If you work with segmentation data sets uh they are slightly more complicated to prepare them uh for training and for model ingestion. And in terms of training hardware, I did this on my laptop. So I quickly ran into issues. So if you are dealing with complex data, big data sets, you need to uh consider how you are going to uh get access to compute power. So the wins though it's easy to set up, it's easy to get started, you have lot of different types of models, data sets and tasks you can explore on hugging face. really curiosity is the limiting factor and a lot of different resources are available. Hugging face itself is a good resource. YouTube is a good resource and I will do a little bit of self-promotion. I have uh all of the work that I have shown here as well as some extra work is in the form of these transformer tutorials on GitHub. So these are all notebooks that you can open in collab and run um yeah if you so choose. So I think that's the end of our talk. Thank you so much for your time