W&B Paper Reading Group: DETR

Key Takeaways

um let's get started hey everybody uh welcome to another paper reading group at wits and biases and today we're going to be looking at the endtoend object detection with Transformers so this is a um shift from what we've been doing at the paper reading group so far uh we've been looking at mostly uh common architectures like efficient net we2 we've looked at Transformer architectures like Vision Transformer um kite paper and mostly we've looked at classification kind of architectures and classification uh related papers but I thought um computer vision is a big field and object detection is one part of those is is a is a major part of that computer vision field so I thought maybe this could be a good shift and that we should also start looking at object detection papers given that we've already looked at a lot of Transformer papers so it only made sense to see where else Transformers have been applied and uh one place where they've been applied very efficiently has been in object detection as well apart from language and apart from just computer uh just classification type of problems so uh if you could maybe uh just double checking that everybody can see my screen um and if you could please just post in the chat that you can see uh 1 db. me/ so if I go to my browser can you see my browser as well please so if I go wb. me so this is the same as uh every paper reading group but for those that are new um to this for those that are it's the first time in this paper reading group if you go to that link 1db I've just post I'll just post that in the chat as well so 1db me so if you go to that link it will take you to this paper reading group D and uh this is where you'll end up so just go in there write a comment so as we're going through the paper today uh what we're going to do is we're going to um because we also share we also live on YouTube and it's difficult for me to look at the zoom chat it just makes things a bit easier if we post all our comments over here and then I can address those questions and I can keep coming back to them as we go uh something I do want to point out is that uh so this uh the paper that we're looking at is again uh object detection so that's the paper end object detection with Transformers um and one good thing about this paper is that they have open source code so um if you to that URL which is this repository here you'll see that Facebook research has uh open sourced the D code detection Transformer and this is not a new Transformer this is not something that's just come out recently like we've been discussing very Cutting Edge papers recently that have come out uh within the past two two and a half months or when we were looking at the MLP mixer paper um but this is more of a paper that's now been STA and has been there for a while for I guess a year and a half and uh one thing I do want to highlight is if you're interested in sorry one second um one thing I do want to highlight is this annotated D um is this annotated D can you sorry one sec I'm just able to see this uh background the PowerPoint in the background okay that's gone now that's better um so if you go to that blog post link I will share this in the chat or it would have been shared with you guys as part of the reminder anyway um but I've just put that in the chat um so anything that we're going to learn today if want to see how that's been implemented in that particular repository uh from code uh then this is called the annotated D and the simple idea is we go paragraph by paragraph in the paper and then we look at the code that's corresponding to that particular paragraph So if I go for example uh let's see which one should I look at let's see if I go to the Transformer um and you will see how this is mostly stuff from the D Transformer so you'll see how this is the figure that's related to that Transformer and underneath this there's code on how the Transformer has been implemented so anything that's been for example the Transformer decoder um anything that's been directly copied from the paper is in italics in this paper in this block post and then you'll see how the Transformer decoder has been implemented in py Doge so if you want to have a look at the code we won't be uh looking much into the code today um we'll mostly be spending time just reading the paper um but I just want to highlight this as a resource for anybody that's interested in in um looking at things on how they implemented in py to so that's that that's just the introduction about the ETR um but let's let's see now uh let's get started with with the paper sorry give me one second all right um so we're going to look at this end to end detection paper today uh so let me bring up my one note uh my best friend which is this software okay so as you'll see in this uh part of uh in the end to end object detection with Transformers as you'll see um I'm I'm also I haven't been doing object detection for many many years but I have been doing this for quite some time now a few months and one thing I realized is when I was cing catching up with the literature I saw how object detection has also Advanced quite a bit so we started with one stage and two stage detectors in the past so we used to have things like rcnn or we used to have things like YOLO which are still very relevant in today's world um so for General object detection you'll see libraries like and this is just me providing context so you'll see libraries like detecton true uh detecton two which is Again by Facebook research uh which is a very popular library that uh has pretty much a lot of the models that you'll see from from a bit older like retina net or all those models and then there's another one which is mm detection so I think these are both really good repositories to get started with object detection but with Transformers uh there's all of these separate repositories that are there um in this world well in this Dage right now but I think that might change in the future but anyway so then what's the difference in object detection with Transformers and all of these past uh methods that I just showed you that are part of those repositories U one of the main methods is that first um this uses Transformers so that's the main one I guess um in the you're using Transformers but another one is that this uh paper looks at object detection as a direct set prediction problem so what does that mean so until now um I'll give you some context again until now what used to happen is you used to have things like anchor boxes or we used to have object detection used to depend on some prior knowledge so we used to have like uh we used to have region proposal networks so what would happen uh before this paper came around or typically uh is that you have an image sorry that's a bit too uh that should be better um so you have an image and let's say um here's an object in the image here's an object in the image and what used to happen or would would happen before this is that you would pass this through a region proposal Network I'm just going to call it rpn and what that rpn does is it finds regions of interest in this image so that could be a region of interest that could be a region of interest and then based on that you start doing classification so you say okay this belongs to class one and this belongs to class 2 so then you're able to do object detection and classification but you can see how these region proposal networks have been quite a bottleneck for quite some time because you'll um mostly they used as a blackbox and also you don't have an end to-end pipeline what I mean by that is you're dependent on something that's not part of the object detection system in itself like this is not something you can change or this is not something you can train and so you don't have a very end to end Pipeline and what that does is it makes things hard to customize so then this paper uh what they say is we look at it as a direct set prediction problem what does that mean it means that we get rid of any extra stuff that's uh any any extra or third party or you know anything that's that's not part of this paper is not used in the datr architecture so they pretty much say that uh datr architecture is like this Standalone architecture that can be used to object that can be used to do object detection in an endtoend manner so from having you just have to provide the input images and the bounding boxes and then based on that this network on its own is able to do object uh detection is able to learn everything that you want to do and in fact uh when you go through the repository you'll see not only is it able to do object detection it's also able to do segmentation um but for this part of the paper reading group I'm mostly going to focus on the object detection part the um segmentation is only a small change in just the network head um but having said that uh let's just have a look at the abstract so I think one of the main uh parts of the paper are just is just the abstract itself so we should actually I actually spend a lot of time uh reading the abstract very carefully so as I said um our approach streamlines the detection Pipeline and it takes away the need for many hand design components so you used to have things like non maximum uh suppression which is nms or anchor generation so what nms used to do is like it's just like a postprocessing as I understand is that when you have duplicate bounding boxes it would just take away the duplicates or you used to have anchors so what happens in anchors is like if you have an image you pretty much divide it into to uh a grid and then these become your anchor boxes and you see okay is there is there like a an object inside or like that just becomes as a reference and so what this paper is saying is that we don't need anchor boxes we don't need nms so we we don't really need any prior knowledge about the task that's the way how this uh detection Transformer uh has been designed and then it says given a fixed small set of learned uh learn object queries D reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel so the word Global I think is is important because when we were using uh region proposal networks then as you can see you only have a look at the uh you only have a look at the regions that have been highlighted or that have been sort of flagged by the region proposal network but in this case you have a look at the whole image when doing object detection uh in the D architecture so then and the main thing is D demonstrates accuracy and runtime performance on power with wellestablished and highly optimized faster rcnn Baseline on the Coco object detection data set so not only is this a in a way a simpler endtoend way of doing things but it is also able to get the same accuracy and it is also able to get the same runtime performance as the current or as the state of Art art networks or as the uh you know as as a very competitive Baseline which has been fast rcnn and then as I've already mentioned uh the code is present here and then it's it significantly out performs competitive Baseline so that's the overall introduction about about the ET um so then there's there's two or three main things about this architecture but let's just go through the introduction so this is the I think I've done a shift from the P paper reading groups and in this one is that we're actually going to go Section by section but uh we're going to go in a different a slightly different order than than we generally do but I just want to point out this thing so modern detectors um or previously until before the you used to have things like you used to have a large set of proposals which I've already mentioned is the region proposal Network or you used to have anchors or you used to have window centers so that is how things used to be and these were surrogate regression pretty much because you used to have like a reference point and then from that reference point you would see okay this is my bounding box prediction how good or bad is it it's not like uh not like classification where you have some predictions from the model and then you directly match it to the actual ground tooth so that's that's not how things used to be but that's changing well in 2021 that's changing quite rapidly um and a big part of that is thanks to the dedr paper okay so then to simplify these pipelines which is again these surrogate regression pipelines we assign a direct set prediction approach uh and then this is an end to end philosophy it has led to significant Advance we've seen like this sort of Transformer or this sort of end to end philosophy which has led to uh advances in other fields but as they mention in this paper that at the time of writing not so much in object detection so previous attempts they have these either surrogate task or there like two stage um but this paper what it does is that it aims to bridge this Gap so as you can see in this uh figure one you will see that this is uh just a general uh overall uh just the general overall idea of how the D architecture looks like sorry one second okay um so as you can see then in this you will see uh just how the overall architectures looks like so you have some input image um and this is how I mean that this is how it's different is that your input image directly goes to a CNN so you get some set of image features what does that mean you know we we can use CNN as backbones so what does that mean is that you could have an efficient at backbone but instead of doing classification instead of you could keep the whole architecture until the point you have the global average cooling so this CNN would then output some features right so if your input is let's say uh if the my bat size is one and number of channels is three and then I have 2 to 4 by 22 4 which is my height and width 22 4 by 2 to4 um and then I could pass this image to the CNN and I could get some output of the image features which could be say uh 256 so these obviously when you go through a CNN your number of channels goes up and your image Dimension goes down and I'm just making these numbers up so don't we we'll look at the exact number of channels and we look at the exact numbers just in a while but I just want to give you a sense of like okay how does how do things really work so what happens is the number of channels will go up and uh as we know a CNN it will just reduce the spatial Dimension so you go like something like 48 by 48 so instead of having 2 to 4 by 2 to4 you've reduced the height and width but you've also increased the number of channels um so that's the idea so once you have that this now smaller Matrix becomes a representation of the image and then you pass that through a transformer in coder decoder architecture we will look at exactly uh what that architecture is but then this is how uh the D is very different from the the previous uh papers that I read about object detection is that you get a set of bounding box or pretty much predictions and you see how these different colors they are like different predictions that you get from the architecture but you can match that using this idea of bip bipartite uh matching loss so you use this bipartite matching loss and you match the predictions with the ground truths directly so at this point you don't need any anchor boxes you don't need anything um and this is how the dedr architecture when we look at it it looks very simple and it is actually very simple because if you when you read through this paper they'll say oh we've tried to make the training pipeline really easy and um the D architecture on its own doesn't need any uh special sort of uh any any special sort of modules you just need pretty much sorry one second um any library that that can do uh convolution and you have the Transformer architecture as part of the library which is pretty much pyos tensorflow jacks or pretty much any library that you uh use for deep learning they have the convolution and Transformer architecture as part of them so they say in in this paper there was some line mentioned somewhere that said oh because of this because the architecture is so simple uh you know that you can actually use any library that you like to implement the D architecture it does not require any special libraries that have to implement these blackbox networks like rpn or you you don't need to worry about anchor generation or all that sort of stuff which is quite clunky and which has been a blocked bottleneck so that's the idea behind D so as we see uh this there's good stuff written here we adopt an encoder encoder decoder architecture so what does this encoder decoder architecture mean well uh that's from a attention is all you need uh Transformer so as has happened with many fields like when the transforma architecture came about in attention is all you need you um Not only was that a big big success in natural language processing and let me see where the architecture is encoder decoder Stacks there it is um so that's the architecture which is the attention is all you need paper which introduced multi-head attention um and we we s started the first time this was applied was for machine translation so sorry language translations from say English to German or that sort of uh that sort of objective but then people realize that this architecture is actually very helpful for all of these other tasks as well or in computer vision then the encoder was taken and the vision Transformer was born so this is me just again giving context and we're not really looking at uh the D but I'm just giving context on how things have evolved so you saw that there's like this images Worth 16 Cross 16 Words which is Transformers which pretty much applies this same architecture but just I believe the encoder um for for classification so I can see where is the Transformer do we have do we have an image there it is so you see how it's just using the Transformer encoder but that's again for classification then for object detection I'll go up um then for object detection you have end to end which is the D architecture so that's the idea that's the encoder decoder architecture again uh I'm just having a look if there's any important bits that I should cover here um so this is the main bit like as you can see in figure one which is the figure I just showed you it's this figure at the top uh you can see how you predict all objects at once and it's trained end to endend with set loss function so we will look at the loss function we will look at what this bipartite uh matching means and we will look at all of these in in a lot more detail in just the coming uh few moments but right now I just want to sort of uh introduce you to this idea of how D is different so this is just me kind of providing you an overall sort of uh objective or that's sort of overall uh idea of how the ETR architecture looks like so you can see we don't need spatial anchors we don't need nms that's just being a repeat and that's it oh this is the bit where I said d doesn't require any customized layers and thus can be reproduced easily in any framework that contains standard CNN and Transformer classes which means you can implement this in tensorflow you can implement this in pyge really really easily and that's why the D could be its own separate repo and when we'll have a look at or when you have a look go through the annotated you'll see that things are um really simple and straightforward to implement this architecture okay um so that's the basics of it uh this is where they say you know how um until now in the paper reading groups I've been mentioning image net a lot uh similarly in object detection you have this Coco data set which is kind of this Baseline um and then that's just used for detection instead of um instead of classification so I believe it just struck me that I haven't really said what object detection is I just assume that everybody who's on this call knows what object detection is so object detection just means if you have an object in classification you say this is a horse this is a cat or you know whatever the object category is in object detection you not only classify but you also provide the coordinates so you say okay this goes from X1 y1 or you know just pretty much you just provide the coordinates in the image that that this uh problem is then object detection sorry that's something I should have done right at the beginning but anyway let's keep going if you're here as part of the I'm sure you'd been you'd know what object detection is um so then uh one thing I would like to uh point out here is that D demonstrates significantly better performance on large objects but one thing that's been pointed out in this paper is that it however obtains lower performance on small objects but uh since this paper was in 2020 uh so let's see I I believe it was around September 2020 what exactly uh 28 May 2020 sorry so it was in May 2020 and um since that time a lot of uh a lot of new architectures have come up like the de formidable D up D or like this these other architectures that have come up based on the D architecture and then they have tried to look at this uh downside of like the lower performance on small objects so that's that and that's pretty much it in terms of introduction to the um I won't look into the Rel related work CU we'll come back to that later I think I won't look at the loss function right now as well so I'm going to skip this this part of section three of actually having a look at the loss function but we're going to do things a bit differently is that we're going to have a look at the architecture first so we're going to have a look at how when we pass an image to this the architecture how does it process that image and then how does it really pass uh how does it really give out the uh classification outputs and the object detection out puts and then we're going to see and then we're going to go back at the loss function and then that's when we're going to see how uh how those outputs can how those outputs can be used to calculate the loss so let me put that in one note one second oh I am in one note one second sorry so if I go and I add a section here so as I said the way we're going to look at is the first thing was the intro then we're going to look at the overall architecture so that's second and then finally we're going to look at the loss so in this overall architecture we're going to see how things look like so if this is my architecture let's say this cube is my D architecture we're going to understand how things look like if I pass in an input image then how does it really give me the bounding box and my class probabilities prob I'm just going to call prob okay so so we're going to have a look at this part of this whole architecture and then we're going to come back and we're going to have a look at the locks function so you're going to see okay now we have our outputs how do these outputs compare to my ground truth and then we can see how to calculate the loss um that's how um today is going to look like but let's see if there's any question on that report so let me go back to that report yes uh that is correct does surrogate pipeline mean using rpn for anchoring or well pretty much surate tasks means just you have a separate task you don't actually just compare the output with the ground truth you have some other third task that does it um in bip bipartite matching how ground truth boxes can be compared while actually trying to identify the same object okay these are questions we haven't really looked at any of this so these are questions we'll come back to later but let's first go and have a look at the overall architecture um so let's see that so in figure two this is where the overall architecture has been defined and I'm going to look at this architecture a slightly bit differently we're going to have a look at this figure too and then I'm also going to show you how the Transformer inside looks like okay and then if there's interest I can also show a overall idea of how um these things can be implemented in codee but we we'll get to that in a while so let's see sorry I've lost the architecture I'm so there we are that's the architecture um so let me actually just copy paste that in a new part so there's SL clunky so let's paste this here and then okay so what happens is this is what happens I have my let's say I have my input image so I'm just saying three channel uh 224 this is 2 to4 2 to4 224 by 2 to4 is just a number that I've come up with it could be 4 448 usually in object detection you have higher size images like 640 by 640 but I'm just um don't code like don't take these numbers take them with a grain of salt and then these are just numbers I want to present just so everybody has a good understanding of what's going on in the over overall architecture first and then you have three channels so the three channels are red green and blue okay so this is my input which is my input image you pass that to the CNN now this CNN could be a reset 50 it could be a reset 101 it could be any CNN uh like an efficient net or could be any backbone that can extract some information from my input image so any backbone that we've looked at in the past could be used to extract information or could be used to extract image features so what this would do is that once I have my CNN what could happen is that my outputs again what's going to happen is that instead of having three channels I'm going to have say something like I said 256 in the last time but let's say I have 2048 or actually let's just keep with 256 so let's say I now have 256 channels and then my feature size gets reduced to 40 by 40 so that's the image features here right these image features are 256x 40x 40 um and then that's my uh that's that's just the output that's like a representation of my image and then I have my position encoding so what happens is uh if you see my input I'm just showing you an I just want to uh sort of highlight an example is like if this is 40x 40 so I'm just uh making a grid of like 40 by 40 then each position the Transformer uh from the attention is all you need paper you'd know that the Transformer doesn't quite know the order or like it doesn't really know what position things are in so if we pass the Transformer expects the input to be in a sort of a sequence so let's say um when we were doing translation if I said I am Amon right uh the Transformer doesn't know that this is position three or this is position one this is position true the Transformer has no idea about the ordering of things and that's why again this idea of positional encodings is coming from that attention is all you need paper so what you do is you pass in positional encodings I will explain exactly what the shape of these positional encodings look like I will explain all of that stuff in detail in just a little while but uh the main idea is then from these features you need to know how like the how The Ordering of things look like so you pass in these position encodings over here which would be the same length as my input so right now my input is in like a form of a tensor Matrix I could just pretty much flatten it out so my input becomes like a flattened image so this is what we do we' we've have had a look at this in know in a briefly in in the vision Transformer just from the idea now my input is again say a sequence and then what you do is that to that input uh to that input image or that input feature which is this flattened image what you do is you add these position encodings so now when we add position encodings we have to basically inject some positional information to the Transformer so the Transformer knows where features are so the Transformers knows okay this is the top left and this is top right or this is or sorry bottom right or this is top right and this is bottom left like the transformer needs to have some information about the positions and these positional encodings are responsible for doing that okay so that's just the basic idea of why we add positional encodings and then one thing you'll see when we pass things through the Transformer encoder it kinds of It kind of have a looks at the image it kind of have it it has it it looks at these position encodings and then it gives some output so without looking at what goes on in this Transformer architecture there's another beautiful beautiful image that will show you what exactly goes on um let's just take this as a black box let's say okay all of this is a black box then I can get my prediction outputs which is pretty much I get like an output Vector that output Vector can go into these feed forward networks which can say okay which can perform the classification which can perform the bounding box so let me show you the Transformer uh what exactly goes on in the Transformer so this is the image okay hm okay thanks for doing that to me um so then that's that okay so what happens is as I've already said once you have your input image that input image gets reduced to a smaller feature map okay so this input image will get reduced to a smaller feature map but it will have a lot more channels so it will look something like that so let's say um as I've already posted in the this uh block post let's say that looks like 256 by 24x 29 so let's say my number of channels are 256 and then my height and width become 24 by 29 right let's say that's what happens and then what I could do is I could flatten it out like this is my 2D grid this could be one vector right I could because this is just one uh 2D Matrix I could flatten it out right so then I could get my outputs in this shape okay all the shapes are now here so I could get my outputs uh please feel free to interrupt or uh post a question in this is the most important part of understanding the D architecture and possibly also the most complicated but be with me stay with me and let's just spend extra 10 15 minutes just looking at what exactly goes on in the Transformer and how the det architecture is then able to make those predictions right so I'm just trying to marry this uh this bigger image at the top uh without looking at this this is all the Transformer so I'm not going to look at the Transformer from here this Transformer is just this image at the bottom okay so right now all I have is I had my input image here I passed that through a backbone which then gave me 696 by 1X 256 okay let's say those are my image features where does that 696 by 1X 256 come from my uh when I passed that through the backbone remember I said we get 256 by 29 by 24 right let's say this is what the backbone is going to do what I could do is I could flatten this out so this uh this becomes 256 by 696 and I could reshape so that's become that becomes my input to the Transformer is there any question about uh I'll just actually have a look yeah in 250 6 by can we say 40 CR 40 is the size of the image reduced from 224 by 224 and what is the intuition for 256 channels well 256 okay so in a in a backbone or pretty much in convolution your networks what happens is you have a bigger image like you have a massive image as input and as this image goes through each of these stages of the convolution neural network what happen is the number of channels keeps on going up and the image uh size keeps on going down so in a way you don't lose any computation you don't lose any information okay so then this 40 Cross 40 is this smaller 40 Cross 40 is the smaller uh representation of the bigger image which is correct and then the number of 256 channels in actuality it is two 2048 but then that gets reduced to 256 but let's just say right now 256 by 40 by 40 or 256 by 29 by 24 I'm just using 256x 29 by 24 because that's what uh the shape is assumed in this image so I'm just going to assume that when I have some input which is say three channel 224 by 224 the when I pass that through the backbone the output becomes 256 by 29 by 24 so this is a lower uh resolution feature map or like a smaller feature map so instead of my image being massive it is now smaller but it has a lot more channels okay that's just that's just passing through a backbone like that shouldn't be that's not something that should be confusing at all cuz um that's just part of any standard classification so that's the idea so that becomes my input to the Transformer okay see here uh let's go at the bigger image so you can see how things get passed to the CNN and before things are passed to the Transformer from there I flatten it out so then my input to the Transformer is this so it can be thought as I now have a sequence of length 696 where each token is 256 Vector long okay that's the idea like now after this point it's just after this point it's just uh it's like machine translation or it's like language translation CU I have my input image converted into a sequence okay that's just how we can look at these things so then what happens is this is my Transformer encoder like uh this is just again I'm not going to go into the detail of the Transformer encoder it's just multi-head attention followed by a feed forward neural network again this is exactly the same as the attention is all un need paper so where is the attention is all you need paper so you can see how that is exactly the same as the attention is all un need paper the one thing we do is we adding these positional encodings so as I already said um positional encodings are then the Transformer needs to know where the relative position or the position of these objects or these different classes or these different labels are in the image so you have these positional encodings and they're going to be the same size as my input right why um if this is my input then I need to have a 40x40 grid or pretty much I need to have like that's exactly how many points or that's exactly how many positions I need to cover for so I can have my input uh as as like 696 and then each position encoding could be um again this is if this is confusing about position en coding again this idea is straight from the attention is all you need paper so the positional encoding is instead of it being one number it's a vector right and because right now we have converted our input image into a sequence of length 696 then this is how it looks like so let's say this is 1 2 3 so on 696 I should make this a bit smaller so my input is now 696 then for each of these positions right 1 2 3 so on until 696 I need to have a position encoding so that is why my position encoding is say 696 by 1 by 256 CU each position is then represented by a 256 length long Vector okay so that's the idea so you input both of them you input that to my Transformer you get some output from the Transformer encoder which is going to be the same shape again this is just using the Transformer architecture as is you get some output from the Transformer which we call it memory okay then on the decoder side of things what happens is now we haven't really uh done any object detection so far right we just passed in my input image as a feature we passed in my spatial position encodings which is just uh useful for the Transformer to know the relative position or the position of all of these 696 sequence um but we haven't really provided any object queries so what the uh authors really did which is really really smart of them to do is what you can do oh sorry one second uh what you can do is I can have uh in pyos you can have something as nn. embedding well that's just that's just an embedding so what that means is I can start with random 100 position so they the authors of D they said okay I'm going to assume that there's at Max 100 objects in my input image okay so my input image right now is this 69 96 long uh V this 696 long sequence where each part is like a 256 long Vector okay so what they said is I'm going to assume that in my input image there's going to be at Max 100 objects right so what I can do is I can input 100 cross 1 by 256 so I can input that as my position embedding so let's say this is my object positional embedding okay um so the idea is then I'm going to input these 100 random positions in my my in my in basically in my Transformer decoder so this is what happens when I pass in my object queries over here these are just right now the network hasn't learned anything these are just random 100 positions in my input image and then what I say is okay uh in my in my input when I when I answer these random when I basically input these random object queries firstly these object queries will start to perform self attention within themselves so this is what's happening this is why this is called self attention because now the objects or like these 100 as the network will learn more and more it needs to know like okay where the objects are or what the relation is between them right it needs to know okay the cat is there the fence is there or like in an image it needs to know all of these different things but there's there's mostly a relation between the objects themselves as well right so what it's going to do is right now because we have these 100 we just provided the model with random 100 uh starting points what the model is going to do is as it's going to learn it's going to learn this relationship between these random 100 uh again when I say 100 that's what the number the uh bet paper authors have chosen it could be a different number as well but this is what they thought they said okay at Max in Coco there's going to be 100 objects in one image so then you have this 100 objects sort of interacting between themselves but we haven't really looked at the image so far right that happens here so once you have this self attention which means these 100 objects these random 100 starting points interacting between themselves you get some output which is this output over here you mix that output with the output from the encoder so remember now I have my interactions between the objects themselves I mix that with the output from the encoder what the encoder has done it already took my input image it already took the positional encodings for the input image and it has been able to basically look at that input image and has the information about the image so then we can mix the information about these 100 random objects with the information of the image so now the Transformer decoder actually learns to uh marry the various object positions within the image itself I hope that makes sense because this is exactly what's going on in the D architecture so you do that over and over and over and over again you start doing this over and over again few too many times for uh say a 300 or 350 eox by the time by the end what you have is you have this D architecture that's then able to learn the position of for each image it's able to predict these 100 uh object queries it started with random positions in the image itself but by the time you finish it's able to tell okay uh the first object is over here the second object is over here the third object is over here and so on and it can say the 100th object is over here so that's why you get this output of uh 100 by 256 which is just the output forget the six that just uh means it's Ox uh it's using ox Closs so what that means is in deep learning terms because the decoder is six layers in the D it takes the output from each of these layers so if you take the output of 100 by 256 from each of the six layers your total output becomes like 6X 100 by 256 I hope that shouldn't be confusing but this is exactly what's going on in um in the D architecture you start with again I'll summarize I'll summarize using the going back to the paper and then I'll take questions if there are any so let's see D so let's use this this image now to try and summarize okay so I start with my input image over here this is my input image the input image goes through the backbone the backbone will reduce the input image to a smaller feature size which could be any you know depends on how how deep your CNN is or depends on all that stuff but let's say it is a smaller feature map of 256 channels and then some height in with 40x 40 or 29 by 26 it's just the idea is that the backbone is going to reduce my input image into a smaller feature map and then for each of the position in that feature map I'm going to supply a position encoding to my Transformer because the Transformer has no idea about the relative positions of objects in my image or it doesn't even know how that image is formed so that's why we're providing these position encodings and then the Transformer encoder will do its magic what does that mean that just means that it's going to take my uh backbone output it's going to take my positional encoding output it's going to mix the two and then because the now the Transformer has information about the position of these various pixels in my image and it also has the image information itself it's now able to give some output which I call let's say memory this memory is this output from the Transformer encoder which is the Transformer way of encoding my input image into some Matrix inside so that's just a way of like let's just call it an intermediate output it's just a representation of my image so that the Transformer knows where each of the different objects are in my image it's basically just the Transformer reading my input image and then storing that information it's just like we humans do we we look at an image and then let's say we store that information in our eyes that we know okay uh when we look at an image for example in front of me I can see okay the laptop is over here the door is on my right so that's just like a a map of what that input image looks like and then uh in the Transformer decoder side of things this is what this is where all the interesting fun this is all the interesting stuff happens all the inter interactions happen is that now if I'm trying to do object detection I start with some random positions of these 100 different objects so for example in my room I could say okay I think the Des desk is going to be on my right it doesn't matter if the desk is on the left okay it doesn't matter I can say okay I think the desk is going to be on my right and then I think the bed is going to be on my left I think the TV is going to be in my front I start with some random positions then I mix the information that I have well okay it doesn't make sense that the TV is in front of the bed I I hope that this makes sense that it it you know when the TV is in front of the bed that doesn't look like a good position so there has to be some interaction between these various different objects so that happens the first time and then once you have a uh an interaction between the objects like you know okay uh where the TV would be related to the bed or where the table would be related to the TV and the bed then you mix that information with that previous map that I had in my head of the room so I knew that the door is on the right I knew that the screen is in front um so instead of now starting with these random positions of these uh 100 objects I start to learn the position of these 100 objects so that's what the decoder is doing it's mixing the information that we get from the encoder and it's mixing these 100 different uh it's mixing these 100 different random object locations so now the decoder as we go through over and over and over and over again and we do these multiple multiple epochs using the loss uh it's it's without telling you what the loss is um at this point then the model is actually learning the position of these 100 objects okay there might not even be 100 objects in the image that is fine we will look at that later but for but just have a look at this that it's just starting with these 100 positions but it's now able to mix and match and it's able to interact um based on this information and then finally you get some output from the decoder uh which I've already said looks like uh as you can see the output looks like this 6 by in terms of shape 6 by 1 by 100 by 256 you could pass that output uh where are we uh you could pass that output to uh any feed forward Network to get my class and bounding box class and bounding box class and bounding box that's how the D architecture overall works like um okay I've been talking quite a bit and now is the time to take some questions so let's go to questions oh sorry one second all right let's let's have a look I could understand position encoding for 696 as the size of the image but how position encoding can happen for 256 channels what does that mean well um in position encoding I think I would refer you D to the attention is all you need paper uh it it's not like the position encoding is happening for 256 channels think of it this way a position encoding is a 256 long Vector itself so for each of the 40 okay uh let's do it this way so your input image is like this right your input image is this is my input image so let's say this is uh sorry this should be fine let's say this is 29 let's say this is 24 just as an example right so this is that this is that now for and this is 256 okay this is just my input image right now now for each of these positions this position this position this position this this so on all of these positions I have my position encodings as a 256 long Vector okay that's that so then I can add my position encodings to this input image and then I have like a the Transformer has some idea of looking at the position of these various different points in the in the um in the image I hope that helps and if it does help could you maybe you please reply to this comment and just say okay thanks that helped how did they get the positional encoding didn't understand that part oh sorry which part of position coding so in uh you could you could just start with like with position en codings there's two ways of having a position en coding in the attention is all you need paper you can see how you have sinusoids and cosine which is just uh each position gets a representation of a 256 long Vector but you could also learn position in coding this I think uh this question is coming from more from the attention is all you need paper then from then from d uh but the main idea is you can have some representation some 256 Vector long representation for each of these different positions in my image uh how are we mixing the Transformer output of 69 6 by 1X 256 with 100 is it doing an attention sorry this is a I don't get this how are mixing the Transformer output do you mean to say rames the backbone output uh yes the encoder output uh your encoder output is 696 long sequence right and then you use that in the in the in the decoder um yeah object or the query Vector which is 100 by 1 by 256 right when it go mean you mean all of this here right this is where the mixing is happening yes okay well it's just learning to pay attention to the 696 well it's just uh like a 696 you could think of that as a as a sequence so it's just learning to pay attention to like these various uh objects and it's learning to pay attention to the memory so you can see how my uh you can see the query qu and uh query key and value uh of the inputs of this multi-head attension so I would recommend you without because uh if I go into the details no it makes sense to me makes sense to me I add one more followup question on the same thing um is U does it also uh you have M layers in decoder does it take the final output of encoder and feed and use that in all M layers of the decoder that is correct so it will take the out output from the encoder and then it will go into the first layer of the decoder which goes one by one by one into the layers but that's the that's the idea instead of I think you're thinking it differently you're thinking of these decoder layers as being separate but they're actually stagged so you have your first layer then you have the second layer on top then you have the third layer on top then you have the fourth layer on top uh but you can just pass this output from the encoder to the very first layer and then you have these other layers stacked oh so it doesn't pass to all all y layers only to the first layer of decoder uh yes that's correct it passes it to the first layer of the dec decoder and then the second decoder layer gets the output from the first layer of the decoder the third layer of the decoder gets the output from the second layer of the decoder and that's how this whole system works again uh this is a question more from attention is all you need than from the D paper but uh for all of these questions I would recommend just going back to the just understanding the Transformer architecture quite deeply um but that's the idea so uh durka I'm still waiting for uh if if that explanation that I helped about this explanation about uh position en codings does that help or or not please uh let's see if there's more questions is all of the uh oh excellent thanks uh or are is all of the learning visual spal I don't even know what that means um or are colors able to in be incorporated I don't remember any fixation talk as far as I know what in the TV in front of the bed is neon green I don't really understand this part uh so I'm really sorry I don't think I can help answer that question more about that was just trying to help create an intuition um in terms of colors because we have reduced the three channels to 256 channels you don't have to think of the colors as being three separate channels anymore the whole image is now 256 channels 40x 40 right so it's now a lower resolution Matrix now what those 256 six channels are that's just up to the Transformer to learn so it's a very deep l