18: Object Recognition (cont'd), Texture Perception

Key Takeaways

Uh today we're going to wrap up talking about object recognition and get into texture recognition. So the um remember um object recognition is a thing that we can do. You can look around scenes and name objects. Okay. Um, it's mostly effortless for for people, but a pretty challenging computational problem. In humans and other primates, object recognition is something that we we typically do when we foviate objects. So, it's kind of dominated by central vision. Uh, and so oftentimes when you're looking at a scene, you'll be making a series of sacads around things in the scene, recognizing whatever it is that uh that you your eyes uh focus on. So, the problem is hard for a couple different reasons. One is that there's a large number of things that you can recognize. Okay, this is just a small subset. Um and then the second issue is that the same kind of thing um can produce totally different images like depending on the viewing conditions, right? the the viewpoint from which you view an object or the illumination conditions or what the background is that the object is on um or whether it's oluded um or whether like if you're talking about a biological organism whether we're non-rigidly deformed you know I can do all kinds of weird things with my body right totally changes the image okay >> [snorts] >> um so you get wildly different images depending on the viewing conditions okay um so we talked last time about how there's lots of evidence that object recognition is um largely mediated by the vententral visual stream. Remember dorsal vententral streams. We talked about evidence from lesion studies where you leion the temporal lobe. Um and monkeys in humans end up with recognition difficulties. We talked about agnosia. This is this uh inability to recognize objects that occasionally results from brain damage following strokes. Um and then um we talked about the the vententral visual stream and evidence that object recognition object recognition is fast and thus the inference from that being that in many cases object recognition is mediated by largely feed forward processing. Okay, so this is in the macac visual system. We got lat latencies of about 60 milliseconds at B1, 100 millonds at it. Um in humans it's a little bit bigger because the head's bigger. Um and then we we talked about various pieces of evidence that recognition is um is a quick thing in some cases. Okay. And so this is this study where people looked at series sequence of images and these are event related potentials in response to images and you can see that they diverge for categories of images in this case animals and non-animals um at a relatively short latency of 150 milliseconds which is you know on the order of the the latency of the early response in it. Okay. And we talked a little bit about um the the differences that you see between different stages of the vententral processing stream uh which much of which we've we've talked about at earlier stages of the class um in to to varying extents. So in V1 you you have relatively simple tuning orientation selectivity and then in simple cells and in complex cells you get some um tolerance to position. question there >> um for the feed forward processing in recognition is that that that works for like these broad categories if you ask for more specificity like rather than recognizing a face getting a person does that require feedback or does it just require a longer chain of feed forward >> so I think that on the whole there's evidence that to a large extent even more subtle distinctions can often be made um with largely feed forward processing. Um I mean there's also lots of evidence now um especially over the last five to 10 years that like hard rec images that are hard to recognize for whatever reason like maybe maybe an object is oluded pretty substantially you know or that are otherwise kind of unusual in some way where the information is degraded that those might require recurrent processing. Okay. But I think um in in some of the things that that um I'll get to in just a moment that we talked a little bit about last time um where you look at whether you can decode object information from brain responses. The brain responses that are used to do the decoding are um are often kind of fairly early responses that you would think would be kind of mostly feed forward in origin. All right. And so um it's not just this distinction. In other words, right? I agree this is a pretty coarse distinction between animals and non-animals. So there's certainly things that are more fine grain that I think you can also account for with a fairly feed forward mode of processing, but it's clearly not the whole story. All right. Yeah, good question. Okay, so um you you've got selectivity for fairly simple features in primary visual cortex. Think of that as early vision. Um you go up to V4, things get a little bit more complicated. Um maybe harder to describe. And then in it, which we we spent most of our time talking about last time, um you see that it's dominated by the central visual field. So most of the projections into it are from the central part of the visual field and how there's like lots of these examples now of pretty complicated neural um tuning um often for things that are kind of behaviorally meaningful, you know, things like hands or faces and so forth. talked about how there's some functional organization within IT um and how there's this other major organizational principle that you see throughout the visual system that is also very prominent in IT which is that the receptive field sizes increase um as you go from the the beginning of the visual system up to the sort of deeper stages. Okay. So even within it you see a gradient of receptive field sizes. Talked about how individual neurons in infratempaloral cortex show some degree of invariance to position and size. um and and uh then we transitioned into to sort of trying to think about h what these properties mean in computational terms, right? So these are all like different sort of clues um to the basis of recognition in invertemporal cortex. Okay, but but how can we actually turn this into sort of working models and a computational understanding? Um and the kind of key thing that we talked about here um was this idea of thinking about images or stimula in general um as being represented by populations of neurons. So you can think of the the the image is being represented um in this highdimensional space where every axis of the space is the response of a neuron. Okay? So every image gets represented as a point in that space. Okay? And if you're dealing with different images of the same thing, you have a set of points, right? that u will form what we often call a manifold. And we proposed um that what a a good recognition system ought to do um is generate representations of images that have this particular property whereby all of the images that correspond to one type of thing would be in one part of the space and all the other images would be in another part of the space. And the way that you kind of quantitatively assess that is by asking whether there is a plane or a hyper plane that can separate the those different sets of points. Um, and if there is such a plane, then you can use that to build a linear classifier that can take the brain response and tell you whether you're dealing with Joe or Sam in this particular case. Okay, so we talked about how linear classifier is based on a projection operation onto a vector which in this case would be perpendicular to that hyper plane. And so the crux of the recognition problem, the reason that recognition is hard is that in the input representation, the classes that you care about, like you know, one kind of thing versus another kind of thing, they're all tangled up, right? So the points that correspond to one person and the points that correspond to another person, they're all sort of intermixed in a complicated way in the pixel space, right? That's kind of why you need a brain. Um um and there's this proposal that we tried to evaluate that what a sensory system might do in this case the vententral visual stream is perform a series of transformations on the input that would end up with a representational space that would have this property. Okay, where all of the representations of one kind of thing are one part of the space and the representations of another um kind of thing are in a different part. And so then we talked about evidence that this is the case. And so um this is the experiment that I was alluding to in my answer to the the question just now. Um so we looked at experiments where u responses from a big set of neurons in infratempaloral cortex were measured to a big set of images. Okay. So here we have all these images and all these um different recording sites which you can think of as neurons or groups of neurons. Okay. And some neurons respond more to some images than others. And the question is whether or not the representation that is implied by that data would be good in the the sense that we've discussed in making object identity explicit which again is evaluated by asking whether the the two different objects are linearly separable which again you evaluate by um fitting a linear classifier and then measuring classification performance. Yeah. >> Also test like different orientations and rotations of the object. So, in this particular um experiment, I'm not sure if they looked at different views. Um this is showing you different sizes and positions. Um that's certainly been done um and I think would have a similar outcome, but I don't remember about this particular study. Okay. Okay. Um and so the outcome of this experiment was that you can classify objects pretty well um using with a linear classifier and using the responses of of neurons in it. And so this graph plots the number of recording sites versus the classification performance. And once you have a couple hundred uh recording sites, you do you do fairly well. Okay. And we saw that this was at least somewhat specific to it. So if you do the same analysis in in infratempaloral cortex and in V4 which we can kind of think of as like the area of the ventral stream that sort of precedes infratempal cortex um classification performance is worse. Again these are measured as as a function of the number of recording sites um and also the consistency of the classification judgments with humans um is a lot lower. And so the conclusion from this is that it neurons mediate object recognition by making object identity explicit where what it means for something to be explicit is that it is easily read out. And what it means for something to be easily read out is that you can build a linear classifier that will enable you to classify the bit of information you're interested in. In this case, object identity. Okay. Okay. So that's where we left off last time. Um, are there any questions about that before we move on? And so I'll note that in um this description of the basis of object recognition, we haven't really talked about like the mechanistic details of how these of what these transformations are at each stage, right? We've talked about the outcome of those transformations, right? in terms of a property of the representation. Okay. Okay. And those transformations they might be complicated and not that easy to describe potentially. Okay. All right. So another kind of important aspect of our recognition machinery um is that there's evidence for specialization of this recognition machinery for particular classes of objects um that are maybe especially distinctive or especially important. Okay. Um so in particular neurons that are that are responsive to certain special classes of objects seem to be segregated in the brain. Um, and we talked about our colleague Nancy Camwasher who's done a lot of of work on this. And so one particular class where there's especially good evidence for this um is faces. So there are lots of patches um of cortex in both monkeys and humans within which the neurons respond a lot more to faces than to other kinds of images. Okay. Um and so Nancy came um discovered one of these diffuser form face area. There's there's a few others in humans and this is the result of an fMRI experiment in um macak uh where they are comparing brain responses to faces to brain responses to other types of objects um and there's a bunch of these patches that kind of emerge. Okay. Okay. So, and then in this particular study by Doris Sao and colleagues um they then used the fMRI results to direct electrodes. Okay. So, they put electrodes um into these face patches and now measured the responses of individual neurons. So, each row here is a neuron that they recorded from in one of these patches to like a big set of images. Okay. So, there's 96 different images and the images are in different categories. faces, bodies, fruits, gadgets, not sure exactly what goes into gadgets, but I guess gadgets. Um, hands, and I think that's scrambled images. Okay. Um, and so what you're supposed to take away from this, and the color here represents the firing rate of the neurons, right? And so, um, the vast majority of the neurons in these face patches are responding a lot to all of the images of faces and much less to all the other kinds of images. Okay. Um so here's another bit of um evidence for the same sort of thing. Um this is a plot of lots and lots of recording sites. So over a thousand um in macakian fortemporal cortex and each recording site is a dot here. So they have like lots of electrode penetrations. Um and then they're color coding the electrode sites as a function of the extent to which they prefer faces versus non-face objects. Okay. Um and so the point is just that like you kind of see these these regions where there's a lot of red dots, right? There's one here and there's one here, maybe one up there. Okay? And so this this these various lines of evidence, they um address this kind of long-standing question in the realm of visual recognition, which is like whether faces are are kind of special in different ways. And there's sort of four pieces of evidence that I think are all kind of consistent with um the notion that there is some degree of specialized machinery for recognizing faces. Um the first is what we've already discussed, which [clears throat] is evidence for a brain region that responds a lot more to faces than to non-faces. Um this is in humans the fusoiform face area um mostly characterized with fMRI. So it's right there. Um here's another view. Um the other another piece of evidence that there's something special about face recognition is comes from inversion effects. Okay. Um and so it turns out that face recognition is much better when the faces are upright than when they are upside down. Okay. Um and this is not necessarily true for um other objects. And the classic piece of evidence for this um is something called the Thatcher illusion. Okay. And so it's called the Thatcher illusion because um the original example of this was performed on the face of Margaret Thatcher because it was it was done by somebody in in Britain and Margaret Thatcher was the prime minister of uh the UK at that time. Okay. Um and so the illusion here and so here it's done on the face of of somebody else. This is just what I happen to have handy. Um, the illusion here results from the fact that the face is distorted. Okay, so there are little um there's there's a few sections of the image of the face that have actually just been locally flipped upside down. Okay, but when you look at it upside down, it just looks fine. Looks like a face, right? Um but what it actually is when and and this is becomes very obvious when it's right side up is um is this, right? So you can kind of tell that this is not how the face is is like supposed to look, right? So that and that are exactly the same um just with with the exception of being being rotated um 180 degrees. Okay. Um and so there's there's like lot and this is not the only piece of evidence. There's lots of other evidence like you you can measure the ability to recognize or discriminate faces um and compare that for upright versus inverted images and there's a huge advantage for when the the faces are upright. Okay. Yeah. >> Do you notice how thisffect? >> So the question is do does this affect facial recognition systems? So are you asking whether like a computer vision face recognition system would like exhibit the same phenomenon? >> Yeah. Does it mold for the illusion or does it say like this is messed up? >> Yeah. So my guess is I actually don't know whether that's true or not and this somebody has probably looked at this. My guess is that they probably would because we sort of think that like this comes from the training data at some level, right? Like the na the naive interpretation of these inversion effects is just that most of the time faces are are upright. Um and so your face recognition system is trained up on all this data. I mean again we don't know whether this is over evolution or learned via development but either way most of the the training data essentially for the recognition system is in one orientation um and so and and not and a lot of objects are not like that right so you know you see chairs from like all kinds of different viewpoints um and um are less sensitive to that um but yeah my guess is that probably machine face recognition systems will show something kind of similar um yeah and so there's and you know this is sort of part of a a long-standing debate about the extent of viewpoint invariance of human recognition. Um so one of the really interesting things about object recognition um is that it's pretty invariant as we've been discussing and one of the most impressive forms of invariance is invariance to viewpoint. Right? So I can look at this cup from this direction or this direction. you know, I'm going to be able to tell that it's a cup, but the the image that is produced by the cup is like totally different in all these different cases, right? Um, and so there was there's like a a long history of people thinking about the basis of that viewpoint invariance. And so like, you know, some people have postulated that you actually infer 3D models of objects and that's the basis of recognition. Others say that like you have all these stored 2D views, but they're kind of linked together. Um and there's been lots of experiments on new kinds of objects like where you expose people to particular views of the objects and not others and then sort of show that they don't generalize perfectly and you know that suggests that viewpoint invariance is not automatic. Um so I think there's various pieces of evidence that um you're only as invariant as your training data kind of forces you to be in a lot of cases and faces maybe kind of an extreme example of that. Okay, but at any rate, like this is this is not something that is as pronounced for most other classes of objects. So, one piece of evidence that face recognition is special. Um, another uh piece of evidence um is proipagnosia. Um, so there are uh these examples of brain of of people with brain damage that end up with selective impairments in recognizing faces and that's known as prozipagnosia. Um, and so the most striking cases of this come from people who've had a stroke. Um, which is the typical cause of brain damage in people. And so here's a a quote from a case study of this person. Dr. P says, "By and large, he recognized nobody, neither his family, nor his colleagues, nor his pupils, nor himself. He recognized the portrait of Einstein because he picked up the characteristic hair and mustache. In the absence of obvious markers, he was utterly lost." Here's another uh report. This is of a per a person describing themselves at the club. I saw someone strange staring at me and asked the steward who it was. You'll laugh at me. I've been looking at myself in a mirror. So people can't ne even recognize themselves. Um and nowadays so these classic um examples kind of come from people that have pretty serious brain damage. Um there's now I think growing evidence that face recognition, probably a lot of recognition abilities, they lie on a continuum. Okay. Okay. And some people are actually really good at recognizing faces. Others are less good. Um and at the extreme end of that that um of the less good end are people who might be considered to be prozzagnosic but just can generally proipagnosic. Okay. And so um and there's a there's thought to be a genetic basis of this. It runs in families and so forth. Um so um but proipagnosia is classically a neurosychological phenomenon. Um and so and the inference of that from this right is like um and this is sort of the power of of all of these examples of brain lesions is that if you um can selectively disrupt a particular ability with other abilities remaining intact, that's evidence um for modularity in the brain that there is some bit of the brain that is specially involved in that function. And if you're unlucky enough to have that particular bit of the brain damaged, you end up with a deficit in that particular thing. In this case, face recognition. So we saw this with acromatopsia. Um and uh here here it is with with face recognition. Um and then we've just saw seen these examples of face cells. Okay. So a variety of pieces of evidence that there's some degree of of specialization of the recognition machinery for faces. All right. Any questions about um face recognition or this particular idea? >> Mhm. >> So the time series here or in the responses it like recognizes it and then stops recognizing it. Um I'm a little confused as to why it's a lot more spread out the bottom one. Like it seems like it's contining to later. >> Oh yeah. >> Yeah. I mean, so these are like responses for individual trials and I mean neurons are sort of stochastic devices, right? So um yeah, I wouldn't over interpret this. I mean, I think if you repeated the experiment, like the exact pattern of the spikes is going to be different every time. >> Yeah. So the main thing to take away from this is like there's a lot of spikes up here, kind of less here, you know, some here, less there. That's sort of the point. Okay. Um so this large scale organization of selectivity for different kinds of categories that sort of shows up in these face patches and also things like the region that's selected for bodies other things. um that is evident in what we call representational dissimilarity. Okay. And so this is um now a pretty well established and widely used method for looking at neural representations and um so it's worth kind of walking through just curious how many people have encountered representational dissimilarity before. Okay, a few in NY's class. Yeah. Um okay so the um the essential idea here um so let me tell you how how you construct this okay so these are what are called representational dissimilarity matrices okay um so each the matrices here consists of sets of points okay and each point represents the dissimilarity of the brain response to two images Okay. And so the dissimilarity um is essentially one minus the correlation between the brain responses to the two images. So what is the brain response? The brain response in this case is measured with fMRI um from a whole bunch of different voxels. Okay. So this is an experiment where you present this large set of images. Looks like there's a couple hundred here. Okay. To every image you measure the fMRI response in this big chunk of visual cortex. So it's a whole bunch of voxels. So for each voxil, you get a response to each image. Okay? So you can think of the response to each of these images as like a big vector. So let's suppose there's like a thousand voxels in the region that you measure. Okay? So you get a thousand numbers for every image. Okay? Which is the response of that image in every voxil. Okay? And so then what you can ask is for a pair of images, how similar overall is the brain response? Okay. And so you can measure that as the correlation. Um, and here this is dissimilarity. So it's just one minus the correlation. Okay. And so blue here means that things are are not at all dissimilar. So they're very similar. Red here means that they're very dissimilar, very different. Okay? And so in this experiment, there were like 200 different images of objects and other things. Okay? Um, and so there's lots and lots of these pairs. Okay. All right. Um, so what what do you what do you get from this? Okay. And then and it has to be symmetric, right? Because the the um the rows and the columns represent the same sets of things. Um and so the and also the diagonal thus has to be zero because you're just comparing the response to one thing to um itself. Um okay so these are this is what you get and this is a particular study that um compared the representational dissimilarity in monkeys and humans. Okay. Um and we'll talk again about why you would want to do that. But let let's just first get into what what is actually shown here. Okay. Um and the other thing I should tell you is that the images that are presented in the experiment are kind of organized here sort of according to like categories of images. So um the first little um bunch that you run into here are images of humans. So these are faces, those are bodies. Um these are images of animate things um that are that are not human. Um again faces. So these would be animal faces and these would be animal bodies. And then down here you have inanimate things. And that's divided up into natural and artificial. So got a banana here and like a gun, you know, it's just all kinds of random images. Okay. All right. Um and so when you look at the um at the matrix here, um there's a few things that kind of pop out. Um the first is that there's there's what we call block diagonal structure. Okay? So if you step back and squint your eyes, like there's kind of a blue square here, right? And kind of a blue square here, and then red here and red here. Okay? And so that's an indication um that the the images of the animate objects. So that's the first half, right? Are kind of producing similar brain responses on average, right? And the inanimate objects are kind of producing similar brain responses on average, but that the animate and the inanimate images tend to produce pretty different brain responses. Okay? So you can also see some really strong block diagonal structure right here. And so what is that? Well, that's faces, right? Um, so you've got human faces here. So that that's really solidly blue. So all of the face images are generating very similar um brain aggregate brain responses. Um there's also and and then if you look down here, there's another kind of blue square. All right? So that's all the animal faces. Okay? So those are also generating very similar brain responses. But then you also get kind of a blue square here. Okay? So what does that what does that tell us? Anybody want to tell me? >> Yeah. So the animal faces and the human faces are also producing pretty similar brain responses. Okay. Um Okay. So this is sort of one way to kind of characterize the brain's representation of objects. Okay. Now one of the things that this was used for and this was really the point of this particular paper um which introduced this method but then what they were really using it for was to try to compare the monkey visual system and the human visual system right and so the monkey and the human visual system have you know there's certain sort of homologies in terms of the areas but really like you know they're they're kind of different right so in in humans like the visual system is kind of scrunched into the back of the brain because like you sort of use your temporal loes for you you know, language and stuff like that. Whereas in the monkey, the visual system kind of extends into the temporal lobe, right? Um, so the actual sort of physical substrate is kind of different. Okay? So it's not really totally obvious like how you would actually make a comparison of the human and the monkey visual system at some sort of really fine grain quantitative level. Okay? But this particular analysis method gives you a way to actually make comparisons across brains even if it's like different species, right? So you can ask um is the the structure of the representational space as measured in this particular way of like the similarity of the responses to different images is that shared between monkeys and humans and what you're supposed to take away from this is that the thing on the left looks a lot like the thing on the right. Okay. So um in both species you kind of see this core scale structure of it like different responses to animate and inanimate objects. faces seem to really stand out as being kind of special. Um, animal and human faces seem to be represented similarly both in monkeys and and humans. Um, [snorts] yeah. Okay. So, the take-home meth me message here is that this is one way to kind of analyze brain representation. It's called representational dissimilarity. It's in pretty widespread use now. Um, it's based on kind of population level responses. So in this case it's like the fMRI response is from a big chunk of brain okay to like a pretty large set of stimula um and we believe that the structure here is sort of mostly driven by the large scale organization of category selectivity. So like like the fact that you know there's this fusoform face area that responds a lot more to faces than to other stuff. And so what that means um is that if you look at the response of the the entire visual system to objects, faces are going to tend to all produce pretty similar responses because there's this one set of voxels that generates a huge response to faces and not much of response to anything else, right? And you see kind of similar stuff for bodies. Um so this large scale category um selectivity that we certainly see at the level of fri um contributes a lot to this kind of thing. Ask me questions. Yeah. >> What's the benefit of using dissimilarity for like a normal correlation? >> I'm not sure that there is much of one. I I think there's some reason for it and I forget what it is. Yeah. I mean, it's it's got the same information in it. Yeah. >> So, this seems like one extra step. >> Yeah. I think the idea Well, yeah, it's sort of supposed to I guess the idea is that it's sort of more like a distance, right? And so people often think of this as like capturing geometry. And so just dealing with like a distance-based measure sort of is appealing for that from that standpoint, right? But I think you could draw all the same inf a lot of the same inferences like without doing that. >> Yeah. >> Can you use this to draw any information on differences between monkeys and humans? Like for example, like the human one looks to be like a lot more well defined in terms of borders and also like like for example there's in the monkey one there's like not there's high dissimilarity between like human face and not human body but you don't really see that in humans like I don't know. Yeah. >> Is there some explanation for these kinds of things? >> Yeah I mean there I I think your observation is correct. There are some subtle differences. I mean exactly what I mean what that means and why it's there you know it's not so totally clear um but yes you can conclude that the representations are not identical you know um and you know without something like this it's not even clear how you how you how would you go about evaluating that right you know so yeah there's like some similarities but they're also not exactly the same >> yeah has this been done for animals other monkeys and like is there any comparison between like the different animals and humans? >> I'm not aware of this having been done with any animal other than monkeys. Um yeah, I mean it really and like yeah I mean the it's interesting that the the animal models that are popular in neuroscience has kind of changed over the years. So you know in the the early days like the the the 60s and 70s like there were lots of experiments done on cats um and some on monkeys and then you but really hardly anybody like does anything with cats anymore. Um and then nowadays really because of the the molecular genetic tools that are available in rodents like there's lots of people who work on kind of mice um and rats um but they their visual system is very different from humans. So um I I don't think anybody has done an experiment like this with them. I would presumably would look totally different, you know, big area for cheese or something, right? So um yeah, >> I'm curious if the similar type of representation of the similarity matrix has been done at um different levels of development. So like if like these associations are more ingrained or they're like learned through time. >> Yeah. So I'm I I I don't believe it has. Um and the reason for this is that doing an experiment like this in um children is really hard. Um because really the number one thing that determines the success of an fMRI experiment is whether the participants hold still and um it's kids are just not as good at staying still. And so it's really hard to get the kind of power that you need to do an experiment where you have like 200 stimula. So there I mean there are people in this building in like Rebecca Sax's lab that are working on doing um infant fMRI to look at questions like this. Um but it's because it's so much harder to kind of get good data like you know typically the experiments will have maybe four kinds of conditions rather than 200. So I mean you could look at a very very coarse um representational similarity matrix. In fact they I think they may have done that actually but um but nothing like this with like 200. Yeah. So yeah, so it's it's a really interesting question, but they do find that um you you kind of see evidence for face selectivity in really young infants. Um and I think also maybe body selectivity. So Heather Kasakowski is a grad student who did a lot of that work. Should check out her papers. Okay. Um so we talked about how about about this idea that we can kind of think of recognition systems as having a computational goal, right? Which is that you take this image and you want to transform the representation of that image into a representation that makes object categories explicit, right? By kind of put putting them in sort of different parts of the representational space, right? And so we talked about evidence that if you look at the representations in infotal cortex, they have that property. Um but we didn't really talk at all about how you would go about kind of achieving that property. Okay. Um and so one kind of interesting development in this field um is the the advent of artificial neural networks that can recognize objects from images pretty well. Okay. Um and this is something that just didn't really um it just wasn't the case even 10 years ago, right? So this is a pretty recent kind of development. Um um and so this has led to like a lot of sort of interesting analogies between in this case the vententral visual stream and a neural network an artificial neural network that recognizes objects. Okay. Um so this is a diagram of one common type of artificial neural network called a convolutional neural network. Um and these are so these are machine systems that consist of cascades of simple operations. The operations that are in these kinds of systems are arguably loosely inspired from things that people initially observed or theorized about in the brain. Um so they perform filtering operations, um thresholding operations, pooling, um normalization. Um, and these are each individually simple operations, but then when you stack them, okay, um, you can get pretty complicated behavior out. Okay. And the critical thing that kind of makes these architectures work and makes them useful, um, is that they're pretty easy to optimize. All right? So people, you know, [snorts] for a long time were not actually able to successfully optimize big systems that looked like this. Um, but nowadays we just know how to do it. Okay? Okay. And so and and the essential way by which this works um is uh is really pretty simple and I'll tell you about it. But um as I said the most common architecture for sensory tasks um in the and the one that's really kind of useful for kind of making comparisons to uh biological sensory systems is what's called a convolutional neural network. So remember we talked about the about the operation of a convolution. Right? So what a convolution is is you've got a a filter and you apply that filter at kind of all the locations in your signal, right? So in in an image that would be all the the X and the Y um locations, right? In a in a sound, this would be all the temporal positions. Okay? Um and so we talked about this idea that in the context of vision, you can think of the convolution operation as representing what a population of neurons that have the same kind of receptive field just at different spatial positions. So like think of like center surround receptive fields in the retina or orientation selective uh receptive fields in V1. What they would essentially do to an image um when they're all acting in concert. Okay. And so one of these neural networks is performing that operation. So each one of these sheets kind of represents the response of a filter at all the different possible locations in an image. Okay? And the different sheets kind of represent different filters. So like this one could be horizontal, this one could be vertical, this one could be diagonal and so forth. Okay? All right. Right. So you have multiple layers of these filters separated by nonlinearities um and pooling operations um and um what makes it what makes them useful what makes them really work um is that you can learn the filter parameters using gradient descent in order to maximize the performance of a task, right? Um and really a very popular task and one that's very widely used is object classification. Okay. Um and so the um these systems are are trained using these big data sets with lots and lots of images where people have gone through and with every image they label the images say that's a cat, that's a dog, that's a chair, you know, that's a building, so on and so forth. So you have this huge set of images. Um you can present the images to to the model. The model will give you an answer for what class it thinks it is. Initially like the weight the filters will be randomly set and so the answers will mostly be wrong. But the gradient tells you how should I change the parameters of the filters in order to have the model make fewer errors. Okay. So you do gradient descent over many many iterations. Um and the model moves into a place where it can perform well at the task. Okay. Um and so this has now kind of become sort of the dominant one of the dominant methods in engineering. Um and object recognition was like a a key in early success story. Okay. So, um, some sense this is the realm of computer vision, but these systems, you know, because they're solving a a problem that we think is also important for biological vision. They're they give you a a useful computational model that you can propose as a model of the of biological vision and you can ask, well, does this model actually do some of the same kinds of things um that humans do? Okay, so this is now sort of a cottage industry of people building these models and like analyzing them and comparing them to human sensory systems in different ways. Um and so one kind of thing that you can do um is measure representational similarity. Okay, so these are representational dissimilarity matrices. Um this is the kind of thing that we just looked at. This is from human it using fMRI. So again you get this block diagonal structure. Um, and this is that same quantity measured in a um, HCNN is hierarchical convolutional neural network that's trained to recognize objects. Um, and you can see that they're not exactly the same, but there are some common features. So again, you see this pretty clear distinction between animate and inanimate objects. You see that faces are kind of special. Um [clears throat] it's interesting that you know in the neural network maybe you don't sort of see this kind of quite as strongly as you do in human IT. So there's some differences too. They're not exactly the same um but there's some similarities. Um so this is kind of one way that you can kind of make these model brain comparisons. Um, another very popular way to compare brains to models is to use the the features in the model um as what's called an encoding model. Okay. Um, and so the way this works is that you try to model each neuron or or each unit of measurement in a brain as a weighted sum um of unit responses in some stage of of your model. Okay, so here's just um a schematic to kind of illustrate this. So imagine you do the kind of experiment that we've been talking about where there's a big set of images um that gets presented to a person and you're measuring their brain responses. Okay, so let's say with with fMRI. Okay, so here's a voxil and this is the color here is supposed to represent the response of the voxil to each image. Okay, so every voxil gives you a vector which is the response to each each image. Okay, so here's our neural network. Um consists of a bunch of stages. Um you can kind of think of each stage of the model as consisting of a bunch of units, right? You could think of that as like a filter at a particular point in the image for instance. And so each one of those units will give you a numerical response to any image that you present it with. So you can take all of the images that you used in your fMRI experiment and you can present those to the model. All right? And you'll get some um big complicated matrix of responses. All right? And so one way to ask whether the model is representing the same information that is represented by the brain is to try to predict the brain responses using the model responses. Okay. Okay. And the the standard way to do this is to take the model responses um and to fit a linear mapping here that's f between the model responses and the brain responses. Okay. So you take um the model responses to some subset of images and you optimize this linear mapping. So it's a weighted combination of the of the units in the model that allow you to predict the responses to some subset usually half of the of the images. And then what you do um is you you take that um that that matrix that linear mapping and you evaluate the predictions on another set of images. Okay? Okay. And so you ask, does the the um predicted response that you're getting from the model features actually match the measured response from that little bit of the brain? Okay. And so you can think of like what the of the linear mapping is sort of it's like kind of helping you align the coordinate systems, right? So it's telling you uh how to kind of rotate and scale the model responses to kind of um get them to match up with the brain. All right. um but that doesn't guarantee that you'll be able to make a good prediction. So if the if the features that are in the model are kind of totally unrelated to the kinds of things the brain is measuring um the predictions might be terrible, right? Sometimes they are. All right. So here's just an example of what um would come out of this. So um these are example IT neural responses to like a big set of images. So every point here is the uh response to one image and these like shaded regions are kind of categories of images. So the black line is the actual measured response um of a particular neuron in in it and then the red line is the predicted response using the neural net the artificial neural networks features. Okay. Um and you can see that there is not like a perfect match but um the prediction here is good in the sense that it's kind of high for the images in this particular uh region which happen to be images of faces and kind of low for everything else. Okay. Okay. So the outcome of this exercise um is typically expressed um as variance explained. Okay. So there's some variance in the the brain response and then you ask whether that variance is matched by the predictions. Okay. And that gives that gives you a number and that's a quantitative metric of the extent to which the model is replicating the brain's representation. All right. So here's one finding that kind of comes out of this exercise. um and that is that particular stages of artificial neural networks tend to be best at predicting particular stages of the visual system. Okay. And so this is one early example of this. So the graph on the left is showing the predictions of V4 responses by different sets of models and different stages of this artificial neural network. And the graph on the right is showing the same thing but for infratemporal cortex. So remember V4 we kind of think of as an earlier stage of the visual system than it. Okay. So the y-axis here is variance explained. So bigger numbers mean better predictions. Okay. zero is like the about the worst that you could do. Um, and the gray bars here are a bunch of kind of like old school models. So, some of the like this is a model of V1 that's kind of based on Gabbor filters. Remember Gabbor's it's our best model or one one model of V1 simple cells. Um, SIFT is like old school computer vision um features. Um, HMAX is a early model from this building from 20 years ago. So on and so forth. Okay. Um, and then the red bars here are the different layers of this artificial neural network that's um been trained to recognize objects. And so what you're supposed to take away from this is two things. One is that the in the best cases the red bars are are pretty far above the gray bars. Okay? So the neural networks are giving us better predictions of the brain responses. And in V4, the best predictions kind of come from the middle layers here, layer two and layer three. Okay, if you look at it, the red bars are still doing the best, but now the red bar that produces the best predictions is actually the deeper layer, layer four. Okay, so it's consistent with this idea that the hierarchy that we believe exists in the visual vententral stream where it consists of this sequence of regions and as you go from one region to the next like things change, right? The receptive fields get bigger, the responses kind of get more complicated in sort of an intuitive sense. Okay. Um object information becomes more explicit, right? Right? So remember we talked about how you can read out object information pretty well from it, less well from V4. Right? So things change. We think that there's this series of transformations going from region to region that culminates in representations that can mediate object recognition. And in this artificial machine system, the artificial neural network that is also optimized to recognize objects, there's some evidence of of an analogous hierarchy, right? In particular, the middle layers are giving you the best predictions of kind of a middle layer of the vententral stream and and the deep layer is giving you the best predictions of of the deep stage of the vententral stream. Questions about that? Yeah. in like current um research being done on like CNN's or artificial networks to model um or like to perform object recognition is like the focus more so on uh editing the actual models to like make them more replicated of the brain or like uh changing the data that we're giving. >> Yeah, it's a great question. Um, so the question is like if we if we want to make these models better, do we change the training data or do we introduce architectural constraints that would make them more brain-like? Um, and I would, you know, I think there's a lot of interest in both. Um, and it's sort of, you know, I think it's an open question as to what the relative importance of those things will be. Um, you know, there's one perspective that really like the task and the diet kind of predominantly determine the the characteristics of the representations. Um there's another perspective that um anatomical constraints really kind of you know matter a lot. Um so I mean the other and the other thing I should say is that you know when we talk about training a neural network right so training often kind of means adapting the weights via gradient descent um but it can also mean optimizing the architecture. So anytime you build one of these models, there's all these choices that you make about like the number of stages and like the operations that are in each stage and the sequence of the operations and like how big the filters are and how many there are and you know there's a million choices that you make, right? Um and those choices can kind of matter. Um and so that sometimes that can actually be the subject of an optimization process. So you search over like a big set of architectures um to find something that kind of works well. Um and uh you know that can that may also be one way to may potentially find um architectures that are computationally promising and maybe that also end up kind of resembling the visual system in some abstract way. Yeah. So these are kind of I think these are open questions that like lots of people are interested in right now. Okay. Okay, so this is one kind of early um piece of evidence that there is um some tendency for artificial networks to replicate what we think of as the hierarchy in the vententral visual stream. Um this is a another more and this is this is from recordings in monkey v4 and it this is another more recent piece of evidence using fMRI responses in humans. Um so this is a comparison between a