1: Introduction to Perception

Key Takeaways

All right. Um, let's talk about perception. So, perception is the task of determining what is out there in the world um, from sensory input. So, as an organism, you kind of need to know what's happening around you, right? And and things in the world, they give off different types of clues to their existence. And organisms have sensory organs that detect these clues, right? And the nature of the clues is different for the different senses. So in vision, photons that originate at some kind of light source reflect off objects and and are absorbed by the eye. And the pattern of photons gives you information about the objects that they reflected off. In hearing, objects cause vibrations in the air. They travel through the air and they're absorbed, measured by the ear. With a sense of touch, we bump into things intentionally or unintentionally and that stimulates receptors in your skin that respond to pressure and other things. With the sense of taste, we lick things and molecules that are contained in the substances that we lick interact with the taste receptors in your tongue. And turns out they also interact with the alactory receptors in your nose, as we'll learn. And with a sense of smell, substances in the world give off molecules that float through the air and interact with the receptors that are in your nose. Okay, so we've got these sensory organs that measure these different types of clues that are coming from the world. Now, the task of perception is to take input from these sensory receptors and then with that input figure out what is out there in the world. So the first important point I want you to take away from this lecture is that perception is deceptively hard. Okay now normally you just open your eyes or you listen and you effortlessly apprehend the world around you. Right? I look out at at at you all and I'm seeing all these chairs and people in them and people looking at me, right? And um I don't really have to try to do it most of the time, right? It just kind of happens, right? Um and this this often kind of makes it like sort of non-obvious like all of the complicated stuff that is going on that actually enables us to derive that kind of information about the world. Um and one way to get some some perspective on this is to actually view the sensory input in a slightly different way. Okay. And so here what we have is an image. Okay. But instead of representing it like a normal image where there will be like pigmentation on on u the the screen at a particular point here we have numbers right so this is a grayscale image and the number here um represents the gray level at a particular point in the image right so you can kind of think of this is like you know if you if you snapped a little picture with your phone um and if it was a black and white camera um this is sort of what the CCD u from your your um phone camera would output All right. It's a it's an array of numbers. Okay. Okay. And so your your task here is to take input like this and then to figure out what's out there in the world, right? And so when you look at it like this, it makes it clear that well, this is actually kind of nonobvious, right? Because normally what happens is when you look at the image, well, your brain is doing its thing, right? So there's all this complicated stuff that happens and that causes you to see. Okay? But it all starts with this. Okay? Now when you you you talk to people on the street and like you tell them that you you study for instance vision um they often think that you must work on the eye. Okay. So here's the eye. So the eye is amazing. The eye is u this incredible device um that evolved to form an image and to and to measure that image. Right? So we've got a lens there. light passes through the cornea and then the lens and it it gets focused onto an image on the back of the eye. It's called the retina. Okay, so this is a close-up schematic view of a little piece of the retina. Okay, so the light rays kind of come through here. So one kind of weird thing about the retina is that actually is in some a certain sense wired up backwards and so the light has to pass through a whole bunch of stuff, but then it eventually gets to the things up at the top. Those are the photo receptors. So the photo receptors are a special type of cell that absorb photons and then take that absorbed photon and and um turn that into a change in voltage. So they take light and they turn it into an electrical signal. Okay. All right. And so this is happening across um the entire retinal image. And so you effectively get a spatial array of voltages, right? So it's really analogous to this array of numbers here. Okay? Um except that it's happening in your eye. So you can think of your eye as something that is measuring light, turning that into a bunch of numbers, okay? And then sending that um through the optic nerve to your brain. Okay? Okay. So that's just the start of it though, right? So um the eyes just measure light. Um they don't interpret it for you and that's really the job of the brain. Okay. Um and so one piece of evidence that this is a pretty complicated thing comes from the fact that really a large fraction of the brain is um devoted to seeing. So roughly 50% depending on how you measure it in humans um more in in in monkeys. There's another pretty big piece that's devoted to hearing. Um so there's very similar issues um with audition. So just listen to this particular sound signal. Okay. So, what did you just hear? >> Yeah. Just shout it out. >> Some background noise. Yeah, there was some background noise. Yeah. What else? >> Question. >> Yeah. Somebody asking a question. Yeah. How many people were talking tonight? Two. >> Two. Three. Okay. Yeah. What kind of um where do you think that was recorded? >> Restaurant. Okay. All right. So, just from listening to that, right? Like you just kind of immediately know all these things that are happening in in the world, right? Um but the sensory input that feel that the sensory input that you received was a pressure waveform, right? So, um there was a sound signal. Well, it started off in my computer and it traveled to the speakers in this room. All right? And there was a diaphragm that wiggled back and forth. Sound wave traveled through the air and it caused pressure variation at your eardrum and that made your eardrum wiggle back and forth in some particular pattern. Okay? So, you can think of this waveform here um as the eardrum displacement displacement as a function of time. Um but really at some level it's just a it's a time series of numbers. Okay? Okay. And so from that time series of numbers, you were able to determine all those things about what was going on in the world to cause that signal when it was recorded. Okay? And that's your brain doing its thing. Um, so the ear, kind of analogous to the eye, is a really remarkable device for transducing the mechanical energy from sound into electrical signals that get sent to the brain. Okay, so this is a schematic of the ear. We got the eardrum here. Then there's an organ called the cookia um that ends up turning the the sound energy into electrical signals. But your brain then so th this is the auditory nerve analogous to the optic nerve. Um and your brain then gets the signals from the auditory nerve and does all this complicated stuff to cause you to hear what is there. All right. The second important point that I want you to come out of this lecture with um is that perceptual problems are usually illposed. Okay. So illposed means that there's not enough information to uniquely determine the answer to the the problem. Okay. And and most perceptual problems actually um are of this nature. So one one classic example derives from the fact that the world is three-dimensional and we usually are pretty good at correctly perceiving its its threedimensional structure. So I can reach and grab this coffee cup, right? And that requires that I know the shape of the cup otherwise I'd knock it over or my hand wouldn't be closed tight enough so forth, right? Um and that's primarily um something that happens visually. Okay. Um, similarly, you know, I know roughly how far away each of you are from me and so forth, right? So, we we're pretty good at perceiving threedimensional structure, but the input to perception particular division is two-dimensional, right? We form an image on the back of our eye. Now, we have two images, one for the left eye, and one for the right eye. And that that actually is part of the the um the solution. Um, but you can close one eye and depth perception is still pretty good, right? All right. All right. So the depth information um that is there in the third dimension is lost in the projection that forms images. And so there's lots of different shapes for instance that have the same threedimens have the same 2D projection onto an image. So each of these is like a different threedimensional shape um but they line up in just the right way um and would all cause the same image. Okay. There's lots of other examples. This this is um another one that's kind of interesting. So this is a bunch of moving dots. Um, and the same dot motion um is consistent um with a whole bunch of different possible objects. Okay, so here we have triangles. Here we've got some squares. This is something else. Uh this is something else. Okay, so it's exactly the same the same motions, right? But they could be grouped differently and either represent sort of fixed points of objects or things that are moving along other things. Right? Um another classic example um is that of auditory scenes. So usually in the world um there will be more than one thing that's making sound at the same time. Um and um the vibrations that are caused by different sound sources. So like my voice and the rustling of your neighbor turning the page of of their notes for instance. Um they sum together at the ear, right? So the signal that you get at your ear is a mixture of the sounds that would have been caused by the individual events on their own. Okay? Um but as an organism typically what you want to hear and what you need to hear are the individual sounds. You need to understand what I'm saying. You know you want to understand whether somebody's turning the page or walking close to you or whatever it may be. Um, and so the problem that you you really have to solve there is is akin um to me giving you this equation and asking you to solve for x, right? And so if I put that on the exam, you're all going to complain because there isn't a unique solution, right? There's lots of different combinations of x and y um that could sum to the same number, right? One equation, two unknowns. Okay? Um but that's exactly the problem that is happening when you have a mixture of sounds and you have to understand one or more of them. Okay? Okay. And so somehow in the in the case of this auditory scene problem, we can usually hear the constituent sounds um reasonably well. Maybe the classic example of this is what's called the cocktail party problem. Um how many people know what movie this is from? >> Graphics of Tiffany's. Yeah, you got it. Yeah, good. It's classic. Audrey Hepern. So, at a cocktail party, you're often trying to talk to somebody, right? So, somebody who you want to understand, okay? And so, maybe it's somebody saying this. >> She argues with her sister, >> right? And next to that is a is a picture. It's a way of turning a sound into a picture. Uh, very similar to what's called a spectrogram. So, we have frequency on the y- axis and time on the x axis. So, there's all this like structure in that speech signal that allows you to understand what the person was trying to say. Okay. Um, but the problem that you you might encounter is that what actually enters your ears might be this. >> There's another person talking there. Okay? And so now that picture is kind of complicated, right? Because it's got these two sound signals that are sort of on top of each other. Okay? Um, but it could also be that there's even more people talking, >> right? Or even seven other people talking. So, by the time you get down to here, it's really a pretty serious mess. Okay? But you could hopefully probably tell that you had a pretty good ability to actually hear out that target voice um throughout these examples. Okay. So, in general, um in many cases, speech remains intelligible despite the presence of other speakers. Um this is a problem that humans still solve um substantially better than machines. Um, so present day speech recognition algorithms like in your iPhone, they work pretty great now. Um, if you're in a quiet room, but in a situation where there's a lot of other u people talking, um, they'll they'll typically still perform poorly. Okay, so um, that's another example of an ill-posed problem. Okay. So, one of the amazing things about perception is that despite the fact we we're constantly confronted with these illprosed problems, which means that there's there's not usually a unique solution, usually we arrive at a single unambiguous interpretation of a of a stimulus, right? So, you just sort of open your eyes and you kind of see what's there and it's usually correct, right? Again, this is what enables you to pick things up and avoid running into things and so forth. Okay? Um there are however um interesting cases um where perception can be ambiguous and those often kind of suddenly give you this insight into oh yeah you know it's it's not always completely determined right so this kind of looks like a cat wait actually no hang on that's a that's a crow right I don't know which it is could be either um this one kind of makes your brain hurt um so is the which way is this person facing I'm gonna say they're facing up. >> Could be. Yeah, could be. But I kind of feel like the person's looking that way. Wait, actually, no, they're looking that way. Okay. Yeah. And if you look at this for a while, it may it may change. Um, what is this? Is it a dog or is it a person? >> And are are this person's legs shiny? >> How many people think this looks shiny? >> Yeah. >> Okay. But I think what actually happens is this is like suntan lotion, right? Okay. >> Okay. So these are again are are kind of interesting examples in the sense that they um they highlight the fact that these problems are illposed, right? Because there are these two interpretations of the image like things that could actually be happening in the world that are consistent with the image, right? So this person could have put suntan lotion on their legs or their legs could be like wrapped in cellophane, right? Um, this could be a photo of a dog or of a person kind of running into the woods with some kind of backpack on, right? Um, this could be a face looking forward or a face to the side, right? And so on and so forth. Okay. So, there's also these unusual cases where individuals may be um confident in their interpretation, but they'll disagree with with other people. Okay. So, this was a an example that took the internet by storm um several years ago. Now, it's the dress, right? And there were all these fights over whether the dress is actually white and gold or blue and black. Hey, how many people think it's white and gold? How many people think it's blue and black? Okay. Yeah. Disagreements. Okay. Um so this is actually really interesting because um the fact that people disagree about this um indicates or suggests that there are different assumptions that the white gold people are making about the world compared with the black blue. Okay. Um and and these assumptions are made to resolve the ambiguity. And in fact that you know that's kind of like one of the central themes of perception is that the way that we are able to solve these opposed problems is by making assumptions about what the world is like. Right? Okay. And that constrains the solution space um hopefully enough that you can reach a unique solution. Okay. The third important point I want you to walk away from this lecture with is that perceptual systems must be invariant. Okay. And this is because the sensory input that is caused by a single type of thing in the world typically varies enormously. Okay, so these are all images of a car. Okay, car viewed from different distances and different viewpoints and on different backgrounds. Okay, they're all cars, but the actual image that is here and here and here is totally different. So that that that array of numbers if you think of this that image as an array of numbers it's going to be totally different right so the question is like how do you actually build something that can take that array of numbers and tell you that that is a car given that the numbers are changing so much from this instance to this instance to this instance. Okay so that's like one of the key problems of of invariance right and so somehow or another you're you're immediately able to look at these things and tell that each of these things is a car and each of these things is not. Okay. Same problem exists in speech. Um, so I'm going to play you an example of what we call dry speech. So without a whole lot of reverberation. >> They ate the lemon pie. Father forgot the bread. >> Okay. So his this this now is the second thing is going to be speech and reverberation. >> They ate the lemon pie. Father forgot the bread. >> So that's like in a subway station or something, right? Or a really big bathroom. um the sound waveforms there are now very very different. If you actually look at at spectrograms you can see the effect of the reverberation is to kind of smear the structure out in time. Okay, so it's like massive distortion. So totally different sound signals but you can listen to them and tell that that they're saying the same thing. Okay. Now the ill-posed nature of the problems that your perceptual systems have to solve and the difficulty of acquiring the right type of invariance is why perceptual problems present a computational challenge. Okay. Um and so for really many decades um these were computational challenges that were insurmountable. Right? So I taught this class when I was a PhD student which was in the early 2000s. Um, and one of the things that you would always comment on, um, is how amazing human perceptual systems are compared to computer vision systems or speech recognition systems and so forth. Okay. Um, and things have changed a little bit. Um, so in particular, in the last 5 to 10 years, contemporary machine perception systems have become pretty good at certain types of perceptual tasks, specifically classification tasks, right? Um, so object and face recognition now work re pretty well. There's sort of arguments over like do they work as well as humans? Um that those are interesting questions we could talk about. Speech recognition works pretty well. Again, most of us talk to our phones all the time, right? You can dictate emails and texts and things like that. Um um so that's that's remarkable and kind of a gamecher. Um of course that that's been a big deal in the world of engineering. Um it's also been a very interesting development for the study of perception because the resulting systems um now give kind you can treat those as models of perceptual systems um and they exhibit many interesting parallels with um human perception. And so this has given given rise to a new generation of models of perceptual systems. Um and so that's going to be a theme that we will talk about periodically throughout throughout the class. Namely, can we obtain better models of of the brain in particular of the sensory systems in the brain using contemporary uh technology from AI. Okay. Um and so one of the main engines of all of this is artificial neural networks. Um so these are systems that consist of the repeated application of pretty simple operations. All of which were kind of loosely inspired by things that people saw in the brain. Filtering, pooling, normalization. We'll talk more about what each of those things means. Um, and we now have really effective methods to optimize the parameters of systems like this um to cause them to correctly classify input signals. So, just to give you a little uh a little example of um something that was like really unimaginable back when um I was a student, I'm going to show you a comparison of speech recognition by an uh humans and by an artificial neural network. Okay? And so this is an experiment where humans are played short excerpts of speech superimposed on background noise. Um and they just have to say what the words are that the person is saying. Okay. Okay, so the y- axis here plots the proportion of words correct and the x- axis is the signal to noise ratio. So as you move from left to right, the speech becomes louder relative to the background noise. And so you expect that people will get better. Um and indeed they do. But you can see that in this experiment um there were four different types of background noise. Okay, so um the green one is music. Um for instance, the purple one is what's called speech babble. That's like crowd noise. And you can see that some types of noise for humans are like much easier to recognize speech in than other types of noise. So that's just what people do. Okay. And so next to it, um this is the results of of running a neural network model on the exact same experiment. And there's kind of two main things to kind of take away from this. One is that um the model is doing about as well as humans. That's the thing that was like kind of inconceivable um 15 years ago and nowadays it's like pretty common place. But the other thing that's sort of interesting is that the the conditions that are easy for humans are easy for the model and vice versa. Okay. Um so sort of like the phenotype of of speech recognition seems to be shared um across um humans in this model. Okay. Um and so the question is well can we use these things to model sensory systems and that's is something that we will um talk about throughout the class. >> We have a question. >> Yes. You mentioned that 50% of the brain is the relevant. Is that by mass or volume and is that the entire brain or just the neoortex? >> I was actually talking about the cortex. Um that's a very approximate number. Um and it would be mo both mass and volume. Um so it's mostly it's pretty much like the sort of the back half of the brain. Um more or less this is very very crude. um more or less is involved in vision. And again, that it's a little complicated to give you a very precise number there because the question is what does it mean to be involved in vision? Um and there's lots of parts of the brain that respond when you're looking at things um but have other functions as well. Um but yeah, roughly roughly half is what I would say. Thanks for the question. All right. So the the fourth important point that I want you to walk away from um is that perception is unconscious inference. All right. So we talked about how one of the the kind of key things to know about perception is that the problems that are we're solving are illposed. Okay. So that usually means there's not like a unique solution. Okay. So the information and the sensory input does not uniquely specify the structure of the world. And so the consequence of that is that the brain has to make its best guess as to what is out there. Right? And this is inference. So when you see or when you hear we think your brain is kind of choosing the most probable interpretation of the sensory input that you are getting. Okay. Now so this is inference but you're not aware of the inference, right? So it's very diff different from like some kinds of inferences you make consciously like you might reason about a problem to sort of work out what what might likely have happened to explain something right so the inferences that your perceptual systems make they just sort of happen automatically like really without you being aware of them and so that's why we call it unconscious inference so that is a term that is due to Helm Holtz. So Helm Holtz was um a giant of um 19th century science um did lots of stuff um and had made many important contributions to perception um including this this idea that perception is unconscious inference. So let me give you um an example or two of that. Um so I'm going to play you um a bunch of moving dots. Okay? Okay. And so this is a little bit like that thing that I showed you a little bit earlier in the sense that um the moving dots because really all that there all that you can see there are these moving dots. Um there's lots of potential explanations of the motion. Okay. Okay. So you kind of look at this and you can sort of see the dots moving. Um may not really be completely obvious what what caused those dots. Um but now what I'm going to do is show you the same thing um flipped upside down. Okay. Okay. When you see it in this orientation becomes quite obvious that this is a person walking. Okay. And so in fact this stimulus is an example of a very famous type of stimulus known as a point light walker. And so originally the way that they would um generate something like this is by putting these little lights on the joints of a person then putting them in a dark room and then filming them. So now of course we can do this with computers. Um um and so the really remarkable thing is that when the orientation is like correct, which means kind of what you're used to, right? You can take that pattern of motion and then perceive a form, right? Um but when it's not what you're used to, um that's a lot harder to do. Okay, so what does this mean? Well, okay. So, it's an example of an ill-posed problem, right? Because there are lots of potential explanations for this motion. Okay. So, one explanation is that there's a person walking upside down, right? And we verified that because you saw it right side up and you can tell that that could be caused by a person. But another explanation is that these dots are just kind of moving around like maybe they're fireflies or something, right? You know, they're not necessarily related to to a single thing. Okay. Um, and there are these multiple interpretations and then your brain is choosing one of those interpretations based on what it thinks is likely. Okay? And so when there's an interpretation that there's a person that's right side up, that seems to be what you see presumably because usually when people walk, they're not walking on the ceiling, right? They're walking on the ground and you're mostly right side up. And so most of the time you're seeing people walk right side up, right? Okay. Um, so that in in some sense is an unconscious inference. Any questions about that? >> Yeah. >> Is there sort of a nature nurture thing going on in the fact that unconscious versus conscious? Like is any of this um things that we are sort of hardwired versus learned by experience of like watching thousands of people walk? >> Yeah, that's a that's a great question. Um so you know you you will be able to answer ask that question of many of the things that we will talk about in this class. Um and most in most cases we don't really know the answer very clearly. Right? So there are there's bits of evidence um that young infants um can [snorts] experience some of the kinds of phenomena that we'll talk about. Um it's not that easy to actually determine whether like a baby can tell if this is somebody walking because they can't talk, right? and so on and so forth, right? So you have to do really clever experiments to kind of um assess that. Um and yeah, so we often, you know, the these assumptions that your brain brings to bear to to constrain these illos problems, um they could be something that you acquire through evolution that you're born with. It could be something that you acquire over development. It's probably some of both. In most cases, we don't really know. Yeah. um we will talk about there are there are a handful of cases where there is very clear evidence um that you can that you can learn um assumptions. So like in some cases and so part you know part of why this is like sort of hard to study, right? Is like it's not that easy to like have somebody inhabit a world where people walk on the ceiling, right? Um so then just in practice like the interventions that you would need to do to do the experiments are impractical, right? But there are cases where those interventions are less impractical and in some of those cases people have done them. Um and we'll talk about some of those. >> Yeah. Um, so here's another um kind of interesting example of an unconscious inference. Um, so I'm going to play you a sound um that consists of something called a sound texture. This is the the sound of a lot of people clapping, applause. Okay? Then that will be interrupted by noise. Okay? And then there'll be a little bit more of the texture. Okay? Um, and the amazing thing about this particular sound is you're going to listen to this and you will have the sense that that texture continues during the noise. Okay. But in fact, it's it's completely not present. Okay. So, listen to this and see see what you think. Did you hear it continue? Yeah. >> Again. >> So, it's so crazy that you might not believe me. You might say, "Well, hang on. How how do I know that that thing is not actually there?" Um, so this is a variant that will hopefully convince you of this. So in this particular case, it's exactly the same, but there's a little gap here between this texture and the noise. Okay. And so now I think your experience will be very different. Okay. So it wasn't there then. Okay. All right. So what's going on? Well, it it's it seems to be the case um that this what we call illusory continuity, the fact that you hear that sound kind of continue during the noise. One second. Um is an in an unconscious inference about what most likely is happening during the noise. Okay, you had a question. Yeah. >> Why can't you hear the noise in the texture afterwards? >> It's a very good question. Um the presumptive reason is that the noise is actually um more intense. So the sound level or the intensity or the loudness um is higher. Um, and so what we think actually happens in these examples of illusory continuity is that the reason that you actually hear the texture during the noise in this example is that the noise is sufficiently intense that a phenomena called masking would occur if the texture were there. And specifically what that means is that when two sounds play at the same time, okay, if one of them is loud enough relative to the other, you will not be able to detect the presence of the other one. Okay, that's known as masking because that's just a that's just a phenomenon. Okay, and so um the consequence of that is that when the noise is very high in intensity like you actually would the stimulus would not be any different whether or not the texture actually was there. And when I say would not be any different, I mean it would not be any different from the standpoint of the ear. The ear the the stimulation at the ear would be the same due to this phenomenon of masking. Okay. So and that's that the consequence of that is that this is the situation is illposed. Right? So the stimulus that you would get if the texture was there is basically the same as the stimulus that you would get if it wasn't there. Okay? And so now the brain just has to make its best guess as to what is happening. Okay? And presumably with these texture sounds, they're of the sort that they just tend to kind of go on and on and on and on. Um, and so your brain infers that it's a good bet that that sound actually continued. You had a question there. And then there >> Yeah, I was wondering like um does the type of noise like let's say it was like a trumpet instead like we still have the texture like let's say but it's like a different So are you asking whether the effect would would be the same if this was a trumpet? >> No. If the noise like if the noise was not like that. >> Yeah. Okay. Got it. So the thing that seems to be really important for this illusion to happen is that the frequencies in the noise kind of have to overlap with the frequencies in the textures and they have to be kind of loud enough that they could mask the texture. Okay. Um and so in in fact what happens is that if you actually make the noise quiet right if you lower the amplitude of the noise at some point this this stops happening. Okay. And you stop hearing the texture there. Okay. You had a question. No. >> Uh in the back. >> Um if you were to just play the texture then play the noise for a long time would you eventually stop hearing the texture? >> Yeah. In fact so Richard McWalter he was a postto in my lab he did these experiments and so the amount of time that the texture lasts depends on the texture but in some cases it can go on for like three or four seconds um but it see sort of fades out eventually. Yeah. Okay. Oh, yep. >> This question is asking about it, would I hear the same gonna tell me that you will keep hearing? Thinking back to before page 25 when I was listening to the seven people overlay, I didn't hear the initial sentence before realizing that the sentence was saying >> um yeah, good question. So for this phenomena the priming pro I believe the priming has very small effect and the reason I think that is that we've done lots of experiments where we didn't tell people that they were going to hear this thing in the noise and we just sort of we just in fact we don't even tell them that the texture is not there right sometimes the texture is there sometimes it's not and we ask them whether it's there and when it's not there they say it's there essentially right so this the illusion is real in that sense um and it's certainly true that with that cocktail party effect um part of what helps you hear um that that what we call the target voice. So the very first one that I played, part of what helps you hear that is that you you heard it in isolation the first time, right? So that is priming, right? So in that situation, the priming actually does have a fairly substantial effect. Um so that that can definitely be important in a lot of settings. Okay. Um so the fifth important point I want you to leave the lecture with um is that illusions illustrate perceptual mechanisms at work and can help us study them. Right? So illusions are fun, right? We love looking at illusions because they they're they make us realize that the world is not always how it seems and um they're lots of fun. Um but they're also scientific tools, right? Um, and so a lot of what what happens in perception research actually utilizes illusion. That's partly why studying perception is so much fun. Um, and so throughout this class, we're going to um be looking and listening to um and feeling um a lot of perceptual illusions, and you'll be making some yourself. Um, so here, this is an example. Um, this is a classic illusion, one of the one of my favorites. um where shadows um are being used to manipulate your perception of depth um and thus your perception of motion. And so what is going to happen here when I play this um is you will see this ball move across the screen. Okay? It'll go from here to here. Okay? And you're going to see two different versions of this. And the the versions will be differentiated by the trajectory of this dark spot that is supposed to be supposed to look like a shadow. Okay? And so in one case the shadow is going to move in one one direction and the other case it's going to move in another direction and that will cause you to perceive a totally different trajectory from this for this ball even though the physical trajectory along the image is exactly the same. Okay, so check it out. And this I'll loop this a couple times. See, it's on the floor, right? Whoa. All right, let's do it again. On the floor up in the air. Okay. Right. All right. So, what's going on here? So I think I I love this um particular effect because it illustrates a whole bunch of important things. Um so one of the things that it illustrates is the ill-posedness of depth perception. Right? So the image and the trajectory of the ball is exactly the same for those two totally different trajectories in the world. Two two different threedimensional trajectories. They project to the same two-dimensional um image sequence. Okay. Um, however, there is a relationship between the location of a shadow and an object. So, in general, if something is sitting on a surface, the shadow that it casts will be kind of right next to that object, right? Just sort of like the geometry of the way the optics work, right? Um, if something is way off of of a surface, then the shadow will will be more distant in the image. Okay? Um, and so your brain either over evolution or development has kind of learned the relationship between shadows and depth. And you just kind of it just automatically kind of estimates the the threedimensional structure of the world using um the shadow in its in its knowledge of the way that that optics and geometry work. Okay. Okay. Um, and so and and this illusion kind of really, you know, it it demonstrates the role of shadows in depth perception in a way that, you know, you wouldn't really have known otherwise, right? Because you have this stimulus that's otherwise the same and you just manipulate the shadow and it can it changes what you see. Um, another great example of of this is um lightness perception. Um so as organisms we typically want to infer what things in the world are made of and that is partially signaled by their pigmentation. Okay. Now the problem that we face is that the light that reaches your eye from an object. It depends not just on the pigment of things in the world but also on the amount of illumination. Okay. So right now the light level in here is kind of moderate and so this surface is white and I'm getting a certain amount of light that is reflected off of it. That's a function of the fact that this thing is white and that means that it reflects a high proportion of light but also of the fact that the illumination here is modest. So if we walk out into the center of the atrium where the light level is much much higher the number of photons that will be coming off of the surface will go way up. All right. Now the amazing thing about your visual system is that the color of this thing doesn't change by and large. Right? Most of the time the that we have what's called color constencancy or lightness constancy. Okay? And so in order to do that, your visual system has to somehow discount the illumination. Okay? And so illusions um are one way that people have gotten a lot of insight into this. And so my PhD adviser Ted Aden um did a lot of the pioneering work on this. Um and this is one of my favorites. Um so this is an illusion because these ovals and these ovals um are the same shade of gray. Okay. You look skeptical. Okay, so they are. And so this little um test patch can be dragged over here and then dragged up here to verify that they are actually the same. Do people actually want to you want to see that? >> Yeah. You don't believe me? Okay. Well, we were going to have to satisfy your curiosity. Okay. >> All right. So, let's see if I can do this. Okay. So, we got our test patch here. That's the same, right? Okay. Oh, I tell you the truth, right? I only say true things in this class. Okay. Um All right. So, so why do they look different? >> Yeah. Yeah. So that's basically the the the the way that we think about this, right? Um that this contains a lot of evidence that the illumination level down here is very very low, right? Um and there's a lot of evidence that the illumination there is actually very high, right? So it's on the top side of this thing that you can see a little highlight here kind of indicates there's a light source. this is sort of in this kind of lower region of this thing. Um so all this evidence for different illumination levels and so the same physical amount of light in these two different cases um is best explained by there being very different colors of paint on the um on the object in these two settings. Okay. So again it's an illusion. Why is it an illusion? It's an illusion because these things are physically the same but they look really different. Okay. Um, and it it teaches us something about the way your visual system works. And so, you know, you often kind of think of illusions as in some sense your your perceptual system is making a mistake. Okay? Um, but as a perceptual scientist, we typically think that the illusion represents a sensible engineering solution that is in your brain, right? That most of the time is going to cause you to see the world as it is, right? So we think that like you the reason that this works the way that it does is most of the time this is actually helping you to correctly tell that you know this thing is white. Yeah. I'm just interested in knowing kind of like or like seeing your opinion and like how much like of the other information plays a role in like making like causing that illusion because like you're like squiggly lines around it and you're like stuck on the like on the ground and so like how much did that really contribute to like like if it was just a plain oval like would we still have that sense of illusion? >> Yeah. So, like you know, Ted and other people who who have worked on these things have like looked at things like that. And in general, like there's lots and lots of like little cues in these displays that kind of add up to create this like incredible illusion, right? And so, if you take away some of those clues, it kind of it'll get a little bit weaker um and weaker and weaker. In fact, when we start we'll have a lecture on the problem of of lightness perception in a couple months. Um, and we'll see, you know, the original version of this that was discovered 150 years ago is way simpler, right? It's essentially, you know, you take the same gray square and you surround it either with kind of white or something dark. Okay? And there you get a small effect. That's called simultaneous contrast. Okay. Um, and this effect is huge because there's lots and lots of these other cues that have been kind of added to the thing. Okay. And so your brain is just kind of using all the different bits of information it can um when it does this, as far as we can tell. Yeah. And actually going off of her question, um I'm wondering is there because for the top dots they're on a light portion of the wavy lines whereas for the bottom dots they correspond to a dark portion. Does that contribute anything or is it >> uh Yeah. No, that that's one of those cues that will that will make that will have some effect. Yeah. Yeah. So there's sort of these local cues. There's these geometric cues related to shape. um like the fact that you know there's this blurriness here because a lot of shadows tend to be blurry that also helps you know there's a lot of stuff that that is working towards this. Okay. Um here's here's another example in kind of the motion domain which I will will show you real quick. Okay. So here we have um some line segments. Okay. And it looks like there's these two pairs of line segments that are sort of moving more or less independently, right? Okay. So, what I'm going to do is I'm gonna add some other shapes to this display. Um, and it's going to totally change the motion that you see. So, now you look at this and you probably see a diamond that's kind of moving around in a circle, right? People are are people seeing that more or less? Yeah. Okay. And so if I get rid of them, you're going to go back to seeing these line segments. Okay. Okay. So again, it's another example of an illpose problem because it's the same image motion and it can be explained in two different ways. All right. Can be explained by these two pairs of line of line segments that are kind of moving separately or by a single shape. It's kind of moving around um in in a in a circle. Okay. >> And in these two different settings, your brain is preferentially choosing one of those two interpretations presumably based on what it thinks is most likely. So in this particular case, um you've got these surfaces that could be oluding the corner of a diamond. Um and so the the the diamond percept seems pretty reasonable, but without that seems like the most likely explanation is is something different. Okay. Okay. And so again, what do we learn from this particular illusion? Well, this shows us that when interpreting motion, your brain is kind of seems to be taking into account information about occlusion, whether there are what what other surfaces there are and their potential depth relationships to things. Okay. Okay. Um so to give you some firsthand a firsthand taste of um the role of illusions and perception um we have these illusion laboratories. This is a web page that we made um for the version of this a couple years ago. Um and so what you will do is you'll you'll be in these small teams um and you will make your own illusion to try to answer some kind of question about perception. Okay? And so usually what people do is they kind of riff on things that they encounter during the class. And so over the course of the class um you'll encounter lots of these effects. And if you so many of you have asked me questions about how these things work. Well, that's like an experiment that you can do, right? So you can kind of try this out yourself and then you look at it or you listen to it and you ask your friends and then um you learn something about how you see or hear. >> Um >> question. >> Yeah. >> Would a person who has an unknown broken link between sound processing? So someone that could generally hear finterprets what they're hearing, would they still be able to perceive an illusion the same as a typical hearing person? >> I don't know. Um so sorry the question is like somebody whose ears work fine but maybe like the downstream auditory system is not working normally. So yeah that that to my knowledge that has not really been studied very much. So there's um the main thing I mean one of the main things that can cause the downstream auditory system to malfunction um is brain damage that would h typically happen after a stroke um or if you hit your head really hard potentially right um and so the you know there is lots of instances where people report difficulty understanding speech um and it might well be that if you tested people on some of these kinds of examples that would also work differently, but in practice you most of the time you're worried about trying to get them to to communicate better and so like you sort of focus on the speech um deficit. So I'm not aware of that having been studied very much to be honest. Okay. Okay. So um why would you want to study perception? Um well one thing that's kind of cool about it is it's it's very relevant to everyday life. Um, so there's just lots of things that you experience kind of on a daily basis that really sort of relate to to perception. Um, and then as a consequence of that, like everyday experience can give you insights. You know, you'll be walking around and you'll see something and it looks a little bit funny and then you you realize that it's not necessarily um how how it seems. Um, there's also lots of exciting applications. Um, so one kind of important application domain is prosthetics. Um, so we're going to talk about this in a few weeks, but lots of people lose their hearing um, as they as they age in particular. Um, and so hearing aids are a huge industry. They work okay, but not great. We'd like to make them work better. There's also coar implants um for someone who who's um, deaf, retinal implants for someone who's blind. um and and those are really now um you know there's lots of new exciting new directions in in all of those domains because of some of the developments that have been happening in terms of computational models of sensory systems. Um also very relevant to the design of displays. Um right now in particular is kind of an exciting time to be working on perception because there's lots of interesting analoges to AI systems. Um um and the other thing to note is that sensory systems um they they have homologues typically in non-human animals. Okay. So lot there's you know other other things that we do with our brains you know are fairly uniquely human right like most animals don't talk you know you know not clear how much they think right um but most animals see and hear and smell and taste and touch right um so sensory systems have classically been model systems um for neuroscience and so knowing about perception is is important in that sense all right so just to summarize um what we talked about um we talked about how perceptual systems measure energy from different sources in the world. Um, and then try to infer what caused that in the world. Talked about how perception is is deceptively hard. So, you've got these sensory organs that measure energy, but then the rest of your brain is there to really take that information and figure out what caused it in the world. We talked about how perceptual problems are usually illposed. That means there's typically not a unique solution. Um, and as a consequence, perception is unconscious inference, where you are unconsciously trying to choose what we think is the best explanation of the sensory data that you get. We talked about how illusions um illustrate perceptual mechanisms at work. So, they're fun to look at, but they're also powerful tools that we can use um to help understand your brain. Okay. Um, any questions about what I've talked about so far? >> Yeah. Well, we talked about like perceptions of conscious inference, but sometimes like especially with like illusions sometimes when we notice an illusion or see it like we can understand it like consciously as well like in a different way. And is that like is that why it's in parenthesis like [snorts] >> that's not what I meant. Um um I was just sort of um point yeah it's it I could take away the parenthesis and that would be equally appropriate but the point is perception is inference um but it's a particular type of inference that is typically unconscious right and so um sometimes when you're made aware of an illusion you can kind of then think about it and try to understand it. Um they're often actually quite hard to understand and part of that is because the perceptual systems are often fairly what we call encapsulated. you don't really have a tremendous amount of conscious insight into like, you know, why does that look light and why does that look dark? You have to sort of think about it and and stuff, right? Um so, um that's a that's just an interesting characteristic of the of perception that you're yeah, it's often fairly impenetrable. Um but there there are kind of interesting cases of what people will often refer to as top- down effects where maybe what you think um then influences what you see or hear, right? So there there are kind of interesting cases of that. Okay. So the sense of hearing um exists to measure sound and use it to infer what's happening in the ear around you. Right? So sound is produced when objects in the world vibrate. They transmit acoustic energy through a surrounding medium in the form of a wave. The task of the ears is to measure sound and transmit that to the brain as an electrical signal. And the task of the brain is to interpret that signal and then to use it to figure out what is out there in the world. So sound waves are longitudinal. So that means they consist of regions of high and low pressure that move away from the sound source. Here we have a tuning fork um that's been whacked. So it's vibrating. um and there'll be this longitudinal wave that moves um away from it with regions of compression and refraction. All right, so some facts about sound. So one is that it needs a medium to carry it. Um how many people have seen this movie? Yeah, every year it's a couple. Yeah, you should all go watch this this movie. It's one of my favorites. Okay, so this is um Alien is a movie that came out in 1979 and it's um it takes place far in the future. There's a whole bunch of people that are on this spaceship. Um, and they get like an alert beacon from this planet they're passing and they they they go down there and some bad things happen and they end up with like this kind of monster on the spaceship. Okay. Um, and then they have to deal with that and um, lots lots of bad things happen. And so this is um the the poster that was originally used to publicize the movie. And you can see that it's got the slogan in space. No one can hear you scream. Okay. So why is that? >> There's no medium. That's right. Space is a vacuum, right? So there's no sound in space. Okay. So they won't be able to hear you scream. So sound needs a medium to carry it. Okay. Um so sound travels through the medium. Um the speed of sound is proportional to the density of of uh a medium. Um, so sound and air at room temperature moves at 343 meters/s. So in water, because water is denser, the speed is faster, 1500 meters/s. In solid objects, it gets really fast. Okay. Um, so um, the intensity of a sound is the energy that's transmitted per second through a unit area. Um, but the speed is independent of a sound's intensity. So whether you kind of speak quietly or yell really loudly, the sound will travel at the same rate um away from you. Okay. What does change within um with distance is intensity. So we often it's often approximately the case that when something is making sound, the waves that travel away from the source are approximately spherical. Okay. Um and so the consequence of that is that as you move away from the source, right, by conservation of energy, right, the energy that's in the sphere kind of remains the same. But the sphere gets bigger and so the energy per unit area drops as you move away from the source. Um and it drops with the square of the distance. Okay? Because the area of that sphere is proportional to the square of the distance. Okay? And so that's known as the inverse [clears throat] square law. Okay? So there are these very predictable relationships between how close you are to something um and how loud it is. All right. So last thing we're going to talk about today um is the decibel scale. Okay. So sound sound level specifically is measured in decb. All right. So one of the remarkable things about the auditory system is that uh it can deal with a huge range of sound intensities. Okay. So typically instead of talking about intensity directly we use a logarithmic scale of the ratio between two sound intensities. Okay. So a bell is defined as a ratio of 10 to one. Okay, so if you want to know the number of bells that differentiate two sounds, one with having pressure P1 and the other having pressure P 0, you take the log to the base 10 of their ratio. So it turns out um that bells are impractically large, right? So a lot of the differences that actually matter in sound are are smaller than that. And so we use the decibel scale instead. So a decibel is a tenth of a bell. So the number of decibb by which two sounds are different with pressures P1 and P2 is 10 to the log 10 of that ratio. All right. So um if you increase the sound level by 10 dB and I should this this stands for power not pressure. I misspoke. Okay. So this these are power measurements. So if you increase the sound level by 10 dB 10 dB that means a 10-fold increase in power because log to the base 10 of 10 is one. Okay. 20 dB is a 100fold increase in power. Okay. So typically um we will describe particular things um in terms of their sound level. So I might say well I went to a rock concert last night and it was 120 dB. Okay. Um whenever we say that we are implicitly measuring things with reference to an agreed upon reference sound level. Okay. here it's called P 0. Okay, so there is a Bureau of Standards that's responsible for all these kinds of references. Okay, and if you go to the Bureau of Standards and you look up this this reference, you'll find there's this particular number that's been chosen as the reference level for what are called sound pressure level measurements. Okay? And that particular value P 0 is chosen to be close to the minimum detectable sound level for humans. Okay? So 0 dB is defined to be that reference sound level. So if you have a sound that is at zero dB, that means that that ratio in there is one, right? Because log of one is zero. Okay? And so that means that that sound is is at that power level. Okay? And so that's intended to be the the the quietest thing that you can hear. All right? Of course, this is all sort of approximate, right? because everybody's hearing is a little bit different, but roughly speaking, something that is zero dB SPLPL is supposed to be just the the quietest thing that you could detect under perfect conditions if you were in a really quiet room. Okay? Okay. And so when we measure sound levels in this way with respect to that standard reference, so like if you have a sound meter like we have in our lab, um we will say that the sound is in this case 120 dB SPL. Okay? Okay. So when you say dB SPL it means that you're using this special reference that is close to the the threshold of human hearing. Okay. All right. So here are some example SPL values. So as we said zero is sort of designed to be close to the threshold of hearing. Um normal breathing u would be around 10 dB. A very very soft whist would be 30 dB. Quiet conversation 50 dB. busy traffic on Massachusetts Avenue for instance might be around 70 dB. Somebody shouts right next to you be close to maybe 90 dB. Um when once you get into the hundreds of dB you're getting sound levels where prolonged exposure can cause hearing loss where you should really be wearing um earplugs. 120 dB is a propeller plane at takeoff and 140 dB um is roughly the sound the the sound level of a jet at takeoff. And supposedly that's a threshold of pain. I never experienced that myself and I don't intend to. Um um okay. So this is the last thing that we're going to um we're going to talk about and then I'm going to end. Um so another reason um to use the decibel scale, right? And sort of the main one that that it's very very common in perceptual science is that human discrimination thresholds so that means the smallest change in intensity that you can detect. Human discrimination thresholds are roughly constant when you measure them in decb and usually they're on the order of one decel in optimal conditions. So what does that mean? That means that if we have a pretty quiet sound, so like say 40 or 50 dB, right? You'll just be able So if it's 40 dB, you'll just be able to detect the a change in intensity if I move it up to 41. Okay? Similarly, if I have a pretty loud sound, 90 dB, you'll just be able to detect a change in intensity if I move it up to 91 dB. Okay? And so, the decibel scale ends up being pretty convenient um because the changes that you can detect are pretty constant um as a for as a function of sound intensity. Okay. All right. So, I'm going to quickly play you some demos that illustrate the decibel scale. Um here you go. The decibel scale. Broadband noise is reduced in 10 steps of six dibels. Demonstrations are repeated once. Oops. Okay. Sorry. So, what I wanted to pause this is to say, so this is going to be six decel steps, right? So, I told you that your threshold is around 1 dB. So, these six decibel steps will be very very obvious. Okay. Let's let's do it. The decibel is reduced in 10 steps of six dibels. Demonstrations are repeated once. [snorts] [snorts] Okay. Now in this example, the next one I I think this is in one decel steps. Okay. So these steps should just barely be discriminable. And a lot of times you might not be 100% positive if the level is changing, right? But you'll be able to then hear it over a couple steps. Broadband noise is reduced in 20 steps of one decel. [snorts] Okay. So that's very close to your discrim. >> [snorts] [snorts] >> All right. So, those steps were very close to your discrimination threshold. And you can probably just barely tell that that was changing. Okay. So, that's the decibel scale. Um, that's how we measure sound intensities. It's always based on ratios. There's got to be a reference. Um, and we use it because it's pretty closely related to discrimination thresholds. We're going to end there. Um, welcome to 9:35. We're happy to have you here. I'm going to have office hours upstairs. You're welcome to come by. No obligation. Um, and we will see you Tuesday.