[MINI] The Vanishing Gradient

Key Takeaways

one quick announcement before we begin the show New York City New York I'm going to be there next week we're going to have a little informal happy hour on Wednesday July 5th at around 6:00 p.m. we're going to meet at a place called super fine in Dumbo and the address is 126 Front Street 11201 at the moment we're not planning on making a reservation come if you want but if you would please join us on slack in the New York City Channel and let us know if you're planning to attend if the number gets big I want to give them a heads up maybe have some sort of reservation or whatever but either way even if it's a last minute choice I'd love to meet some of you guys once again we'll be at super fine in dumbo New York City Wednesday July 5th we'll get there about 6 p.m. data skeptic is the official podcast of datas skeptic.com bringing you stories interviews and mini episodes on topics in data science machine learning statistics and artificial [Music] intelligence l today's topic is called The Vanishing gradient now that's something that'll require a little bit more introduction and I'm going to lead up to that let's just first by start talking about some basic machine learning for example what if we wanted to build a regression that picked or tried to predict the price of a used car what are some useful features we might want to know about that car how old is it how many miles yep have there been any major maintenance issues who owned it what kind of lifestyle did it have maintenance issues are two parts like right it's that car specifically what has its history been across the board but also like the individual car you're talking about does it have any accidents or stuff like that yeah what we're discussing here is feature engineering and we don't actually do this I imagine if I talk to someone at one of the companies that does that they tell me that there are more powerful signals about car pricing like you know how far the person lives from the dealership and local tax rates all these things play in but when you go to answer that problem you have to think very specifically and bring in a lot of domain knowledge to help get a better the most accurate model possible obviously you could build a model that just looks at mileage and that's it or mileage making model it will be accurate to some degree better than random but to get a really precise model you keep coming up with new features and modeling them so that's sort of your traditional machine learning approach you can look at other problems like trying to do language analysis you know the issue of there there and there this being an audio podcast people can't tell that I said three different words t h e r e how do you know which one to use they are the apostrophe one is a shortened of they are mhm t h e i r is possessive mhm t h e r e is location like over there right so those are the rules and they get applied totally based on context you can't just look at the one word and decide if it's the right choice you have to look at like the whole sentence or at least several words around it so that's a pretty well studied problem in natural language processing going back for decades they were trying to solve this there's a famous paper by brillan banko out of Microsoft research what they looked at was what happens when you use the same old algorithms but you just increase the amount of data and what they found was at least for that specific problem there there and there coming up with a fancier algorithm didn't help much what really made their system more accurate was simply more data so extending it to extremely large corpora is what solved the problem and I'll have a link to that article in the show notes every data scientist should read that that's an important paper what does that tell you about problem solving in general the fact that algorithms didn't help more data helped I'm not sure what you said what that study showed so when they gave it let's say a thousand or 10,000 examples of different usages of those three words the model was I don't know 80% accurate something like that and then if they gave it 100,000 examples it got to 85 and the more examples you gave it the more and more accurate it became until it started to get like at the level of human accuracy in picking the right one once it had enough examples to follow are you making up these numbers that took that long to figure it out I'm making them up but they're like they're within an order of magnitude those are close to the right answers yeah it's like it's millions of documents kind of solves the problem so they had to scan each document well the computer does it doesn't complain that's how a lot of problems get solved especially in the the quote unquote Big Data world the same basic algorithms can be used something as simple as logistic regression maybe or perceptron or something like that but if you give it enough data that gives it a breath of examples where it can learn the actual Ling system but then sitting on top of even that to me is the next step deep learning where you give it tremendous examples no feature engineering and you ask it to learn the individual layers of abstraction you know like as we've talked about in previous episodes a neural network might first learn edges and from edges it can compose shapes and shapes become noses and eyeballs and those things can be composed into faces and the system kind of automatically does that through the magic of back propagation which we've also talked about previously now I wanted to talk about Vanishing gradient today because it's a good followup to our last mini episode on activation functions oh yeah so let's jump into an analogy let's think that you work at some big Factory and you work on this control panel you go into a room and there's you know 10,000 knobs on it and there's a big light that's uh or a scoreboard of some kind telling you how good the configuration of knobs you have is and then you go up and you start turning certain knobs and certain knobs they don't seem to do anything other knobs when you turn them your score gets higher this is uh the basic idea of back propagation right you want to see the derivative that is how what is the rate of change or the gradient of one aspect of the network with respect to its loss what's the gradient the gradient describes how much improvement you get relative to a change in some parameter you control is that like a number or how is that expressed well it's a derivative so yeah It ultimately comes down to being a number think of it like the gradient of a hill right if it's very steep that has a high gradient which would mean you would roll down very quickly if it's very flat it has a low gradient or a zero gradient and then you don't change your elevation by moving around much so if you always want to be hill climbing you know get to the highest point or actually as it tends to be the lowest point we usually minimize the error you would want to move kind of proportional to that gradient if the gradient says Hey small changes in this area change the value a lot then you want to take small steps right that's like stochastic gradient descent we've talked about that before it's a good process for tuning these things but what ends up happening is you have so many layers that are all dependent upon one another the highest order layers the things in a neural network let's say doing facial recognition that might start to recognize oh this is a face or these are two eyeballs next to each other those rely heavily on all these other layers of nose leading up to them that do the lower level detection what's lower level detection that would be like you start out detecting edges once you're able to successfully detect edges you can detect shapes so not the thing you actually W want to detect the dependent exactly right so you don't code those intermediaries you hope they emerge naturally through the training process and usually they do through back propagation however at the very beginning when you know you initialize the network to random weights and it's just kind of a noisy experience the back propagation checks the derivative between what you would like to have out puted and what you actually outputed which essentially happens at the very last step in the network so if those nodes aren't yet well trained so back to our control room example where you have all these knobs and it's your job to tweak them and move them in the right direction now realize that next to your control panel in another room is the next layer it's someone else's job to tweak that layer and see how their knobs change can affect the overall score and what they change affects you of course so you have all these independent layers is all kind of dancing around hoping the other ones start producing useful signals and You' to learn from those useful signals in this sort of constant stochastic state of dynamic change as the last layer which has the most signal cuz it sees the actual loss between what you predicted and what you would like to have seen you can tend to get that into some extreme values maybe depending on the activation function you use like a sigmoid that's the one that goes between zero and one and extremely high and extremely low values tend to get pushed very close together by close together we mean that the gradient is very small so the difference between like oh what if I turn the knob all the way to the right versus almost all the way to the right there won't be much change so in that case when the layer before looks at that you might be in these extreme positions so almost no information can be pushed backwards in the network to measure anything further that back propagation every step you're multiplying weights by weights right that's how you kind of add these things up if these weights are all between zero and one do you know what happens when you multiply two numbers between Z and one what happens it always gets smaller right 1 * 1 is 1 but any two values less than one let's say 0.5 time 0.5 it gets smaller it's 0.25 oh yeah yeah sure so that is basically the essence of the vanishing gradient as you go backwards through the network trying to do back propagation this the possible signal there goes to zero very quickly so then the first layer of nodes might not get much signal but those are actually the most important nodes to begin with because they want to be able to learn these very primitive fundamental features in the network like the edges so this for a while was almost a reason why you could say neural networks won't work because of this Vanishing gradient problem I think it was identified in the 90s this was somebody's PhD thesis as I recall it said that the cause for not being able to learn some of these networks is that the signal is is diminished as it goes backwards through the network so that there's not enough left at the earliest layers to do any real learning and if the earliest layers can't learn they can't push any useful signals forward to the later layers well a couple of breakthroughs happen one of the key ones is introducing new Concepts like long short-term memory that's a topic for another mini episode the activation function railu that we talked about is another good one that is the activation function that sets the value to zero if it's below zero and leaves it alone if it's above zero in that way values can get really really large potentially they're not bounded between zero and one so you don't have that same problem so a lot of architecture and deep learning is all about trying to make your network so it doesn't fall into these traps such as the vanishing gradient where the signal gets lost as you try and learn the system what happens if you give it enough time and enough data ah well that's a good question yeah in theory maybe you could just wait longer and it would work um in fact you know we initialize all of our neural networks to random weight weights so if you are the luckiest person in the universe you might just uh initialize your network into the perfect place or very close to it and your network converge fast enough so yeah one way would be to just roll the dice every time and hope for the best and if you wait an infinite amount of time you'll have a good network but most people do not have an infinite amount of time or a relativistic computer or anything like that so we try and speed up that process and you speed it up by finding little tricks that allow the network to learn something in a practical amount of time so yes more data is still a good idea that generally will help but more time while I can't tell you that isn't the solution I mean I guess I could cuz maybe you could fall into some local Optima where you seem to have a good value but there's a better value elsewhere and you have no incentive to go look for it that could happen but mostly it's because the vanishing gradient means you're never going to get enough signal propagated backwards the earliest layers don't get enough signal you have to do something extra to augment the network so that the earliest layers can learn something and contribute to later points in the network how do people identify where the problem is in this neural network oh that's a really good question that I don't necessarily have the answer to um and I'd love to do another show on this because to me this is really a deep the difference between like a senior and a junior person in machine learning a junior person just says well let's throw the algorithm at it and hope for the best maybe do some feature engineering the senior person in their mind they dream up the way that theoretically the model could work right like how the signals could propagate and how information could learning how learning could happen and then you have to go back and say like okay I have an expectation about what and how this network could learn for example it's very reasonable to say in image detection the first layer should learn Edge detection that's a reasonable expectation not only because we know it happens other people have published on that but it just makes sense right it would be easy for very simple models to do Edge detection so if you're counting on your network to do that and it doesn't work go inspect the earliest the first hidden layer and say do these things look like Edge detectors if they do the problem is elsewhere if they don't now you have to which would be the case in vanish and gradient you have to ask yourself why isn't the first layer learning Edge detection and that's the open question then now you say well how could it and then you go through and and iterate and try and figure out what's preventing it one good reason that prevents it is the vanishing gradient so you say all right well how do we build out a workaround for the vanishing gradient and we'll talk about some of those workarounds in future episodes what uh some people must have discovered was that when they looked at why the network wasn't learning they saw that most of the weights by the time back propagation got to the first hidden layer were at extreme values you know very high zero so the vanishing gradient is when signals get very small and weak it would be unreasonable to expect the earliest layers to learn from them because the later layers aren't passing back enough information mostly because they're not uh relying on the earlier layers enough so if you don't rely on an earlier layer it's hard to give that earlier layer good feedback I feel like there's a deep analogy to corporations there somehow how do they know they have enough data anyways each layer oh well the layers work with whatever data they have and they're not really self-aware in any way of like I do or don't have enough data they're just sort of following whatever procceed algorithmic or otherwise that their creator coded them to do so it's up to the the architect of the system to figure out what they expect from each layer and to decide if it's doing its job they just little cogs yeah uh every neuron is just a cog is is very much just a cog and you actually want it that way because if one neuron were too important then it's likely you've overfit or something is peculiar about the problem thanks as always for joining me Linda thank you Kyle and until next time I just want to remind everyone to keep thinking skeptically of and with data data skeptic is a listener supported program to support the show visit datas skeptic.com and click on the membership [Music] tab