Understanding Deep Neural Networks with Dr. James McCaffery - #13

Key Takeaways

[Music] hello and welcome to another episode of twiml talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam charington if you listened to last week's show you know that today Friday May 3rd is the last day to register for our O'Reilly strata Hadoop World Conference giveaway you've got until midnight Pacific time to register and you can do that via our new Facebook page whether or not you're interested in attending strata Hadoop we'd really appreciate you taking a moment to like our Facebook page as well as subscribe to our new YouTube channel we'll link to both of these in the show notes last week I mentioned that I'm working on an event called the future of data Summit and I'm excited to share some of the details of that event with you now the event is part of a larger IT industry conference called interop ITX and I've worked with a team at ubm that organizes the event for several years now at last year's conference I presented a workshop called the it leaders guide to machine learning and based on the strong response to that session they asked me to work with them to do something bigger this time around the result is a two-day future of data Summit that will bring together noted experts and practitioners to discuss the future of Enterprise data from a variety of Technology perspectives we'll be exploring the Innovation and opportunity being offered in areas such as of course machine learning in Ai and cognitive services but also iot and Edge Computing augmented and virtual reality blockchain algorithmic it operations data security and privacy and more I've handpicked the speakers to both Inspire Summit attendees with a view into what's possible as well as to provide practical insights into how to get there to give you a taste of what I've got planned here are just three of the 16 great speakers on our agenda for the summit well first off you remember Josh bloom a former guest on the podcast whose startup wise. was recently acquired by GE well Josh will be joining us to speak about building AI products from idea to production Intel's Assaf Iraqi will give us a view into the next five plus years of compute storage and network innovation in his talk titled how the future of Hardware enables the future of data and Diana Kelly Global Executive Security adviser at IBM will be discussing the future threat landscape and how to protect cloud iot and Big Data Systems I've got more information about the event as well as a preliminary agenda posted at twiml AI ./ futureof dat on that page you'll also find details for registering for the conference and a code offering a special discount for TW listeners to give you a bit of a sample of the type of content you'll get at the event Our Guest on the show today is James mcaffrey who's a research engineer at Microsoft research James will be speaking at the summit on understanding deep neural networks and that's the focus of our conversation on the podcast as well we had a good time with this conversation and even if you know your way around a DNN I think you'll pick up some interesting tidbits enjoy the show and check out the event page at twiml ai.com future of data or the show notes page at twiml ai.com talk3 for more information on James or the summit and now on to the show hey everyone I am here with James mcaffrey James is with Microsoft research and we've we've got an exciting show for you uh this time and we're going to be spending some time digging into deep neural Nets uh James why don't you introduce yourself Hi Sam thanks uh thanks for having me today um my name is James mcaffrey I work at Microsoft research uh before working at research in the research uh division of Microsoft I worked in the product groups uh so I have some experience sort of on the pragmatic side of things and before joining Microsoft I was a university Professor uh in mathematics and computer science so um uh I I made that transition from Academia to Industry at Microsoft my area of expertise is uh machine learning and in particular um neural networks I'm uh one of one of the things I uh do here at Microsoft research or my role is somewhat of a hybrid uh at Microsoft research we have I'm going to guess maybe in the neighborhood of um 300 uh serious researchers worldclass guys that are uh have very uh specific domain knowledge and uh my my uh role is uh because I have enough uh mathematical knowledge to understand these guys uh and also um my software engineering uh background I act as a interface between the engineering groups here at Microsoft and the research groups and I do of course some research on my own interesting interesting so the way we got connected was in fact uh you're going to be speaking at an event that I'm organizing uh as part of the interop ITX conference uh in May and that event is called the future of data and uh you're going to be speaking there about understanding deep neural Nets uh that's a topic that you've been spending quite a bit of time on of late isn't it well yeah sort of interesting yes and no um I'd say I've been uh looking at neural networks for many many many years but we're sort of in this area of uh the third wave of artificial intelligence and in particular deep neural networks and uh there's no clear consensus on exactly why uh deep neural networks which is a sort of I'd call it a subset of artificial intelligence or a tool uh that enables artificial intell why they're making this giant comeback again um maybe some of your listeners can remember back in the 80s the uh first wave of neural networks that uh held great promise at least theoretically but they tended to overpromise and underd deliver and then for a long time artificial intelligence the phrase wasn't really used because it had sort of gotten a stigma attached to it but then um here here here's the analogy I always use for people I I think many of your listeners might remember some of the uh speech recognition software that was popular in the 990s uh uh dragon uh was very well known in fact they still really are well known too uhuh but then all of a sudden about two two years ago two and a half years ago uh seemingly out of nowhere uh we had Siri and Cortana and Alexa um from uh Apple uh Microsoft and Amazon of course uh and the speech recognition just seemed to take this gigantic Quantum jump in Improvement and I'm I'm fortunate enough to be working directly with those guys that created that Quantum jump in fact it really was and it was all due to deep neural networks and now uh you mentioned speech recognition that's that's clearly one of the big application areas um um there was some news recently I think within the past three months or so about a group that um I guess just hit a new uh kind of passed a new bar in terms of speech recognition accuracy I think it was 95 or um High 90s per uh was that at Microsoft yeah it's interesting because there's uh uh several different benchmarks and right now there's tremendous amounts of excitement uh I'm sort of beating around the bushes and my my bottom line answer in a second will be I'm not sure it could well have been um but there there's um literally breakthroughs I was sitting in a talk just a few weeks ago where there was sort of like an arms race on some of these uh Benchmark problems and therefore um uh speech recognition uh text recognition uh basically any kind of you know input type things and literally uh one research Group after another is improving and jumping over the others on a uh week by week basis I uh it reminds me the uh the early days of jet aircraft in the 1950s when there seemed to be a new speed record set every few months or few weeks or months well we're sort of in that that same area of frantic activity um that doesn't sound quite it's not so much frantic activity but uh significant advances are happening uh weekly now in in uh several areas of uh Ai and most of them are directly related to deep neural networks so maybe let's take a step back I'm curious how do you define and describe deep neuron Nets to people this is very interesting um I have a way I I I get asked this question so much so excuse me even among my colleagues who have phds in all different kinds of fields and my uh peer uh Engineers who are some of the best engineers in the world um it's very very difficult that to explain what deep neural networks are without a picture of some sort I I found that the only way I can uh uh describe completely uh to the satisfaction of anyone is to use a diagram um and that's what I uh do in many of my talks including the one I'll be doing for you so using vocabulary it's it's not but the main difference is this or I I'll try to uh uh Express as best I can whenever I try to explain what a deep neural network is I start and say and it kind of makes sense you have to have an absolute solid understanding of what a so-called regular neural network is and because the distinction until recently when you said neural network you meant what is now called a single hidden layer neural network they're the simplest forms of neural network and deep neural networks can actually have several different meanings at the basic level a deep neural network is simply I mean it it's really simple it's just a more complicated basic neural network uh multiple hidden layers if I can interrupt you and go back to the single hidden uh layer neural network we're talking about a neural network that uh will have an an input layer and then this hidden layer and an output layer and basically each of those layers has a set of Weights assigned to them and using some math and algorithms back propagation for example uh you're able to based on throwing a bunch of training data at these neural networks come up with a quote unquote optimal set of Weights uh which really is what defines the neural network is that like is that uh a good way to describe what this single that's absolutely correct um put another way um in the end a neural network is just a very complex mathematical equation that can be used to make predictions the number of inputs is determined by your data suppose you're trying to predict um the political party affiliation of a person and that could be Democratic Republican or other um so that's the what you're trying to predict and your uh uh features which means the uh variables you use to make the prediction uh suppose that could be a four things the person's age their annual income their level of education and some other uh metrics so far Therefore your neural network would have four input nodes and it would have three output nodes but the the hidden layer processing nodes uh this hidden layer is where the processing most of the processing is done the number of those nodes has to be determined by trial and error but in a regular neural network there is one such layer so it might be maybe 10 hidden nodes but with a deep neural network you just add multiple layers you might have three hidden layers of 10 processing nodes uh 20 processing nodes and then 10 processing nodes and in fact neural networks have been getting much much deeper than three layers of late is that right quite right the um until relatively recently there there's been a com two two things have been occurring that have led to these dramatic increases uh across multiple areas of artificial intelligence one one of is that we're just getting more raw horsepower to process these things um it turns out that they get exponentially more complex and so it turns out that we're you know we're just getting more and more uh processing power but the second thing is uh combined at the same time is that we're getting very uh clever with architecture and that is combining these different hidden layers in very clever ways uh instead of doing it naively this the analogy in this case reminds me of the advances in computer chess programs where computer chess programs all of a sudden got very very good better than any human being uh uh somewhat unexpectedly and it was it was not due merely to more uh processing power and it wasn't due uh simply to uh better algorithms it was a combination of the two so we're getting quickly to an area that uh I find really interesting uh and that is the architecture of deep neuron Nets um I I have I have a ton of of questions about this so um I'm I'm excited that we get a chance to chat about it uh I know I I guess as a as a as a preference to this um I know that Microsoft's research has been you know one of many research organizations that's been kind of pushing the the frontier here uh and in fact in 2015 they authored a paper on what's called Deep residual learning uh that won the the image net competition that year um and so uh you know I guess what I want to talk about is like what is uh deep neuronet architecture and you know what is deep residual learning and what are convolutional layers like you know so take us from this description of a deep neuronet and layers through the how those the architecture of those networks has evolved and you know what are what are how do we think about all that right now okay I'll I'll do my best to describe these again when I do describe these um I almost always have to use a diagram because when we're talking about architecture it's sort of going to be a bunch of nodes and how they're connected so let me sort of uh uh talk about all of these things that you've mentioned are are closely related they're they're somewhat cousins to each other um let's let's take the um uh the the residual neural network that you just described now this is more a a an exotic uh variety and in my mind at least uh the the residual neural network is very very close to a close cousin to uh a type of neural network called a recurrent neural network uh they're usually abbreviated rnns right now what what makes a recurrent neural network special and by the way there's uh a ton of research activity on all of these things that we're talking about now but a a standard neural network does not maintain State you feed it some inputs and it produces some outputs then the next set of inputs come along the the neural network is essentially wiped clean it doesn't maintain state from that previous set of inputs and outputs a recurrent neural network has memory an internal memory so to speak and that manifests itself with just some extra nodes um if you can imagine uh a a uh regular neural network with a hidden layer of nodes say 10 nodes in there there's going to be an a recurrent neural network has a second group of 10 nodes that maintain the memory of the previous input this allows this just intuitively you can tell that this makes the neural network much more powerful and smart because it has uh in an English word it it has context and this means for instance suppose you're trying to predict uh you're coming along and um your input are words in a sentence and you're trying to predict what the next word might be uh this is something that you might see on like a uh uh a Smartphone when you know you're typing a message and it tries to predict what your next word might be although there's some pretty pretty rudimentary right now now if you just used a regular neural network to do that um e you know input is separate and you wouldn't um have any context but a recurrent neural network would have a a shadow of the memory of the previous inputs and it would be able to make a better guess at what the next word is because the next word in the sentence is clearly going to depend on what the first words were so these uh recurrent neural um sometimes they're categorized into short recur shortterm recurrent neural networks where they only have a limited ability to remember quite recent inputs or they can be the one of the most exciting areas of research right now is these long-term recurrent neural networks and they just maintain more memory and these things have the potential to be super powerful and to get to sort of close the circle here um in my mind I view uh the Microsoft residual uh neural networks as one of these long-term recurrent neural networks with some special archit Ure features thrown in sort of custom so you mentioned long uh long-term memory and short-term memory uh and in fact on this show I've talked quite a bit about applications using uh lstm RNN so which are long short-term memory how does that relate to longterm and shortterm they um the uh to tell you the truth the vocabulary is not very standard ardized okay and um they they all from a um so maybe these long short-term memories are what you're referring to as longterm and yes that's okay thank you okay you precisely said what I was trying to say got it got it okay so yeah I I have been we have been hearing tons about different applications of these lstm networks uh you know often relating to the example that you use which is your um predicting uh trying to predict words or or things like that from a sentence um which kind of brings us to uh maybe the difference between predictive networks and generative networks oh okay uh very good this is if I had to pick one area where there's more excitement intellectual excitement in the research Community than any other it's exactly we said these generative neural networks um they're called G one of the most popular forms of these is called G and a generative adversarial Network where it it's really conceptually a little bit difficult to grasp in here's how I think about it um a generative neural network does just what you might expect from its description is it doesn't doesn't try to make a prediction based on input it more or less tries to create new inputs in some sense which is uh a little bit hard to to grasp now I got I'll be the first to say that I don't fully understand these things like everybody else um they've only been around uh really the the biggest name is a guy named Ian Goodfellow MH who is the best known name in this area and these things have only really been around for a matter of months now uh by that I mean maybe a year and a half to two two years or so uh so a lot of us are still trying to figure it out the the classic example at least that I used on my uh blog post is that you can train a neur you can feed a neural network a bunch of Van go paintings and then that uh generative neural network will be able to generate and create paintings based on the style of Van go and in short what it's doing is it's sort of separating out it's it's learning to separate style from content well this is all you know very difficult for me to get my uh head around and I'll say that people who are much much smarter than me uh figure that this is something that could lead to tremendous breakthroughs in the future mhm uh and for folks that want to dig into that uh that last uh use case I believe the paper is called neural artistic style transfer or at the very least if you Google that or being that uh you'll be able to find uh lots of information about about that application right you're exactly right uh but the yeah so so there's generative networks and Gans in fact I just had an opportunity to hear Ian Goodfellow talk about this last week uh I was at a an event uh deep learning Summit uh rework deep learning Summit in San Francisco um and the basic idea there as I understand it is you've got um as you mentioned the network that is kind of trained to uh produce or approximate inputs uh and then you feed those feed the stuff that it spits out to another Network that is uh I think called a discriminator Network that's trained to basically measure how close those inputs are to uh the real life thing the thing that you're trying to approximate and then you basically have a feedback loop between these two that that's correct there they're called adversarial because really under the covers there's two neural networks going on where one is trying to generate um information uh and fake out the other neural networks so they're adversarial they're working against each other and uh that that this relates more to the uh architecture the engineering architecture but as you said the the real goal is to generate information and the idea being there that if you if a neural network is smart enough to generate uh information then it's also smart enough to uh understand and discriminate information MH uh uh so we talked about uh rnns what about convolutional neuron Nets what how are those different from uh rnns and other types of deep neuron Nets uh it's funny that um we're talking about convolutional nwks you know they're usually abbreviated cnns that now they seem like they're um just sort of old news but in fact they're quite new still uh the the main problem with deep neural networks as I described a basic deep neural network which is a a simple architecture but with just lots of nodes in multiple layers the problem there is the training um the number of weights and biases that you have to compute or you know using your optimization algorithm just becomes intractable um in a simple neural network suppose you have um uh five inputs uh six okay five inputs six hidden and three outputs the number of Weights you have to do is 5 time 6 + 6 * 3 + 6 + 3 um well as you expand the number of nodes uh these i i in English you'd say it increases exponentially that's not mathematically correct let's just say it gets really big really fast it gets intractable does it drive you crazy when people do say that and it's not actually exponential it depends on how much I've been drinking no just um uh you know I I try to um see because when I um here here at Microsoft I I speak to different audiences I'll speak to Business Leaders I'll speak to engineers and I'll speak to mathematicians and when you're speaking to anybody but the mathematicians if you try to phrase yourself too carefully and be correct you you mess up your argument but when I uh when I say for instance that the output of a neural network a classifier are probabilities oh my math colleagues will go nuts and go no they're not no they're not I go okay yeah I know they're not but but anyway back to uh convolutional uh neural networks because a uh straight forward approach just isn't tractable computationally the idea and uh let see This was um uh oh I I always have trouble remember Yan leun was is the big name here he created an architecture where the really the the main idea of this architecture was to make these things tractable uh cnns are used almost exclusively for IM processing um imagine a uh uh this is an area that I'm not too familiar with I mean I'm familiar with the math of it all but imagine you have an image or a set of images and you want to classify them um the classic example is called the mnist database where there's a a data set of ,000 handwritten digit characters that were called from um IRS tax returns um and and digitized um and so suppose you want to classify you know what what is this is it a digit one is it a digit two or so forth well even a very small image is going to have thousands of pixels and each pixel is going to be one input now if you get go up to like a seriously large picture or even something that a uh a smartphone can take you've got millions of inputs and millions of inputs you just can't deal with that in a a a basic way so the the the brilliancy of convolutional neural networks is to simplify it still uses the same basic ideas of neural networks but it uses them in very clever ways by slicing and dicing the image up and sharing weights instead of having to calculate a million time a million which is whatever that is weights you can break it up and there's a part of the uh Secret SAU is shared weights where weights in a particular area of inputs meaning a particular area the image are shared and there's a lot more to it than that convolutional not works were really a uh a remarkable achievement of architectural design and they're now considered more or less standard many of the tools um that you can find in particular uh Google's tool whose name I can never remember cuz I I uh I don't use use it it runs strictly on Linux do do you know which one I'm talking about uh Google's tensorflow tensor flow thank you thank you yes anyway um so Google's tensorflow can do CNN and Microsoft has a recently released uh basically uh same idea called cntk not one of not not a real it doesn't slide off the tongue really easily there but these things are are now well known but I always like to point out that it took a lot of researchers a lot of years in fact the CNN version that's in common use now is called you know CNN version five or something like that which means there were you know many major iterations in tons of work that went on so in short to summarize you know these cnns to the best of my knowledge are used almost exclusively for image processing but they are the stay the art um however they have some really interesting s that uh a lot of there's a lot of thought about some of the limitations of CNN can you speak a bit to those yeah sure there was um uh a very uh interest a fascinating paper that came out of Google research um what was it called it was called the um intriguing properties of neural networks something like this um and the key takeaway is and I like to use this example um which I don't think was in the paper but other people followed up on it you can suppose you uh train a CNN um to recognize images you can feed it a uh a picture of a school bus and it's clearly a school bus and the CNN will recognize it but by uh cleverly messing up just a few of the pixels the image is completely unchanged to the human eye however this exact same classifier now sees the school bus as an ostrich so it's the bust to ostrich effect well this is very troubling um in a lot of ways uh it raises but by the way you can't just throw you can't just randomly you know mess up the picture you have to do it in a very clever way right but it raises some important issues um one of them is it it leads to the whole question of comprehension does a CNN really understand things if you can hoax it this way it also raises questions of if people are going to and they are using these cnns for uh things which have security implications um or imagine you know Medical Imaging where that it has uh implications for health and safety or law enforcement exactly if these things have this inherent weakness maybe there's something wrong with CNN's it I you know this is all just the speculation that's going on no one really knows but it it leads to some very interesting questions and the the research goes on at just giving uh more interest in research um in particular um some of my colleagues are working on trying to go back to the very very early days uh where instead of just using raw math and raw processing we're going to try to do some symbolic and some sort of a deeper level of understand can you elaborate on that what is that uh what is that mean in this context and how would we apply uh symbolic uh symbolics here yeah because we just hired uh a per the person who's considered the leading guy in this area and he only started here here is uh Paul and I'll spell his last name name s m o l n SK k y Paul smolinsky uh we just hired him out of John's Hopkins University and he's been um what many people including me consider the leading researcher in this area of symbolic reasoning and machine learning uh I got his book and I'm I'm a fairly bright guy I have a PhD but this was a complicated book um he he he's thinking at a different level and he he trying to um I had an interesting chat with him in the uh the hallway the other day uh he he sits right behind me and the analogy goes like this when I was an undergraduate um uh my very first degree was in cognitive psychology um which through various thing you know that led to math and that led to computer but anyway uh when I was in my cognitive psychology days I worked with a u a a brilliant researcher R Duncan loose and he his goal was to create a complete mathematical framework and description of certain areas of psychology the human mind in other words try to map cognition how do people think um you know because still we we still don't know how people think well to to map that Paul um is attempting in some ways to create a meta framework for symbolic reasoning and and logic mhm um this is right now deep neural networks have been remarkably effective in doing what I call um the sort of sensory aspect of artificial intelligence imagine the five senses that we have um Vision uh you know vision and pattern and image recognition they're really good at speech recognition they're really good at um even the robotics manipulation they're really good at but the one thing that they just we're we're not even close right now is the reasoning aspects of it and that's what the symbolic type of process is designed to do or one area you know is it's it's one attack on this so I know that was a little bit vague and fishy excuse me but maybe you can get Paul uh in a future one of your uh podcasts to talk about I'd love to I'd love to hear what he has to say okay awesome yeah that would be great uh so we've got uh CNN's rnn's uh and I I still um I still want to probe around this the the idea of network architecture and uh residual learn what um there was a a blog post uh by a guy named uh Steven merid who's uh at at Salesforce now um he came in Via one of their uh recent acquisitions and uh if I remember correctly he wrote this blog post and the title was something along the lines of network architecture is the new feature engineering meaning uh in traditional machine learning uh you know a big part of the job was trying to figure out how to uh how to massage your data and how to uh uh create you know whether you know natural or uh man-made uh features that Express the underlying properties of your data in a way that your machine learning algorithms can easily train on those and produce accurate results and in this new world um you know defining the network architecture of your deep neural Nets is kind of the moral equivalent if you will it's the it's the new thing that we need to do to kind of massage our data and our solutions to produce accurate results and I'm trying to I'm wondering if you can help us uh wrap our heads around like what that process looks like and you know what are the things that um you know researchers or Engineers are thinking about as their you know they they start with a problem they say I've got this data set and I think uh deep neuronet is the way to uh to solve this problem like how do they then get to oh well the optimal answer is something that I'm going to call a deep residual you know Network that has 150 layers and you know these convolutional layers and every fifth layer is a residual layer and that whole that whole process is that something you can speak to well yeah I'll talk about this because um the sadly the the bottom line is there's no good answer to this that if if sort of the the phrase that everybody's heard a million times is that machine learning and Ai and deep learning and all this is still as much art as it is science and that's has been true and it still is true um very there you know there's some incredibly bright people who work in this field I I'm fortunate enough to work with with some of the greatest Minds I mean you know they're they're world famous and and leaders but when we sit around you know drinking coffee and chatting about this there's so much unknown um even the the the brightest skies in the world are learning daily new stuff and for instance the another uh related thing here is um that another hot area is reinforcement learning um which is you know how and that how does that fit in and you know even among my colleagues were talking about you know knowledges are I mean we're we're knowledge Junkies you know we're just constantly trying to soak this information up but things are happening so fast and there's so much unknown um the the the area you're talking about Network architecture that's one way I mean that would be a good surrogate term for exactly what's going on in all of research now it's almost all related directly or indirectly to the architecture now I'm a pretty s i i you know I believe in Simplicity and for me Network architecture deep architecture is really simple in the one hand where it's just how you combine your processing nodes and not so much input output in different ways and it boils down to think about the human brain there's been some interesting work done by uh of all things DARPA the uh Defense Agency uh in conjunction with IBM where one of the projects they have and and Microsoft has a similar project that I don't think I can talk about now its name it's still under wraps but I can give you a rough idea uh of what we're doing by talking about the IBM and the Department of Defense thing where the idea here is that instead of it's almost too simple instead of using the approach we're using right now which is to get very clever with very specific types of architecture very you know just think of a blueprint instead take the approach that the human brain may have and that is just make your architecture a bunch of a bunch of nodes totally connected in other words like the human brain and then uh instead of using supervised learning where you have to have labeled data you have to have known correct outputs with your you know inputs uh use unsupervised learning and I'm sort of tossing out a uh schmorgus Board of terms here but unsupervised learning is another incredibly hot area of research right now where we realize that methods that require labeled training data which is just the way to say uh data where you tag what the correct output is that can only take you so far it's just not going to scale to the um you know kinds of things that we want to do anyway back to the DARPA IBM thing they're they're creating this thing where their goal is to create a uh a processor in Hardware because you know IBM is known for that that kind of work that is you know scales to biological levels and as far as I can recall from last time I read that an article on they believe that they have successfully created in Hardware um uh a neural network and they're not calling it that that roughly simulates the complexity of the brain of a honeybee and then okay the question here is not okay so how does it learn and you know that wraps around back to symbolic thing so any sorry that was kind of a rambling answer here but I agree that you're right that it's right now if you wanted to summarize all of the or most of the areas including these generative uh adversarial networks the uhu uh long short-term memory uh networks the residual networks it's all about the architecture MH uh so to to maybe further summarize um I guess the way the the way I kind of take what I would take away from what you are saying is you maybe on the one hand um you know to ask you know how do we create new network architectures for a given problem it we're just too early to to right now we're in the stage where the fact that we come up with a new architecture for a problem that is that works and is useful like that's a big deal and and we're going to have to do a lot of that before we can say oh this is the process for creating new architectures for given problems let let me interrupt by saying that uh sorry for interrupting you but you just recalled to my mind um a a very well-known paper um it's actually not even a paper it's a um basically a blog post but it's extremely well known uh in the you know in our field it's called the unreasonable effectiveness of recurrent neural networks um with the idea being that you know there's no obvious connection you know for the person who I I tried to look up I could not find too much history on recurrent neural networks um there was some indication but it's not exactly clear who thought of them first but it's not at all obvious you know you have these recurrent architecture they just work unexpectedly well so you know unreasonably well so the point is none of the stuff is obvious and and we're still in the very very early stages of figuring all this out exactly what you said and then uh I guess along those lines if I am a listener and I'm building I want to I want to create a solution you know does it stand a reason that you what where would you start if you were building something like would you even would you even try to build your own deep neural net or would you use some off-the-shelf implementation would you you know use a service would you if you thought that you like how would you know if you needed to build your own thing um that's an interesting question and there's a a there's I want to say controversy but um clear differences of opinion here um my personal opinion is that whenever I'm going to tackle a problem for instance my um one of the problems that I'm fascinated by and I've worked on for many years is predicting uh American National NFL uh football scores and I like that it's an interesting problem uh because it's concrete it's practical and you can determine your you know how good you are right away and I originally started using sort of standard canned um approaches um I started with sort of regression techniques and then I started using regular sort of neural networks from a tool like weka like tensorflow and things like that and I got up to a certain level of accuracy or goodness and I just couldn't get better no matter what I did I couldn't get better until I threw it all away and created my own neural architecture from scratch where it was custom designed for this problem in much the same way that convolutional NW works are absolutely custom designed for image recognition with um you know the they're optimized because an image has pixel values and the RGB or red green blue values well anyway to cut to the Chaser to reiterate I totally believe that at least now with the tools that we have you get the best results by far by creating your own custom version and I do now the problem here is that I write code every day and these things are not easy right even for extremely Advanced developers and it's very timec consuming and very difficult so there there aren't you I'm not trying to sound boastful but there aren't many people like me who can spin up a um customdesign neural network in you know uh to two days or a week so I think it's going to boil down to eventually boil down to problems one of the things that we talk about a lot at Microsoft um is the democratization of AI or machine learning call machine learning and the analogy here is maybe uh uh you and some of your listeners can remember the days the very early days when spreadsheets Lotus 123 uh were just becoming popular and a lot of people were saying what what why why are companies including Microsoft uh making these spreadsheets why would people why would normal people ever want to use a spreadsheet this is just something for accountants so but then uh Lotus 123 and later Excel and the others uh democratized numeric processing with spreadsheets and then all of a sudden all kinds of interesting good things happened uh from that in much the same way um the goal to democratize um machine learning is the idea that if you give some basic machine learning tools and knowledge to millions of people they're going to find interesting ways to use it and solve problems that we haven't even thought of that said though I still believe that just like you can only do so much with Excel um in numeric processing you'll only ever be able to do so much with a canned program or no matter how powerful the tool is and that there's always going to be need for um uh machine learning Artisans I don't know if I said that word right that go in and create custom models and custom prediction models for particular problems your description made me uh prompted me to ask myself what is the VBA for deep neuron Nets right exactly and [Music] um ah let's let's skip that Rat Hole for a second and um before we leave yeah I want to talk to you about uh applications including the stuff that you do uh around NFL scores uh but before we leave that um there are a couple of areas that I wanted to dig into around uh and these are all things that I I noted that you wrote blog posts around uh one of them is around uh I made this comment about Network architecture being the new feature engineering uh but in fact it sounds like there is some of the old feature engineering that's still important and that uh needs to be done around data encoding and normalization when dealing with uh neural Nets and deep neural Nets and I was wondering if you could speak to that uh and then I wanted to ask you about uh Dropout and cross entropy error as well okay so I didn't quite follow the first part of what you're asking exactly oh you wrote this blog post uh about data encoding and normalization uh and um I didn't dig into that post uh in a lot of detail but I was wondering if um if there are specific techniques in the in those areas related to neural Nets and deep neural Nets uh beyond the kind of things that you do in traditional machine learning uh very very interesting um this is something that I I'll answer indirectly as you as seems like I always do um the the bottom line is that okay neural net Works no matter how you um slice and dice it currently they they only understand numbers they're they're they're number crunchers now very very interesting complex number crunchers so all of your input data eventually has to be or not eventually has to be right away turned into some kind of numeric form so it can be uh understood by the neural network and people who are new to the field this is often one of the most discouraging parts of learning machine learning is that at it seems that there's an endless number of um data transformation techniques and just all this data massaging before you can ever get to the really interesting part and it can get quite depressing new people but I always tell um my audiences when I'm doing training and things like that that fortunately there there's only a discrete number of these things you have to learn for instance there are four real ways I mean four major ways to normalize your data so it's all scaled to roughly in the same range or so now once you know those four and once you understand when they're used and when they're not used have a few examples then you got it but at first you know when you're first trying to learn it it seems like hopeless oh man I've got to worry about data normalization I've got to worry about data encoding in the same way that there is um uh only a few ways uh for data encoding now so the answer is that oh and then and none of those things have changed with deep neural networks got it however I'm always cautious to say because that's sort of the accepted generally accepted truth but I love to you know take whenever I hear something like that in fact I hadn't really thought about it until you you asked this question I always like to go back and look and go you know what is this really true just because everyone says it's true it just sort of like creates like this viral thing here here's an example where I argue all the time with my colleagues it's something very basic suppose you're trying to use a neural network this is going to be a little bit technical but you're suppose you're trying to use a neural network to predict something that can only take one of two values for instance you're trying to predict whether a person is male or female based on a voting behavior based on Age based on income based on all these other things so in other words it's a binary classification problem now the standard and totally accepted by everybody except me technique is to create a neural network that has only a single output note and that single output node is going to be a number between 0 and one where values less than 05 are going to indicate one of the two uh outcomes may say and values greater than 0. five are going to indicate the other female so and that is mathematically efficient as opposed to the alternative of having a neural network that has two output nodes explicitly where they sum to one so you still get the same result in other words it it'd be if uh your your listeners know about a a multiclass classifier you just use the exact same architecture but with two output noes in other words when you're doing classification you will never ever see a two output node neural network classifier because the idea being that if you're trying to predict one of two things just make it a single not well I tell everybody I go you know okay yeah it makes sense but I haven't you know explain that to me you know prove that to me that um one node is exactly equivalent to two nodes and so anyway my point I'm I'm getting fired up because I like I'm I'm passionate about questioning common knowledge so back to your thing so it's common knowledge now that the data encoding and normalization techniques uh that were commonly used in our being used for standard neural networks we don't need anything new for deep neural networks I'm not so sure well before we leave the the specific example are you are you excited about questioning the fact that uh a two- node Network in this example is uh inferior to a one node Network or have you demonstrated that uh there are some cases that a two node network is Superior to a single node Network or for some um external Reasons by external reasons I mean like you know maybe they're the same in terms of accuracy but implementation wise one is better than the other um what's the source of your excitement around this question there's um there's two I say a couple or at least two reasons okay one and primarily I think it's hard to sometimes to be you know self-evaluate I think it's probably psychological on my part where in my world knowledge is power knowing more than someone else is considered you know our Mark of success you take I I work with a lot of guys who who work in in some form of sales and for them you know I mean everybody's competitive but for them a measure of success for them is how much money they make because that's the external kind of manifestation of their their goodness in some ways well in you know uh research and stuff your measure is knowing something or coming uh understanding something publishing something first that other people didn't so I think that there's a psychology there where if uh uh most people like me you know we competitive in in in some sense of the definition where if everybody is saying this everything this and I'm somehow able to prove everybody else was wrong I'd get great satisfac faction out of that so that that I mean it sounds kind of terrible but I think that's part of it now the other part is from an implementation point of view um I know that working on the code end of things every implementation that I've seen has a completely has sort of two different code bases for neural network classifiers one for binary classification and one for all other cases but if when you're classifying uh doing a binary classification and you have two output nodes then you only have one code base in other words the neural network is the neural network where the number of output nodes is the number of uh classes that you're trying to predict so from that engineering point of view it's very um appealing there's an Elegance to having the same Solutions at the same code base for independent of the specifics of the problem or to not have the exception of the the single class or the two class prediction and then regarding training deep neural Nets um what's the what's the state-of-the-art there um and I think my sense is that that that training techniques or tell me if this is true or not that training techniques are tied very closely to architecture at this point meaning the research papers that talk about new architecture um are also talking about specific training techniques for for those architectures or is that not the case uh and then talk about uh Dropout which is uh I think that was Jeff hinton's group in 2014 go ahead if that's enough to get going with no um well first of all I agree with you for the first part of your question and that in general there are a few except but in general if you create a custom um Network work architecture then you'll have to use a custom Training uh algorithm optimization Al now there's I I'll make a parenthetical remark that an area that I believe has great promise and again I'm in a ve very much of a minority view here is that there are certain optimization algorithms and techniques that can be applied to any network architecture and in general they're called swarm intelligence optimization algorithms U particle swarm optimization and so forth there's there's some others and basically they just use absolute brute force uh whereas most optim they're not Bas the Swarm techniques are not based on calculus and gradients and things so that's you know most optimization uh algorithms are based on calculus and you have to calculate derivatives and the derivatives depend on the architecture so that's why you got to basically in most cases create a custom Training uh algorithm if you have a custom Training AR Uh custom neural architecture and then as I mentioned uh I'm intrigued by the idea of applying the Swarm optimization to these things I've made a few stabs at it but like anything else there's just uh not enough time uh going back I remember in the previous discussion we were talking about you know my two node versus one node binary class I just haven't had time to uh look at it would take you know I'd have to dedicate a week or two to that and I it like like all of us you know I mean I I've got more more things that I have to do uh than things than time to do um so in short I agree with you that U um uh custom Training algorithms are needed um with the possible exception of swarm optimization which in my few stabs I haven't been entirely successful but I'm not ready to give up on them now with regards to Dr Dropout is interesting there's a whole bunch of um not a whole bunch of techniques but uh quite a few techniques and Dropout is one now I remember Dropout training uh which is closely related to uh jittering uh input jittering and so uh are all designed or mostly designed to prevent overfitting uh during the Train That's sort of their motivation in most cases and Dropout training was everywhere say two to three years ago it was a very hot area of research um uh a lot of excitement around and that sort of faded out um for reasons which aren't clear to me I I have this nagging suspicion a lot of times that Trends in research even you know very high-end mathematical research are subject to uh to Trends and Fashions just like a lot of things are and sometimes things fall out of favor for no apparent reason uh an example of this that I like to point out is that there's a u a uh neural network training algorithm uh called resilient back propagation it's a a form of well obviously a variation of back propagation I did some uh experimentation on it where I generated artificial uh data sets very large artificial data sets and the resilient back propag ation algorithm what I mean clearly outperformed um normal back propagation now I have to say that with an asterisk the problem with it's almost impossible to compare Al training algorithms because they all have so many hyperparameters typically the learning rate momentum rate um uh regularization you know L1 regulation that there's just too many parameters you're you're you're not completely completely comparing apples to oranges but you're comparing two different kinds of apples perhaps so it's very difficult to talk so anyway Dropout training is something that just seems to be not be in fashion but is there and um uh I'm I'm a a believer in in Dropout training but you know it's kind of funny and now that you asked this question I'm I'm thinking back to recent neural networks that I've done and I haven't been using Dropout to tell you the truth um because it is it's in my world you know I I spin up custom neural networks myself and uh they're they're quite difficult to implement it it creates a lot of extra work and so I take the often take the easy way out and what is that easy way out is it I mean besides from not using Dropout are there things that you're doing with your data or there are other algorithms that have the same effect of avoiding overfitting or is it you know your standard you know uh data segmenting validation sets that kind of thing yeah you know I I I got to to be honest I don't really have a good answer to that question you know I'm not sure uh to to be perfectly honest um for for one you know here I was talking to uh the Chief Architect of um Microsoft's uh cntk tool which is our it's released to the public you can find it on GitHub it's our version of tensorflow um uh deep uh neural networks including convolutional neural networks and recurrent neural networks and stuff and I was talking to him one time cuz he's uh not only a he's a he's a great the main architect very bright guy and uh named Frank and we were talking and I uh asked him a question I I I I saw some really weird behavior that I didn't understand I don't remember what the weird Behavior was and so I I saw him uh in the hallway and I said hey Frank you know I and then I described the phenomenon and he go I said can you think of anything uh that would you know cause that to happen by the way it later turned out it was just a weird it was just weird random randomness but uh you know Frank thought about it for I goes you know the only thing I can think of is that you've got a bug in your code and that was I mean he was and and I tell you you know when I saw that uh when I saw the behavior that I described that's what that was my first pro go man I I must have a serious bug in my code somewhere well it turns out that it wasn't a bug at all it was just sort of bizarre behavior and but the conversation led us to talk about and then you know our conversation sort of meandered and and I said yeah I remember I told him the story how how I spun a a uh neural network this is a few years ago and I was using it for you know work it was I was actually using it was performing very well until one day I was looking at the code dusted off the code and realized I'd missed a com i' completely missed updating one of the bias values um in other words I completely was ignoring one of the constants in the uh equation and yet the the neural was performing well so the moral of the story and so I mentioned that to Frankie Goes yeah I I've done this many times myself so what's happening here is when you create a neural network it's really really hard to tell if if it's good or not because you can get you can get good results and have a seriously flawed uh implementation so in the same way by I'm coming back to this Dropout thing where adding Dropout or moving Dropout you you'd think it'd be a relatively easy to tell is this helping me or hurting me it's not at all easy to determine in in total the there are a few things that have been really interesting uh for me about this conversation but one of the most uh is the like the the hard definitive stand you took on the need to craft your own networks and uh I think how that what that relates to here is I don't know I guess the the idea that uh deep neuron Nets are you know they're kind of magic black boxes right and uh they're particularly magic they're magic black boxes even if you built them from scratch that's and they're particularly they're going to be even worse if you are using something out of the box that you don't fully understand absolutely and um and I think this also relates to you know there's there's always this question around um you know using these outof the-box tools and you know for many types of problems you're trying to get from0 to 80% and you know the researchers are trying to get from 95.2% to 95.7% and so that's kind of an argument for well you know just use the tool MH um you know but if you you know even if you're just trying to get to 80% if you really need to understand uh what's happening or you need to be able to understand what's happening in the case where uh it generally works great but for whatever some variant in your input data produces outlandishly wrong results like you have to know what's going on under the covers quite I mean I I I think you I think you phrased that really really well I guess how are we doing on time are you still you have I'm about I have a meeting that started right now uh oh okay so we'll have to wrap this up I'm afraid okay so we'll wrap this up maybe I can just ask you to quickly tell us about you you mentioned you do your own research you've done uh some projects like NFL scores like what's your what's the project that you're most excited about and maybe give us a quick overview of that and uh if it's something that's public where we can learn more um well the project I'm working on right now interestingly enough is that Microsoft recently launched What's called the AI school this is a big deal Microsoft uh is a large organization and we create products and services Microsoft has made a massive investment both MoneyWise and um sort of culture-wise where our senior leadership believes that putting intelligence real intelligence into every product and service that we do is critically important um uh so some of our um uh senior leaders I've seen say something where they believe that this wave of adding artificial intelligence and machine learning intelligence into our products is every bit as important as what you know the internet uh came to pass so towards that uh Microsoft created What's called the AI school and I was um uh ired from my uh previous to help run uh the AI School uh because I had a background in education and I'd say that I'm um U pretty relative to most of my peers I have a pretty broad uh knowledge of many areas of machine learning AI Al I'm not nearly as deep uh as they are of course so that's what I'm working on right now I'm trying to spin up uh trying to determine how to transfer knowledge of all these things that we just talked about in and plac that knowledge into the hands of the software developers that we have the project managers that we have the business decision makers that we have the salespeople who sell our products and and generate the revenue that you know keeps me employed uh try because everybody I was surprised we we sent out an announcement uh to this like um oh we're creating the AI school and we we thought we'd get a you know maybe a couple hundred uh messages of interest uh exclusively from uh engineers and developers but we got thousands and thousands literally we I mean we were overwhelmed by the response and not only just from Engineers I think Engineers see pretty clearly that machine learning and artificial intelligence skills are quickly becoming must have skills for them um in other words they're going to have to know how to put logistic regression in or they're going to have to know the difference between this kind of classifier and that kind of classifier right but so that made sense but we were surprised by the number of people um uh designers uh UI people uh literally across the organization people excit so that's what I'm working on now and I'm very uh you know passionate about this and very interested in it and uh trying to uh deliver this knowledge while at the same time in fact I remember the U the researcher who hired me uh to run this and and he actually in charge is one of the most famous names in speech recognition he basically created uh the technology behind Cortana which is the same as technology behind sir I mean extremely famous guy but and he told me uh when I was uh interviewing for this position from my my old position just upstairs by the way he he told me you know how are you going to manage or how are you going to balance doing you know your job of creating training classes and delivering classes and doing videos and stuff with the need to stay uh up toate because things are rolling out on a you know weekly monthly basis right and I said well you know that's that's the challenge so you know I I'll conclude by saying you I'm really excited about working on the Microsoft AI school but also really excited about all the all the things that are going on generative adaptive neural networks and and um uh all these other things and now you've got you certainly got me excited about this AI school and probably a lot of listeners as well is this primarily an internal resource or will it be a public resource that micro Microsoft is promoting yeah that's a good good question something that we've talked about and we're really not quite sure you know our our Mandate of course initially at least is to um provide this internally it's not a secret or anything but we don't have any uh externally facing kind of information very much but on the other hand a lot of people are saying hey you know I mean we want to the the content that we develop could be useful to everybody right the the deal here is that there's a lot of content out there already um what we're trying to do is find our sweet spot where how can we use our particular areas of expertise we don't want to just rehash and redo say for instance um most of your listeners uh probably know about Andrew in out of uh Stanford his excellent uh online courses which which I think are uh probably pretty much state-of-the-art we don't want to just try to replicate that for a couple reasons we'd be wasting our time and we probably wouldn't do as good a job so we're trying to find areas uh where we have our internal expertise and sort not only the the knowledge but the method of delivery once we figure that out then I fully believe that we'll be able to share that with every great great with that James you've been very gracious with your time thank you so much uh and look for look forward to uh keeping in touch and to our you know when we meet in person at the the future of data Summit well thanks Sam it was a pleasure chatting with you and thanks for your time all right everyone that's our show for today once again thank you so much for listening and for your continued support please remember that we want to hear from you you can comment on the show via the show notes page via the @ twiml AI Twitter handle or my own @ Sam charington handle via our new Facebook and YouTube Pages or just via good oldfashioned email to Sam twiml ai.com please do show some love to our new Facebook and YouTube pages though your likes and subscribes there will really help support the show and remember if you're catching this podcast on Friday you've still got time to register for our strata Hadoop giveaway the winner will be announced on next week's show the notes for this show and all the links I've mentioned will be posted at twiml a.com SLT talk sl13 thanks again for listening and catch you next [Music] time m