Natural Language Understanding for Amazon Alexa with Zornitsa Kozareva - #30

Key Takeaways

[Music] hello and welcome to another episode of twiml talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam charington I apologize in advance for a longer than usual intro but we've got a bunch of news and announcements that we wanted to share this week thanks to everyone that listened to shared and commented on last week's show based on your feedback so far it's pretty clear you're enjoying both the industrial AI series as well as our more technical nerd alert shows I am too so you'll definitely see more of both this week though we're starting a two-e break from the industrial AI series I've got a great show for you today and then next week the week of July 3rd we'll be foring our usual Friday release and experimenting with a shift to a Monday release schedule for at least the rest of the summer when this podcast drops I'll be in New York City for the O'Reilly AI conference where I'll be interviewing speakers like Douglas eek from Google brain ra Al kbi from affectiva Ben VOD from gamalon and naen raal of Intel Nirvana our O'Reilly AI series will be posted on Monday July 10th for your binge listening pleasure the following week I'm in Germany in Hamburg Berlin and possibly Munich if you're in or near one of those cities and you'd like to connect definitely give me a shout out as we've mentioned over the past few weeks we've been planning and now have finalized our very first twiml happy hour we've partnered with our friends from the NY meetup group and we'd love for you to meet us at answorth Midtown in New York City on Thursday June 29th starting at 6 p.m. right after the O'Reilly conference for a few hours of drinks conversations and networking I'm looking forward to being back in my hometown and sharing a drink with those of you who can make it make sure to RSVP at tm.com slny Meetup to let us know you're coming and now about today's show our guest this week is zorita kosara manager of machine learning with Amazon Web Services Deep learning where she leads a group focused on natural language processing and dialogue systems for products like Alexa and Lex the latter of which we discuss in the podcast we spend most of our time talking through the architecture of modern natural language understanding systems including the role of deep learning and some of the various ways folks are working to overcome the challenges in the field such as understanding human intent if you're interested in this field she mentions the AWS chatbot challenge which you've still got a couple more weeks to participate in of course a link will be in the show notes which will be posted at twiml ai.com talk3 I had a ton of fun chatting with zor and learned a bunch and we couldn't wait to share this conversation with you [Music] enjoy all right everyone I am on the line with zorita karova zorita is a manager with AWS deep learning and we are going to be talking about deep learning and natural language understanding and I'm super excited to have her on the line how are you our need likewise thank you Sam it's a great pleasure to be here and to be part of the show I'm doing really well awesome awesome well why don't we get started by having you tell us a little bit about your background and how you got to where you are yes so currently I'm a manager of the AWS deep learning group at Amazon that focuses on nature language processing and dialogue systems my background and phds are in the field of natural language processing which focuses on Building Systems that understand what humans mean for a couple of years I wore an academic hat at the University of Southern California I was an assistant professor there for the computer science department and I focus on different types of research funded by DARPA and a arpa and after that I moved to Industry where I decided to tackle the same challenges but at a much larger scale and with a bigger impact to humans and Society do you currently do research or are you primarily focused on product oriented work I'm focused on both like we do a lot of product development inside at Amazon but at the same time I'm making sure that I continue participating and serving the scientific Community we do have scientific work as well as I regularly serve on the program committees and I'm an area chair so I'm trying to do both but at work definitely the focus is on building products got it got it you're currently working on the Deep learning and natural language understanding systems that power Amazon Alexa and Amazon Lex I'm pretty sure everyone knows what Amazon Alexa is we've talked about it a bunch of times here on the show uh and in fact I demonstrated a few times how to you access the show on Alexa so folks are familiar with that but I'm not sure everyone knows what Amazon Le is so can you maybe give us a high level overview of that service it was just announced last year at reinvent right that is absolutely right well let me walk you through like how we ended up with Amazon legs so if you think about it we live in the artificial intelligence era and we see the development of very smart systems from self-driving cars to internet of things but at the core since this conversational assistant that enable the communications between machines and humans and it has been a dream for developers and for Sciences in general to be able to build such assistance but if you take any developer and they have to build such kind of a system it requires them to know a lot about natural language understanding and speech recognition which are very tough and you either should have like a PhD or should have spend significant amount in these areas developer had to worry about how do I build systems that are really scalable how do I enable testing and make sure that my bots are going to be working what the users need how do authentification how to integrate the business logic and all of these challenges were kind of uh blocking the development at a faster space of all these applications so we introduce Amazon Lex which is a new service for building conversational interfaces for your apps using Voice and text and there are multiple benefits to that one is like Amazon Lex is very easy to use it's built for developers and we at Amazon 2 all the heavy lifting in terms of the infrastructure we take care of the models and the developer just has to focus on what is their customer use case and how they want these applications to look the best part is like we have very high quality text and speech language understanding and they are powered by the same deep learning technology as Alexa so that's why we have the short Lex it's short for Alexa the good part is also that any developer can seamlessly deploy and scale if you build your application for example for specific platform like let's say Facebook Messenger you can very easily Port it to many other platforms and you don't have to worry about that and at the same time we have like AWS mobile hub integration that allows you to do many things like synchronize data analyze user Behavior track retention integrate your B and so many many things so this new service Amazon Le allows people to focus on what matters to them and literally like solve a particular user need whether it's booking a hotel or it's opening a bank account and the most uh exciting part is like we're organizing a challenge right now it started in April and it will end in July so those folks that are really passionate and want to build their Bots I encourage them to have a look at our web page and just register for the challenge oh is that some kind of is it like a prize for the best Amazon Lex app or something yeah there are different rewards there's a monetary rewards there's an AWS credits and also folks can come to reinvent 2017 there there will be ticket and as I say the focus is like build a chat bot that engages users and at the same time fulfills a specific need that you have like booking a hotel or any other thing that you might have in mind okay nice nice so with that in mind let's maybe talk about what are some of the biggest challenges in working on systems like this like what are you currently researching well my focus is in natural language processing and the most hard part is to build systems that actually understand what the humans mean this involves like can we understand what intents people have in mind can we identify the slots that enable us understand how these intents should be fulfilled and for most people that haven't worked in this area it sounds like oh that's pretty straightforward but actually language is very ambiguous and very very hard to understand if we deal with very explicit intents let's say cancel travel to Miami I just literally said every single thing that that I'm planning to do canel is my intent and Miami is my destination but imagine I'm chatting with someone and they're asking me are you coming to the birthday party and I suddenly say I'm on my way if the conversational assistant pops in and says hey zorita should I send an Uber your way that's amazing so building such applications is really hard it requires the system to understand implicit intents which are very hard to detect it requires the system to know what your user preferences are maybe I'm using specific means of transportation the system has learned it over time and it's making automatically the recommendation and the ability to generate such kind of replies automatically instead of using templates that are already pre-specified these are all very very complex problems that are open-ended and we're continuing to invest both in terms of Sciences as well as in in Industry how do you solve and make systems being able to handle all this complexity so maybe what's a specific area that you've been focusing on there in terms of your research my research focuses on building this natural language understanding capability anyone anytime an utterances comes in uh we focus on extracting those slots to fill in the template as well as understanding what these intents are such that when you pass that information to a dialogue manager or a component that communicates with the backends we understand like what the human meant and more correctly you extract such information and the most accurately you populate it in these specific templates then the the better your system will be and humans don't have to repeat the same question over and over again hm you mentioned a dialogue manager do these systems have standard components you know independent of the implementation to kind of all of these systems have dialogue managers and you know what is the general architecture of these types of systems yeah that's an excellent questions while conversational assistant can live in different shapes and forms like either on your phone or in your home device even in your car the common part is like they have very standard like procing blocks input is like either speech or text and the first component that gets hit is the So-Cal natural language understanding component that is the piece that focuses on understanding the intents and the slots of the users once that information is extracted it gets passed to the dialogue manager the task is to take these pieces of information send them to the back end and a back end think about it if you want to book let's say a flight maybe I just say book me a flight to Miami once you extracted the Miami term and you pass it to the dialog manager it communicates to the back end let's say your favorite travel website and it says well now I need to know what date you will be traveling do you have any price restrictions do you want it to be like a direct flight or not it sends back all this information to the dialog manager says hey I need to have all this information filled the dialog manager J like passes the slots and then they hit a natural language generation component that says back to the user hey can you tell me like do you want a direct flight and do you have any price constraints that output could be both text or it could be speech it gets sends back to the user and then the user says well actually I don't have any constraints find me let's say the cheapest flight so this Loop constant Loop between these three components natural language understanding dialogue manager natural language generation is what drives the whole conversation between a system and a human how you implement them it depends on you as a how do the developer how you decide what Mo models to include but on the high level these are kind of like the three building blocks that you act like everyone has to focus this on on building okay okay and then even within the nlu component there we can even break that down further I know um or I recall that Google a while ago I guess about around a year ago announced like an open source parser their parsey mcar face and that's just one of the different pieces of the nlu component itself like what does that part look like well think about it that the natural language understanding components they're as I said they're different and depends what your need this and the the tools that you you quoted they're kind of like a high level dependency parsers or just a parel you can use that information to facilitate how to build this kind of natural language understanding components but at the core is like you have to think about how to define your slots or how do you define your semantics how do you define the space over which your system will operate and would semantically understand what a human means and these things typically are C slots they're the entities and semantic bits of information that capture what our request is you can build them from scratch you just need to know like what are the right machine learning models for that or if you want as I say in Amazon leg we have like pre-build capabilities such that a people people can just choose from a drop down menu so that enables like a person who knows nothing about machine learning to be able to to build them for anyone that is from natural language processing field and has dealt and worked in semantics and information extraction it is much easier for for people with such kind of skill set to to be able to build their component from scratch and to know how to define these semantics or how to define the space that the machine Learning System should operate over and make predictions and how do you characterize that space you know you're starting with typically you know some audio wave form like what's the process for getting that into you know some pars set of slots that you can then operate on further back in the system uhhuh yeah so once you have the speech as I said the the input could be both speech that you transfer from speech to text or it could be the text itself think about it like a text messaging once you have that text in what you do is like you have to make the design of what the intents and the slots are and and often times people come and ask me well where do these intents and Slot come from and the answer is they're the designer's choice you can take them from an existing large ontology just pick the pieces that you care about let's say in travel you might care about cities and countries and so on you can automatically learn them from a lot of unlabel texts or you can even manually create them so how do these slots look like let's say we're building a shopping bot then we can decide that our slots are the things that are most important products and vendors and Brands and models and product families and the intents or which are the actions on top of which we can do operations are buying them selling them recommending them tracking them MH what we do is like we formulate this type of problems as as a sequence prediction like if I give you this cat ories the products vendors Brands and I give you some text of length M like can you find segments inside the text that can be labeled with these specific categories and the moment you do that then you're having this Rich representation that that that you're caring about and that your dialog system can ingest and actually build on on top of so you have to annotate your data in the correct way meaning if I say purchase added a shoes quick to and Nike Pegasus say each word or how say segments of words they get tacked with this specific information and then the machines just ingest it such that they can build prediction models on on top of that and that's one way you can can do it yeah okay okay and so one thing I'm inferring from your description is that you consider or it's generally considered that the speech recognition component of you know a broader system like Lex and the nlu component of a broader system like you know system are like two different things is that the way you tend to think of it like this you're getting speech that's already you know turned from audio signals to some set of symbols whether that's you know language or something else and then that's what you're working to understand is that right that is correct yes think of them like different building blocks that you need to piece together in the right way such that you can build a complex system that is going to be able to drive this dialogue so in my case I just focus on the natural language processing component and we have amazing colleagues that do the same thing they focus on speech or they focus on generating from speech to text yesh so when you were talking about the intent and the slots and looking at that problem as predicting a sequence that makes me think of deep learning models like lstms do those corre come into play there talk about the the impact of deep learning on all this yes absolutely you have very good intuition and lstms are very useful in solving such type of problem it's a structure prediction problem and as I say until recently people used to employ a lot of the traditional supervised learning once they pick their favorite algorithm being a CRF tagger or being like learning to search algorithm most of the focus was on how to do feature engineering and find what are the right features and how do we iterate over those features but now with the big wave of deep learning we see that one can build such kind of systems if you're an expert in the field one can build them much faster in terms of like picking the right algorithm you can train your wording Bings on a larger unlabeled data and also get much more powerful results so lstm is great for solving this kind of structure prediction problem and typically we see very good results if you have the last layer with conditional random Fields can you elaborate on that what are conditional random Fields yes it's very very standard they're like discriminative classifiers it's very very standard buil for many decades and using this kind of task they're use as I say for sequence prediction can code relationships between the observed data like typically the current and the previous word does not model longrange dependencies while in the lstm you have these deep learning models which are more powerful as I said they can learn powerful representation given enough data they can capture the long-term dependencies not only in a sentence but even they can span between sentences and paragraphs and as I said the best part is like you don't have to to focus on feature engineering you can just pass the row data as input mhm so if you're if you're looking to build a system like this what are the primary considerations that you're that you need to think about to architect it correctly and to build a system that meets you know a specific set of needs I would say if you are familiar with machine learning if you use these sets of methods or methods that are appropriate for structure prediction your problem will be kind of solved but the part that one really needs to focus on is the data because the data is going to drive the quality of how your system performs and behaves and the semantic representation this means is like how are these labeled spaces going to look like in terms of like the machine learning algorithm themselves there's like plenty of Open Source platforms that that people can use at Amazon we have the mxet platform which is an open source machine learning platform we have tensorflow people have a wide variety of choosing which machine learning toolbox they can use and as I say that's pretty easy to pick up but the hardest part is like if you don't know how to get your data if you don't know how to represent it right then even if you have the best toolbox on Earth you won't be able to build your applications in the in the right way so I encourage people to uh look at both things not only at the machine learning side but also on the side of like how do I design my system how do I collect the data how do I drive all of these processes together MH and you said making sure you represent your data right is one of the big challenges what exactly does that mean and what are the considerations there yes well as I said the biggest problem is like semantic understanding of like what humans mean and how should the computer encode and understand that and and that's a challenge and it's been a challenge for many years there are theories but yet building such a system that understands us humans is very challenging so we have these basic representations that can to some extent work but that doesn't mean that really understand what a human means it's very hard for the systems to pick up like sarcasm how do you take it represent it and even like model it these are much much harder things we have to account for right now we have these basic more flatter or slightly more structured representation but that's definitely not the way to go forward I'm trying to put that into context if I'm thinking through you know building out a system to understand language you know and and you're advising me to start with the the data collection and representation you know I get collection obviously it's going to be or depending on the situation it may be difficult to collect the data but assume that I can collect a bunch of data you know say you know let's assume it's in text form from you know bot requests or something like that the next step you're saying is the representation where does representation come in before I start throwing my data at the machine learning algorithms and put another way like part of what I'm expecting or hoping the machine learning algorithms to do is to kind of parse the data and you know help me figure out how to represent it is that right or well there are different ways one can do it I'm also hearing a good thought from your side which is like can you even build the the machines or can you build a machine machine learning algorithm that can given given just some chat B can automatically infer what the slots should be what the intent should be and definitely that's that's possible it's not going to be very accurate because it's almost like when you do clustering over documents right you can end up with these different representations and the same thing is going to happen here so that's why typically you start with the Notions of you defining your your slots and intents this so when you build the machine learning applications typically you have the classes right that you want to Output meaning if I give you a large document collection and I say well I just want to know is this low is this medicine and so on okay you already have these categories so it's the same thing for building these natural language understanding components you need to know what categories or what is your space over which you're going to operate if you try to learn that automatically as I say you're going to end up with this Co grein very high level you define like meanings and it won't be sufficient for you to build a meaningful application right but if you sit down and actually write it yourself you have much higher chance because think about it just to to build a flight booking system if you open the different travel website and you see like what are the minimum sets of inputs they require you from destination to Destin time location and so on and you just Define literally that to be your space or the slots over which the system would operate you're going to be able to build much much better system and to be honest yes one of the of the pushes in in science both and in terms of Industries like can you build systems or conversational assistants that can learn from human interactions and that you can teach right now if you ask something to a system it will tell you I'm sorry I'm not quite sure about that or like I'm still learning in reality it would be really awesome if you make those system learn such that anything you say and based on how people have replied plus like request for new let's say for new information that you haven't seen before if the system can learn it and starts asking about it or sees many people are talking about it then this is the the place where we want to go because humans cannot include all the knowledge in the world and where I'm headed at is like if you look at how all of these applications are built they are operated over different domains which means you have a movie domain you have like music Domain Shopping domain and so on and each of these domains have their semantic representations or so-called slots and templates that you can call and you can fill in but a human cannot sit down and represent the whole world in these frames right it's too time consuming it's labor intensive it doesn't scale and most importantly is like we humans don't operate per domains right I and say order my pizza and book a flight right I just call two different domains and I express multiple intents and that's the main reasons why it's how you say like the the current technology is great and it soles real use cases but there's like many many more things that have to be done and and we have to focus on figuring out well how do you build NN system that can operate for any domain that can operate for different languages and that can continue to help assisting us and so what's the state-of-the-art there are people building you know a bunch of individual domain knowledge and then using some deep neural network to kind of identify which domain is being asked about in a particular utterance or are they building you know kind of bigger you know flatter models is there any particular direction now I think yes one of the most common Trend that I I have been seeing over time in this field is for people to try to focus on on the different domains and for each domain focus on the slots and sometimes I've seen people trying to transfer Knowledge from similar domains let's say I know everything about reserving restaurants and I have a little bit of data for hotels now can I learn how to reserve hotels by knowing how to reserve restaurants so that's the most common approach and as I said it's it's a challenge because it doesn't scale it just doesn't work the IDE ideal scenario is like can you build a system that doesn't have these boundaries and can can just like operate over anything that we we have in mind yes in terms of like what algorithms is a lot of people do use deep learning as I said different parts of the components yeah yeah and I guess deep learning isn't really fundamental to my question as much as is the best practice to or you know is the research Frontier if I'm trying to build a system that can handle hotel reservations and pizza orders and buying cars and you know multiple things am I likely to be better off getting a bunch of data for each of those building out specializ representations for each of those and training models that perform well for each and then use some kind of discriminator that can figure out which model to use or is there some other kind of technique that at that meta level you know to pull it all together yeah if you're building a production system that has to work and satisfy user and customer needs then that's the right way to go it will guarantee you one that that your system is going to be doing the right things and it you have higher Precision which is what's very important for users nobody likes to hear the same question twice like what did you mean I I didn't get your request in terms of research I think people can have much more flexibility and they can use all kinds of techniques they can use like transfer learning to see like how do I transfer my B that I buil let's say for restaurants into flights or into something else and I think this kind of transfers are more successful for things that are closer to each other there are some paper that I have just seen on General like entity recognition and sentiment analysis but the thing is that they've been evaluated on much smaller data so ideally what I really would like to see is like people doing it on a larger scale with lots of data and having many of these domains and actually showing like how possible it is and and where are going to be like the biggest like challenges also you can build the system in a different way like if you want you can build these individual components and then you have something on the top that actually ranks your results and tells you what's the most likely domain sets of intents and also slots or you can build one big giant neuron Network architecture that you can incorporate let's say if you have your lstm you can make it learn both your slots it can learn your intent as well as the domain and over time with lot of data you can learn these constraints that in certain domain let's say like movies it's very less likely that I do an action that is let's say possible for for a different I would say that if you are building a real application then then stick to what works if you are passionate about exploring and pushing the Frontiers and doing like lot of research then definitely there are many more like open doors in terms of like trying to see like where is the boundaries and how far can we get great great let's talk about the data a little bit are there standard data sets that folks are using for these kinds of problems like there are in the image recognition side of things I've seen like there is some challenges that are called the dialog State tracking challenge they do focus a lot on the dialog manager component and there are also these attempts with these different same data challenge that people can can do the the knowledge transfer that I was talking about I've seen people use different types like to chats and that's like if you just want to build like a high level chatbot system that doesn't it's not go oriented and then I've seen a lot of work from from Facebook AI that drives in that area and they have a data said that's called Bobby it's I think Bobby okay I'm not sure how the right pronunciation is b a b okay but it has like different tasks like 20 20 task and it's annotated and and people do use it for conducting research so this if somebody wants to like start in that area and wants to play around maybe these are good places as I say depends like do you just want to build like chartable that that doesn't have any goal fulfillment or do you want to focus on building a real working system that is going to fulfill your goals for you as a human and now if you're if you're building a system like this and none of these data sets work for you you've got to figure out some place to find that data and I noted that you've got some research experience into large scale knowledge extraction from the web yes are there any techniques there that you might use to help you collect a data set well you can do different things people have invested a lot of effort in in building these wizard ofos systems it means like you can build very quick application that simulates a scenario let's say one person will pretend he is the the booking expert and the other one will be the the customer and they can drive the conversation on their own and the booking agent will just try to fulfill that intent so if you creates their scenarios and record all the data with different peoples interacting that could be one easy quick way to start building your application another way could be that you can just to to drill into that what your suggesting there is that you define out these scenarios and then are you thinking like go to someplace like a Mechanical Turk and say Hey you know you play the booking agent and you play the customer and you get kind of random people to explore these scenarios or do you mean something else by that yeah it's perfectly feasible yes you can exactly do that like kind of decide what you what the application you want to build you can set up everything being on Amazon TK or being even let's say you're a startup and you don't want this information to to be leaking then you can set it up even build your own very quick in-house application that just can record these conversations and once you have that data then you can start building let's say a first prototype you can even launch it and then test it on real users when the data comes back you can start iterating and and once you have this kind of nice Loop you can keep iterating and improving what you have done that's one way second way is like completely even focus on let's say like just Amazon Mechanical TK you can either have predefined questions but then the conversation will become very artificial or you can play the game that you have just explained and then the on your questions like can we do something if I give you the whole web what can I do to facilitate building of such system like at the core of these systems is like a knowledge component you have to have knowledge bases that help you extract this information much more accurately that help you get World Knowledge and one ways that you can get that World Knowledge if you don't have your own Knowledge Graph you can get that World Knowledge by literally extracting all of that information from tons of web pages on different topics and you can integrate it inside your models right either as additional features or additional signal and help Drive these applications to be more precise and and more accurate H interesting interesting you did a talk on on dialogue systems actually that's coming up it looks like is that in the same kind of domain of the things that we've been talking about thus far or are there different aspects to that yes that was like kind of the core component of like if people know machine learning and they want to build everything from scratch theirselves is like what components should they focus on what machine learning algorithm they should work like pick up and then how to get the data and at the same time it's like what's the expectation some people have the expectations that this system should be like 99% accurate and I was like oh not really we're doing these baby steps in the field for some very specific basic things you have high accuracy but that doesn't mean that you have solved the problem it just shows that the solution of such task is possible but doesn't mean that it's a it's a solved problem so you bring up accuracy and that raises some interest interesting questions I think on the speech recognition side you know there are some standardized measures of accuracy and you have folks like you know Microsoft and others reporting their their accuracy and like Microsoft recently reported or they I think a number of organizations have recently reported human parity and recognition it seems like it's maybe not quite as straightforward to report accuracy on the nlu side of thing is that true or or no well actually no think about it like when we annotate the data you typically have your training set your development set and you have your test set and for each of those we always measure what is your Precision what is your Reco what's your F score we go on the level of each of these slots and we also record other things like how hard is the task for a human we measure the cap agreement the cryp andorf Alpha and the idea is that if you see how hard the task is for a human then expect your system to be 10% less than that right so if two humans have really hard time annotating your data with your representations this means that it will be extremely hard also for for that system to to learn about it my point was that building the natural language understanding systems is is much harder than than building a let's say image understanding system and the challeng is like language is very ambiguous we have to deal with a lot of SL that people might be using or specific like metaphors that people have in mind and so the building of a system that can understand and operate on top of that right it's very hard that's why I was saying that it's important to have your knowledge bases hooked up so that the system can can get think about like a cheat sheet some extra information that that can help it facilitate understand better like what we mean right yeah I think that that's what I was getting at with the accuracy question I I you know can think of many occasions often happening right here at home where I definitely understand the words but I'm not sure I totally understand the meaning and while I can you know I can certainly grade my systems accuracy relative to you know some label set it's harder to capture you know even the right metrics in terms of you mentioned this earlier in the conversation like are we even do we even have representation for you know nuance and and sarcasm and you know things like that that you know if I had some Uber metric of hey does you know does the system understand what was being said it just seems way more difficult to really capture what that even means yes and you know like back in the day when I was like with my academic had I had the Grant from a arpa that focused on can you build systems that understand metaphors for four different languages it was Arabic Russian Spanish and English metaphors metaphors got it yes and you had these four different languages like Spanish and Russian and Arabic and English and the idea was like we just pass any text could be news can you find those metaphors can you interpret them and also assign the sentiment that the person had when he said that metaphor right so if I say my lawyer is a shark so we know that yeah sharks are vicious but it's really good for me to have a lawyer like that because it means that that the lawyer is going to do the right thing and and protect you but if I'm saying this to you then for you that is going to have a negative connotation right so it was really hard to build systems that just given any free text in in these languages it can identify the metaphoric expression interpret what they mean and yeah we focused like two years on on just building and trying to solve it then unlike other task in natural language processing that have matured over time and that have like significantly higher performance during such kind of system that understand metaphors was very hard it's like in the in the 50% it's really challenging yeah again going back to my previous point it's even hard to before we get to building the system to think about what performance even means in this context right I think even that statement about my lawyer as a shark can be you you know can probably have either positive or negative sentiment depending on the circumstance that is correct exactly but you know the the best part is like as I say language is very hard and there are a lot of people trying to solve these challenging tasks for some of them we've made tremendous progress others are open-ended and and we're still working on them and that's what excites me because it means that we have a lot of work to do a lot of efforts to to put into into building such systems absolutely absolutely what are some of the other areas that you're tracking that you're seeing exciting work happening in in natural language processing there is like a lot of focus on on question answering and building both type of systems like if I give you a knowledge base can you do question answering over this knowledge base and do inferences on top of that and we have seen open-ended question answering where you have a document and just somebody comes in and types a question like this on the Spore of the moment um then can we find the corresponding answer in in that document MH they're both very challenging and exciting so this year I was area chair for AIA which is Association of computational linguistics conference our top tier one conference and like the biggest natural language prosting conferences and the areas that were training a lot for information extraction and question answering and there's like a lot of effort some challenges that are coming up and people have just been given the data and given the ability to to think about how to innovate and and how to solve them I know quora is among the folks that have data sets for question answering and there's another popular one at least another popular one that I'm forgetting the name of right now yes that's absolutely correct like Kora has a nice data set you have squat who is coming as a data set developed by University of Stanford perang group and that's one of the it's a very good data set it has lot of training data magnitudes larger than than previous data set and it has different types of categories of questions so that kind of mostly simulates or approximate real case scenario and that data set focuses on on the second question answering type that I was talking about you have like documents and then you have a question like can you find the answer in the document and it's much challenging because people can ask the question in any form using different paraphrases and then finding these pens with the correct answer is is way harder than than traversing knowledge base H and so are the core techniques that are used for these two different types of question answering tasks the same or are they dramatically different well I've seen people that use let's say knowledge graphs to answer questions to use more graph based algorithms and I'm seeing a lot of Trends with the Deep learning for the second kind of problem with this sad data set which is more like machine comprehension from text and there people have different architectures of how they solve it but let's say for for the machine comprehension one I've seen very common is like people try to represent the question into some kind of an embedding or vector space and then they have the document they try to use like a tension mechanism on top to pick up entities or pickup spans that could be good match for the answer and then they kind of try to have some similarity between the question and and the answer so yes both of those different types of question answering have wide variety of methods that have been employed but these are two things that I'm noticing are kind of trending when people publish their work H so for the knowledge graphs I'm imagining there that you're using some kind of technique to identify well maybe a precursor to that is what types of representations are you using typically on your documents that you're trying to do this question answering for are you you know doing things like trying to do semantics and like identify you know nouns and verbs and that kind of structure or are you operating on a a lower level than that oh well actually yeah that that's what I was saying that this data set I like it because one the magnitude is much larger than any previous data set and two you have to focus on extracting different types of of bits and pieces and sometimes they could be just word the phrase they could be combinations of like just like a non phrase or or much harder it's not like a single answer like when was Barack Obama born and you just say the the ear right that's what a knowledge like a question answer over knowledge base does this one is more open and that if I say like hey what were the symptoms for people who have Cardiac Arrest maybe the answers were contained in different paragraphs and you have to find those different paragraphs and and the exact spans with the answer and that's what makes it much more challeng in but at the same time much more useful because most of the information that we need and the questions that we have they lay in this unlabeled documents that are being on the web or or that you as a corporation might have and and you might want to just search for the information so I personally prefer this type of question answering work because it's as I say more real case scenario and second it's like more useful to us and so just to take a step back so with these question answering data sets in particular the Stanford one the data set includes the base document and then a set of questions and answers corre from that document and are there multiple answers for each question and to what degree to the questions overlap um yes that's a that's a great question so typically you have something like one question like I'm just reading from the official paper that that was published which governing bodies have VTO power and then you have a whole document that let's say is talking about something right there is one specific sentence that can have this answer for example the European Parliament and the Council of the European Union have powers of demment and VTO and so on so given that question then the idea and this document idea is like can you find the paragraph and more specifically the sentence that contains this specific type of a of an answer and I haven't like the deeper in terms of like understanding like how many of the question the exact let's say question could be f with the exact answer inside but I do know that the the creators made sure that when that data was annotated that they they asked people was like if you read this article can you ask a question using paraphrases which means like different words or different ways that you can you can ask about it so it doesn't have to be the exact same how to say exact same phrasing if you had the exact same phrasing then the problem is much more simpler right but definitely that's how you say much more challenging data set the way it was created and the way it's annotated so I think they did a good job on making sure it's created the right way it has different complexity yeah so that helps me that helps clarify for me like what you're fundamentally trying to do is you're given a question and you're trying to essentially index into this document the sentence the particular sentence that answers it you know which is a totally different problem than or at least it is a narrow more narrow problem than you know synthesizing an answer based on you know multiple sentences in the document or summary type of problem or you know trying to pull pieces from you know two different sentences that are required to formulate an answer yes that's more like a summarization what you are descri in and there the goal is a little bit different if I give you one or multiple news articles about the same topic it's like can you find sentences such that you can summarize the text in a much more compact fashion and for a human the moment they read it they get the gist of the of the facts and at the same time the whole summary is coherent and has natural read and flow there are data set that that has been created on that on that area is just the problem is called more like summarization right right but there's some overlap right so if um you just to construct a simple example if I've got a document that says somewhere in it you know my favorite color is red you know and then the next sentence is you know but I also like blue and yellow right and if you ask a question what colors do I like there there's got to be some synthesis or summarization in there somewhere and you know a is that you know typically part of the question answering ing is there a you know a set of work in question answering that's looking at those kind of more complex scenarios or are there folks that are trying to combine the question answering and summarization pieces to you know answer these more complex questions yeah if I'm not mistaken there are people who are trying to do both I've seen how to say mostly like focusing on one but not at the two but I'm sure that there are people who work on that I'm just not aware of where I don't have them on top of my head for such papers but like there's a researcher called Sasha rush like he has a paper on like how do you do this summarization that you were describing with attention mechanisms so I'm sure that very likely there could be a work combining both of the things that you're describing so someone wants to dig into this more do you have any go-to resources for getting started yes for example depends on what your aim is if your aim is to stay on top of the natural language processing field a great place is just to go to the ACL onology and all the different NP conferences they indexed there you can see the latest papers organized by years and by tracks so you can pick your favorite truck being question answering being parsing being machine translation whatever excites you and it great to just sit down and like read through these papers and such that you can get on a higher level what is that people are working on and and how far they have advanced if somebody's just a beginner and they're trying to learn about the field I encourage you you can look at the Stanford's class on deep learning like natural language cring with deep learning it's taught by Chris Manning and Richard zoher both of them are like leaders in the field and it's a it's a good place to start and just kind of learn both about like what are the probl