AI-Powered Conversational Interfaces with Paul Tepper - #52

Key Takeaways

[Music] hello and welcome to another episode of we'll talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam Cherrington the show you are about to hear as part of a series of shows recorded in San Francisco at the artificial intelligence conference which was hosted by our friends at O'Reilly and Intel Nirvana in addition to their support for the event itself until Nirvana is also our sponsor for this series of podcasts from the event a huge thanks to them for their continued support of this show make sure you check out my interview with Naveen rau VP and GM of Intel's AI Products Group and Scott app 'land director of Intel's developer Network which you can find at twil Malaya comm slash talks last 51 at the AI conference Intel Nirvana announced a dev cloud a cloud hosted hardware and software platform for learning sandboxing and accelerating the development of AI solutions the dev cloud will be available to 200,000 developers researchers academics and startups via the Intel Nirvana AI Academy this month for more information on the dev cloud or the AI academy visit Intel nirvana dot-com slash dev cloud my guest for this show is Paul Tepper worldwide head of cognitive innovation and product manager for machine learning and AI at nuance communications Paul gave a talk at the conference on critical factors in building successful a I powered conversational interfaces we covered this and a bunch of other topics like voice UI design behavioral biometrics and other interesting things that nuance has in the works and now on to the show all right everyone I am here at the AI conference in San Francisco and I'm with Paul Tepper who is the worldwide head of cognitive innovation and the product manager for AI and machine learning at nuance communication is the enterprise division of nuance communications in particular and I had the pleasure of meeting Paul at the AI conference in New York just a few months ago and he was kind enough to volunteer to jump in the hot seat it's a welcome Paul thank you good to be here absolutely absolutely why don't we start by having you introduce yourself to the audience and talk a little bit about your background and how you got into machine learning and AI sure well for that a year now I've worked at nuance communications but our enterprise division I lead a team that has a few functions one of our main functions is identifying high value problems for which the company doesn't yet have a solution and working with our large corporate research division to see if we have new forward-looking research that could be sort of productized or prototype to get in front of customers as a way to kind of move innovation in product give the product management teams across the company new opportunities and things to look at there I did my PhD at Northwestern my focus was on computational linguistics particularly in dialogue I did a lot of work on nonverbal behavior gesture in particular and I also spent a few years prior to nuance working at a start-up not around anymore what's called heady bond and we focused on building cloud based NLP platform that really focused on human-in-the-loop computing and crowdsourcing as a way to build quickly build data sets out to build custom models for NLP ok I don't think I realized the north-western connection I'm a PhD dropout it yeah engineering home I said were you engineering or yeah I helped found this program called the technology and social behavior program that was a joint degree in computer science and communication studies of all things but that's where we ended up in communication studies doing a lot of work at that time okay super interesting super interesting do you miss Evanston I'm a nice guy my undergrad at Rutgers and I spent a year in Scotland my Master's out there so I travelled around quite a bit but I'm definitely an East Coast guy nice nice so I think the you know folks will get a little bit more about what you're up to now if you maybe spend a few minutes talking about nuance and what nuance is doing yes a nuanced is a fairly large complicated company with about 14,000 employees and several different divisions we have a division the focus is strictly on mobility mobile products an automotive products so we have ASR a lot of cars that people drive do recognition for cars the in-car systems we've got a healthcare division that is some of the top systems for dictation for doctors to do electronic medical records imaging division and my division particular focus on enterprise communications and we have two main product areas or areas of focus and that's one IVR systems or interactive voice response and that was one of the first commercial applications of speech recognition systems that you can call into and instead of doing kind of dial tone menus you can just say what you want conversationally and more recently our focus has been on the digital side now they're called chat BOTS but we've been calling them virtual assistants for a long time and actually there's a lot of overlap in those two technologies so a lot of the technology that had been that has been built out for IVR so to recognize intent to recognize concepts and do entity extraction inside sentences and then also the dialogue build out the dialogue flows whether they're graphs or trees to build through a dialogue a lot of that actually works similarly in the chat world however in the IVR world the utterances tend to be a lot shorter the tower o people tend to talk a lot longer so different complexities there that's an actually an active area research for us so how do we use user experience how do we use UX to get people to talk longer an IVR so we have more because the natural language understanding or NLU as we call it has really far outpaced what people actually say today in IVR systems oh that's an interesting technical challenge for us yeah yeah I think of when I think of IVR I think of call center and I think of the objective on both the person that's enter the callsign in the company that's that's making it available is to keep the interaction as short as possible and it sounds like the technology is allowing us to go the opposite direction I think that the technical term is the industry are things like containment so keeping the user contained to the IVR as opposed to transferring to a live agent these live agents are more expensive and not just expense it's actually hard to staff and maybe if you have all the resources in the world it's hard to hire up call centers that are big enough for some companies to even handle a check if they even with unlimited budget so containments a big one another ones first contact revolutions modular KPI in the industry then you can have a you don't the call back you know your resolves within that first call so it's not always about short but it is tends to be about can you build self-service so being able to let the user call in Iraq with a system that they can solve their problem automatically I'll talk about that in my talk a little bit about some of the statistics that have come out recently showing that today especially with like younger generations and consumers people really don't care whether they're talking to a machine or a human they just want to get their problem was solved and in some cases there's even some prefer it I prefer to talk to a human you know sensitive sorry I prefer to talk to a bot maybe a sense of situation you know you have to call in and talk about you know I need a change of flight because of a death in the family or something in some cases like that you don't it's sensitive you don't actually want to talk to a person even though people have this feeling that emotional intelligence is so important sometimes it's actually you don't want to get into it you just want to get your transaction done yeah yeah so you mentioned your talk what's the title of your talk okay I put a long title these critical factors in conversational interfaces design and conversion arrays the whole thing memorized so kind of lessons learned and best practices that you've come across along the way yeah it'll be a mix of lessons learned and best practices as well as some of the newer things we've discovered and developed new ones okay so walk us through walk us through those yeah sounds like a fascinating topic putting me on the spot I don't think they have them all memorized before yeah I think like this you're talking got about I think ten at this point then we're going to talk about yeah I think I had previously done a version that stopper had six and this talk was 20 minutes so I thought okay thirty five minutes I've got to beef it up a little bit yeah well just expand upon some of the ones that i briefed 3v1 through so I kind of go there's like a high level of things like context and personalization so if you're on a website and you're browsing let's we we do a lot with like banks and insurance and those kind of enterprises and you're in the auto insurance part of the website and a chatbot pops up and says can I help you actually you know that's the typical thing they're gonna say okay help you but at this point in the website basic UX you know will let you know you should say hi so-and-so hi Paul how can I help you with auto insurance those basic principles of UX have not totally been carried over yet into chap out world so it's those kind of things like building those integrations into the systems the websites and stuff a lot of times these chat bots are come from a different platform so you have to actually build those integrations between the company's website and the chat BOTS and those kind of things basic UX but they're not totally there yet in the chat bot world it's funny you mentioned that I've been kind of on the DL evangelizing this idea that that a lot of you know what we've learned so much about user experience design in the web world and in mobile and kind of been these other interface technologies that hasn't really made it into or we haven't formalized to the degree yet or in and around AI and artificially intelligent user interfaces and I think the you know so I I call it intelligent design but the ideal the ideal replacement for that one but as certainly you know chat bots are the kind of the tip of the spear so to speak but even devices like your nest thermostat or something like that there's I think there are like unique things that you need to take into account to you know make sure that users are comfortable with the fact that this thing has some intelligence and also kind of signal to them how to interact with the intelligence as interesting now nuance has been doing we call vou a design for a long time or voice user interface design amid the people who design many of them backgrounds in linguistics PhD in linguistics even who designed the flow of the conversation and will do you know user testing a/b testing etc to figure out what the rest flows are and you'll these people who the people with this experience are being scooped up now by Google and Amazon so you look at the people now Google has a whole vous a design team now for a system as is Amazon a lot of former nuance people because we've been doing this up a long time in the IVR world and it turns out like a lot of those a lot of the basic principles of how a conversation works without a screen transfer over to these note these kind of like IOT devices and it's different when you have a screen when you screen we actually know a lot about a lot about web design UX and that's another problem right like there's this area of overlap between an omlette thing of a Venn diagram of like conversational user interfaces on our voices universe design web user interface design and that little spot in the middle the overlap in the Venn diagram where this weird world of okay well the conversational principles aren't going to cover it completely the web principles or credit cover completely so you have to kind of merge those two together and figure out things like okay well on this part of the website the person's browsing you know looking at whatever product the conversational agent needs to know about that stuff that's one of the things I'll be talking about another thing is security making it easier to pull to either authenticate black traditional password I says so as well as anyone says voice biometric product so he could only transfer you over to say like my voice is my password use a voice print to identify I don't get the sense that that's very popular out of I haven't not yet no no it's it's really new cutting-edge and it's it's it's easier than mobile devices from the mobile devices where we're building Virtual Assistants they're where the whole interface is spoken it's a little more you know part of the flow but we are devote prototyping systems now where you can actually I've always prettify for a bank on the web oh wow yeah yeah that's a huge it's a huge friction point like I think of the places that that recognize my number and based on my number you know associating me with a record like the airline's for example have done a really good job of coming up to the speed on this in the past years but then there are other places where they're like okay type or say your password I'm like well you know it's like 26 characters and it's in my password safe thing or there's no way I'm either gonna type it or say it yeah so and this is an arts a big area of AI machine learning you know is these systems are all I and machine learning driven these voice biometric systems there's also face print identification now we do some work there and we'll start to see other kinds of biometrics too like behavioral biometrics I've heard of this where the way a person interacts with a website the way they move their mouths this is the cadence at which they type etc that also creates a unique fingerprint of that person very very hard to spoof fingerprint I think my first experience with that is with Coursera like when you when you taking a Coursera course like the engineering you know deep learning course like you there's an honor code and you have to type out this honor code thing that says that you know this is you and when you're taking one of their exams you type this passage out again and it uses that to verify that you're you oh and seeing that I don't think if you don't if it doesn't think you're you you have to take a picture with your webcam okay that's really cool yeah so that's that's and Larry gates of reducing that friction point and I'll be spending a while talking about some new techniques that were using to try to incorporate unsupervised learning into our pipeline you know this is kind of a frontier area for AI right now the majority of it's all supervised at this point but we're looking at methods where you can take in a set of chat logs or voice conversation transcripts and the big thing with chat BOTS and with conversational systems is the first layer is the NLU the intent classification so what are the what's the intent of the burden was saying such that then you can then you know respond to that intent whether it's like check my balance or pay a bill we're working on looking at datasets and extracting those intents automatically through unsupervised learning hmm you kind of double-click on that and you give a little bit more detail there yeah sure so unsupervised methods one of the big things I tend to do is group things together automatically clustering or hierarchical clustering or various kinds of methods that bucket things together so I can't give out all the secret sauce the hair that part of it yeah that's a part of it involves that and part of it involves other steps that can actually identify what the intents are of those clusters automatically so as a first pass this used to have to all be done manually are we call them speech scientists and also our data scientist would have to go through the data to figure out what the intents were in a large data set as well as interviewing subject matter experts at a company now is the first pass but today we've managed to cut down a process that you know use take hundreds of hours down to a few days months down to a few days by using this new process that we call intent discovery whereby you can bucket the data and then automatically identify what the intents are in the data then you can use those data use those that bucketing to automatically label the data so you're gonna first pass that like a label data set and bootstrap a model or train a model off of that put a model online now the idea here is that the system won't necessarily have the same accuracy as it would with a hand-tuned system so we're spending a hand can just simply 85 90 percent accurate one of these bootstrap systems might be like 65 70 percent accurate but then what we do is we put that system online and for all the questions that the VA or the bot virtual stone or the bot doesn't know we can pass off for one turned what we call hidden agent a person a human in the loop they can then check what the intent was for that and send it back in the system so the user ends up with this seamless experience of they're just talking to a bot and B with like a five-second delay when it goes to the person or 10 second delay where it says hold on I'm checking with you know checking with one of my partners then we can use that data then train in the future and then we won't have to have that you know that missing date or that loop in the future that makes them good it makes a ton of sense it I'm wondering does this create a new business model for nuance where we're in previously I'd imagine you you selling an enterprise some set of technology whereas now it's you know the technology but also the service like are you providing the agents really does this or are the you providing a you know a console that they can have their own virtual agents doing the checking yeah we're both oh yeah a few years ago about a year two years ago we acquired a company called touch Commerce which is a live chat platform they provide a whole suite of live chat they also you can provide the agents to you or they can just provide the interface to you so we have that which is really cool too because now we can if a VA doesn't know if a virtual assistant doesn't know the answer a chat pod doesn't know the answer we can we could if a company wants to partner with us transfer that way transferred directly to our live chat but we also work with lots of companies who have their own lab chat platforms a big one at Salesforce they provide live chats with really popular one these days transfer to those agents we have a console we can provide to allow companies to have this to use their own pool of live agents to do this human assistance step or we can provide the software and the agents of all different ways of working on it's very you know we can customize and call different ways with that not to be too salesy here but you know we have all them ways of working on it and yeah it is it is a new kind of a new line of business a new way of thinking about it okay or what else Oh targeting that's another big area so when do you you have a chat window actually pop up you know I think based on the way it's implemented now always right yeah and it seems like and when I use e systems I just didn't know where I'm a reason before I flicked a lot of times these systems it's just this a clock or something right and like if you put on the page longer than 30 seconds it pops up if you're visiting the site on your mobile phone do it you know immediately secure everything else yeah see there's the USB worked out yeah we we have a targeting engine that's constantly undergoing new prototypes and stuff but that's aimed that like when are you interact with the user how do you interact with the user in what way what language do you use what even down to like the colors and stuff on the interface and that targeting engine is based on all kinds of things like how long has the user put on the page it's certainly one of them but what have they put in their shopping car like they put something in their shopping cart and walked away so there's all these different inputs of the system that you can use to then tune when you're going to pop up that you know pop up that chat bot to ask if they want to talk to an agent or talk to Oh Nina's the name of our mo just to talk to Mina talk to a chat bot and assuming it's machine learning driving that ultimate target decision how transferable are the models from one customer one customer one website to another like are you training these models on a customer by customer basis or is there you know you have industry models or is there a defense model right out performs just show the chat bottle yeah so there are we do have generic generic setups you know for various verticals but when it comes down to it a lot of this is a very difficult subject right now these days for us because a lot of our customers really feel very strongly they don't want the data share there anything what the models shared you know so it's a new it's it's a it's even for when their chat bot pops up on their website anything it's it's a it's a very tricky area that I think that our company I think a lot of companies today are having to deal with which is like how do you both be a competitive AI machine learning company while at the same time protecting the data of your customers in a way that they're comfortable and they can like dealing with all the kind of compliance issues Duke's a lot of our customers are banks so maybe that legally you can't you know unless there's actually specific consent for example we deal with a government agency in Australia and they I was reading one of their privacy policy document earlier today and I was saying look unless you have explicit consent from a user hypothesis I had consent to use this or you signed some check box or something they can't use that data for anything else tuning a model even you know so this is really I think we talked about all you know all the exciting stuff in AI today and all the amazing things that are happening when it comes down to it for a lot of businesses these are the real problems today it's not how do i scale a deep learning model or how do i productionize this system and get it working on you know going from one cluster to a thousand computers or whatever it's like the legal problems how do you actually deal with the data in a way that's safe for the company and for the users you know it's really what the huge stumbling block right yeah I was just reading I forget what I think it was I think it was a article on someone his newsletter or something like that that was talking about how there's huge gray area around copyright and datasets and you know basically everyone who's using for the most part these public datasets is kind of flying under the radar but there's this you know potential exposure where there's no established you know precedent around you know the extent to which copyright on a data set flows into the model for example and so potentially someone using this copyright data set to train the model could be creating a model that's you know has potentially owned or has some copyright liability with someone else so that's an example of this kind of met a concern that that your customers are thinking about do you have any perspectives on the you know the rise of the consumer voice interface devices the Virtual Assistants your Alexa's and your Google assistants and things like that yeah it's interesting I think a lot of that stuff it has to do with their API is you know with those customs we right now we have our chat BOTS integrate so it's like Alexa integrates with Google home but there's certain things we can do and certain things we can do so some think one thing we would really like to do a lot of our customers I guess center banks right and what Thanks this voice biometrics has become very popular recently but one thing we can't do right now is voice biometrics over Google and Alexa because they only send you text so at this point there they don't actually send you the audio second they do all the ASR for you speech recognition and the TTS for you the Texas speech and they have no way of sending out those signals so I think this is something that we've had huge strides in terms of the openness and the compatibility these platforms with Alexa and now with Google home but for years this was a big problem with Siri right with Apple if they had no ability to do 30 third-party integration you couldn't say to Siri like you know play a song on Spotify for me so this is this is definitely like a big topic for new ones right now this idea of we call it cognitive arbitration where you have agents that sit like kind of uber agents that sit in the middle of all these different IOT systems and coordinate those for you and bring that data to bring the system sealer those api's together to talk to these over distance I know this is also a vision of Amazon's as well with being able to sit and like have their agent that has goes out and there's always different skills that they come to so this is kind of this is the newer topic for for new ones that I'm working on more recently is how you can use intelligence and reasoning and other kinds of machine learning to build agents that can sit in between all these different IOT devices and talk to each other and do you see is there a role or any emergence of like standards or standardized approaches like for example you know there's tons of work that's been done around federated identity in the web and now this is you're introducing a whole other layer around biometrics does any of that kind of work transfer here do you think or we just like to or no no no I think it will I think that this is word in very early days here so it's basically like I don't think the very beginning of the internet when these kind of standards were being we're being built out you know like what's HTML and how does it work that's where we are today with AI so this issue's the thing that you brought up with the data that those kind of issues haven't been solved yet like how do you there's no you know apache license for data at this point you know whereas that stuff you know to season software engineers and product matters now it's just kind of like second language they know what the licenses mean they know which what they can reuse which code can be leveraged which can't what you had to declare what you don't yet cetera but that stuff doesn't exist for data bits an interesting point you know I hadn't thought about it but yeah I think that that'll come about and similarly with with these standards around interoperability I think that's something that we have teams that nuance who are working on and trying to kind of reach out to different partners because this is not a problem that's going to be solved by any particular company just in the same way that like you know JSON standards can't be solved by Google you know they can't just make a really great JSON parser and everybody take it and put it in their browsers or put in their systems it has to be you know standards based and community based so I think we are we are thinking about these kind of interoperability standards at this point very early days though mm-hmm right any other things that you covered in your talk that you want to share with us yeah there's a couple of things like in the front matter the cover the oven talk that I was talking about which is that in this world of chat BOTS and virtual assistants it's surprising by how much of it isn't AI and isn't machine learning and we really there's a lot of companies they'll have like you know I can dot AI and their name but everything is still like based on regular expressions and rules and that sort of stuff so more people or people yeah are people big time that's that's not always necessarily a bad thing but it's it's often part of the company's strategy in terms of you know having everything the beginning be run by humans as a way to gather data and then over time learn from it but that's something we try to help educate our customers about is how much of this actually is AI and machine learning and how much of it isn't today most of the AI and machine learning for us happens on the language understanding side of things so when the input comes in be able to categorize it using statistical models and LP models in order to route to the right response you know to give you the right response to a question we can also do things like energy extraction and extractor concept so if you're saying something like you know I'd like to order a pizza or I'd like to order a large pizza a large vegetarian pizza with pepperoni be able to identify both that the intent was order a large pizza but the toppings and the size and that sort of stuff pull out those categories that kind of technology now some of that stuff is standard but there's still quite a few you know different platforms out there for doing chat bots that don't offer that kind of you know that level of a machine learning NLP but a lot of that played with that stuff in the past like API that AI it's a very manual and tedious process to build out those those entity trees exactly and so your it sounds like what you're describing is a way to learn some of that from the data itself yeah exactly so that the when it comes to when it comes to our intent discovery and bootstrapping process for learning the intents automatically and building out kind of you could call it an ontology the group of the intents then the various concepts that are associated with those intents but yeah we try to just like encourage people to be discerning and try to figure these things out when you're looking at platform today I think the next frontier is going to be on the other side though there really isn't any system on the market today including ours that can automatically learn how to answer the questions especially if they're complicated back-and-forth dialogues so if it requires like a back and forth conversation those tend to be you know built out manually either as an enterprise and the enterprise you know by talking to subject matter experts or when people are doing it themselves by figuring it out themselves that way that I think is different this is the next frontier for this technology is learning how to answer questions and dialogue and a lot of now there are tons of people who say they can do that there are a lot of questions what they tend to do though is you have to feed the system question-answer pairs mm-hmm and then it can learn to map new questions to those answers hmm forgot it this question answering but it's not dialogue that's if something exists sort of an FAQ can you identify if the question takes a different format that it actually mapped to this question that's in the FAQ that's about a format I mean it could may not be I think it might be like a list of a thousand questions and answer pairs or whatever but I mean cutting-edge research today I know from there with like the Stanford squad data set that can I do yeah Q&A data set where it's trying to answer questions on Wikipedia articles Newsline pretty cutting edge research at this point there's research teams across the world competing in this kind of thing but even that still is focused on you know the answer is in the article you know we're talking the kind of stuff he wants is working on today well that is one thing I forgot to mention we've got a product a project we're working on now called mean and knowledge where you can basically push a button to ingest a website or a set of documents then start doing questionnaire answering on it yeah and it it does leverage from the technology that's being used to answer like those to work with that squad data set but it also leverages technology from information retrieval search so it combines a few different areas machine learning as well as NLP and question answering so that and I like to think of this as low-hanging fruit really because these kinds of questions that you actually have just a one-shot answer to those are the candidates for automation today the things that we still really are still very early days in the research in is how do you actually learn a back and forth conversation that requires multiple questions and feedback going through a conversation with somebody from data mm-hmm I think it's the next frontier in the case of this of Nina knowledge so what degree is it you're doing like transfer learning off of a model trained on squad and applying that model to the website that's being ingested or is this process including like training up a new model on that website yeah the process is yes sir Klee we start from scratch you know on each one each time we haven't we haven't gotten transferred longer like that into production yet today but those are certainly areas of research or working on is so one area where I don't know if I'd I don't know if you'd call it transfer learning or not but you can certainly do things like learn word vectors you know from one data set and apply to the earth that's that is certainly like a form of transfer learning most people thing about transfer loading I think today they're thinking okay I built this big multi-layer neural network and I'm extracting a piece of it and then they're using another data set I think it's still early days for that sort of stuff but certainly when it comes to word vectors or other kinds of vectors paragraph vectors document vectors etc that stuff can be transferred and it can be very helpful to improving the quality of a model pretty quickly where are we in terms of you know we've talked a lot about identifying the intent of you know an utterance or something that's typed into a chat bot but where are we in terms of you know more you make more realistic kind of dialogues that have multiple intents or hidden intense or you know things like that you know and this is maybe a slightly different direction for the question but you know for awhile we've talked about the IVR systems being able to identify the emotion and you know change or escalate the way the call is handled based on the emotion you know when I'm frustrated I try to make sure the IVR knows I'm frustrated and it never makes a difference like is that you know I really love this kind of this question particularly the emotion part of it I really love that question because we've been working on it a lot lately of what do you do with text at sentiment right so that's basically tend to be in the seminary although there are some models now that classified things into like the five base emotions anger happiness frustration senses like these you know old models of this there are some models that claim to be able to do that I don't know how accurate they are but sentiment that's at least one that you were pretty accurate we can get pretty accurate on that for the most part elite global sentiment of a sentence that doesn't always sufficient because there can often be two sentiments in a sentence you know I love this but I hate this what are you there so getting off on a tangent but the reason I think this is really fascinating is because the real question is okay great so what do you do with it what do you do when you know someone's pissed off what do you do when you know someone's frustrated and that's a really hard question because it tends to be what a company when a customer wants to do is to figure out a way it's about containment right so how do we handle this without escalating to a person because that's going to a person is expensive but when you have a frustrated person the only thing we know how to do to them were the only thing we tend to do today is escalate to a person so I think what it's going to come down to you is understanding that there these aren't black and white things that there's just like we have no confidence scores and probabilistic models that give you a gradient of how sort of of confidence and say you know a right answer there's gonna be a gradient in terms of like sentiment so if a person is saying something like you know I'm having trouble with this then you map out to say okay I'm sorry you're having trouble with this I really you know wish you were doing a better job there but you know can I offer you this you know tool this troubleshooting step you know it is troubleshooting dialog whereas the person is really upset then yeah you probably just want to say maybe you want to offer them at least to say would you like me to have a technical support person to contact you directly you can usually like hold on them will call you or do you want to proceed with art you know troubleshooting online yeah so I think although we still aren't there under percent with recognizing these things the question like what do you do with it once you have recognized it might be even a harder problem to deal with no because great we can get like we end up with a hundred percent of 99% accuracy of knowing when some how angry someone is about how do you what how does that help you what your customer does not want to escalate all of those to you know their most senior rep or manage that right so how do you know what do you do with it for an enterprise and how do you actually use that data in a way that's interesting again this doesn't actually connect an AI problem that comes on it's like a user experience problem or a design problem yeah I wonder if there's like you know company is all over of glommed on to Net Promoter Score as a way to measure customer satisfaction that I wonder if there's like a net ivr anger scores yeah yeah this our customers ask about this stuff a lot so it's really it's it's on tuck the front of people's minds but you do to that you can you go down that dialogue when you go down that path it's you know okay so then what yeah awesome so Paul thank you so much for taking the time to sit down with me I'm looking forward to catching pieces of your talk and how you know there ways that folks can kind of find out about your research or connect you on Twitter or anything like that yep my twitter handle handle is my name with the first initials switch so okay salt pepper tau L pepper on Twitter that's pod that's way again I mean these days okay nice nice well good luck with your talk tomorrow thank you Thanks [Music] all right everyone that's our show for today thanks so much for listening and of course for your ongoing feedback and support for more information on Paul and any of the other topics covered in this episode head on over to Twilio comm slash talk slash 52 for the rest of this series head over to Twilio comm / AI SF 2017 and please please please send us any questions or comments that you may have for us or our guests via twitter @ 20 written or leave a comment on the show notes page there are a ton of great conferences coming up through the end of the year to stay up to date on which events will be attending and hopefully to meet us there check out our new events page at Twilio comm slash events twi mla Icom slash events thanks again for listening and catch you next time [Music] you