From Particle Physics to Audio AI with Scott Stephenson - #19

Key Takeaways

[Music] hello and welcome to another episode of twiml talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam charington once again thanks to everyone who sent in their favorite quote from a recent podcast another batch of stickers got mailed out this week and please keep those quotes coming we've had a blast getting your perspective from each talk if you're new to the twiml family you too can get a sticker just send us a comment via the show notes page tweet us at twiml AI or Sam charington or share your quote via a post on our Facebook page before we jump into Today's Show which I'm sure you're going to enjoy I'd like to take a moment to remind you all about my upcoming event the future of data Summit which will be held May 15th and 16th in Las Vegas Nevada I'm really looking forward to the summit and I hope you can join me there you'll hear from industry leaders and Technology users on how they're taking advantage of emerging data Centric Technologies like iot blockchain deep learning and more you'll learn a ton I'm continuing to add new speakers to the lineup including Jennifer preny from Walmart labs who will be joining us to talk about what she calls data Mixology Jennifer currently leads a team of data scientists and Engineers working on improving the online experience of the Walmart customer by integrating Walmart stores data with e-commerce data she also manages the metrics and measurements team there a group in charge of creating metrics to measure the impact of all new features and algorithms as well as solutions to automatically control and manage all machine learning algorithms in production she'll be talking about the importance of finding infusing data to making machine learning predictions in production and describing the evolving platform that Walmart's put in place to facilitate all this Jennifer is just one of a whole lineup of great speakers I've got for the summit and the Summit is just the first two days of the inop ITX conference which offers a week full of great educational content for folks interested in data and analytics in addition to a dedicated data and analytics track at the conference there will also be an AI theater and demo showcase dedicated to the practical application of AI deep learning and machine learning for traditional Enterprise tasks such as Dynamic pricing and insurance perfecting the retail customer experience optimizing job costing Transportation expenses and more case studies will be presented by businesses employing successful AI strategies and will provide strategic guidance and suggestions for attendees to utilize in their respective businesses accompanying the AI theater is the demo showcase which will provide a Hands-On lab demonstrating realworld AI Technologies if you have any questions at all about the summit don't hesitate to reach out to me and to learn more visit twiml ai.com futureof dat the code I provide on that page which is simply my last name charington provides twiml listeners with a 20% discount when they register for interop ITX and now about today's show my guest this week is Scott Stevenson Scott is co-founder and CEO of deep gram which has developed an AI based platform for indexing and searching audio and video Scott and I cover a ton of interesting topics in this talk including applying machine learning techniques to particle physics his time in a lab 2 miles below the surface of the Earth applying neural networks and deep learning to audio and the Deep learning framework core that his company's open sourced I know you're going to love this one and now on to the [Music] show hey everyone this is Sam charington and welcome to another episode of this week in machine learning and AI uh this week I've got Scott Stevenson on the line with me Scott is co-founder and CEO of machine learning and AI startup deep gram Scott say hi hi thanks for having me Sam uh it's great to have you on the show I think the best place to get started is to maybe uh have you spent a little bit of time talking about your background because you come out of physics right not uh audio speech uh and all that yeah absolutely Ely we are um you know deep gram is an audio AI company but uh our background or most of the technical people in uh deep gram our background is in particle physics or at least some form of you know deep physics and um for me and uh my co-founder Noah Shetty we both were doing uh particle physics before we started deep gram which was uh look searching for dark matter and this is about uh you build an experiment that sits uh around miles underground essentially you know like one mile two miles underground the deepest labs in the world are 2 miles underground and we were our experiment was in the deepest lab in the world in western China it was called Panda X and um we built this experiment um over the course of four years with a lot of other people you know with around um two dozen other people and with the help of the Chinese government and that's that's an interesting story um but yeah that and and the techniques that we learned in building this experiment um we figured out we're like really applicable to audio and the reason is that physics is very um at least particle physics at the very hairy edge of research is still analog and you you still read um you still look for particles using analog detectors they're called photo multiplier tubes and the signals that you get out of these are um just a a waveform it looks a lot like an audio waveform but it's at a much higher sampling frequency and um those those waves uh that you that are contained in the output of these photo multiplier tubes the signals that are in there tell you the signature of the particle that you're seeing so you might see a single Photon you might see a a big splash of a muon in your detector um but it's all contained in those waves and so what you have to do is extract that information from the photo multiplier tubes and then make a make a guess you know did you see dark matter or not and so we got very good at doing that um and it's actually really interesting because the the state-of-the-art in particle physics is essentially a lot of humans sitting down figuring out how do you extract information from these signals and you know maybe you've heard of uh like the higs bosz on being discovered a few years ago that a lot of people yeah and uh that was done by thousands of scientists sitting down and saying hey I really want to find the higs BOS on how how do I how do I make how do I make cuts um in my data how do I process my data to figure out or to make my signal to noise you know good enough to find these particles right and um this is actually a lot of this um hard manual labor of figuring out these Cuts is um is done by U machine learning now so people will build boosted decision trees or kind it's that's not necessarily the realm for neural networks um but uh boosted decision trees and other um statistical machine learning techniques people are figuring out how to sort of automate this but that's like the world that we came from essentially and we were on that you know we were in that area thinking like man this this sucks let's let's automate this you know how do we automate it and um because I I feel like a machine already just going through like here's another plot did this work well did that not work well like what went on and it's like this could all be done with a machine and so uh a lot of the people that are in deep Gram now we were all sitting at University of Michigan thinking you know how can we make machine learning uh or how can we Jam machine learning into physics particle physics and then like extract something out of it and did you how far did you get down that path did you actually complete the jamming or did you end up uh leaving and and going off to start deep ground before you fully applied ml to the particle physics yeah so it's it's um varied for the different people in deep gram so I did finish my PhD I finished about two years ago and uh Noah who is my co-founder uh went to a PhD program at Stanford like actually didn't even start it um he he came out to the Bay Area and uh we were working on deep gram at the time and we got a little bit of funding and he was like you know what let's not do this physics thing um and anyway uh but the a couple other people in deep gram um either finished their PHD or you know got a masters uh and left after that but uh we were definitely successful in uh sticking machine learning into a particle physics uh analysis Pipeline and uh in particular for dark matter what we did is reconstruct events uh in a 3D uh position uh the reconstruct the 3D position of the events that are happening inside so the the way that this works I mean the it's like not to go too deep but uh a particle detector is just the at least for dark matter it's just a tub of liquid and it's a tub of cryogenic liquid so it's very cold but and it's put 2 miles underground and it's under a shield and it's all very you know like particular but it's all it's just a tub of liquid and that tub of liquid has a lot of um atoms in it and those atoms happen to be in this case Xenon and uh you you choose Xenon because uh it doesn't interact with other things it's a noble gas so just like a helium or neon um or argon it doesn't really interact it doesn't form molecules it doesn't really have like a chemical Decay to it nothing like that um but but it's really big and scientists surmise that dark matter particles like really big particles essentially so you find the biggest non-reactive particle that you can you put it into a tub and then uh that tub is just sitting there with um tons of atoms and it's just waiting for a dark matter particle to hit it it's just sitting there just you know that its sole purpose in life is to get hit by a dark matter particle and almost none of them have a dark matter particle hit it um but but that's what's happening and those uh those signals that that atom that got hit as soon as it gets hit it uh flies through the other Xenon atoms that are sitting around in this liquid and it creates a puff of light basically and stir up some photons essentially and it's like 10 photons you know it's a very small number um but you have these detectors that can detect the single photons and those single photons are bouncing around inside your detector and they're finally getting picked up by these single pixels essentially and you need to figure out like where was that event right okay and so you're talking about like really low statistics and pattern matching and how do you make some kind of algorithm that says hey this is what the pattern that I got where do I think this event was and that's that was really successful and um compared to what a human can come up with essentially and the other is determining the energy of the particle how how big you know how big of a splash was it that's a little easier to do but uh machine learning you know made it a little bit better and what about uh boosted decision trees lent themselves to application for this problem domain yeah so boosted decision trees one thing is that they're like fast to train so um that that was really good they also when you have a low a low number of Statistics or at least a fairly low number of Statistics um they can do pretty well um compared to like a neural network that would overtrain or something so we were very hot at the time on doing boosted decision trees for that type of uh problem and I still think it's actually very good um and uh people are still using this you know so so yeah it it lent itself very well to doing that uh the the con the Reconstruction and it also lent itself really well to doing um uh a determination of whether it is a signal or background was it a dark matter particle or not because actually because the signal that you see in this photo multiplier tube in the waveform it actually does look different it's like a little bit noisier and it's a little fatter and things like that but it's only kind of statistically noisier and statistically fatter you know and so these machine learning algorithms can do a better job um than our just standard hard cuts um a human can still do better than the than the machine learning algorithm you can look at all of these but you don't want to look at billions of events interesting interesting do you have a sense that there are maybe more uh that physics people with a physics background are somehow disproportionately represented in the machine learning and AI Community I I'm only working on a small data set here sample size but I recently interviewed Josh Bloom who uh is an astrophysicist and did a an AI startup have you seen that at all yeah I I I do think so and um I I run into a lot of people that have a phys phics background and I think the reason um that these these two fields are kind of coming together is that AI is kind of just information physics the way you the way you try to solve a problem is exactly the same as how you would solve a particle physics problem like get a ton of data try to figure out what the trends are try to see what the outliers are like how do you how do you actually tackle a problem it's identical I feel like I'm doing the exact same thing as physics but I'm just doing it with voice now super interesting um so before we get too far from it you mentioned uh a lot of your work happening in China uh and that there were some stories there uh how did how did how did that happen what's the connection yeah this there was just an upstart uh experiment as I was starting graduate school this experiment was just sort of being circulated as a possibility it wasn't even there wasn't really anything going on yet but people people were like hey there's a spot underground in western China it's 2 miles underground it would be the deepest lab in the world and it's not a lab yet but we're trying to petition the Chinese government to turn it into a lab so that we can put a dark matter experiment there and I was like I am so in you know this whatever whatever that is that you just said let's do it you know and and that's actually what it was I I went to my advis or you know my new adviser at the time I was like let's do it he's like you know this isn't really a thing yet and I'm like I don't care let's do this right and it eventually turned into a thing um it involved um China is an interesting place so I'll say that um but the at first the there were failed talks to turn this into a lab and the they went something like this well okay to back up just a second like why was this lab there anyway or why was this uh spot there they were building the world's tallest hydroelectric Dam right next to it and they have all these tunneling machines and these mountains are made out of marble and Marble's really easy to Tunnel through so they like swiss cheese the mountain you know they just cut a whole bunch of holes in the mountain and um there just happen to be a tunnel that is 2 miles underground um and this is where they were diverting water through so it's it's actually a new type of hydroelectric Dam where they have a really tall Dam part and then they extract energy there and then they go where there is a river that would go around a mountain like a long distance around a mountain instead they just tunnel through the mountain and so the the mountain itself is now a secondary Dam essentially oh wow yeah you can look this up it's the Jinping one and Jinping 2 Dam in China and it was just completed like a year or two ago um but yeah they that's what they were doing and so we were like hey you know this is a great spot to do it and we went there and said hello hydroelectric damn company um we're like 10 scientists and we'd love to put an experiment down here and they were like No And so we were like hm how can we work this a little better and we were uh in In Cahoots with uh Shanghai jao Tong University which is you know one of or a very um well-known Technical University in China and uh we went to them and well we were um in collaboration with them and so like along with them and the leadership there they they said okay we'll talk to some people essentially and so the the educational system in China has a ton a ton of power and so they went to uh the hydro the the like leaders of the educ education Community went to the hydroelectric Dam company and said hey do you guys want to do this and they said sure So So I have some sway there it always helps to have the right people involved yeah absolutely and so so that's how that all got started and from that from that yes moment to having a lab to actually work in it was less than N9 months they had to you know get Dynamite blast out a lab like turn it into an actual thing with cranes and uh like yellow railing so it looks like a James Bond layer you know they had to do all of that and they did it in in less than nine months and we were in there building our experiment right after that and were you physically uh in China in this lab or virtually connected to it uh from Michigan so I I would physically I would physically go there like um four months out of the Year essentially okay yep and that was again like for me I grew up in a really small town in Michigan and um I I kind of figured out that like the world exists you know outside of like a 3800 person City or city I shouldn't say City you know Tiny Town and um I you know was it I went to you know undergrad in in Missouri at University of Missouri St Louis and then I went to graduate school at University of Michigan and the the world is just getting bigger to me you know and and then now I'm like in China halfway across the world to toally different people totally different uh culture and I was loving every minute of it I thought it was great and that's awesome yeah and there was there there was so much to learn um from being over there but one of the biggest is that uh they have very few roadblocks essentially if you want to get something done and you have the resources to do it you will get it done very quickly and and so that's why we could start the experiment uh well you know that's why they could build a lab in 9 months that's why they could build a Stam in 5 years that's why um we could start our experiment like design the experiment build the experiment put it in run it Etc do all of our data analysis all in under four years you know because they they were just like so helpful in removing roadblocks so so that was that was really nice to see coming from the US that's awesome that's awesome you uh so you worked on this project finished your PhD um connected with your did your co-founder was also from Michigan so you guys each other before yeah we were working on that experiment together um he's a decade younger than me so so essentially he came into University of Michigan at like 15 or 16 years old and I was you know like an old graduate old jaded graduate student you know at that point and uh started working in our lab and you know he is just very good and we hit it off and would spend a lot of time together um outside of physics too like mining Bitcoin or building drones or whatever you know just kind of having fun and doing technical stuff together and yeah yeah uh and so you guys uh you guys started this company the you you talked about how the problems were similar but how did the what was the Genesis for you know the idea behind what you're doing at Deep gram yeah so so Noah definitely has to take credit for this my co-founder so I was a little naysayer at first um where he he wanted to start recording his life so he built a a little device out of like an Intel Edison um or a Raspberry Pi one of these that it just had a battery Wi-Fi antenna microphone and a you know processor on it essentially and he set it up so that it would record 24/7 and basically every minute make a new audio file and just sort of dump it into um storage and then whenever it came in contact with Wi-Fi it would upload all of it so so essentially he was backing up his life backing up his audio life at least uhhuh and I was like what you going to do with all that you know you're trust me your life's not that interesting you know in real time you're not going to go back and listen to it again right and he's like yeah whatever right and so he recorded like two weeks of his life uh you know several hundred hours and you know you're sitting there at the end of it thinking like oh yeah okay now this is a real problem how are we going to go back and find anything that we want to hear and so we started we did the first thing that you would think of um hey let's turn this into text you know speech to text and um when that didn't work very well because uh this that's just like the state of speech DET text essentially where if if you have a microphone that is not super high quality and it's not right up in somebody's face and the person is not enunciating really well then you're going to get pretty bad word error rate meaning you know it's not going to be that accurate right right and and so so that's that's a situation for like almost all of the world's audio it's not great audio um but in this case all of the audio was like that and really like okay this speech detex thing is not going to work um can we do something better maybe maybe somebody in the world out there like Google hey they know how to do search they know how to do audio maybe they made some kind of API or something that can do this audio search right and so we started playing around and looking there and we just couldn't find anything and uh we found papers actually papers from like 2008 2009 where Google wrote about this type of thing like doing search and audio um their technique was to turn it into text and then tried to search in it and it and it worked pretty well when it was on a reduced domain so they did like uh political speeches and yeah and they had Fairly good results but you know when you try to generalize it doesn't work very well because everybody speaks a little bit differently and political speeches are well recorded and you know everybody's enunciating you know so it goes pretty well then um but if you try to do this into General sense it doesn't work that well and we we emailed or um you know set up a Skype call um with one of the engineers on the paper and said like what are you guys doing now because um you know some years had passed you know surely you guys are working on this problem right and they're like no no no we gave up on that we're not doing that um we're we're just trying to make our speech to text better and um we're like huh okay so it's still not very good that doesn't help us um so so we started just working on ourselves we were like okay you know we know signals we know uh machine learning we know how to deal with lots and lots of data and uh let's see if we can make some kind of search engine for audio and over the course of about four or five months we went from you know very poor accuracy meaning like maybe 20% of the time you'll find what you're looking for to uh 80 or 90% of the time finding what you're looking for so that's like going you know state-of-the-arts 20% you're at 90% you're feeling pretty good um so so that's uh the Genesis of deep gram essentially we're like whoa this has a lot of applications and at the time you know we were coming out of Academia we're like hey students can use this for lectures and whatever but when we start to think about it a lot more like you know what the world like the big audio Source in the world the huge data Lake of audio in the world is like recorded business calls so like customer service calls and things like that just call center stuff and they're all really low quality and nobody knows what to do with them essentially and we're like man like deepgram could be a massive company um if we do this right yeah well I think the other killer use case is uh I don't know if you're married or not but my wife and I always have this convers ation where oh I just told you this or no you didn't say that or whatever if I had his device and recorded all of our audio and then could easily go back and prove whether I said that or not oh there's got to be a huge market for that I I love I love this use case for deep gram because yes I feel the same way actually we've run across a lot of people though that are like oh no this will make things worse it probably would because my uh I think I probably think I'm right more than I actually am much more than than I actually am yeah exactly but um but no I I think there's a regularizing effect there that if you know that there's something uh recording maybe before you say I know I said this you'll actually go back and check or something you know so I I think I think it could work out um and actually we do we do kind of think about this problem um I think that Marcus not really ready for it yet you know if you just put a recorder on everybody and said you know now we're backing up everybody's life and everything you say is recorded uh I think people right now are going to be super into that but it might come um and and I kind of hope it does like like for me I wish I I had it and we do have these devices like laying around our office from those days you know where I think man if I was just wearing that all day every day um then that would be great of course they're very heavy so we don't do that now but you could make it much smaller and yeah they're probably a lot lighter than they were then yep um interesting so you describe the the frustration that you were having with uh trying to do this and the state-of-the-art being turning the audio into text and then running a search engine against that text is the implication then that you guys are not uh using text as an intermediary yes that is true so um we we do build an index just like you would so you know this this is sort of the terminology that you would say before you built an index out of text right well we're not building an index IND out of text anymore we're building an index you know like actually out of activations in a deep neural network you know uh so so activations in a representation that's deep in a deep neural network um the best way to think of that though is that you're doing something kind of like searching through phones so phones are like the uh the alphabet of speech right um but they're kind of a human um there like humans have assigned those like those these are the 40 phones or these are the 53 phones like you don't have to do that you can just have it be defined by the data and um so that that's what we do but if you look at the if you look at what these match up too it's very similar to uh phones so that uh statement that you quickly uh went by kind of blew my mind a little bit you're indexing you're building an index out of activations deep in a neural network elaborate on that a little bit for us yeah absolutely so the the output of a neural network is is again just an activation essentially um and what you're doing is you're forcing it to like in the case of a speech recognition you're forcing it to guess like what is the word being said at this time um or or in the case of like doing some multiclass um uh uh Network that's trying to guess you know is this a handwritten zero or handwritten two or something like that then it's try toess decision yeah and so if you just sort of back up one layer from that it's still pretty close to to that output you know it's just some like linear combination and that has been stuck through an activation um to get to that output and so so you know you can you can think of um images where at the very top it depends on how you think of neural networks but at the at near the input of the of the neural network um there's it's looking for edges or round things or something like that right and then as you get deeper it's looking more for like faces or trees or something like that right and the same thing is happening in audio and now it's looking for things that are kind of wordlike or kind of like portions of a word like right and um so so those activations are what we're interested in and how deep are the networks that you're typically using and how do you know which of those activations to tap into or capture and uh is that static or dynamic does that change uh based on the utterances or how how do you figure all that out very very question um you can so there's several ways to go about this problem you can just say literally let the data Define this and um go to town um that that does work pretty well um you can you can sort of get a boost in accuracy a little bit of a boost in accuracy if you if you pin it at first like when you're training you might pin it to phones or you might pin it to some uh larger subset and uh then then remove that restriction so it's sort of seated that it should have learned something like this and then now it sort of builds off that knowledge um so so that's kind of how you can guide the network you know you're pointing it in a Direction that's approximately correct but then you relax that and allow it to pick the things that are that are working best um the things that we're working on right now to make this even more accurate are instead of uh a word Target uh you are searching for like or sorry not searching you're you're trying to force the network to guess like a topic or a part of speech you know is it a noun or a verb or something like that and you do all of these things at the same time so your network is sort of branched out and it knows you know I'm I'm trying to Guess Word fine I'm trying to guess the topic I'm trying to guess all of these things and so when you when you force that um uh restriction on the network then it it actually it it you can control kind of what the activations are are the information that the activations are holding you can kind of control that by the Target uh so how uh can you say how deep your networks typically are they what you would think of is you know very deep you know tens hundreds uh or more layers or are they relatively shallow yeah we're in the tens um in the tens category and um yeah we so we we have had a lot of success in that regime if you go deeper you need um a lot more fire power on the computational side and and you actually and you need um more engineering on the like where do I put put my skips and where do I put like everything like that you know architecture becomes more complex yeah exactly and so um so we're uh a little we're like an eight- person team so we don't yeah we we we try to um minimize some of those uh engineering bottlenecks as much as possible um and we've seen very good well I guess I guess another way to say this is that if the state-of-the-art is 20% and you're like in the 80 and 90% range with the networks that you have you know then like you could spend a lot of time turning it into like 92% accurate um by making your um making your network a lot deeper or something like that but uh we yeah we we don't Focus too much on that yet until we have the pressure um to do that okay okay and so with the network in the tens of layers then it sounds like it's you have a pretty good sense based on your architecture of where uh you know what layer is the phon name layer versus I don't even know names for the the other um we don't know names either honestly like this this is it would be really um this hasn't been explored as much as images you know there and I you know there are sort of reasons for that like where are the data sets to do this you know um also humans I think are not as interested um or I guess it's harder to to ingest audio than it is images images you know you like instant see it you're also entertained by the images that you see that you see but the um if you hear audio that you're not super interested in um you're like you you your eyes glaze over right away right and so labeling the audio data is expensive in time consuming and you can't just like click through a bunch of things and now you've labeled a ton of images uh you know you can't just click through a bunch of things and label a bunch of audio files you know like that work and so so those those are kind of the problems that audio faces but also uh you can't point to things and um you can't you can't point to things and say like that little Edge feature right there is important because you're hearing it right it's it's just not we don't have that visualization capability and so uh it's a little more challenging in that regard so so yeah we don't have names for this stuff either you know um there there are linguists out there in the world that probably have some some names you know but but we just look at like a 2d fft um you know a spectrogram of the audio and say like oo I can sort of see what's going on there and you just kind of treat it like it's an image and uh think of it that way okay uh so the other interesting um the interesting use case that occurred to me when I first saw your stuff was you know I've got a bunch of audio in the form of podcasts podcast interviews and uh I'd love to find a way to make that more accessible and one idea that I've had for a while is hey it would be great if I had a a bot like a Facebook Messenger bot or chat bot or whatever where you know you you pull this thing up and it's a and you can ask it hey I want to find Twi episodes about conv convolutional neuron Nets and it will you know look in its index and tell you oh check out you know episodes you know 13 and five at times x y and z uh it sounds like you're saying that your stuff could be a part of building that so what would the process be to you know to get to what I'm talking about forget all the bot stuff but you know just in terms of the voice search engine piece yeah that's exactly right the um the the me mentions is a part of it and topics is another part of it um and that's exactly the type of problem that deepgram solves so you have uh you have a lot of audio right and you have you have listeners that don't want to sit through you know 20 hours of audio trying to look for that one little tidbit um of information that they're that they're interested in that time and um that is what deep gram solves and that is actually um kind of a a a really interesting interactive experience once you finally do it you know so like when we first built deep gram and we made you know the search engine and it actually found moments where we were like whoo there I this is exactly what I was looking for you know it's a totally totally different experience then what you would feel if you had tried to do this with speech to text and so so yeah we're we're very into solving that type of problem and and the way that you do it uh with deepgram is we um we provide a an API so you go and sign up at deep.com and um create an account and there you upload your audio and you give us the query that you're looking for so if you're looking for convolutional neuronet or cnns or uh convolution you know put those keywords in send it to the API and you'll get the results back of all of the uh mentions that were in your audio and uh if you want to make that go even deeper to it's where it's more topic based rather than keyword-based then you like sort of work with us uh for a custom API endpoint to build a model that is more um tuned to the topics that you care about and that is essentially you guys working in a professional services or whatever capacity to build some um I don't know what is that what does that process look like from your perspective what are you actually doing behind the scenes to make the the topical stuff work yeah yeah good good question um the so the general stuff kind of works right off the bat hey you you want to search for a keyword you know we have a lot of data that we've already trained on and even if your keyword isn't in there we can find it because it's essentially a fuzzy search um but if you want to do this topic modeling uh then what you do is give us labeled data and so in your case it would be um I have I have all these episodes and they were the contents of these episodes uh you would have to know this um you know yeah we talked about convolutional neural networks in this one um we talked about you know recurrent neural networks in this one and boosted decision trees in this and you know we talked about productizing in this one and uh you just put those simple labels in and um then you train a u model it doesn't have to be a deep neural network either at that point because you have such a small amount of data you probably do some something like a boosted decision tree um but you train this model to do that topic prediction based on the search in your in your audio I this is actually getting really into the weeds but the way that it actually works is we utilize the search for that topic modeling so our our search is is the best in the world it's the most accurate um and and if you train a model with a Target that says here are these topics and I want you to be able to predict these topics and I want you to be able to generalize to other audio files then our our model that we're building what whatever it is but that topic modeling uh model it is going through and it's not doing any humanistics whatsoever it's sort of exhaustively searching all the possible phrases and saying like which ones help and which ones don't and it's recursively eliminating the ones that don't help and and it's arriving on things to search for that are very good and it doesn't have to be a single search term it could be 10 sing you know 10 Search terms and it also might be the non-existence of a term ter it's like you know if if you're talking about these 10 things and this other one then it's some other topic you know and Etc and I don't even know how it works right that's that's how it works and then in the end you look at the output and you say like these are the things it's saying search for and these are the things it's saying don't search for the or you know you don't want these to exist and that's how the model works you know and and it works extremely well so um so yeah that's that's sort of you know how the sausage is made and how much how much data do you need in order for you to to start getting uh interesting results both from the basic search and the the topic modeling or is the fact that you guys have already trained your model like well let's start with that question um is the fact that you guys have already trained the model on the data that you already have um does that mean that I don't need to have some uh some specific level of data about my domain in order to get good results yeah in in some cases that that's how it works where you know if you have sort of like a broad topic like is this about sports or is this about politics you know um then then that data is sort of already lying around and and incorporate it into a model but if you're talking about n Niche markets you know um then you probably want to supply something and uh then we'll you know build a model on top of that but that doesn't the data that you supply probably isn't the only training data that we're using you know we're just we're just supplying that it's essentially a form of transfer learning and then we're like changing the target of the model MH and so what am I supplying then yeah you would Supply like audio with some tags um so like here's an hour long audio and here are the 20 things we talked about or here are the 10 questions I asked or something like that okay and the the neuronet can just figure out what's relevant and what is and kind of map those to the tags exactly even given uh you know I would imagine a fairly limited or what how much how much audio and tags do I need to give the model for that to be useful for it for it to be useful to a like a consumer that's searching for something um because they're fairly error tolerant actually right like if you get if you get one out of two wrong that's not too bad as long as one of those is okay right as long as one of those is what you're actually looking for so you probably need something like um 50 or 100 um different labeled files and you'll get results that are similar to that if you have around a, labeled files then um then you get results that are more like you know 90% accuracy and these are hourong files or shorter it it kind of depends on what you're searching for but um they're generally in like the 10 to hour long range um you'll need more files if it's only like a minute long okay yeah oh interesting interesting and so the so then what you guys are offering is a a service um and that exposes an API but you also have done some work around uh an open source framework um let's talk about that for a little bit so uh why don't you walk us through what you've done there yeah we're we're extremely pumped about it it's called cerr it's open source you can go contribute to it it's on GitHub um but the the idea behind KR is that uh years ago you know around a decade ago gpus came onto the scene for neural networks um data started to become available and it became a really smart idea to start training neural networks on gpus and the people doing it back in the day knew something about gpus you know in order to accomplish that task you you had to have some like domain expertise in order to pull it off in a good way and uh Nvidia started to pick up on this and said hey uh we're going to make Cuda a framework that allows like C developers to get a little handle into the GPU and be able to train things that way or be able be able to use the GPU for matrix multiplication and things like that and so that's like another layer you have the bearer hardware and then you have like Cuda and uh then you have um like uh Brien cat and zaro making CNN so that it even works better for deep neural networks right but this is still all very um lowlevel uh type stuff and then uh you know out come other Frameworks like theano or uh cafe or um um tensor flow and those are another abstract layer on top they make it more human palatable right and they're they're much more palatable for the um developer or computer scientist that you know like really knows their stuff right and um and that and that's totally fine and they work very well um what we what we have found out though at working internally um at Deep gram is that working in those Frameworks still is very it's very slow for us to uh to try to do experiments essentially to try new model AR texures and then and then see the result because we are not we're not necessarily training time limited we're uh we would really love to be training time limited we were engineering limited you know and so it's like the time that you're putting in to sort of cook all of this up and make sure do all the error checking and whatnot to make sure it's like training the way it's supposed to we're like man why don't we just make some other framework on top let's just stack another framework on top that is more abstract that sort of doesn't care about the backend and it it doesn't care you know if you're using Theo or tensor flow or whatever you can just switch it with a flip with a flip of switch and um if if something's faster fine whatever but like all we really want to do is describe our model we want to say hey the input is going to be this audio then we're going to have a convolu a 2d convolution and another 2D convolution with a batch norm and then we're going to have some a few recurrent uh layers those are going to have batch Norm to and then we're going to stick those out out into you know a DSE that then predicts at every time time slice which character is going to be there which word is going to be there and so that's how we want to think of it we don't want to think about like python code and like how do we like put all of that in there and so so that's what um we started working on at deepgram is like how do we do that so that we can multiply our engineering effort essentially you know um I just go in and I change a few settings and and then start running my network again and then go off and do something else and that's that's exactly what is it's a descriptive deep learning framework and um you know we have examples on how to do image classification examples on how to do speech recognition um with a network that's very similar to Buu's deep speech model and uh like Lang uh language modeling and things like that and and it's all in this descriptive format um it's still not like don't get me wrong like deep learning is still not easy because um because there's a computational problem there and there's a data problem there like how do you get it into the format that you need you know how do you collect the data how do you clean it uh everything like that but the model part you know once you have your data and once you're all set up the model part is now like so much easier for us using cerr and so so we thought you know um this is kind of a competitive Advantage yeah it is um but we have gained so much knowledge by um using open source software and talking with people very freely sort of in the Deep learning community that you know this tool is been so valuable to us let's just release it like we're we're probably going to get more you know than if we just kept it to ourselves you know and we actually have like we we released um the curve framework and within like weeks somebody had added multi-gpu support you know and I was like whoa oh wow these people are serious right and so so yeah and and this is also another way to like um if you're if you're uh an AI company out there in the world and you want to hire engineers you know this is another way to find good ones and they could be across the world and so so it's like we're sort of we want to we want to be giving to the community we also just want to be um we want to be part of that uh conversation because I think we have a lot to add essentially and um I I think that the Deep Learning Community there's so much demand to um there's so much hype first of all um but there's also there's a ton of demand for on talent and there's a T of demand for the the type of critical thinking you need in order to solve these deep learning problems you don't have to be secretive you don't have to be like this is our secret soft that whatever no everybody is like Talent limited they're comp computationally limited they're data limited they're not like good idea limited so right so yeah the you mentioned it's it's declarative that's one of the the main things that's doing are you have you created like a DSL to Define your neuronet or or is it a different uh type of Express differently yeah okay great yeah great question and the way that you interact with cerr is you pip install it and um so so it's you know written in Python and uh you can use it as an API just as you would Caris or something like that where um you know you're programming in Python and that's just how you use cerr and that's totally fine um but sort of the DNA of your model is contained in what we call a curve file and that is or Json and so so that that contains like your hyper parameters your model architecture like how you want your data to be supplied and things like that um and a lot of that is boilerplate that is already um out there in examples essentially and so you just like use a curve file that somebody else has put out there in the world and then just edit it a little bit and for your purpose essentially and so yeah that's how it operates um just basically yaml files and if you want to do um like a deeper uh surgery than you do it in Python using the API H and does it uh can you uh does it sit on top of any of the other Frameworks or can you Le how do you leverage the work that's um being done on you know sensor flow and all the other Frameworks out there yeah absolutely I should have said that earlier it does um it supports Theo it supports tensorflow and it supports P torch and um maybe will support other backends um since since since uh like deep learning 4J is uh is working its way into Caris now uh you know we'll probably be supporting them soon so so yeah that's uh that's that's kind of how it goes um if if you have a high level API that would fit into something like Caris um then it'll fit into Kura as well okay yeah and we but we've already done the leg work for those three you know tensor flano and pytorch awesome and you mentioned uh you men men this already but it's not just for audio it's for images and basically anything that you're trying to use a deep neural net uh with absolutely we all that we do at deepgram is audio that's true um but the but the uh cerr you know and the networks that you can train using cerr are agnostic it doesn't matter you know you can do sequences you can do audio you can do you know text audio images you know it cook it up and you'll be able to do it um we just had a hackathon um last weekend and people were doing all sorts of things you know uh music edit editing videos on the fly so that you know your cat will look like a van go as it's um you know CRA crawling around and things like that you know like they're and they're using they're using cerr to do these things so so yeah it's um it's it's sort of agnostic H uh that's very cool very cool um and so you mentioned uh the by Deep speech stuff was that the the uh was that the inspiration for Kerr or to what extent do you is your uh is the Deep gram work um you know your product based on that research sure so our the models that we use um to to build our indexes and to ingest audio are extremely similar to The Deep speech networks um you have a convolutional stack and you have a recurrent stack and the target is uh characters or words um in the Deep speech case is characters but but nevertheless the the architecture is extremely similar um and so the networks that we Supply like incur or sorry the examples that we Supply in CER um are extremely similar to what we use um but but we we have tried to make networks that people are already familiar with like they can go read a paper and figure out how it works right so that so yeah and we'll you know we put that into the example file as well saying like hey if you want to read about the architecture this is where it came from so so that's yeah we we're not trying to um confuse anybody about like how it works so so we sort of stick to the things that you can go out and look at a paper for umam has has not written um a paper on what we're doing yet um it's kind of in the works always um but you know we we really would love to but you know that when you have when you have like businesses to uh you know to take care of and uh you know customers I guess to take care of and um and only you know eight people then then you that kind of gets thrown by the wayside right right are there any standout use cases for uh for this approach and deep gram anything that you're seeing uh as you know kind of coming to the for in terms of what people want to do with it absolutely um from from the business side um there's uh fraud fraud detection is a really big one uh where people will call into financial services companies and try to um you know try to get money from them essentially try to take money out of your account or use you know get a credit card sent to the wrong address or something like that so that they can take advantage of it and there are sort of patterns um in this and you can the these companies have you know uh M hundreds of millions of calls every year and they're trying to find they're trying to correlate these things and say like hey we know that fraud happened on these calls Etc can you like help us find where that's happening you know like every day people are coming calling in trying to fraud us can you at least give us an alert so that we can look at those harder you know and and that's just that's that's one of the channels um that that like provides a lot of value essentially to the to the world you know because if if there's lower fraud you know then uh everything becomes less expensive for everyone essentially um but there's also like a quality assurance aspect to this and and compliance aspect um again this is still in calls where you know are people just saying the things they supposed to say you know are they having good responses you know do customers have uh nice interactions um and having a the way that companies deal with this now is they pay humans um to look at maybe anywhere from 1 to 5% of the calls and in generally this is outsourced where uh you send them you know a random selection of your calls and then you say like tell us what happened in these calls and they'll report back with a rubric of maybe like 10 or 20 different things and the quality will be pretty low and that's like the only source of Truth for these companies about their customer interactions that happen through phone calls and so that's the type of thing that you know deep gram is trying to help with we we take like that QA data that you've sent out to have humans label we'll help you figure out which of those labels are actually good and then we'll build a model based on those labels to predict all of your audio essenti you know to predict the uh contents uh of all of your audio and then you can take your QA team or your comp team or whatever they're doing you still have hundreds of these people like listening to all these calls you point them in a New Direction you know so they aren't doing the same like rote thing over and over they're like actually using their brain to do things that humans are really good at which are like creative things like figure out you know a new way to to find fraud rather than just sort of listening and uh hoping that they detect it randomly you know um so we this is this is like um where deep gram has a huge impact I think or has a uh ability to have a huge impact is just automating that entire process um so for for the consumer side we are um we're not putting as much effort into making a product like like Google for the internet but for audio I I wish we were like actually this is a product and somebody should build it using deep gram but um but and we have built demos of this uh where where you just scrape a lot of YouTube videos and then you're able to search it um using deep grams Tech um but but I think um from a like company Health perspective like in other words deep gram not dying in two years um and not having to raise like 200 million in the process of that death um you know it could go a lot of ways but but essentially um I think that product is is is like available and it's something to test I think that um the just the greater human world right now is not necessarily ready ready for it at this moment but you know in the next couple of years uh that's we're going to expect we're going to expect that all of the content in the world is searchable that you know when I think of like a movie quote or when I think of something that I listen to in a podcast or when I think of some interview that I saw on YouTube and I think like oh yeah they talked about this and they talked about that I should be able to search for it and I should be able to find it and people will start demanding that soon um but yeah we haven't spent we did some um some market research on this you know um to to figure out is is this an area that we should spend our effort right now but it just isn't yet um maybe maybe maybe we'll start doing that you know in the next couple years but uh it it it's not our Focus yet awesome awesome um well this has been a great conversation we we talked about Kerr and that's open source and we'll put a link to the GitHub in the show notes anything else folks should know to uh look for or how to get in touch yeah you can you can get in touch with deepgram at uh on Twitter at deepgram Ai and or you could send a message to me um at Scott deep.com I am not shy about throwing my email out there so if you want if you want to contact me just just go ahead awesome awesome well thanks so much Scott it's been great having you on the show Sam I appreciate it thanks all right everyone that's our show for today during this interview you may have heard me mention my previous interview with Josh Bloom whose company wise. IO was acquired by GE to help them permeate machine learning and AI throughout that company if you haven't already listen to that show which was number five you should because it was a great one but even better you should plan to attend the future of data Summit because Josh will be speaking there on building AI products and running them in production really you don't want to miss the summit so check it out at twiml a.com and if you work on machine learning learning in AI for your company and you think you've got an interesting story to share don't hesitate to reach out I'm finalizing the agenda for the summit soon but I'm always looking for interesting user stories this podcast is full of great quotes don't forget to share your favorites for one of our twiml stickers you can share them via the show notes page via Twitter and Via our Facebook page the notes for this show will be up on twiml ai.com talk sl9 where you'll find links to Scott and the various resources mentioned in the show thanks so much for listening and catch you next time [Music] [Music]