Practical MLOps // Noah Gift // MLOps Coffee Sessions #27

Key Takeaways

i found out about noah on his youtube channel where he has got a wealth of information and i highly recommend everybody listening that if you are looking to learn anything about computers go and check out his youtube channel he's got so many different avenues you can go down and one of them is ml ops and he's just very very generous with the content that he provides and his knowledge so check that out today i am joined by none other than vishnu and as usual i say this all the time but i really believe that we are in the presence of one of the greats today with noah he is absolutely just uh just an incredible person and so that you all can understand how prolific this man is he is teaching courses at uc berkeley and northwestern and he's got at duke and that's just like a few of them that's not all of them there's a ton of courses he's got all kinds of certificates with all the major cloud providers and then he has been in roles from cto to head data scientist to general manager i mean the list goes on the bio on this guy i think i could spend the whole hour just talking about what you've done noah and it is incredible that you are taking the time to talk with us today and just uh dive into this ml ops so without further ado noah thank you so much for coming on here and joining us thanks yeah i guess that's the advantage of getting older because you you do a lot of stuff compound interest right yeah yeah and and if anybody wants to check out everything that he's done you can head over to his website which we'll leave in the description below or you can just go to noaa gift.com that's an easy one it is easy to remember so you've got all kinds of courses you've got all kinds of videos that you're putting out books i forgot to mention you are putting out a book what is it um for ml ops this year yeah there's there's a book that i'm uh writing currently that should be done in the next two three months called practical mlaps and that's with o'reilly and i'm the co-author is alfredo uh daisa who who also did um python for devops with me for o'reilly nice and so one of the i guess the main theme that vishnu and i were thinking about kind of framing this around is because you have your company right now right pragmatic ai labs and i love the title pragmatic i love that word in general and the idea of this o'reilly book also practical mlaps i think it's really easy to get lost in the weeds and go down rabbit holes these days and then the rabbit hole leads you nowhere especially when it comes to something like an emerging field like ml ops right and so i'm wondering for you let maybe you could just start off and give us your opinion on how we can stay pragmatic when it comes to ml ops yeah so i i teach i'm actually teaching today i'm teaching a course at duke in cloud computing where we we do mlaps in the course it's called cloud computing for data analysis and my recommendation to students is to to think about machine learning almost like like a mobile app or a database or you know some component where it isn't necessarily the whole thing that you're solving the whole problem that you're solving and i think what i've seen when i when i see a lot of interest initially into machine learning is you know i've had a spectrum of students and also when i when i worked full time was that it's easy to get caught up into kaggle for example or predictive value and there's a really good example of this in in the last probably month or so you know um the the documentary if you haven't taken a look at it yet called um was it the social experiment is that what's called social experiment uh and and they talk about how dilemma social yeah yeah so social dilemma where where there are literally i don't know i know i know at least two myself people that have phds from berkeley now i know three people that have pc and computer science from berkeley that work at google that work on things like recommendation engines and yet there are they served out more traffic to alex jones flat earth theories and q anon than they did to like new york times washington post wall street journal combined and so that i think is is an indictment of the approach that was used and i think it was an incorrect approach and it's that if you're so stuck up on the predictive power of what you're trying to solve and not thinking about the real thing you're trying to do that's how you get into a situation like that so i think in fact the deeper you go into into predictive power and you only stay there this is the result and i think the result is very bad too for society and so the practical approach is that is that you you you kind of treat all things equally where hey what you know do the customers that i'm building a solution for are they going to like the product you know is it ethical to build it i mean it's easy to get kind of hand wavy and say ethics you know oh you know but but i mean come on you're literally indoctrinated indoctrinating people to believe in flat earth and in fact we just saw you know that a bunch of people truly believe that they were helping overthrow the government and people died and that's that's not squishy that's that's a very that's a specific outcome that was focused on only you know the predictive power of a problem so i think i think it's really not giving the machine learning the center stage i think is is the solution to be practical because it's one of many pieces in a problem and and if you can't solve the other 10 pieces like you know user experience repeatability data drift you know all those kind of problems then you're really building something that even if it does have extremely high accuracy could be catastrophically bad yeah it makes complete sense what you're saying and it really comes to this realization i think that we're all having now which is it's not just how good a model is it's equally as important as how that model is used that is what the operations component in machine learning and mlaps is supposed to be how the model is used and you mentioned one thing about machine learning systems needing to look more like traditional software systems like databases and and mobile apps at least you know conceptually because of that need for reliable operations that's something we talk a lot about in our community on the podcast on coffee sessions it's it's it's kind of the mantra of our group my question to you is in your courses how do you teach your students on what to prioritize to learn when there's so much content out there about you know modeling and predictive modeling and the value of working on that versus you know maybe the tradition how traditional software engineering principles apply to machine learning is less clear how do you give them a framework for that in your classes so what i what i've gravitated towards is is is um telling students to to take a model that's already been trained somewhere so go to kaggle just download it and or or use auto mel go to uh you know google automobile vision train a model download it tf lite you know or go to apple has um what is it again the the the gui that that you can just click you can just drag stuff on to it um i think quorum core ml create email create ml right just you just literally just drop it on there and then just get that working get get the feedback loop working um before you before you get too deep into the weeds on the machine learning model because that's to me that's the basics it's like you know it's like if you're into running or riding bikes or swimming or something if anyone's ever run a marathon you know you can't just start going hey i'm gonna get six minute miles for 26 miles that's not how it works like you you you first can you even do 26 miles before you know and i think that's the that's the machine learning component of it is that it is if you first make sure you understand how to put it into a web app or put into a mobile app and then also you have a feedback loop where there's a where there's a continuous integration system as part of it that that to me is is really the foundation that's necessary and that is exactly what i teach in every course that i've built is that you need to get continuous integration can do continuous delivery set up and you really don't necessarily at first need to know anything other than it's a the machine learning model is a file or it's an endpoint then once you get that set up then i think sure you can go in and you know tune the knobs and get more in depth in it but yeah i would even say that it's interesting because there's like a there's a disconnect between a lot of people that are interested in data science they want to be a data scientist and then what happens though is that if they don't have a foundational knowledge of devops they could be really in trouble at an organization because they they require let's say a dozen people around them to get anything done and that's you know for anyone that's had let's say five years of work experience or more you realize that's basically the the kiss of death for your career right if if for you to be productive there has to be a group of people around you at all times that's not a good way to start your career so i would say the the the antidote to that is if you have an understanding of devops then it's going to be much easier to do data science at scale i i think the principles that you're teaching and that you're mentioning that you clearly impart to your students are great uh one of the community members in our group laszlo he has a great quote which is you know ninety percent of mla's problems are really our data science writing bad code and it's just facetious but it goes to show that if you don't necessarily have those you know those principles in place or have a good understanding of what that looks like in the context of software delivery you can end up creating more problems than you're really solving and and that's not the position you want to be in related to this on a principal level this this makes a lot of sense in how they work with machine learning how do you reconcile how early some of the tooling for these for these for these principles are and how you know perhaps even immature they could be considered how what do you what kind of tools and stuff do you tell your your your students to use in their courses yeah i i i'm a little bit skeptical of um of some of the tooling currently you know if you know i think there's some nice things about things like scikit-learn for example and that it's you know it's a couple lines of code and you can create a classification model or you know do clustering the the issue is that at the academic level a lot of times people assume that that's what they're going to be using in their job and psychic learning pandas is really not going to work for many companies it's just it's not a big data tool and so now you've got yourself in trouble is you you you assume that you can use those tools maybe you will but a lot of times you have to use a platform like aws sagemaker or microsoft azure ml studio or or google ai platform or spark or something like that so i think that the tooling is still pretty weak and i think that is that is an issue where i would recommend that if you're working at an organization that has let's say 20 or more people in it probably you should use a cloud-based tool you know or or a open source framework like spark ml just because you're gonna you're most likely gonna have less pain and suffering if you're dealing with data at scale if i guess if you're just doing um dashboards or you're doing you know ad hoc analytics on small data then sure the academic type tools like second learn are reasonable so that's one component of the question the other one is that i also am deeply skeptical that a lot of the work people are doing currently will they'll even need to do in the next two three years i mean most i'm i'm 45 so i've been around for a bit and what i've seen in my career is that everything is automated i mean in fact i worked as a teenager i worked in television and i grew up near um hollywood and there was there was editing systems that were linear editing systems you had to have two tapes to make a dissolve and all this i mean it's we're so far beyond that just in 25 years that it's hard to even describe the changes but if things became digital and automated and and now you can edit things on your phone you know you don't you don't need any of that stuff and you could do 4k editing on your phone but same thing with machine learning it seems really unlikely that all this stuff people are doing on kaggle for example will even be related to what they're do in the workplace like how could it possibly make sense that you're gonna do all this like hyper parameter tuning and like cleaning and all this stuff where there already are tools where you click a button in a transit model so i think that's the wake up call that many people that are doing data science are probably don't want to hear but should hear is that if you're focused on the the nitty-gritty details of like the technique and the technique is is very is is focused on something where it's it more than likely is going to be automated your skills won't be that relevant but if you're focused on the solution then you don't care how you got to the machine learning prediction or the machine learning system so i that's what i suspect is the automl components are going to come very fast and it just i don't know my intuition tells me from seeing so many different things automated in my life or the same with cloud like i remember i'm old enough to remember when when i had bosses that told me that cloud computing is a bad thing to do you know our company can't do it and they're like fortune 10 companies like oops can't do it you know cloud's bad see i hear the same kind of talk about auto ml it's like well i'm pretty sure you're wrong i'm pretty sure that we're going to click buttons and that's how we're going to do machine learning the quote that you just gave us i think the work we do now is not the work we'll do in the future i think i think that's a great that's a great point and i think it's something that's been mentioned at the margins um but not anything i've heard of such as much clarity as you've put it and i think i think i think you're right in large part i mean even when we look where we were five years ago with just machine learning right we were working with very very different frameworks i mean tensorflow and pytorch became popular you know now i'm in pre-2014 i don't think anybody was using them they were using things like cafe right and so so with that in mind and keeping that in mind i think i think it's a great point to really focus on the solutions and not just a technique and not just can you you know do this one step really fast uh which i think a lot of times we focus on yeah and there's so many you've been dropping quite a few bombs here and i've been taking notes because it's really nice to see the idea of like hey just figure out if you can run 26 miles before you even go out and try to get the strongest and the best equipment for marathon and you buy your like new suit and you download the app that's gonna make you like check your times and all that just like go out and see if it can get there and is the model doing what it needs to do and then you can figure out the rest and and like you say the the goal like not forgetting about the bigger picture is so it's so easy to say it right but it's so difficult to do when you start getting into the weeds on that and i think that goes back to the whole idea what i am assuming is the pragmatic part of this whole ai like being very pragmatic in hey does this work and how does it work and then asking some really difficult questions in the beginning like around ethics and and that stuff and i wanted to just mention that my neighbors told me today so i'm in germany my neighbors just told me i think they're going to close the gas stations and the supermarkets so go gas up on your stuff i was like what have you guys been watching how are they going to close the gas stations and supermarkets then how are people going to eat but that's what happens when you get too many bad algorithms recommending too many bad videos on flat earth and all this so anyways that was a bit of a a tangent there and i i wanted to to jump in sorry i cut you off vishnu i don't know if you had anything else you wanted to mention i think i think the the thing i was going to say is actually uh you know i'm actually a student in one of your courses on udacity on the cloud devops which is a great program and i'm really excited and and and able to apply those lessons every day um i think your point about the cloud providers and what they're doing with automl and these tools are spot on i mean in your opinion we talk a lot about this in the community and i know our listeners will be really interested in this if people have different gcp aws azure they're using different platforms you've worked across you seem to have experience across all three right the two degrees on udacity that you that you've put out what's your opinion on the relative merits of each and where have you seen particular strengths i think aws definitely is the leader and it's it's a pretty safe choice to to focus on aws uh and i don't see that changing anytime soon uh so i think that's a you know probably the place to start if you if you haven't started anywhere and then if you get a certification for aws you know they they pay very well in fact you could be a high school student and get a aws solutions architect certification and probably get a job for six figures uh which is you know interesting and i think not a horrible career pathway to to consider doing again i mean i'm teaching that in college but i i think we should be open to letting people do whatever they want and start making money early i think there's a lot to be said for that the or even you know working while you're in college if you want to stay in college but i'd say aws is probably the the number one now i think microsoft is doing some stuff that is i like the azure platform uh some of the tools like azure ml studio is pretty good it's it's a pretty good product and the machine learning automation they have is pretty good too uh the sdk all the stuff that they have so i'd say they're probably number two and then google is is is is in last place and there are some things that are nice about google i i think the google app engine is is nice and the auto some of the automotive stuff they have yeah big queries pretty nice like they feel like they're more like a la carte offerings where you know maybe you would use bigquery alone and use it with um let's say you know google data studio and and and but but maybe your core platform is microsoft or or aws so i i would say that might be the solution if people really like one product on google i don't see it being the end game you know it does everything but some of their products are good like you mentioned like i think bigquery is a good another good example of a really good product i'm wondering you you kind of said that really quick like aws is is the best in your eyes what makes you give it that the crown well a couple a few things i think that are that are interesting about aws one is that the culture of aws is different than the culture of google and and i don't i can't comment on working at any of i have not worked at any of those three companies i know people that work at all those companies i don't know if i would be a good fit to work at aws but but but uh as a customer of aws you know their their their culture is basically make the customer you know have great products google's culture is not that google's culture is like hey let's try some products out oops oh or you know there's a sharp edge there oh well yeah we may or may not fix google hangouts in the next three years you know like so i think the big difference is that the most of the sharp edges are done or are solved on aws i mean they have products that that do have issues like um you know they'll be in beta stage and things like that but for the most part things work on aws also uh it's a comprehensive solution so you know there isn't going to be a gap in virtual machines or networking or you know they they pretty much always have everything covered uh and then in terms of pricing as well the pricing really is pretty clear with aws and it's it's a lot easier to to to see what's happening versus google or azure interesting yeah yeah and as far as like um we had luigi from mlm production on here a few months ago and he was talking about how he studied what sagemaker was doing and what aws was trying to do with sagemaker just to get a better idea like going back to this having that devops mentality and one of the nuggets of wisdom that you dropped earlier of like if you need five people to do your job that's the kiss of death for what you are trying to do so as a data scientist if you don't know what is actually going on and you're just relying on all of this other stuff or other people to help you out and make you valuable at your company it's never going to work and so i'm just wondering like when you're looking at sagemaker and just aws in general i guess it's more the ecosystem that you're looking at right you're not even looking at one of these tools you're saying how everything if you just had to choose one cloud throw it all on aws i think so i mean because you also can bet on the fact that they're going to keep improving so for example excuse me one second aws you know that they're serverless offerings like aws lambda step functions all these things are always going to get better there's this like constant you know let's keep making it better let's keep lowering the prices approach where i don't see that with let's say the google cloud again there's some great you'll get surprised excuse me you get surprised with um let me just drink a little bit of water yeah no worries yes so you'll you'll see that with google cloud where occasionally they'll come off with like an incredible product like uh google bigquery like incredible or um google auto ml vision incredible product but it's not like you can bet on that where aws across the board it's just consistent like you know you know product releases product releases product releases product releases every product they have gets better so i think that's really the core difference between you know looking at aws in particular and looking at google now i would say that microsoft is going to give google a run for their money in that you know if you look at the ecosystem around the azure cloud itself like github is pretty good i mean that's a great yeah uh company the github actions the sas product they have is great i know a lot of people say bad things about microsoft teams but i think it's actually got some promise to it there's definitely some things that are nice about microsoft teams so so if you look at the eco linkedin is another one i mean they have some awesome tooling and awesome products so i think that that's probably going to be the the big competitor you know they got the contract with the government the for defense contract i i think that it's a viable option i think i think the microsoft azure is a viable option i think google will be in you know a niche product which again could have some great tools but it it'll be the part of a multi-cloud solution for a company but it's not going to be everyone i know that's had only the google cloud has not stayed with it yeah i think the the key point is whether the people at google feel it or not that you and i feel this way about google's reliability speaks volumes and i think that that's the issue and i i think you're completely right when aws releases a product you know that that will have consistent updates you know that it will have a certain level of documentation you know that each widget looks a lot like the other in terms of how they make it fit in to to a pipeline or into a you know a number of boxes we use we use aws at work and as i was listening to you talk about the different cloud providers you know one thing we talk a lot about in our community at high level is just architecture right system architecture and thinking about that what different you know tools and what different processes should link together and i kind of want to ask you at a very general level what advice would you give to ml engineers who you know have spent the last couple years really building models and are now starting to think more about you know think more about the ml piece and think more about the um the architecture piece and they're you know perhaps into their career how would you advise them to go about starting to get into architectural thinking about machine learning systems with your perspective as an educator yeah i think that this is a good question the um that let's just since we're talking about aws i think it's not a bad idea or microsoft let's take those two right those are the top two clouds is is it's not a bad idea to get certified i think that a lot of times someone has a master's degree or a phd and then they're like oh that's beneath me i shouldn't get you know aws solutions architect i would say you should uh and the reason why is it'll give you a a test of of your knowledge or at least do it once get you know get one of those certifications because what you'll realize is that there are some concepts that are actually pretty important like elasticity and you know high availability and and you'll learn about those those concepts and also building things that are non-machine learning applications like just building a um maybe like a flask app that you deploy or a serverless you know aws lambda application just having building some apis and just getting an understanding of it is is really going to be helpful i i think it's a mistake to to think that that's not my job in fact when i was cto that was the thing that bother or engineering manager that's the thing that bothered me the most when someone were building something they're like oh that's not my job that's devops it's like oh don't say that like because it's like it's like saying you know the framing of the house that's not my job you know it's like we yeah if the house falls down of course it's your job everybody's job is to build things that are reliable so i i think it's important to think that way when you're doing machine learning is that you do need to understand there's there's decades of experience that people have with building software and they know stuff that you don't know and you should listen to what they're doing and probably you know if you're if you're if you're really motivated you could just study what the mobile developer does in your company or study the web developer and just listen to to what they're doing for six months to a year and you could you could reverse engineer that for machine learning and they have a lot of wisdom like for example mobile well i can't think of any professional company that does mobile development that doesn't have a build server right like you would be it would be literally um you know completely insane to build mobile apps that are production mobile apps that don't automatically get built you know so so you know i mean if you had a developer ad hoc build you know like an ios app once a month and and they're the one person in the company that can build the ios app and then they put it to the app store that would not work for much for a very long time but then people feel comfortable doing the same thing with machine learning like oh i built this model it's on my desktop and look like wait you can't do that you have to do the same approach that people are doing in other parts of the company and i think that's where the shortcut could come is just follow what a professionals web developers mobile developers are already doing and then apply that to machine learning yeah and then you are able to take those best practices i think you put it so perfectly there how it's there's a lot of experience in this whole software engineering world that can be transferred over and you don't it's not necessarily a machine learning problem it's more of a best practices and how can we learn and take from those best practices and then apply them to machine learning yeah i think that this whole like calculus linear algebra you know part of of machine learning is like this good excuse to to to not do the whole job and it's like oh well there's calculus so that means that i don't have to do all the real hard work like no you do like that's just that's a it just happens to be the thing you're working on requires uh some some math or some knowledge of math but that doesn't mean that you you get to say that's not my job for the rest of it and and that's pretty much the the worst excuse that you can give somebody is that hey i just do the stuff that is like hard intellectually but like the part that makes it work that's your job i mean that's i wouldn't if someone gave me hints of that when i was hiring them i would not hire that person yeah i i completely agree with that i mean you see a perception and i think this is common to see on hacker news threads in our community all over the place where people you know talk about how there is a there is a resistance on the part of people who do pure modeling to do anything else besides that and then to focus on the elegant work so to speak of of building models and then saying everyone else figured i figure everything else out yeah it's similar to in academics you see this a lot is that that they'll be places that say well we we don't do vocational training we just teach you theory and and there's obviously nothing wrong with learning theory but that's to me that's a cop-out to say oh and yeah nothing you learn from you know a master's degree at duke i'm not saying anyone at duke does this by the way i've it's probably the opposite but but it but uh you know if you if you're like hey we got a master's degree and like we don't do anything that gets you a job it's like what like how could why would someone get a master's degree if if they don't know any and the same goes with the modeling like you can't just say oh i do all this like theoretical work and then yeah the part where like you put into production and then we know you know we we tested and then we see if the customers want the product yeah that's just somebody else's job yeah i mean well the fact that you're that duke has brought you on and that you're at duke tells me volumes that they care about making things useful for their students um i think one thing that struck me especially when you made that comparison which is nobody is a mobile developer sitting there and saying hey you know this app it needs to be built in a specific way they're built continuously and they're built with the principle of good software design and software delivery and you know i think what struck me there is that that growth and that evolution is driven by need right when companies started to realize okay a mobile app is essential to what we need to do in order to reach distribution for customers to do things so that we can actually support our business those principles of good software delivery for mobile and for databases and for all those other areas of software design started to evolve you know that growth was driven by need in your experience working at companies consulting i mean what principles or what realizations do you see trigger business leaders to really realize that okay it's not enough to just invest in one star data scientist or one star modeler but to build a culture of software delivery in the machine learning organization yeah i think that's a good question i you know i think the real issue that i learned being a manager was that the worst possible way to think about a software team is to focus on individual talent and even though that's the myth and there's all these articles you know the 10x software developer and you know google is a good example hey we'll whiteboard you for nine months and look where that got them but but the i i think that what you don't want is people who think they're special and that won't work with other people you want to have people that work in a team and so i think that in itself forget even the domain that you're working in whether it's mobile web machine learning is if you hire people who think they're better than everyone else and uh they're smarter than everybody it's it's a toxic work environment and i've been there many times and it's the worst hires i've ever made in my life were people that wouldn't work with other people because they thought they were better or smarter the rules didn't apply to them so i think that's the big caution i would say and i'm not saying that data scientists are like this that that's the kind of data scientist my i would hire would be someone who not only you know knows about data science but also is very willing to learn all the other things and is willing to do whatever you know whatever the job is they're willing to do it like and and i've definitely hired people like that and it's just amazing when you hire somebody that's like hey what would you like me to do i'll do anything you want i'll stock the fridge i'll dig a ditch i'll also do machine learning also do you know mobile development i'll do web development you let me know how i can help your company that's the kind of person that i think you want to hire to if you hire someone who's really open to to helping everybody and it's very team oriented then you'll have a great approach you'll have i think you're then your product will will be a manifestation of that if you're only focusing on like whiteboard interviewing and credentials and and elitism i think the toxic stew will be very bad in your organization you might get away with it at google because you know they have so much money and so much prestige that they can get away with it but i again i've never worked at google but i would suspect that it probably there are sections of it that are very toxic just because of my experience in hiring people where i've only focused on hiring the smartest people i bet it's not fun i bet parts of google again i don't know but i'm just my guess is so same thing with when you're hiring people in a company is really personality and character i mean we just i mean i i think anybody whether you're republican or democrat would you know there's a lot of people are saying this now is character as destiny you know if you're a liar or you cheat or you think you're better than everyone else it doesn't just go away it will manifest itself throughout your organization so i think i think people's character and teamwork is probably the number one character given a basic level of competence is probably the best thing to look for yeah i think we had todd underwood the director of machine learning sre at google come in and speak to us a couple weeks ago and one thing he said is that look we're figuring this out just as much as you guys are at google we're figuring it out and everyone else is figuring it out and we're learning from you just as much as you're learning from us and that dovetails really well with what you're saying which is a lot of times when we when we in the machine learning world are sitting at our jobs we're thinking wow google has this amazing infrastructure they put out these incredible papers and they put out incredible work and they do but that doesn't mean we have to take everything from them and they wouldn't say that either and particularly that point on the cultural uh the cultural fits for for for machine learning work and for particularly to do you know interesting new kinds of work the reality is is that it's not that glamorous on a day-to-day basis and i think what's interesting if you look at it at you know for a long time the the tech industry has said everything google does with hiring is great and i think we now have a different data point again i think that google does lots of things that are incredible but i think we have a we have a now another data point which is that literally some of the smartest people in the world were hired for a decade decades decades and they produced a product that was so toxic that it actually brainwashed a huge percent percentage of the population and made them believe in things like anti-vaxx alex jones alex jones i think alone was served out um 15 billion times i think according to the social dilemma and you can't just wash that away you can't pretend that that was not a data point i i if i worked at google i would have be having a reckoning with with with here here's the the thing where the teamwork comes in is i really wonder if there were more regular people who are just kind of like hey let me how can i help i want to be you know like just kind of regular like maybe who had read books that were non-computer science books like you know john steinbeck or history or like the rise and fall the roman empire or where they would say hey hold on hold on did you ever did you look at youtube like we're we're serving out more traffic to flat earth in anti-vax than we're serving out like like this happened before in the past like you know so i do wonder if they should have a reckoning in that company don't even get me started on facebook but but with google the reason i'm saying that because i like google i like the the the company and i think that there's a lot of good things that they've done and they have probably the ability to to really think deeply about what they did but hopefully other companies as well think about this with you know facial recognition if you've trained the model and it's basically trained on people that were uh you know discriminated against and put into prison at a higher rate than other you know segments of the population and then you're using something that's got all this bias built into it you know is is that really something your company wants to do you know those kind of ethical things it's easy to kind of pretend like oh that's just you know that's just ethics you know or like we're trying to get customers but but externalities which is you know the fancy word for it and and you see this a lot with startups in the in the bay area yc combinator startups all these startups is they you know if i had to i like a lot of things that paul graham writes and he obviously a very intelligent person went to harvard but i think he's got a few week weak points in there in some of the the arguments that he makes which are specifically he likes to pretend like externalities don't exist and a good example would be you know airbnb would be one where you know hey great we we have um you know housing now is available as a service but then let's say you're in a neighborhood and then they you have a family and you've been living there for 10 years and now your next-door neighbors red harleys and they do donuts all the time or or they have wild parties you that was free right but the neighborhood was safe that was free you just stole that and then now you're selling it and i think that's that's a lot of times what you see in the startup world is that the externality is what what is it that's free like let's take the scooters you know like um the sidewalk was free you just stole it and then you just polluted it and then and then your your your growth hacking like i think i think those those kind of things just those are ethical issues that maybe in the short term you know someone would be okay with with with doing because but i think in the long term it eventually will have repercussions and so i think with whether it's a higher or it's the the machine learning thing that you're working on that the the externalities and ethics do matter even if you're very very technical and hopefully companies think about this because i i think that's the way that you you can um you can get a good reputation is is by by having integrity uh and and hiring people that work well as a team yeah this is music to my ears i don't really mention this much but i also do a ai ethics podcast shameless plug for that but the the stuff that you're saying goes along perfectly with what we talk about in that and it's so important as we are moving forward and we're looking at what we want technology to do that we don't lose sight of that right and it's really interesting this idea that you talk about if we took what was free and then we basically just polluted it and there's these side effects that people don't look at or we don't think about when we're first starting and then boom it's too late and that's been the like the story over and over for the past decade right it's like oh we didn't see this coming and then how could we have known and so i i've even said this before like i don't know maybe read a book that isn't involved with technology right like the history books or the fall of rome you mentioned so i want to shift gears a little bit i know we're running a little bit lower on time so i wanted to go into the ecosystem of tooling that is you mentioned before like go with a cloud provider or maybe go with some open source and so first of all i wanted to just get your opinion on why you thought those and maybe not some kind of managed service that is a startup and seeing how there's billions being thrown into the mlops startup space right now i want to hear your take on that and then the other thing is like how you feel about open source tooling for ml ops and where that's going yeah this is i mean it's an interesting question like the startup mlaps versus the cloud provider i i think it just depends on like you could think about it like investing in stocks where you know if you bought tesla in april or or march you quadrupled your tesla right that's that's lucky right that's lucky to to do that most people buy an index fund because you get eight to ten percent you know over time i think like the cloud providers are like the index fund where you're going to get great results and it's very predictable but if you wanted to take a risk and invest in individual stocks like an individual component of an ml op startup you could do that i personally at this stage in my life have taken lots of different bets and many of them have just not worked out so you know i i i guess as i've learned is like the index fund approach where statistically you get a good result over time is the approach i would choose and so if you you know in the case of amazon for example or microsoft they're so big you know you're going to get predictable results the the the downside is very low just again with the index fund the downside's pretty low i mean even if there's like a a global event like covid19 an index fund goes you know s p goes down 40 that's not horrible right like if you bought an individual stock it could go down to zero right that and so the same thing with that ml ops you know component it could go to zero so you you you invest in something and it turns out it just doesn't work i i think it depends on the stage of the company if you're a big company and you're already using aws for example sure why not try it out try to try some other tool and see what it does i personally would not bet my whole startup on some other startup i mean who knows what could happen again this is that's just me i think the startup tools are awesome and they can like splunk is is a good one where i remember when their first a startup company and i started using their product and that did turn out i got lucky and i did a lot of great stuff with splunk and and now they're just gigantic company but i i just again my approach would be be skeptical of betting your whole company on one particular startup product you don't know the management of the company you don't know what's happening i think it's better to treat those as like nice to haves and if it is some particular soft problem that it solves and it saves your time great go for it but don't make it the core component of your of your company i would bet on the big companies because they're not going anywhere this is my opinion people would disagree but that's my opinion i think that i think that makes a lot of sense it's it you know there's a dash right don't take a risk or you can lose your shirt and if there's a if you know um with the startup when when you know stars go bust all the time right and and and if their core part of your infrastructure if you're dependent on them to store some data or something that's a serious serious risk to your business in the long term and it's funny because sometimes people don't even take that into consideration when they're thinking about their risk analysis right when they're you know teaching to investors or thinking internally and saying hey let's do our risk analysis well what if what if what if this provider of this service for us is not is not there anymore that's going to cause a lot of destruction and delay um so i think it's it's it's a really really good point there and then all and then here's the other the the dark dark side of you picking something that's like a newer technology that's unproven is are you sure you can hire people to support that technology a good example i'll give you one in my career was erlang the language i did a lot of stuff on erlang worst mistake i ever made because just nobody uses it just nobody uses the language you can't hire anybody you got to train people it's just whatever advantages it had which there's some cool things that our lang does it's just not worth it so that's the other thing is if you're using some really small product and there's a bigger you're like let's just take sagemaker versus joe's sagemaker right or whatever it is like everybody knows sagemaker but they don't know joe's sage maker that could be a problem because if you need to scale up people very quickly what documentation do you point them to you know so again i think it's early stages you have to be very careful along those lines i was just wondering about how there was one question we had around like the languages that data scientists should know and i think you you've touched on the idea of hey just that t shape and be able to traverse and then go deep if you need to i'm just wondering because go is popular these days do you feel the same about go as you do about erlang no i i think goes a little bit different in that it goes salt i would say go as a replacement for lang so go go actually i know a lot of people that used to be erlang and then they switched to go because it is sponsored by a large company so google is actively behind it that's a good sign right and and then the other one is that uh it does appear to have a large amount of people that are go developers so i think from infrastructure there's nothing i think go is a great choice probably for many companies and especially if you're in the kind of web you know web type ecosystem or you're building back-end services i i think go would be a great choice now something like julia i wouldn't touch it with a 10-foot pole because i don't in 10 years maybe awesome julia would be awesome but at this point in my life i just you know life's short i don't want to spend five years then later it turns out that nobody uses the language and then you can't hire anybody at your startup that's horrible it's i've been there and it's just it's it's not fun that's some wisdom right there speaking from experience and i can tell the pain i can hear it in your voice of man we spent all this time we did all this stuff and then when you need to scale like you said like you what happens when you want to scale up quickly and you need resources you need talent and then somebody has to find the people that are out there and maybe the market yeah they don't exist because the market's moved on and so now it's like all right we're in a bit of a predicament so that's that's a great gem right there too yeah the one i was going to ask about was julia and so it's helpful to hear your thoughts particularly given your experience with our lang i think you know just to kind of wrap up you know we've talked a lot about how machine learning is is going to look a lot like other software fields um i want to kind of flip the question what do you think about machine learning and ml ops will always remain separate why will mlab stay ml-ops and not just devops well that's a good ques i don't know if it will i think there's a very good chance that machine learning becomes commodity and that in fact it's just something that everyone has to have some understanding of so i'm not convinced that it will and that would actually oh sorry vishnu that was that was one of the questions that i had too was like do you feel machine learning is going to be commoditized and how do you foresee that happening and is it going to be like because again i'm referencing luigi he was talking about how you're going to have machine learning be a commodity but then you're going to need the subject matter experts if you really want to do something and so the people who have deep deep knowledge in their respective fields then you throw some ml on top of that and you can have a product but the machine learning may potentially be it's like a machine learning as a service how how do you foresee that yeah i mean if you look at certain problems for example like classification problem i mean really how much do you really need a data scientist if you've seen some of these auto mel tools you you give it you know a few columns and you click a button and it tells you every single thing that's important it tells you area under the curve i