ML Platforms - The build vs buy question

Key Takeaways

and so when you were looking at what you needed how did you go through this build verse by process because I know a lot of companies are looking at should we get a third project party or should we jump on it and start to build it ourselves yeah always the first question you want to ask I think we also asked ourselves the same question too and I think what really helped us figure out whether we should build or buy is we returned to our team we say like hey you know if we had the ideal platform where every problem that we're facing today and maybe for the foreseeable future is taken care of what would that look like what would the requirements did that look like and so we had I remember something like a multi-day brainstorming session where we're all just sitting down and writing like okay in the ideal world I break this down into we we broke it down into three groups or three pillars and actually realized that these three pillars aligns really closely with what Phil had talked about last week on this meetup as well which is we talked about the data side then we talked about the services and the serving side and the last we talked about the management side and although those three pillars are not the exact same words that say Phil it shows I think is where data software and governance I think they match up pretty closely though and for each of those we ended up defining okay what are our requirements you know for data we need some way for our data scientists to easily find data within our organization we need some way for us to reduce training serving skew we need real-time live updates for our machine learning features then on the serving side we needed ways to have feedback pipelines be built out either real-time or you know in a slower batch moving way we needed to record every single prediction that took place or inference across all of our models as well as be able to handle that within our a be testing and experimental platform and then on the management side being able to manage all versions and new models that are coming out model projects and then all sorts of data that's coming through this and so as we started writing everything down we said okay you know there is not one platform that can do all of this for us and then it becomes a question and we keep pinching things well given the way that development moves in this kind of space I would so rapid you know we might choose that we're gonna go with this platform today for the data side but then tomorrow we might see it's not compatible with our upgrades on the serving side or on the management side and so that's where we sat back and said okay I think you know it makes sense to build here and we had the expertise in-house which was really great for us to convert some of our general software engineers into machine learning engineers and data scientists that could help us build this out from within and you know we did some hiring of course but I think largely we did have great experience within the organization and and I'm wondering how big was the team that embarked on this journey it's great question I think so we have we had like four or five data scientists and five to six machine learning engineers and just to be really specific the difference between these two groups of people our data scientists and our team are primarily responsible for developing machine learning models whereas machine learning engineers are primarily responsible for developing machine learning services so all the deployment side of things sometimes don't be some overlap because people have some expertise as well and just because as projects grow and expand there could be something there but largely speaking that's the breakdown and actually since then we haven't grown too much I want to say we've probably added one more one or two more data scientists or one or two more machine learning engineers but our output capacity has definitely grown to the point where you know we had a handful of ml use cases back in 2018 and that's exploded to the point where were you know nearing the 30s and 40s where it would take us on the order of say six to eight months to have one use case from inception or like ideation to out there in production but now it's on the order like say a month where we can get something out there and get the ball rolling which is really great