MLOps lifecycle description

Key Takeaways

life-cycle sorry the ml the ml ops lifecycle is just fundamentally more complicated than the software development lifecycle so when you're doing software development and life is easy it kind of it's easier anyway because all you need to do is you need to think about versioning specific versions of your code because as long as your doctor izing things then you ought to only need to know exactly which version of the code was used plus the doc file and dependencies in order to reproducibly build the same or effectively the same deployable artifact and so you can build you can create code you can test that code in CI you can deploy that code into production and then your monitoring system can tell you how well it's performing maybe you need to upgrade a database maybe you need to go and optimize a certain code path and you can go around this loop as quickly as you can that's basically what we're doing with tables but with machine learning in ml ops you've got this fundamentally more complicated thing going on which is that you've got data coming into the system which is a major form of entropy and you've got code that's that's being written to train the models and you've got parameters which are being fed in to the training the models the model runs the training model run the model training runs sorry and all of these things combined and before you even train a model you've also got the data runs which is when you're doing feature engineering or your filtering or splitting the data and so you're executing a certain version of a piece of code against a certain input dataset and creating an intermediate dataset that's what a data run is or you're you're running a certain version of a piece of code against an input dataset like a test training validation set and then you're creating a model as an output and so when you're doing ml when you're developing machine you are doing these data runs and these model runs whether you're keeping track of them or not these might be the cells that you're running your dupes are notebook or the Python scripts that you run on your laptop and these runs are happening you just haven't given a name to them and so what we've realized is that it's really important to introduce this notion of run as a kind of a building block it's a fundamental object in the system when you're doing ml outs it's this version of this piece of code ran in this environment with this input data and these parameters and it created this output and one of those types of outputs like I said is models and then the model itself the serialize model file is the deploy Balazs fax it has some metrics associated with it like how well it performed against the validations data and then it's that model that you're deploying into production and then you're monitoring in the way that I described that monitoring models is different to monitoring microservices and and then you're monitoring might actually cause you to go back around all sorts of different loops through this diagram so that's kind of the the flow that we're looking at here and so I'm a little short on time so I'm going to skip over the lifecycle quickly and the lifecycle is really just a visual representation of how the diagram I just showed you is cyclic and so you do data engineering you need to keep track of the data runs when you're doing things like feature engineering and then you train a model which means you take a certain version of the data and you you train the model and out of the model development process come serialized models typically docker images with serialize models baked into them and then those serialize models get run in production or staging and then production and then there are these kind of two feedback loops there's the statistical monitoring feedback loop which is the first one where you can say show me the behavior of my model in production in real time based on like the distribution of classifications it's making and then there's also this slow feedback loop which is that you retrain the models with new data as you get new data coming in and of course the model making decisions can change the world and so can also impact on the data that is being recorded in the database so so that's kind of the the ml ops lifecycle and and hopefully I've adequately answered the first part of the talk title now which is what is ml ours my opinion is that ml Ops is not just about operating models it's actually about the entire lifecycle of doing data engineering training models and then getting models into production and it's about being able to implement this process in in your company or in your organization with the same level of rigor that sort of best-in-class DevOps and software engineering teams manage and so