Distributed Training using PyTorch with Kubeflow on AWS and AWS DLC - AWS Virtual Workshop

Key Takeaways

[Music] hello everyone and welcome to our workshop distributed training using pytorch with kubeflow on aws and aws deep learning containers my name is alex jankowski i'm a principal solutions architect for self-managed machine learning at aws and my area of focus is orchestration of machine learning workloads on aws using containers and i'll let my colleague introduce himself as well hey everyone this is kawajit kurmi i'm working a senior aiml specialist solutions architect and i'm with amazon web services for almost three years now and i specialize in containerized and machine learning applications so so let me take you guys through the agenda for today's workshop so so we'll start all the way from basics discussing why we need distributed training and how it works and then we'll look at the landscape of tools and services that aws has to offer to support distributed trading workloads we'll then deep dive into queue flow and build automated workflows leveraging both kubernetes and amazon stage maker in a hybrid architecture and along the way we'll have a hands-on experience on installing keyflow and running all the examples of distributed training as part of this workshop so with that let's get started and delve into the details now let's start with some high-level understanding on what is distributed training and why it is needed now today if you see uh the data scientists and researchers who are building very complex deep neural networks are using open source frameworks like pytorch or mxnet uh or or tensorflow they have found that they get better accuracy if they train their models with significantly large amount of data set especially in the use cases like computer vision and natural language processing however the challenge is that when your training data increases it takes longer for your training to complete and that means it takes longer for you to get it to get to the training results now that ultimately slows down your time to market because you'll have to wait for your training run to complete look at the results and then go back tune your tune your code or change your hyper parameters to kickstart the next run so the entire experimentation loop kind of gets extended now now there are different ways you can go about speeding up your training really the first one is that you can optimize your code however there is only so much that you can do with your code so at the end of the day you'll have to get more compute and and there are two different paradigms when it comes to getting more compute one is to scale up your existing infrastructure that means you replace your existing machines with more powerful machines which have more cpus more gpus and more powerful infrastructure right but however uh there there will be a time when you'll hit the wall and you would need more number of cpu gpus that a single machine could accommodate and that's when you need to scale out right scale out means you need to include more machines into your training cluster which can provide you more number of gpus that could be accommodated within a single instance however the thumb rule here is that you always should prefer scaling up your machine and then if that is not enough then scale out because when you scale out you you're going to bring different machines into your cluster and that means those machines need to communicate with each other exchange gradients and they're going to be network overhead and network latency a lot of work that needs to be done in optimizing that network here right so always go uh you know go for scaling up before you scale out however in this distributed training workshop uh we're gonna be training our models uh based on biotorch using multi-node cluster because i think that's that's the objective for this workshop now with that in mind uh let's take a quick overview on the sort of how distributed training actually works and why it speeds up the training now in typical training you have your data set right and you break them up into batches and i'm specifically talking about the deep uh learning models here and when you have to train those deep learning models you actually have to send these patches of data in sequence now when you send one first batch it actually goes through the forward pass then the losses are computed then you do a backward pass to calculate the gradients and update your weights now the single machine which is hosting your model actually does all those processes and steps with every single batch in sequence right that means if you have hundreds of batches to be processed and those will be processed one after another that means your overall training time will will increase now the alternative is to actually use a bigger cluster which has multiple machines in it and you distribute your patches across this different machine so that they can work on those patches in parallel right and that means you effectively are training with a bigger batch of data more data sets in a lesser amount of time which will ultimately reduce your overall time of training right now let's look at the landscape of tools and services which aws has to offer to support this distributed training cluster right fundamentally you can start with uh amazon ec2 instances which are elastic compute cloud instances uh you can bring them up together and build up your own cluster from scratch if you have the expertise however over time over years we have built uh the services like amazon elastic container services and amazon elastic kubernetes service which actually lets you abstract uh all the infrastructure complexities and they help you manage your cluster in a managed way now these are primarily primarily the container orchestration services um and they expect that your applications are containerized right and to host those containers and you know to version them and to use them effectively we have services like amazon elastic container registry where you can push your containers you can version them and then you can pull them when needed uh and then you can you know modify them extend them or or run them on the shared clusters with your kubernetes cluster like ek service or ecs service managers right and then uh you know when you start entering into the machine learning domain now there are capabilities that you need above and beyond the basic infrastructure capabilities that's where sagemaker comes in right an amazon stage maker is a managed service uh which is a most complete end-to-end machine learning platform which addresses the requirements of every stage of machine learning workflow now the best part here is that sagemaker provides you with a training cluster which is completely managed you can use that cluster to run your distributed trainings you can run your large-scale hyper-parameter tuning jobs and even it goes beyond just training your models it can actually let you do a one-click deployment by writing few lines of code where you don't have to provision a load balancer you don't have to worry about scaling your uh backend infrastructure and overall experience is is pretty amazing uh with amazon stage packer now now the objective here is that uh for for the organization who wants to really build up their machine learning platform on this open source technologies like kubernetes they can continue doing that but at the same time they look they can also leverage all the capabilities and innovations that amazon stage maker service has to provide so that way you actually can get the best of both worlds and you can really address the requirements um you know for for your business use cases now before we actually get into uh the the demo and the and the lab for installing qfloor i just want to take a minute to introduce uh the basic concepts on container now now by definition containers is a standard unit of software that packages up your dependencies and code together into a lightweight scalable portable packages which which runs consistently and reliably no matter in which compute environment they are running now think of your project uh where you are working with different machine learning libraries or frameworks right and those frameworks like tensorflow pytorch mxnet they come with different versions of libraries and dependencies and sometimes it gets really difficult to keep them in in power or inconsistent with your fellow data scientists who is working on its own version of dependencies and libraries however if you package these libraries dependencies configurations code together into one single container image you can actually run it reliably and you can be rest assured that it will produce the same results no matter it is running in your machine in your teammates machine or in on any kind of cloud computing platform now i just wanted to mention here that aws also provides uh its own deep learning uh container images uh which includes optimized version of deep learning frameworks like python mxnet and others that we have on the slide and and these are are fully tested they are fully optimized to run on amazon um cloud compute they're optimized for storage for for io and any kind of compute instance that you will include as part of your cluster right these are these make it really easy uh to pull the required image from the easier we publish these images in our github repository uh you can actually pull them you can extend them and add your own um custom libraries to it and then you can use them for prototyping and testing so with that i'll actually um hand it over to alex to talk about how we can build machine learning platforms on kubernetes how does the user experience looks like and also let alex uh really walk us through the steps uh to install the queue flow and and get access to the queue for dashboard so alex please take it away thanks ken well so um as kennel mentioned we could use the deep learning containers to do uh machine learning and run distributed training jobs directly on kubernetes so in this slide um we show that kubernetes is um gives us a way to describe a desired state using kubernetes manifest and then we can apply that manifest and the orchestrator will schedule containers on the nodes available in the kubernetes cluster and run a distributed job however to be able to do that one needs to be familiar with writing um kubernetes yaml manifests and be familiar with the resources and structures that are required by kubernetes and typically uh that is um not very user friendly and uh data scientists are not experts on writing those kubernetes manifests when you run machine learning jobs directly on kubernetes you get no support for experiment tracking or workflow management and overall you have to be familiar with the low-level uh kubernetes architecture and frameworks in order to be able to do things at that level and this is what is solved by kubeflow kubeflow is an end-to-end machine learning toolkit for kubernetes that provides an integrated development environment that is self-served it allows you to build pipelines and workflows and supports both distributed training and model survey you could use the mainstream cube flow distribution or you could also deploy the aws customized distribution of kubeflow which provides all of the capabilities of the mainstream version and has the ability to integrate with managed services from aws some of these managed services are displayed here on on this diagram in the workshop today we will be using integration with amazon efs and we'll be using deep learning containers from aws so with this let's get hands-on and jump into the workshop we're going to use two main urls for the workshop labs we will use the first url which as a short link is bitly dtw cubeflow event and the workshop instructions are at just the dtw heap flow so let me switch to my browser and i'll type distributed training workshop cubeflow event which will bring me to the launch screen screen for the event and in parallel i'll also do bitly dtw cube flow which will pull up the instructions now the first page of the instructions provides an overview of the workshop this is what we already covered and this is the hybrid architecture that we're going to deploy on kubeflow which we will cover later but the actual workflow steps are linked at the bottom of the introduction page and as i go through the steps i will be scrolling here to help you keep track of which step we're on so we'll start by setting up our aws account through eventbox we'll click the launch button and then the event will provision a temporary account for us which we can access by following the link we need to agree to the terms and conditions and then to gain access to the account we have to enter our email address so a one-time password can be forwarded to us so this is where you would enter your email address and send the passcode this could take a minute for the email to arrive and once it does you would um you would just uh copy the code from the email and paste it into the screen yeah and then i think we will request our moderators to also paste these links in the chat for convenience so my one time passcode has arrived and i am just going to paste it here and sign in to my temporary account so you'll be presented with the team dashboard and uh we'll need to access the aws console from here so we'll just click on the aws console button and then open console to log in onto the aws console as this temporary account so this completes the account setup and now i'm going to scroll to the next step the second step is registering a cloud9 environment in the account we have already pre-provisioned a kubernetes cluster and an ec2 instance that we'll use to manage the entire workshop on this ec2 instance we have cloud9 installed and we just need to register this ec2 cloud9 instance with the aws console and i'll walk through the steps on how to do that so we will need two screens from uh from here if you don't have the links over here or for ec2 and cloud9 you can just type easy to and then it will appear here i'll open ec2 and cloud9 in separate tabs so on the ec2 dashboard you will see that we have some running instances uh seven of them um six instances are kubeflow nodes so they are participating in a cluster and our management instance is the one that has no name click on it to display its details and we'll need its public ip address which you can copy from right here below so i will copy this because i will need this um for the cloud9 registration and then go to my cloud9 screen in the console and click create environment we'll name this environment management instance the name is really not important and click on next step so cloud9 is basically an ide in the cloud that you can provision for yourself but it not only allows you to provision new instances you can also connect to existing instances and that's what we're going to do here by clicking the create and run and remote server the username for the login will be easy to user and for the host we will paste the ip address that we copied from the ec2 console now in order for cloud9 to be able to connect to this ec2 instance we need to copy the public ssh key of cloud9 into the authorized key keys on the instance so we'll do that by clicking this copy key to clipboard button and now this public ssh key is in our clipboard and we will connect to the instance through the session manager link provided in the ec2 console so once your instance is selected we'll click connect and then on the session manager tab we'll click connect one more time this will open up a shell screen into the instance and we will execute two commands the first one is sudo ec2 sudo su ec2 user this is so that we can assume the identity of ec2 user then we'll just type cd to switch to that user's home directory and then we will edit the authorized keys file i will use vi and for people who are not familiar with vi i will spell out the commands that i'm typing so vi dot ssh is the folder where the file is in forward slash authorized keys this will open the file and we'll see that there are no authorized keys in this file i'll press the i button on the keyboard to enter into insert mode and then paste with command v the key that is in my clipboard to save and quit this file i'll press escape colon wq and now the public key is authorized and we can go back to the create environment screen in cloud9 and click next step the cloud9 console will connect to the instance and then once we click create environment it will be able to pull up the cloud9 ide that is already running on this instance if you get any errors you can just disregard and we are done with setting up our cloud9 environment which we will use for the remainder of the workshop to complete the next steps so i will now move to the next section where we're going to use our ec2 instance with cloud9 to deploy kubeflow and the first action that we need to take is clone repository of a new open source project that we've released for this workshop called aws do queue flow which makes it simple to deploy qflow on aws so i'll just get clone github dot com aws samples aws do keep flaw cd into this folder and in here you will see that uh there is a there are a few scripts we're going to use the build run and exec script from from this project so let me run the build script and let it build [Music] the aws tube flow container and explain what this container does aws do cube flow is a do framework project the do framework is a collection of it's a devops framework that has a philosophy and principles of simplicity flexibility and universality and its aims to simplify how you get things done the workflow that you follow when [Music] you implement a do framework project is that you don't tend to install things on the machine and and run things following several steps you put everything in a container and then make that container do the work for you so uh to deploy kit flow this is what we're going to follow as a process first we clone the project it comes with configuration that works out of the box but then you can customize any of the settings and this particular project allows you to deploy either the open source generic distribution of kubeflow or the aws distribution then we build the container run it and get a shell into the container which already contains the deploy script for uh installation of kubeflow we just run that and at the end of the script cube flow is up and running and we can access it just to add to that alex and i think one of the challenges which i have seen with many of my customers is that installation of queue flow in itself is quite challenging and i think do framework gives us a probably a nice path uh an automated path to really deploy keyflow and get access to the dashboard right so that really adds a lot of value to all of our customers right and um another thing is that if we uh we're currently building the aws duke flow container however if we had that container published as a in a public registry then you wouldn't even have to build it you would skip through through these initial steps of the workflow and you can just run it exactly which simplifies things even further but uh just for the purposes of explaining and uh including that as information to help understand how a do framework project works we want it to go through uh the entire workflow so uh the container is already built and uh we can just execute the run script to start it uh with status we can see if it is up or not and then exec we open a shell into the container and here in the container you have all the tools necessary to interact with your kubernetes cluster with kubeflow deploy any version of kubeflow that you like and all that so to show what the script will actually do i'm going to open a new terminal and put it here on the side and i'll open a second shell into this container so now i'm looking at the same container at the same time however on the right i will execute a cubecpl command to watch the uh the namespaces and the pods that are currently in the cluster so you can see there is a default namespace and basically the system namespaces and there's nothing deployed in this kubernetes cluster at the moment and now i'm going to just run kubeflow deploy on the left so in the left screen we'll see the what the deployment script actually does and on the right side we'll see the result of what is happening after the deployment script is executing its uh automation so the first thing that we'll see is a lot more namespaces get created and all of these namespaces are kubeflow related and if you have to do this manually you can imagine there are a lot of steps that you would have to go through to make sure that all of these things are up and running and working with each other seamlessly and the aws do kubeflow project uh abstracts all of that from you and encapsulates it in a container so that you can just run it and not have to deal with that complexity so um the works the the namespaces are created but let's see what's happening inside the kubeflow namespace i'm going to watch the status of the pods inside the keflam namespace and see that a lot of pods have been deployed here and they're one by one uh coming up and starting and initializing in order to bring q flow up i think uh alexis i think it's a great uh uh for a place where we can actually mention that all of these parts which we see in the keyflow team space this corresponds to uh different keyflow components which get deployed as part of the control plane right yes exactly um and also uh if we uh take a look at this slide right a lot of these components they they come as part of the kubeflow distribution but they can be swapped to aws managed services for example the out-of-the-box authentication service using dex can be swapped with amazon code meto and then you can have your logins to kubeflow be integrated with your aws account your menio service which provides the object storage inside kubeflow can be integrated with amazon s3 and then you can store all of your objects on a flexible and scalable amazon managed service and uh for secrets uh we can use aws secrets manager um instead of uh amazon efs we could use fsx for luster and in place of mysql which stores the metadata inside kubeflow we could substitute that with amazon rds and all of these are deployment manifests that are provided by aws and uh open source and available uh for use um right uh and just to add to that i think these uh control plane components which comes with q flow they're pretty heavyweight components right they take up a lot of resources on your kubernetes cluster and if you if you replace them with the atlas managed services that means you are making your cluster light and it also makes it very easy to switch over to a new cluster or your upgrade your existing q4 environments right and so now our deployment is finished and as we can see all of the pods are up and running but just because you have deployed something to a kubernetes cluster doesn't mean that uh you have access to it there is an extra step to expose that deployment to the outside world and we're going to do that securely through our cloud9 ide environment and for that we have a script called kubeflow xpose so i'm just going to run kubeflow here in my right window and this will port forward the istio ingress gateway into my cloud9 environment here and after that i will just click preview preview running application and the kubeflow dashboard the login screen will appear here as a tab and just so we don't lose track i am going to scroll we have we have completed the deployment of kubeflow and now we're on section 4 where we're logging into the kubeflow dashboard so all of this is described here if you want to use this in a bigger screen there is a button in the up in the upper right hand corner of the tab which opens this into a completely new browser window so you can have the login screen and the dashboard in a big window i recommend to do that because we'll be doing a lot in the screen we'll use the standard login user at example.com and the password is one two three four one two three four this comes out of the box it's just the default credentials when you get kubeflow up and running for the first time using dex and now we have uh logged in to the kubeflow dashboard please make sure that the namespace user keyflow user example.com is selected up here and that it appears here this indicates that your pipflo dashboard is working properly and your kubflow deployment is successful and with this i will pass the screen back to canwell so he can walk us through the kubeflow dashboard and the next next steps in our workshop thanks alex now now once now now we have to keep your dashboard up and running right so let's run our first example of uh submitting distributed training on using keyflow on given id stack and if you see my screen we this is the technology stack of queue flow and qfloor training basically is a group of kubernetes operators now these operators add support for distributed training of machine learning models are using different frameworks so if you see on the screen um you know if you're working on frameworks like tensorflow there are operators like transfer operators if you're working on python you can use apache operators and there are operators for rather uh distributed frameworks as well now when it comes to pytorch framework uh these operators actually require a custom resource of type pie torch job which is basically a kubernetes uh customer source which allows you to run distributed training on the kubernetes stack you can actually use that run distributed and non-distributed workloads on on cabernetis and qfloor implementation of python job customer source is basically called keyflow training operator now let's look at the architecture of how it works once you use keyflow training operators for pytorch to summon distributed training on the kubernetes stack now you as a user right the first step is that you have to specify a configuration file which will be of type by torch job because that's the customer source and you have to apply this configuration and submit it to the queue flow api server so kubernetes api server now now the aps server will validate you as a user it will authorize authorize you if you have permission to submit that file and once uh you pass this authorization check it will actually go ahead and save that file to the xcd database and it will acknowledge the user that the particular resource has been successfully saved and at the same time it will notify the keyframe training operator now this operator is running in one of your kubernetes worker nodes and once it is notified this operator will actually read all your configuration from your yaml file which is basically a manifest file where you specify how many worker nodes you need how many master nodes you need and it will read that and spin up those many containers in a distributed fashion so these containers will start running your job in parallel and start working on your data patches in in parallel now now we are specifically using uh pytorch algorithms uh basically the pi dodge distributed parallel library for this workshop and it what it does is it actually allows you to copy your model um in each of the process that you're running in these three containers and your neural network which is part of your model will actually go through the forward and backward pass for every data patch which is executed in these containers now once uh you know the backward pass is done your gradients are synchronized and the master will be responsible to do that and once that is done it the gradients will be sent back to the optimizers running in each of these processes and the optimizers will be responsible to update the weights right one other thing to mention here is that we are using elastic file system amazon elastic file system which is also called efs uh and it will mounted on each of these containers so that once you're done with your data processing or feature engine ring you just have to copy your training data set at one location here and then every single process that is running your training it will actually work on the data set which is in your efs so you don't have to really copy it all over again to your your processes in every container it will pick it up from one central location even your code could be could be kept here so that uh you can actually execute that code in in these containers right so with that uh let's probably run our first lab of uh running a distributed training using q flip it or operators so i'm going to switch my screen and go back to the keyflow dashboard where we left off uh during alex demo so here uh on the left navigation you will see various different components that kubeflow provides let's go ahead and create the volume the elastic file system volume that we talked about and seen in the screen we're going to go ahead and create that here so on the screen we'll click click on the new volume button okay and then we'll name this volume as pfs see now now this name could be any name when you're working on your own use cases but for this workshop there are certain configurations that are dependent upon this name and if you will stick to efs as you claim the same name it will make uh running this workshop a lot easier for you right so let's let's go ahead and then use this name um in in your workstations as well now now we are actually claiming a persistence volume directly from the storage classes because we're using automatic provisioning in the background and we have created an auto a storage class specific to efs so let's go ahead and select efs sc from this drop down and then click on create okay now once you click on create uh this will show you that your volume is successfully created right and then um if i go back to the workshop steps here we are at step five uh we are creating uh efs volume and we have done that the next step is step number six where we're going to go ahead and create our jupyter notebook which will be an ide environment we will be building up our code and we'll be running it okay one point to add here is that you can use jupyter notebooks through the aws distribution of kubeflow but it also supports vs code server and rstudio which are um on that create screen so you have options for the workshop of course we'll use the jupiter notebook but it is really really good that there are other options as well because you can not only provision as many ides as you want you can also pick what kind of idea you want to use yeah it'd be pretty flexible yeah thanks for mentioning that alex yeah so so as part of step 6 let's go on the left navigation and click on notebooks and here we go ahead and create our first notebook by hitting this button on the top top right corner okay and let's give it a name this name could be anything so i'm just going ahead with [Music] right um here uh we will mention a docker image now this is the image which will include set of libraries and frameworks and which will be used to initialize your compute environment that will be supporting and hosting your jupiter notebooks now it's a good place to mention that when you deploy qfloor which is an aws q for distribution it actually has integration with uh with aws deep learning containers and you can actually select uh pytorch and tensorflow specific deep learning containers which are optimized and tested to run on aws environment so right now there there is 1.1 version of pytorch available but these the other version that we are continuously updating and maybe sometime later if you see this you might find newer versions added to it so one other thing you can do here uh these are the ones which we provide out of the box plus uh you can also select your own custom image by giving a mention a url of a docker repository or the ecr repository so again a very flexible solution you can create those images from scratch and bring in and add in your own organizational specific security hardened libraries and packages to work with the id environments okay so for this workshop we'll go ahead and select uh this one image which is pytorch cpu based 1.11 version in cpu let's mention uh one as a cpu that we'll need and five gigabyte of memory so i'll i'll request everybody to just stick to one cpu because that's all we need in our developer environments uh if you try to increase this number you might uh get some errors because the kind of clusters that we have provisioned for each one of you uh they are limited in capacity so just stick to one um so that we can run our example successfully okay so in the gpu we'll just keep them as default we won't mention any gpus workspace volumes will again leave this to default however i'll mention i like to mention that you can use efs file systems and fsx for luster to be added as your workspace volumes as well so just for uh just to uh for the example on how we can use efs and mount it to our jupiter notebook parts i will go ahead and click on add existing volumes here for data volumes and here you will see a section existing volume uh with a small arrow towards the right so click on that that will expand this section i'll go and select the one that we have already created the volume right efs sc clean and make sure that you mention the same name here okay so you can you can definitely have the flexibility to change it later on but then just for this workshop let's stick to this this name so that we can we don't run into any problems and in the configuration let's select uh allow access to keyflop pipeline this will enable our code uh to call the keyflow pipelines we will be at the stage of creating automated workflows okay and after that we'll just go ahead and click on launch now um take a pause here uh just to let everybody catch up to this point right so our notebook is getting provisioned we'll wait until this button connect button on the right it actually gets activated but again um you know just to mention on the on the left hand side there are a lot of different component comes with uh the keyflow you can actually track your experiments you can run automated workflows and pipelines uh you can actually use a cative environment which is one of the components in queue flow which allows you to run parameter tuning jobs as well okay okay now our environment is active so i'll click on connect to connect to our ide environment now in this case uh this is uh this will be a jupiter lab environment and a couple of seconds before it actually brings it up and and for anybody who might have worked in jupiter lab environments before this screen might be very familiar to you on the left hand side this is the place where you will browse through your code and in the center uh you can spin up your new notebooks using python 3 kernels and you can even activate your terminals to really work on bash shells so with that i'll actually go ahead and clone a repository so this repository uh is something that you can copy from the work uh the workshop steps and our moderators will paste that in the chat as well and it's the same repository that we used when we deployed kubeflow exactly so here uh this is the folder that has created uh and just to mention when you will log into the jupyter lab environment you will already see this folder efs sc claim automatically created now this is the folder uh which is basically uh is a is the efs volume that is mounted on your jupyter notebook any content that you will copy in here if you mount the same volume to any other pod that part will be able to see that data okay so right now it's empty but we will copy a data set and code into this particular shared folder okay so click on the aws to queue flow folder and then here click on the worksheet which is kind of a workshops folder basically it's kind of a virtual link to our code repository okay and once you reach here uh let's start our notebook marked as zero initialize dependency and here we kind of decoupled all the installation steps for all the libraries that we'll need to run our code okay so we go ahead and either hit shift enter to run this particular cell or you can just go ahead and click on this button to run it and there are multiple ways you can run every cell right you can go to run menu also and you can run the selected cell from there now now this particular step is installing our keyflow pipeline libraries sagemaker libraries kubernetes and key floor training and while this step okay so we already are finished with the installation step okay once the installation is completed we'll go ahead and move over to the next notebook which is submit python distributed on kubernetes marked as index one so i'll take a pause here uh just to let everybody catch up with me so just to make this clear in the workshop we have these two examples one is going to run pytorch job on kubernetes itself and two is going to run that as part of a pipeline in cube flow pipelines yep yep exactly now yeah i could just mention that alex so so the first notebook basically uh you know we are kind of mirroring a use case where when you are trying to build up and work on a new machine learning use case the first step is experimentation phase right where you want to really experiment building up your code and run it on your local environments or on your local kubernetes cluster so notebook one is is the one which will allow you to run that experiment and use your kubernetes cluster as your uh training environment and once you think your model is is ready uh and you want to run it at production scale that's when in the notebook 2 we'll build up the automated workflow using q flow pipeline and we'll see how that workflows could still leverage kubernetes environment as well as sagemaker looking at the conditional variable now now we we discussed this particular step here right so we're going to use a q flow training operators to submit distributed jobs on the kubernetes control plane um and i want to actually uh take a minute to walk you through a simple pytorch job configuration right that we talked about now at the end of the day this is the configuration this is the custom resource of type pytar job uh that needs to be created and submitted to the kubernetes api server so that your qfloor training operators could pick it up and could initialize your training environment now this particular configuration has a specs in it where you can define master spec and worker spec now you can you can specify how many replicas of master and worker you need so as we've seen in the screenshot before we are actually spinning up two worker containers and one master container and these containers are actually getting initialized using the aws deep learning container images which is pi torch 1.12 in this case and we are mounting the ef with sc claim volume here and this is the code which which includes our neural network that we have built and this is placed in the efs modification as well so we are executing it using python command and passing it few hyper parameters right and similarly the worker container also the similar steps are executed we are running our training code and passing in few hyper parameters and this is the mount location from where we're going to pick up our data set right now now we can definitely create this yama from scratch and directly run it on kubernetes a cli environment however uh data scientists sometimes do not want to work on ml files right because it it needs to be formatted it needs to have proper indentations and it's not like a python friendly uh kind of uh way of building up your training configurations so what we did in notebook one is that entire yaml will be creating it using uh python uh code uh using the keyflow or training operator python sdk okay so that's what we're gonna see in in notebook one so let's let's run this notebook uh so first we're gonna import the libraries so we are importing two kind of libraries here one is the kubernetes client for python and one is the keyflow training client for python okay and then we initialize some global variables our namespace is printed here so we want to make sure that it is queue flow user example.com and here we are copying our code to the efs uh mount uh location and then at this step we are actually downloading the cipher 10 data set so the kind of training uh the model that we are building is a convolutional neural network which is basically solving for image classification problem with the cipher 10 data set right we're going to look at that uh particular file in detail but this is a hello world neural network that we have built here and at some point later we're going to dig through uh every uh specific logic in this particular training file okay so for now let's keep going ahead uh so this is the place where we'll start building up our yml file and this cell creates the volume and then we initialize the volume mount location we define how many resources we need so for this workshop uh we're going to be doing distributed training on cpu instances on kubernetes stack and when we will build up our hybrid pipelines and we leverage sagemaker that's when we're going to use gpus on stage makers so we wanted to show you the art of the possible that you can either use cpus or gpus in either of those environments but we want to give you a flavor that both of them both of those are possible are the possibilities and you can choose any one which will fit your need okay so i'll initialize my container and this is where you see that you know i'm passing in my file and the hyper parameters to it and finally i initialize my master and worker spec here you will see that i'm i'm creating one replica of my master and two replicas of the worker node and finally i create my pytorch job which is referring to that master and worker spec and then it will be used to launch our training job and in this cell i'm just making sure that there is no existing job of the same name and finally this is the command which does the magic right at this point your your python distributed training is submitted to the kubernetes stack and your training operators are building up that environment on kubernetes and your training has initiated so if you want to look at the yaml file that has been created you can actually uncomment this particular cell and you can see here that these are the default values which are all blank but at the end of the day this is the master spec which you have seen and there's a worker spec uh which gets created through that python code okay finally uh we're gonna um you know you can extract the master and worker specification into a separate folder okay the reason we are doing this is that uh these specs will be used in our pipeline components which will request for the master and worker you can definitely create this from scratch all by yourself but we wanted to automate that so that you know it will be a little easier for us to use this when we will be creating the automated workflows okay and finally uh we can actually read the logs right from the jupyter notebook now this is the smaller utility that we have written here which will allow you to query the logs for the master and the worker parts so here uh if you see you can actually mention worker uh and zero is the index so the worker part will start with zero and then we'll keep incrementing uh by one after that so you'll see the logs form worker zero and then similarly you can see it for worker one as well if you want to see all the logs for master and worker together you can just type in over here and it will show you all the logs for master and worker okay now this is these are the uh values that i mentioned here as well uh you can you can refer them and use them however uh though uh you know we wanted to show you that uh data scientists we don't have to log into the kubernetes control plane for anything they you can very well stick to jupiter notebooks and you can do the end-to-end testing from this environment itself however i wanted to show you the notebook the cli environment tool because this gave us a little bit more flexibility in coding the logs and if you see when i query for the parts in my namespace there are three parts of the three containers that have been initiated in the running mode we have the master and the worker and if you want to see logs for the master let's copy this one here i'm going to see it in this namespace so you see that you know you're locked will keep appending and you can monitor them here and just to show you the distributed training parameters so we have a world size s3 world size means that there are three different processors that have been initiated in parallel which is working on the batches of data together and because there are three processes which are currently running in three different containers we have split our data set into the into the batch of three right so one third of the dataset will be processed by each of the containers so 33 of your data set will be processed by every container okay now you will see here that a loss in the in the forward pass uh and after after the forward part is completed is actually getting calculated here and with every epoch that loss is actually decreasing okay so it's after 2.30 and then finally it goes to 1.7 and we are calculating accuracy uh as well after every epoch and you'll see the accuracy is increasing too uh now just want to mention here that this is a kind of a hello world example and we are just trying to demonstrate other ways you can actually make your training environment run and track the training runs once your environments are set up that's when we'll go back and we'll optimize our code right so just for the demonstration that it is actually working and your your loss is decreasing accuracy is increasing but yeah we can always uh fine-tune it uh later on once this environment is already okay so with that uh actually you can we go back uh we can click on this cell uh this will show us true our job is succeeded and all the you know the commands that we have used to monitor the logs uh those commands are typed in here as well okay so this this concludes our first demo where we have used keyframe training operators to submit distributed jobs on kubernetes stack so so with that uh alex uh do you want to talk about keyflow pipelines before we move over to our next example where we're going to build up those pipelines using python code yes kanwal so let me show a couple of slides here and talk about kubeflow pipelines um the kubeflow pipelines is the component of kubeflow which lets you orchestrate multi-step ml workflows it has a ui that lets you visualize your pipeline jobs as they run as well as track your experiments and it comes with a software development kit which allows you to interact with your experiments and pipelines programmatically which we'll see later in the demo the way you build kubeflow pipelines programmatically is by building pipeline steps and each pipeline step is defined by a component components are basically functions that get uh encapsulated in a container and that they get defined uh and deployed as steps into the pipeline and i will let canval dive into the code to show you how this is actually done from the second example yeah thanks alex maybe maybe i'll use uh um one or two slides just to show the steps which uh goes behind creating any any pipeline right so if you see here in this uh diagram so we start with our training code right which is my app called python file so once you are done building up your training scripts you actually have to containerize it and build docker containers out of it once you have those containers ready you actually use a keyflow decorator functions which will convert those containers into the pipeline component and once you have built up those components you will actually create your pipeline bringing all those components together and defining your workflow and finally you will have to compile that pipeline from your python function into a yaml file with the underlying execution engine which is argo in this case will be able to pick it up and execute right and here is the sample code uh so so by creating a pipeline is is very straightforward we we can create them very well from by defining a python function here in this case we are defining that there is a python function sample train pipeline we just have to decorate it with the keyflow pipeline python sdk uh this is the decorator which provides a dslr pipeline and once you do this uh for this decorator every single uh component or step that you will mention in this pipeline which actually be picked up and converted into a workflow which we see on the left hand side right the only uh thing that we have to do before uh executing and converting into this workflow is that we need to compile uh this particular python function uh into a yaml file which will happen in the background and that yaml file will actually be read by the argo underlying engine to convert it into a visual workflow okay uh one other thing which i want to mention is that these pipeline components are basically the yaml specification right which the queue flow community has uh described in detail into the in the documentation so in that specification you ideally have to mention your container which will actually run the code for that particular step uh any inputs and outputs that are going coming in and going out of your container and and the metadata for those containers and and these uh specifications actually allow you to standardize those components and you can build up those specifications and components all by yourself or you can leverage those components uh from um you know any any one of our partners and we provide uh you with one uh of similar components which we call amazon sagemaker component and that component allows you to submit your training jobs right from your pipeline over to sagemaker service only thing that you have to do is pass in the input parameters to that particular component and that component will in the background be calling the sagemaker apis depending upon the parameters that you will pass right so with that uh let's probably switch screens here and look at the demo tool okay so we left uh here at screen one and then we're going to click on our notebook tool which will create the pipelines now now just you know as a refresher you know for the people who haven't worked on sagebreaker or who have recently worked on sagemaker just to understand how sagemaker actually executes their training so so so as a data scientist uh you know the way maker works is that you you call a sagemaker api and you submit that api to stagemaker control plane and in that api you actually mention what kind of training cluster you need to run your job you specify the framework that you're working on number of instance types that you need and the compute power right and sagemaker will in the background will spin up that training cluster now this isn't a family cluster this cluster is only will be activated while the training will happen and once the training will complete this cluster will terminate so sagemaker will spin up the number of ec2 machines with the number of gpus that you will specify in the api and this cluster will pull in the training data set from any of these data sources and we support amazon s3 amazon efs and fsx for luster and the training code basically which is in form of container image will be pulled in from amazon elastic container registry service which will be used to initialize your container and then your training will start on this data set okay now once the training completes the generated artifact which is a model artifact that will be copied to an s3 location that you will specify in your apis as well and your logs and metrics will be shipped to amazon cloud watch so you can always go back in time or even during during your training run you can you can go and look at you know the parameters your system metrics your any kind of logins that you have in your in your script right and then this is this environment is fully managed you really don't have to uh you know worry about creating this automatic dominating this seat maker will do that for you and this will be only built to you for the time the training run is executing so it really uh completely managed service and then it takes away a lot of overhead which which you otherwise will have to go through to create manage configure these clusters by yourself okay now now now once so now provided we have understood how sitemaker works right so it has its own advantages right so maybe the kind of use case where you know come back and say okay fine you know i i want uh to run my training runs on kubernetes environment in the experimentation phase because i just need wonderful gpus and my kubernetes cluster has those but now once i'm ready to run my models at the production scale that's when i'm i need 100 gpus and my kubernetes cluster doesn't have it you probably will have to go back and reserve that capacity or ensure that your cluster might scale up uh and provision those 100 gpus and then you'll have to ensure that it scales back down so there's a lot of things that you'll have to do or you can otherwise say that okay fine you know 400 gpus let me submit this training run to sagemaker and let stage make us spin up that uh infrastructure for me and and run that training cluster uh training job on its own cluster and give me the model artifact right so that's that's where that's what the use case is here and then that's where it makes it a hybrid architecture right based on a conditional statement you can either submit your job in the experimentation phase to your eqs which we have seen in the previous example or you can use sagemaker in your pipeline to submit it on stagemaker specific vpc okay now now let's get started so we will first import all the libraries that we will need in our example uh we initialize all the global variables that we need right and then uh we just do some boilerplate code we assign a role to stagemaker though so that it can have access to talk to the s3 service we specify the region which is usq in our case and we need to initialize stagemaker session which is needed to execute a stagemaker specific apis okay and finally uh we download our data set now the good thing to mention here is that because we are using the same efs volume uh to mount on our sagemaker training cluster as well as kubernetes training cluster we really don't need to download the data set all over again that data set is already there and we're going to use the same dataset to train our models on stagemaker and humanities as part of our pipeline okay and finally we initialize the sagemaker uh parameters which are required by safe maker component so we need the persistent volume claim that we have created uh efs file system id um we need the motor 3 client to get the security group id which sagemaker will use to run its uh cluster and to have a different computer instances talk to each other and this is a place where we are loading the uh component this is a place where we loading the sagemaker built-in components and aws service teams actually own this and they continuously uh keep them updated and upgraded uh so that it it matches with the search maker apis so we are we are downloading components for uh hyperparameter optimization model training uh deployments though we're going to use only the training but we just kept the other ones here just for reference okay and finally we have to push our code which is just a training script to s3 because sagemaker requires that uh in s3 and then we initialize our pytorch operator components as well now these python separator components will basically use the python operator apis that we have seen in the first example and it needs the master and workers pack and if you remember there is was one point where we created those uh files automatically and we are just reading those files here so that we can inject that into our component so this is the place where we are going to load our pi torch job component maybe we can just go ahead and see how that component looks like okay so these components are basically as we discussed our yaml specification where we specify the inputs and we specify a container right this is a container which will actually be running as part of our that pipeline step and these are the parameters that you will specify which will define how that container should behave and should run okay and then there will there could be some output statements as well okay now with that we load the pi torch operators here okay and then now this is a function which uh will define one of the pipeline step so you can actually define your python components as the ml specification and you can create your pipeline components from a python function as well and this is an example of that so i'm just writing a function here and i'm saying that convert this function to my pipeline component okay that's a pipeline uh sdk actually provided with this uh library which actually allows us to convert this function into a pipeline step okay now we come to the uh interesting part where we are actually creating our pipeline so if you remember in the slide uh this is the python function that will have a decorator defined here on top which is the dslr pipeline which will make this uh an actual executable pipeline and here on top we are initializing a variable which will define the kind of training run which we can uniquely identify on kubernetes and the stage maker and this is a place where we are actually controlling what part of a pipeline should run right so here i'm saying that i want to run my pipeline natively on kubernetes but if i specify sagemaker that pipeline will automatically submit the training run on stagemaker service and these are three steps here in a pipeline so we have kept our pipeline very simple uh just so that we can all catch up uh and understand that in and out and rather than making it too complex and adding in two different too many steps to it so there are three easy steps here one is the conditional step which will read the runtime variable and based on that step we will either execute our pi torch operator uh component or a sagemaker operator component okay and each of these components are expecting some variables which we are we are initiating uh you know based on the values that we have created in the past cells okay and here if you see uh um we are actually using two instance types of uh type p3.2x large uh this is the you know the naming notation of our ec2 instances uh p3 is the class of ec2 instance uh which are gpu based and 2x large is the size of that uh instance okay now one thing to mention here our moderators are actually checking in in uh in case they find out that there is not enough capacity for these particular instances they might give you a different instance if they do that make sure you change that instance type here at this point okay and finally we define these three steps and and then we'll have to compile it okay so let me make sure i do that so i compile it uh this will actually execute and convert uh we'll create a yaml file here see so this is a ml file that's created of kind workflow and this is something that is argo specific and then we will execute the pipeline through our q flow pipeline client uh which is provided by the sdk okay and then run pipeline will actually run it on the qflo pipeline and you can look at the run details by clicking on this link okay so here if you see uh my condition step has executed this is the stage maker step so this is has been evaluated as false because we have passed in uh the pi torch uh sorry the q flow as a runtime variable kubernetes so we can look at the runtime variable here okay so it is kubernetes so that's where our fire torch operator job has actually started executing and again as we did in the first step you can actually look at the pipeline part that will be initiated in the in your name space so if you see here there are three containers that are executed two worker and one master on your kubernetes stack now while this step is running we'll go back to our notebook we can actually uh initialize our variable to sagemaker we will run this cell again we'll compile it again and we'll run this uh python execution one more time okay now in this case it will actually be submitted to amazon stage maker okay so it's going ahead checking the condition and then you see pi torch one is evaluated to false it is disabled and the sagemaker one is actually currently running now now now these are distributed training jobs uh it might take some time for these training classes to initialize while that is happening uh maybe we this is a good time to actually quickly see how our training script looks like i think we have we have uh kept it uh for later right so maybe this is a good time to talk about it but but i'll take a take a pause here uh to see if you guys have any questions uh and to let everybody else uh catch up till this point another thing that we can cover a little bit of is we see that we can specify whether we want the work load to run on kubernetes or on sage maker with a simple parameter and that's great hybrid architecture it's very easy to switch between the two but what are some of the decision factors on when you would want to pick one versus the other and you already mentioned one of the use cases is really stable production workload that you want to scale what are some of the consideration that would steer you in the other directions and and that is uh one thing i've seen that i wanted to share is that some customers want to have full control of their infrastructure and full visibility and even ability to debug and and log on to nodes that are executing the workload and look at uh very uh look into details of how things are running and executing and that is why they kind of they prefer to own completely the infrastructure that they're running on this is something that you have to to give up when you're using fully managed services because that infrastructure is taken care of for you but in that case you also don't have to do anything to manage that so it really is a choice in certain cases it is better to go one way and in other cases it's better to go the other and i think uh having the option to switch between the two seamlessly has a lot of value that could be um utilized by uh many of the workloads that we've seen yeah no i think i think you said it uh very well alex and i think um you i think that the decision criteria could be like you know if you want a flexibility control portability choose your kubernetes stack but if you need a kind of a managed environment which is easier to use and to scale without any kind of operational and management overheads uh then you can use sagemaker to get that burst capacity and that number of resources are actually right so with that if you see uh we already are done with our pipeline execution for pitoch operator training job uh we can always go back and see that those parts are now completed you can go into them and see the logs as well so while the stagemaker might still be running maybe let me take a minute to actually walk through the training script okay now now here we are one other thing which you want to mention was we are using uh an open source pytorch libraries there is not even a single api that which is sagemaker specific and it's the same code which actually is compatible to run on your kubernetes cluster and on stage maker basically in kind of a platform agnostic way right it gives you an extreme flexibility and design streams doesn't have to really change your code to make it compatible with one platform uh versus the other okay and we are using a specifically a torch.distributed uh module uh which will allow and support uh conversational uh communication patterns of to support multi-processor cpu training uh here we are building up our custom neural network which is very simple uh image classification uh neural network uh which is built up on com with a combination of convolutional layers uh followed by max pool layer again convolutional layer which is followed by three fully connected linear layers and during the forward pass each of these layers are actually combined with the relu activation function and then we have defined a function to really load our data set we are using cipher 10 data set and we are doing some transformations on the incoming images we are doing data normalization uh we are including some randomization and converting them into tensors and here we are using distributed sampler uh to really uh let us uh get these pieces of data in a distributed fashion and finally data loader will combine the data sampler and data set and will provide us with a single or multi cpu and multi-gpu iterators which could iterate over those patches okay now the training is actually happening here um so if you see we are switching between our cuda devices and cpu instances sakuda for those who don't know is a parallel computing platform which provides application programming interfaces for your jobs to run on gpu if you have the gpus on your trading cluster this particular script will automatically identify it and will use those gpus for for running your training model training and if you don't have it it will actually will run it on the cpu instances so again this script can run on both cpu versus gpu based instances uh seamlessly one thing that you'll have to do is because you're using uh we're using python distributed parallel libraries uh we will use any process group to initialize our distributed environments on each and every instance where we will running our will be running our training in and here we are using a rank which will actually define the index of the current host where this process is currently running okay few important things that i'll mention is that once you have defined your model you actually copied that model to a particular device it could be a cooler device or a cpu instance and if it is a distributed training uh we are actually checking here the world size if it is more than one and if the koda is available then we're gonna wrap up a model with the distributed panel libraries which will ensure that our model is copied to every single process and that the data set is split at the batch dimension and been and being sent to each of those processes running in our distributed training cluster okay then we use cross entropy laws uh we use stock uh stochastic gradient descent optimizer and finally we run through different epochs and go through the forward pass backward pass and finally compute the gradients and let the optimizers update the weights okay and then we we test our models on the test data set to see how or better they are performing than the previous run and finally we saved the models on the master node either to the efs mount location or to s3 in case of stage maker okay and then finally we have all these environment variables that if you remember we are passing in when we are running this particular training script okay so with that maybe let us go back and see whether our pipeline has been executed so pytorch one is executed and sagemaker is still uh still working so maybe it might take a couple more minutes for a stage maker to complete a training run now one other thing is that if you're running it on stage maker uh one uh if you want to monitor what is happening on this speaker cluster side you can switch over to the stagemaker console as well so sagemaker is something that you can type in here uh this will actually bring up your the service you can click on it and you will land onto the statemaker home page once you come here on the left navigation then actually go to training and can look at your training jobs and you can see that this particular training job is in progress which we have submitted from our pipeline and we can see the status of it so we click on new history you can see that you know it has done with preparing the instances it has downloaded the input data set and the training is currently in progress okay and this is this is one of the places where you can actually go back in time and can look at every single parameter and configuration that goes behind running this particular training run a very very useful information for model governance and and visibility you can see here what algorithms were used of what efs location was mounted and finally you can go to your move clicks in cloud watch to see what is happening uh with your training run okay so this new logs will take us to cloud watch screen and talk about cloudwork is a service which actually is a centralized service to track your logs and to monitor your system metrics so every single training round that you do on stage maker cloud watch will be the place to go and look at your logs okay so if you see here because we are using distributed training we have these two uh groups created uh one for each instance type and if i go and click on one of these instances i can see that my training is getting executed on sagemaker and i can see all the uh and the accuracy and the loss matrix being generated right and then along with these logs there is some other system matrix that you can track as well um which will be your algorithm metrics your instance metrics and and search logs and one quick question here uh now on kubernetes we trained on cpus and on sagemaker we're training on gpus but on in the kubernetes environment we had three machines in sagemaker we have two machines how does that reflect on how much data each machine has to process right very good very good point again uh because we are using a distributed uh data pattern library uh depending upon uh number of devices you have in one instance and the number of instances you have that actually gets multiplied so if if i am having two different instances and every instance has two gpus that will be two into four devices that will be available and we'll be uh you know sharing one fourth of the data set to each of those processes in this case in p3 2x large instance there is only one gpu so uh and we are using two instance types so that means uh half of the data set is being sent to each instance and is being processed in parallel okay so that's why in the logs we saw that uh there are five thousand uh iterations per epoch yeah so 50 000 are total number of images and every instance is processing 25 000 images okay let's divide by two right yep and if you see here uh our stage maker training run has finished as well um so so that actually uh will actually conclude our second lab as well where both these training runs are have been successfully executed so alex do you wanna uh go ahead and wrap it up great yes this is a great demonstration um let's summarize uh what we saw in this workshop first the things that we want our audience to remember is that kubeflow provides an end-to-end ml platform and machine learning capabilities on top of kubernetes if you want to deploy kubeflow the aws do kubeflow project is a great way to do that in a simple uh and repeatable manner on aws and as part of this workshop we used kubeflow and pytorch to demonstrate how we can build a hybrid pipeline that performs distributed model training on both self-managed and fully managed infrastructure we are going to leave this page for just a moment so you can take note of some references that might be useful on this topic we are sharing here a link to a blog about kubeflow on aws the kubflow manifests that aws releases to deploy those uh distributions of kubeflow that integrate with managed services a link to the do framework that you do qflow project and uh our uh distributed training workshop which you can execute independently uh as well if even if you don't have a temporary account you can use your own accounts to do that we want to take a moment to say thank you to our colleagues who helped us contribute this workshop to the public and helps helped us put the content together and as always thank you uh to the audience and if you have any further questions please feel free to reach out to us yeah yeah thanks everyone for joining us uh so we really hope that you got a good introduction to qfloo and how you can build hybrid architectures using kubernetes stack and sagemaker please feel free to reach out to us through linkedin through amazon email forums or and please share your feedback through the service so we can continuously uh can deliver these workshops with more even more exciting contents thank you you