Deploying machine learning models for inference- AWS Virtual Workshop

Key Takeaways

hi everyone today we are going to look at generative AI model hosting and we are going to look at how do we host at scale on sagemaker we are going to be looking at a couple of options so first we are going to dig into what are the challenges the key challenges which we Face uh when we are trying to deploy a model for inferencing or hosting as different people tend to call those words and then we will get into what are the potential Solutions how stage maker has addressed some of those challenges and solve them and then towards the end we look at the demo so we look at couple of two different models on sagemaker and we will also cover in French yeah the latest offering to get the better price uh performance ratio so let's look at large models for generative AI has come off age undoubtedly since last last year I would say towards last year it has been one of the most popular topics across the globe and the reason I say across the globe is you know just connecting personally on on this aspect I was talking to my parents in India a couple of weeks ago and my mother was asking about how does she open a chat GPT accounts which tells me that if that generation who has not seen for which electricity was I would consider a luxury if that generation is now asking about chat GPT and open and how do I go large models it has come off age so what are the challenges in hosting the large models right if you look at the large models your key challenge is the complexity and by complexity what I mean is the model size the model size are increasing we have one trillion parameter models now coming up so hosting them and fitting the model in a single GPU is challenging you could have different workflows you could be invoking a series of models before inferencing result comes in you could look at your batch size and some of those expertise need to become so that's the First Dimension right I tend to call this as a trilateral problem the second dimension is of course is your performance because you are trying to have a low latency you look at the chatbots the chatbots are effectively human has uh somebody told me from psychology like two seconds before they tend to zone out so you need to have a response time less than that and you're talking of a large language foundational model uh trying to respond within that time and then you have multiple users so your throughput is is increasing as well and you're trying to balance all that with cost so these are three different dimensions they're pulling in each Direction and you're trying to effectively see how best can you host uh keeping all these parameters in line take a simple example I'll take GPT J so gptj will not get into the training we're going to talk about the hosting but training is roughly about 200 gigs if I talk about model serving NFP 32 weight and I add if I let's say I ignore the checkpoint right so I basically just have the initial weights which is a 24 gig gptj 6 billion parameter I add that that size I'm roughly talking close to about 25 26 30 gigs roughly of memory capacity which I need for a gptj and then of course you know on top of it you have a framework specific memory allocation your Cuda context which need to be managed so all of this put together can vary from between 24 to 30 gigs uh per model which you're trying to host and of course in fp16 you're going to reduce this by half so fp16 would be roughly about 15 gigs right so how do we solve this challenge how do we solve this uh problem so the first problem we have effectively is how can I fit a model in GPU um if I take eight tens a TENS are about 25 gigs in size if I look at a 100s they're about 40 gigs p4ds for p4des are about 80 gigs in size so take that kind of a model uh blue 176 model the first Technique we applies is you know try to apply compression on the model but what by that what I mean is you are able to reduce the number of layers for that specific use case so essentially what you're doing is you're pruning the layers between um the weights so a dense layer like this may actually convert to a simpler Matrix on on the right hand side which is the first technique which will reduce the size of a model which you need to host knowledge distillation is one of the most common techniques it applies to pre-hosting about sequencing training ground so I'll skip that the second technique which is quite often applied is quantization by quantization I mean is you have your weight sizes in activation you have your weight sizes in your um in your last layers as you're trying to go through the forward parts of the model and and most of the models are usually in fp32 uh bit weight so fp32 is a four byte you basically you know multiply that by number of parameters that's what gives you your model sizes you reduce this to an INT 8 you're slashing your model size by four right off the bat now of course what this means is you are going to start to lose precision and so there are other technique there are other parameters available like bf16 which can also optimize but those depend on the model there's a model supporting bf16 is the underlying framework going to support br16 quantization as well assuming we've done all of that you know we are we have been able to reduce the size or we would not agility Society depending on the use case now the second question is okay I've got this model let's take uh a 55 gig model which is a client T5 double XL large model right a 55 gig model I'm trying to fit it onto a 25 gigs of A10 instances uh I will need couple of these to fit in here so the first technique what you apply is what you call is tensor parallel in that what we're doing is we are starting across the horizontal layer so each layer will go and sit on a specific GPU the the downsize or rather the trade-off is you are effectively going to now have Intel GPU communication which is going to increase so how does that work let's take a look at the animation so use fairly are sharding horizontally you the layers go in each one of these GPU devices I get my influence data they move across but I have now trade off a higher GPU communication and so my weights are going till the time I basically get a single result towards the end so that's the first technique the second technique basically is is what we call as model parallelism or pipeline parallelism Where You Are sharding vertically you're basically placing each layer on each one of the gpus now the challenge here is that GPU 2 can only be working when gp1 has finished so now you can see I'm starting to add a little bit of sequentiality in the processing which means I may not be able to optimize the gpus as uh effectively as I can in the earlier State because our goal is to keep all the gpus maximized and activated just one of the few techniques we had looked at how I can take a model which was not going to fit in a single GPU and how do I effectively make use of a multi GPU instance and how do I fit as part of the my hosting challenges right so in sagemaker we've taken a look at this problem and we're going to focus on the deployment side of this I'm going to skip to the slide where I am going to look at the the solution in sagemaker for the challenges we just looked at and sagemaker what we have is effectively what we call as a large model inference container this is the latest offering from sagemaker purposely designed for hosting large models in sagemaker so the the the first challenge we had in sagemaker was okay I've got let's say blooms 176 which is a 350 uh gigabyte in size model how do I take a 350 gig model and load it onto the container so we effectively now are allowing a higher EBS volume so you you can go up to 500 gigs on the EBS value which gets attached to the end point and lets you now fit the model onto the um onto the instance so once you've got the model onto the instance the next step is not loading the model onto the gpus or this gpus and so for that in the large model containers we basically are giving you almost low code low code option and I will show you as part of the demo of how we can achieve model compression how you can achieve model partitioning uh as part of our containers so let's look at this stack if you look at the stack we are talking of a um the section on my left is I would consider still in pre-training post training relevant well what I'm doing is I'm using model compilations to reduce the size of my model so there are libraries and Frameworks available tensor RT deep speed faster Transformer uh being the latest one and in this in this compilation technique you will have a no hit on accuracy and and no hit on performance by what I mean by that is since you're compiling the model to a specific Hardware you are effectively injecting kernels uh for that specific Hardware which are optimizing your performance and optimal rising and keeping your accuracy the same so that's really where the model compilation tends to score regardless of your workload it is something we always recommend always go for model compilations poster compilation now say I've done my model compilation right I'll take an example so I've got a 350 gig um 350 gig model which I've compiled down to let's say I reduce the weight by let's say 200 gigs right I'm just throwing a number out here so again it will not fit in any one of a known gpus so now I have to still look at either quantization or and slash or partitioning so quantization uh and reducing what we looked at earlier reducing the size of the weights in that case you are going to see a little loss of accuracy and so your accuracy is going to go down your performance gains are going to be there so that's a trade-off now having said that uh always recommendation is to do your model functional testing where you are looking at the model outputs comparing them to your known database sets to see um the output is in line with what your use case is and make sure that happens before you proceed further and the second technique once you've done the quantization let's say I've done the quantization brought it down to 90 gigs in size now I'm still trying to host it on the largest a 140 gig GPU instances I have to partition the models in the way we show either I can use tensor parallelism or I can use pipeline realism of both of them in that in that scenario whichever option I choose I will have an increase in GPU Communications that's why my performance and my latency may see a slide dip again it depends model depends use case depends Hardware a lot of variable factors here but I will not see a decrease in accuracy as part of my model partitioning so that's the key takeaway now when you look at all of this and you say hey that's a lot of things I have to do I have to worry about compilation and compression model partitioning and then I have to manage all this infrastructure uh that's a fairly significant heavy lifting and that's where LMI tends to score effectively in sales maker using LMI you are able to just take the you're able to take the all of the table to take the raw weights of the model from this point and you are able to get it down where you can now fit it into multiple 40 gigs gpus let's say I'm able to fit it in two of these gpus with only defining certain properties and it's a no code option uh in in LMI container and then of course you know we have uh Benchmark numbers and and this deck is available on wisdom so we'll be very happy to share some of those Benchmark numbers as well but what I would like to cover is um is is let's maybe we can stop here so this table is what I wanted to show you that you know when you look at the quantization which is one of the most popular techniques there are a few parameters to be aware of whether you're looking at read speed uh tensor RT or faster Transformer and the key point to take a look at it is do they support what we call as bf16 option or not the reason I say that is bf16 as at a high level basically splits your number in two parts so there is a first part which is the mentis and then is the exponent each is basically you know uh one byte each so you are able to take instead of four bytes and you have Precision High Precision you are able to still retain a significant precision and not sacrifice on that by going all the way down to intake right so that's where the bf16 tends to score and they're a lot more support work from the models coming out for vr16 so be aware of that which quantization libraries are you going to use as you're going through uh bf16 quantizations and so this I'd like to leave you with this picture this picture goes into okay here is where the LMI container is able to help you with the partitioning it will help you the quantization and also able to load your models across multiple gpus so that's very good now there was one problem which we uh kind of skipped and now we'll get to that problem and that problem is I've got let's I've got this 350 gig model in S3 how do I get this into the container and so getting this into the Container when I ran the Benchmark numbers took me approximately an hour just to load 350 gigs into the container and you have to load it first into the container and then you will you know use all of these good things to load into the into the memory and so in the LMI container we looked at the problem we effectively found a solution in form of S5 CMD and so if you specify your rates in S3 we are able to use that Library which is massively parallel library to load these weights now in in the testing which I did loading a 350 gig model is roughly about four minutes or so compared to you know 30 minutes which is taking now of course you know if you have your weights already pre-compiled or you have your weights quantized in S3 this time comes down to couple of seconds so most of the models we're looking at is couple of seconds to load from S3 and that's a very significant uh process Improvement which the LMI container brings if and for those who are familiar with sagemaker you are aware that sagemaker needs you to create a model.tart.gz so now imagine having to create a model.gz just for this 350 gigs model when I did this for the first time it took me about four hours just to clear the tar ball itself so imagine if I have to nitrate over that model uh the amount of time it takes to create the tar ball get it to S3 carry to the container and all that is pretty significant right so in lni what we looked at is we said hey we really don't need the tarpaul we basically just need the raw S3 location so you can now have your weights in the raw form in S3 and just point to your DGL container or the LMI container to that and it will automatically take care of loading the weights and making it available in your Edition script and behind the scenes it will take care of you know mounting the EBS volumes correctly mounting base basically where the the temp directory as part of the container because that is where most of the uh almost all the way it's go so enough of uh Theory enough of you know what we have looked at Let's uh look at some of the code and what we will do is we will look at two models so we look at an NLP offering that'll be a flan T5 a double XL model it's roughly about 55 gigs in size and then we will look at neuron SDK as well we will look at stable diffusion on use compiled with neuron SDK and then hosted on on stage maker now I do want to point out this notebook is available most of the code which I'm going to show you is available on sagemaker examples repo so you will be able to just take the file and run it and on top of it what we have also done is worked with the legal team and worked with our our partners in the sagemaker examples repo to make the weights available in public S3 so you really don't have to go to the hugging phase or download the way it's do the compilation we made that available for you so you can just take this notebook and pretty much run through the notebook as is just to get a flavor of you know how would you do uh some of the work so let's take a look at some of the important sections in this notebook right so before I do that let me actually run the model let me run the model and show you how the model looks like and then we can take a look at what it takes to get to this point right so I just basically ran a simple prediction the World Cup has kicked off in United States based on that can we conclude the workplace will take place in United States yes or no um so this is a flan T5 uh it is it is trained on the open source so the answer which I get is yes based on the prompt I've given and actually this notebook gets into a couple of prompts if you're interested in prompt engineering we are getting into Common Sense reasoning that's the example I showed you there is sentiment classification article generation well let's take a look at abstractive question answering so there is a chat which happened between the customer and the agent for an iPhone and the customer is having some issues with the iPhone he's trying to reset this and so here is basically what the chat looks like and then I ask a question to the model saying hey model based on this input can you tell me that is the troubleshooting steps which were taken are they fixed or not so when I run this let us see what the model basically will tell me and based on that the model is able to summarize what happened in this chat and it said that force quit the apps as they're using a lot of battery reset Etc so I kind of thing get picked up so it was able to take that context and you in the host is why am i showing you this is because it is important when we are doing all the techniques or quantizations and of sharding we need to make sure functional output makes sense we are not messing up or having garbled output come out of this second question we ask a series of this was a resetting solve the issue or not and then if I run this it basically tells me no based on the inputs which are given right so this is how the model is now performing so let's look at what it takes for us to get to this point the code which I'm going to show you is basically talking about uh is a low code or low code options by low code I mean is you really do not need to bring your infant script uh there are default handlers which are available in LMI so you're able to bring your infant script and just load from there and let's take a look at some of the parameters so to create uh the magic which we just saw all you need is what we call as serving.properties that's pretty much it you just basically say okay here is my S3 here is my serving dot properties here is my container now go deploy run and I can I can kind of Leverage this in serving our properties behind the scenes we are doing a lot more uh things which are going on right so the first thing is start from the first place I got my model in an S3 location okay move it to The Container what do I need to do I need to basically specify an S3 URL I say here is the S3 URL point to the top level folder where all the weights are and behind the scenes S5 CMD is used to just quickly copy them into the container the second thing which I tell a the LML container is what is the engine which I'm using now this is interesting because I can use deep speed I can use faster Transformer I can use hugging phase accelerate those are very standard uh libraries which are being used to Shard your model across gpus so in here I'm specifically we want to specify faster Transformer as the engine which I want to use because I'm going to use faster Transformer in fact in our testing Benchmark testing we find faster Transformer to be to be the fastest in terms of the inferencing for the same model using faster Transformer D speed and accelerate and then I have an option of specifying the entry point now if I give an entry point it will use this in our case because it's a low code option we will not specify this parameter if you want to bring your own script you have the option of specifying this and the example GitHub will actually go through this so instead of this we are going to specify I'm going to use faster Transformer djlpython.faster Transformer and that is actually going to go to uh the inbuilt Handler and it's going to pull the inbuilt Handler file and then uh let's use that to serve the model so before I show you this class let me talk about the tensor parallel degree first and then we'll take a look at what is the inbuilt Handler doing so tensor parallel degree is what you can use to specify the number of GPU devices or which you're going to partition the model so I'm going to I'm using a G5 12x large in here that's a 4 GPU machine and if I set it tensor parallel degree of 2 that means it is going to effectively Shard my model across two different gpus um now what that also means is that means the default number of workers which I will get will also be true because if I'm using two gpus they're fully saturated by model one and I have four gpus available then I can effectively have two workers so I can increase my throughput based on that so that's the key uh points which you want to take away and if you see over here in this case I'm specifying a tensible degree of 4 so I'm going to have only one worker behind the scenes and I'm sharding my model across all the four gpus as part of this and I use the faster Transformer entry point so let's take a look at the faster Transformers class right again this is part of the GitHub uh deep Java library from our AWS service team and you can take a look at the class it's open source it's Apache license so the key points to note over here is that when you are yeah you don't have to actually write this class you may not even need to look and kind of you know just understand what this class does you really don't need to do all that that I'm just showing you behind the scenes if you want to bring your own model.pi you can leverage some of these properties over here right the key parameter which you're looking for is what is the quantization I'm going to use so there is a d type available if you do not specify anything it is going to default to fp32 now you can do ahead of time quantization which means theoretically you could have had BF 16 weights in your S3 in which case it will it will default to whatever the waves are there but we are using a full precision as part of this and then your key method over here is load model in the load model all I'm doing is taking the faster Transformer and running the init give it the tensor parallel degree and also give the the sharding parameter which I need now this is an interesting line of code it looks pretty innocuous single line of code but if you really look at it in this single line of code I have sharded the model and I've also quantized the model at the same time with the single line of code and that is where the LMI containers really tend to uh you know just start to show the value of sagemaker as we are trying to deploy the large models the the input into the class or rather what the class does is it just needs you to have you want to do your own model or Pi for example right you basically just need to have one method which is called handle and in this method and it's a functional program Paradigm so the entry point is a single method from this single method you're basically just you know uh doing a a fan in and out of various methods classes whatever you want to do now what the reason I'm showing you this is because I wanted to take a look at at this call here when the digital server comes up the first thing it does is it makes a warm-up call to the model on Startup and in which case it is not going to have an influence payload so if the service is not initialized you know create load the model and the model gets loaded and does all the tensor parallelism and so on and so forth and uh and then um for the warm-up call the inputs are going to be empty and subsequently when the user starts to make the request like the way I did you will have your inputs come in and then of course you know there's a whole sort of model parameters which go through some of this so that's how this model dot Pi or faster transformer.i class has been built once we've defined our serving.properties back to sagemaker how we are going to run through a model here is the location of the public S3 weights for plant 25 all we are doing effectively is then just creating the end point and for the end point what we're going to do is create that model.gz and if you remember I told you we don't need to have the raw rates in the tard.g so the tar.gz has only serving dot properties that's it pretty much that's the only thing which we have the dot dot DZ takes less than a second to create this and less than a second to upload this into S3 rest of the steps are the same in what we would do for any deployment in sagemaker basically just create the take the image and the image here is a faster Transformer uh create my endpoint which is I'm going to do a create a model object and from the model object give me instance type which is d512x large for GQ 25 gigs in each GPU kind of an instance and then just do a model.deploy and once I have a model deployed just create a predictor class out of it and once a predictor I can now use the predictions to run some of the inference load which I want to do that's pretty much it uh what it takes to host a large model and there are a lot of examples in here which which are going through so the key takeaway is you have an option now of hosting a large model in sagemaker using LMA container with absolutely no need to bring your own inference file no need to worry about how sharding and parallelizations and all these good things are going to happen all that is taken care of behind the scenes for you by the LMI containers and feel free to play around with this notebook it has some good examples you know try your own bring your own model if you want you know face there are a lot of options available there is one option I did want to point out as part of this you also have the option of specifying the model directly from the hugging phase Hub if you are interested so you don't have to have the weights in S3 that is an option there is a model dot ID and if you want to use the model.id just point it to a hugging face repository uh sorry the hugging phase model object and it will on the Fly download the model do all these steps we spoke about earlier and then deploy our model of course it's going to take longer because you're now going to download the full model and then you are going to of course uh be able to load it in GPU and then continue further right so but that option remains so if you want to do a quick test point to about this model run it see if it works well Point another model run it see if it works well and so on now if you have additional libraries which you want to bring in you can bring requirements.txt file and it will essentially do a pip install uh behind the scenes uh when it when it shows the the PIP install and and it kind of runs through the so once the endpoint comes up I actually wanted to show you how the endpoint may look like when the end point is is coming up so I captured those logs and I will Zoom this because I just wanted to make sure you know I can kind of show you how the logs are going to look like so once I zoom um let me see if I can you know Zoom from here so I'll Zoom this to about 150 percent right and definitely Zoom a bit more let me go all the way to 200 so now when the model starts when the when the container starts out um it will essentially give you the number of CPUs it will give you where the model directly is it will also give you your batch size uh most of the NLP and any generative AI of that size is always something which is recommended so we have we support the batch size but we also have a batch delay standard parameters for any NLP model so it gives you those configuration values right what have you specified and then from the logs you can also see if the model got downloaded where it is getting downloaded from and did it download successful once a detail server starts up Sage maker does a ping you will basically see a log something like okay here is the address it is bound to here is the port number and your endpoint is now up and running if you're more interested in seeing okay what are the parameters of digital server it will be do that as well and It'll point out some of the arguments in your uh when it's kicked off the djl serving so that's about the NLP um that is about how would you host a large model uh any NLP large model using LMI containers let's switch gears now we are going to take a look at stable diffusion and let's see if I can host a stable diffusion which is text to image kind of a workload on LMI containers so in French got launched recently it's been announced and as part of inferentia 2 the LMI container now supports the stigma diffusion 2.1 on Infinity workloads as well so the key benefits you get from the inferentiator 2 one is like like to like when I ran my Benchmark numbers I took a G5 12x large and I took inference here 2 8X large instance uh if I do the same kind of quantization on a stable diffusion sorry if I take the same kind of compilations on stable diffusion model with df16 accuracy I was getting 1.9 seconds my as latency but my course of inferential 2 is almost half compared to g512xlr so that's the gains or we start to see within friendship too and in Fincher 2 actually comes with the if you look at the design of infant share two it basically comes each chip comes with two cores um and each core the very friendship starts to to benefit a lot is each core has uh different engines to perform different aspects of the inferencing so it has a vector engine it has a tensor engine and a scalar engine these are really the key three any Matrix operations which are performed in any sort of workloads right be it NLP be it your image processing workloads so we have specialized areas in the chip which handle this and that's where inferential two starts to score up is able to do the same workload on on less capacity now with AWS neuron you do need to compile the model so the neuron compiler and we will give you the notebooks where you can actually take a look at how the neuron compiler is able to compile some of these models it effectively compiles the model into what we call as nef format which is near an executable file format and behind the scenes it is what is doing is taking our model looking at the kernels replacing them with the neuron runtime kernels and the C plus libraries which are highly optimized for throughput and load latency and that is really that is all pretty much what we need to do for enventia to host a stable diffusion 2.1 version or influential 2. so there is a blog post which goes into the details of what the differential 2 looks like and if you search for achieve high performance on infant share two you will you'll be able to find it on the on Google and it will include the links as well for or some of the some of the containers now let's look at okay how does differentiator really look like so for Adventure 2 I basically have the infinite here to set up on I'm going to show you through vs code this is running on sagemaker uh endpoint and I'm generating a 512 by 512 image using the sagemaker endpoint so this is the this I just gave it a prompt saying mountains landscape and it basically generated this image what you can see as part of this I'll Zoom this so we'll talk a little bit more about the code itself right and we'll this will be available in public stage maker examples repos you should be able to get it from there now the concepts are are almost similar to what we were seeing earlier so the first key concept is okay I want to load my model in how many devices or rather the gpus which are deep equivalent of gpus which are available so I have the tensor parallel degree and I can use that to configure how many devices am I going to use lucky note the key difference that a tensible degree of four is loading it onto two devices because each device comes with two cores so I'm a tensorflow degree of 4 means I'm going to use 4 cores and so it's going to use two devices uh when I specify this particular value so that's uh the key difference between an Adventure 2 instance versus a G5 or ap4d instances on on uh on sagemaker other than that the stack Remains the Same we have a large model inference container where we are going to load the model from the S3 into the container and it's it's running on the Infant share two ec2 instance uh you can pick a variety of instances which you want to do the one which I'm using is an 8X large instance for for compiling the models and they're called hosting the model I'm using and in French yeah just an x-large instance to do this and so let's look at some of the key um let's look at some of the key areas of LMI containers right so again we have made available the bf16 neuron compiled rates in public S3 so you will be able to pretty much leverage those weights and run your workloads this notebook actually leverages that so you don't have to do the compilations but if you're interested there'll be a notebook available which will show you how you can do the compilations for uh infection too so here I'm just basically okay here is where my model rates are and to create the model.gz we are going to use a no code option so we are going to have only serving.properties and if I look at serving dot properties and we'll talk about you know why some of these are important for the infantia when I look at the southern dot properties if you remember from the earlier NLP the demo we saw some of them start to look familiar to you right so the key is the engine we're defining is Python and so what we're seeing is we're going to use the DJ python engine to host uh this specific model entry point in this case we are going to use a Transformers Neuron model which is going to be handling your um your inference workload and we'll take a look at this class as well similar to earlier S3 URL goes into where and how my model weights are that's a parallel degree gives you the number of cores remember this its warrant here is number of cores versus the number of gpus which you're going to use to define your model and then of course is your D type here I'm saying bf16 because I'm I'm wanting to run this on a br16 workload so this is all what you need to really Define your endpoint in sagemaker just this properties file the rest of the code is very similar we create and upload this to sagemaker and the rest of the rest of the code is I'm going to use the cont the inference image which is going to be the D speed one which has a stable diffusion built in create my model here I'm showing you Moto 3 if you remember the earlier demo I was showing you the python SDK you can use either um I just wanted to give you both the flavors so we're using photo 3 to create the model and then we create the endpoint config and then essentially we do the hosting and once the hosting is done then I can pass in the text in this case I'm passing in mountain landscape as my text and when I run this I basically get a nice beautiful image of a mountain landscape in a 512 by 512 instance now you can of course you know specify higher uh Precision if you want to do say 784 you will have to compile the model for a 784 instance now before we look at the inferenced the python file which is handling the inference let's take a look at some of the key Concepts so in a stable diffusion it's really a sequence of three models that's really what stable diffusion is so your key model is the unit model which is handling most of your uh denoising and noising steps as you're going through uh the inference worked out right so when you're compiling your uh weights you will need to make sure you compile for the neuron SDK the unit the VA decoders post Quant and also the clip text so I made a mistake where one time I did not do the clip text and I saw my inference time just jump up so it is a combination of all four you have a text coming in so you need to have that also in a highly optimized way because you're going to create the embeddings which is going to match which is going to be used effectively to do the image Generation Um using the denoising steps and then of course you have your unit model which does all the beautiful work behind the scenes and then you have the vae the second um point to note is for the cross attention scores there is an optimized method which has been used in the neuron LMI container SDK which let's uh which is which brings the influence in town time significantly so doing all of this I'm able to now run a 512 or 512 image in about 1.9 seconds uh or using inference using in French and if I skip and let's take a look at the inferential 2 [Music] um in French here two uh file and then we will effectively wrap it up so here is the Handler which is the default handle for Infinity how is the default Handler set up and if you see over here there are some certain unit uh specific classes which we are creating and then of course we are also creating the text encoder classes and then here is the attention score method which I mentioned is optimized which needs to be replaced other than that the entry point let's start with the entry point the entry point is the same place we basically have what we call as infer and in the info I'm basically getting all of these properties where I'm going to get my parameters The Prompt which I'm trying to do and then I just call myself or pipeline to run through the image get the image and then effectively create a buffer out of it bytes buffer and then add it to the output and send it back to the calling user and for loading the compiled model we are going to be loading those four key compilations which we did so we're going to load the text encoder Quant unit and the vae decoders um now the LMI container does let you also do runtime compilations in case you are just wanted to run some sort of quick test you have the option of specifying your runtime um quantization uh runtime compilation as well in which case will take the raw weights which you specify compile them and then load them onto the GPU and make it available for survey so that option remains but the usual recommendation always is to do ahead of time compilation that's where the save compile method is coming in feel free to look around this class you know you can adapt this class to your specific workload it goes into various compiler options and arguments and there is documentation available around this on the neuron SDK websites and we can we can work from there and with that we would pretty much wrap up our session of being able to host so using this class which is the inbuilt class I'm going to uh just quickly run through the image one more time where if I if I run a Mountain's landscape um or the standard me Runner is slightly different when I'm going to basically run for example uh uh houses uh on snow capped Peaks right so I'm going to run this example um and then of course once the results come back let's see how the image looks like for snow crapping so voila there you go uh zoomed this out so we can see the image slightly better it basically you know built showed me a beautiful houses uh plank twice no kind of Peaks so that's all uh for the element container in today's session we've shown you how you can use LMA containers to host NLP generative AI models how you can host civil diffusion generative AI models uh using the LMI containers we also shown you the neuron compilers and how you can use in French share two to run some of your workloads and we will have the links out there happy learning and that should conclude our session for today thank you [Music] thank you [Music]