Self host Gemma 4: Deploy LLMs on Cloud Run GPUs

Key Takeaways

Hi everyone, welcome to Hands-on AI, where we walk you through AI lab step-by-step. And Annie? And I'm Io. And today you're going to learn how to deploy Gemma 4 model with Ollama or vLLM to Cloud Run. And essentially, you can connect your agent system to Gemma 4. But before we dive into in this architecture on the screen, let's take a step step back to understanding this end-to-end agent system management. So, Io, why don't you start to explain the cost and capacity for >> Yeah, so as you mentioned, so throughout this lab we're going to be learning the pillars of end-to-end agent system management. There's a lot of different considerations that go into that. There's cost and capacity considerations. So, how do you optimize your your resources such as GPUs when you're deploying self-hosted models? Model strategy, when do you select open versus closed models for your use cases? Serving at scale, how do you optimize for you know, multi-throughput and serving multiple users at a time? Security and safety is another one, as well as observability. So, these are all the pillars we're going to be covering as part of this lab and yeah. Yeah. So, before we go through how to deploy Gemma 4, we need to understanding the difference between closed model and open model. You know, closed model like Gemini is great, it's like it's state of art, it's fully managed, easy to start. And open model like Gemma is easy to take control, you can even fine-tune it. So, why we're talking about Gemma 4 today, right? Why would you want to use an open model? Io? Yeah, it's a good you mentioned you just talked about a lot of the pros of using open models. A lot of industries such as healthcare or finance, they're they're working industries where you may not be able to pass data over the internet or you may be limited with how you can kind of handle your data. In those use cases, if you have to kind of uh call these models on premise or in kind of isolated scenarios, running self-hosted models is a really good solution for that. And then as you mentioned, you can customize the model. So a lot of use cases have very um domain-specific data where you can kind of improve performance by tuning and you can do that with an open model as opposed to close models like Gemini. Essentially, they're great out of the box. They're state-of-the-art but really general but you may not be able to customize it as much beyond tuning I mean beyond prompting and you know system instructions of that nature. So those are the benefits of using open models. Yeah and also cost is also another consideration. You can take a look at this diagram that the more you use if you're using a a close model like Gemini, it's the cost is increased per API call versus if you're using open model like Gemma, the cost is not linear growth, right? Cuz it's based on the infra. Yeah. So coming back to this model strategy, um Yeah, there's there's many dimensions to to consider. We talked about a few of them but those are just the ones to keep in mind. But beyond that, it's also important to know that the reason why we're so focused on model strategy is at the end of the day on these models act as the brain behind your agents. So when you're using something like Google's agent development kit, not only could you use Gemini as the brain behind these agents, you can use open models as well. So that's something that people sometimes they don't know. They think you can only use Gemini but in reality, you could pretty much use any model that you want because Google ADK comes with the light LM wrapper that allows you to kind of connect models as you see fit. So later on in this lab, we're going to learn how can you use Gemma 4 as the brain behind an agent. Yeah, so essentially agent is that you're using the model as the brain and then do reasoning to pick which tool you want to pick to make a decision. And as you can see on this diagram, this is agent architecture. So the model you're choosing really like can determine the like the upper bound the capability of your agentic system. That's why it's very important and you want to be smart to choose your model. And when you choose your model, you can think about the performance of the model, the use case you want, and also the cost. You have different angle to pick your model smartly. And so this is the overall architecture we're going to dive into today. As you can see that on the on this right-hand side, we're going to start with this runtime. So today, we will deploy Gemma model, Gemma 4 model, this service via and Ollama. But yeah, I why we have this two different two different way? What is the real difference between Ollama and vLLM? Yeah, they're both LLM serving frameworks, but you tend to see Ollama used more in development use cases. It's really easy to install and get up and running. Um so it's great for experimental POCs. And you can also use it for multi-GPU support use cases. vLLM is great for production use cases. It comes with paged attention. It's great for memory efficiency, um and allows you to kind of do multiple concurrency when it comes to calls and dynamic batching. So again, so Ollama is great when it comes to development. vLLM is great for production. And there's other LLM serving frameworks as well like TensorRT, um LLM, and such. So there's a lot of options to choose from um in the world, but for this lab we're focusing on Ollama and vLLM. And don't forget you can also deploy a model to what is AI through Model Garden. That's also another choice. It's a managed uh service for you. Yeah, so Ollama is very easy to install and it's very good for you to try things in your local development. And let's get start with Ollama. Uh so this is the as as you can see from this diagram, how what we want to do is we want to first pull the model from Ollama, and then we want to build this image with Docker. So once we have this image, we're going to send it to the artifact registry, which is a container on cloud, and you can essentially deploy the image so that you can have this image hosted on cloud run. So, that's essentially how your agenda system agent going to use connect to your deployed Gemma for. And if you if you're interested, let's let's get let's start by going through this lab. If you So, we will have this link on the screen about what is this lab link about. But as usual, you know, like I always usually we wanted to start the equipment challenge with a credit, but today we're going to skip the credit part. >> Yes. Yeah, today we're skipping the credits because this lab does require GPU such as like when we're deploying to cloud run, we will be equipping our cloud run instances with, you know, L4, you know, accelerators and such. Because of that, we won't be giving credits for this lab, but please feel free to follow along. You can use your own credit accounts if you have them or you can sign up for free trial credits on Google Cloud Platform if you're a first-time sign upper. But yeah, please feel free to follow along. As always, we want to start our environment our development in cloud shell on Google Cloud. To open that, making sure you go to console.cloud.google.com and on the right top right there's a button uh it's saying the active cloud shell. Just click that and you will see um this Yes, it's similar to our VS code, but it's on cloud. Essentially, it's the VM that hosting it. Yeah. I don't have anything to add. Yeah, as you mentioned, so the cloud shell editor is essentially like a VS code on cloud. It's persistent over time, so any code that you develop or files that you create, you can come back to it a day or week later. So, please feel free to use it as your development environment, and that's exactly what we're going to be doing for today's lab. Yeah, but one thing you need to notice is you will time out every 30 minutes or so for security reasons. So, if you see any error for like IP like address missing or some weird issue or you want you ask you to reauthenticate and just making sure to refresh the page to refresh the token so that it will keep you authenticated. Uh and to get started, let's just uh set up our environment so that we are ready to deploy and host our Java for model. And as always, let's uh just copy this gcloud auth. Let me zoom in. I'm going to copy this and then paste it to my terminal. So, you can see this is a terminal. Let me paste it. So, that making sure this Gmail account is the same account your you have your uh billing account uh connected. And next, we need to clone those two repo. We we are cloning those two repo and one is the Agentverse Dungeon and another one is the Agentverse uh DevOps SRE. Um yeah, maybe you might explain to our audience in case they haven't watched our previous episodes. They don't know about Agentverse concept. Yeah, so we're cloning two different repos. The first one, Agentverse DevOps SRE, that one has all the template files for like the agent we're going to be uh be deploying later on. The cloud YAML scripts that we're going to be using for, you know, um deploying um our uh images uh to artifact registry and such. Um so, basically it has a a bunch of staging and template files that kind of gets us up and running a lot quicker. The dungeon repo has essentially the the boss um uh you know, dungeon and and uh files we're going to be using for the boss fight later on in the lab. So, we're going to build our agent. Essentially, we're going to be building up to a boss fight where our agent is fighting with another agent in the cloud via A2A. Um so, that one just has files that we can kind of get that up and running. So, by the time we get to that point, um our boss fight is ready to go. Yeah. >> Yeah. Let's come back to this uh setup page. And next, you need to initialize a new project so that you have a Google Cloud project ready, and you can see that it always starts with Agent Verse Guardian. You may have a different number here. Essentially, the script is creating a Google Cloud project for you. And then it's trying to link the billing to this project. So, you can see now how this full setup uh complete. But for uh you If you're running the script, you'll probably notice that you have this When you fetch the billing account, it's has a fetch failure. That's because it's looking for a child account. Um So, what you can do is if you happen to have a billing account with your credit card uh linked to it, um this is what you can do manually in Google Cloud Console. Okay, now you're at console.cloud.google.com. And then, uh in the search bar, so I just click this search icon in the search bar, I will search uh resources. So, when I type resources, I have something called manage resources. So, when I open the manage resources, this is where you can manage all your project and its corresponding billing account. So, you you can select this Agent Verse Guardian uh project, and here you can just uh click this button action and go to the billing. So, now this one it's already attached to the to my account. For your for your case, you'll probably see a pop-up window to see manage billing, and then you can select the billing that has your um has your payment information linked to it. Next is, as always, we want to um to uh config the project. And why we need to config the project? Yeah, we're just making sure that our project is set in the G Cloud uh environment. So, there's two ways you can kind of double-check if you're in the Cloud Shell, you want to make sure that you see your project ID in yellow. And beyond that, you also want to make sure that again, the G Cloud is set to the right project. So, what we just did was making sure that the G Cloud uh is set to the right project, which should match your agent versus guardian uh {slash} {dash} some unique ID. Yeah. Yeah. And next we need to enable some Google Cloud API. As you can see that those are the If you take a look at the API we want to enable, it's including the storage API platform. Basically storage is you essentially you want to store your Later on when we do VM, we want to store our model in Google Cloud Storage. And when we want to deploy to Cloud Run, we want to enable Cloud Build. And we also want to have artifact registry to store our image, so we want to enable this. And yeah, essentially we want to stay our store our API keys, so you want to enable Secret Manager. Yeah, anything you want to mention, Al? Yeah, so all that I would add is that just by enabling these APIs does not mean you're going to be charged immediately for these APIs. Essentially you're just enabling them to be used in the project. A lot of people ask, "If I enable AI platform and Google APIs for example, will I start being charged by the second?" No, you won't. You're only going to be charged when you actually use the API, so that's just one thing to keep in mind. Yeah. And let's coming back to this. Yeah, so now we successfully finish enabling. And next is we need to create this artifact registry repository so that later on we can deploy our image to it. So coming back to this diagram that we want to build the image like we want to first pull the model from All Lama and build the image and later on deploy this to image repository. Yeah, Al, do you know why we want to have this design? Like uh can we Is there any other way we can do? Yeah, this is just best practice when it comes to you know CI comes to Google Cloud where essentially you're building an image, you're storing it in artifact registry, you're deploying it to Cloud Run, and it's essentially this is the build process that we're going to be enabling with Cloud Build. Um that allows us to kind of programmatically specify each of these steps, and then we can go ahead and just execute that Cloud Build YAML file. So, you're going to see that later on once we get to that step. Yeah, and also you can see that the model essentially be directly building cuz we do not have to store model anywhere cuz it's baked in in Ollama. You will see it really soon in our our lab. Yeah. And um Yeah, so now we have this repo create uh like artifact repository created, and next one is set up the permission. And I do know what is service account. Why we need to grant this role to the service account? Yeah, great question. So, every Google Cloud project has a default service account, and um that's essentially going to be like the operator behind many um of your default actions. So, if we're uploading files to Cloud Storage or um you know, uh executing a Cloud Build deployment, you want to make sure that the default service account has certain permissions. So, that's pretty much all we're doing here. We're making sure that the the default service account has the ability to uh interact with Cloud Storage, has the ability to uh initiate Cloud Build jobs, has the ability to uh write logs and read the logs, um has the ability to interact with Secret Manager, which we're going to be using later on for storing our Hugging Face uh Secret Manager key. So, as Annie just mentioned for Ollama, we're going to be baking in the model directly into the um uh container image, but when we actually use V11, we're going to be doing a different approach. We're going to be storing the model weights directly in Cloud Storage and pulling those model weights from Hugging Face. Um so, there's different approaches that are kind of um being enabled through this IAM permissioning. Yeah, one metaphor I can think of is you can think of service account like a robot account Yeah. of yourself, and then you have different robot and different permission, and so that you can manage the project permission. And here we are using the same account for easy demo, but in production you can have different robot account like different service account for different permission configuration. And the last step is to uh warm up. As we need to run this warm up dot warm up dot sh script. So essentially, we want to essentially using GCSFuse for our VLM and deployment. We want to run this warm up script uh here to warm up our GCSFuse cache. And you if you want to know what is GCSFuse and why we want to use that, uh stay tuned. We want to talk we will talk about it in VLM deployment. Yes, yeah. Now it's finished. Let's start host our Gemma 4 on Ollama. Let's get started. All right, so let's get start. Uh so first we want to pull the image from Ollama. As uh I mentioned earlier, you know, Ollama is great for local development. It's great for quickly testing the any model because, you know, Gemma 4 is baked in in Ollama. And you can see it's very easy like first we want to create a Dockerfile, build the image, and we just with only this line of the code, you just need to run Ollama and uh pull the this Gemma 4 from Ollama. So now you can just uh have this model from Ollama. So it's very straightforward. And uh just copy this and coming back to my terminal and paste it. So now we just create a Dockerfile folder. And what's next is we want to create this cloudbuild.yaml file. So I will maybe you can explain um what is cloudbuild.yaml file before we explain um what are inside the cloudbuild.yaml file. Yeah, yeah. So as I mentioned before, Cloud Build is essentially the engine for CICD on on Google Cloud. So, essentially it allows you to programmatically specify um steps you want to run as part of a as a process. So, pretty much all we're doing here is we just created a Dockerfile that defines our container image. Now, we're going to build the container image in step one. We're going to push the container image to Artifact Registry in step two, and then we're going to deploy that container image to Cloud Run uh in step number three. So, that's all that we're specifying here in the Cloud Build YAML file. Yeah, so basically it's like a blueprint file to specify all the step you need to do, and exactly what we're showing on this diagram. So, what we just did is we just uh pull we have this uh Dockerfile to explain how to pull this image. So, you want to build this image, and then and then put push to Artifact Registry, and deploy the image. It's like put It's like a blueprint file to uh including all this process uh we want to have over here. And let's just uh copy this, and uh coming back to our my terminal, and paste it. But, I will Why we want to set the CPU to be four, and why we want to be our concurrency to be four? For all those setting, anything we need to be aware when we trying to deploy our image to um Cloud Run? Yeah, um Cloud Run is a really powerful serverless platform. It gives us a lot of configuration capability. Yeah, so as you see here, we're set we're specifying four CPU um minimum for each machine of this service. Um we're specifying memory to be at least 16 GB, which is really important. And we're going to be storing um Olama or uh Gemma 4 uh model on the image. Um you have to have a a lot of um memory capacity. Um so, we're working with a 2B um um version of Gemma 4. Um so, 16 GB is uh sufficient um memory for that. So, that's why we're specifying that. Um we're specifying that we want to use the Nvidia L4 GPU type. Um we're specifying concurrency to four. We want to allow up to four parallel calls against the service. Um and yeah, so again, uh Cloud Run gives us a lot of configuration capabilities and we're just trying to optimize uh for how we're going to use it in the lab. As you can see at the bottom, we have min instances equals one, max instances equal to one. That's not going to be too common in production. You want to be able to scale beyond one machine. Um but for cost purposes, um um just so you're not kind of allocating more GPUs than you need, we're kind of just minimizing to one machine per service for this lab. Yeah, also we have this allow on authenticated for this uh lab purpose, but you can like make it authenticated for security reasons for your actual um image deployment. All right, so next one is we want So now we configure this Cloud Build YAML file. So next step is actually, you know, follow this this blueprint in the Cloud Build YAML file and then finish this process with Cloud Build. So just uh copy this um basically you put this uh Cloud Build YAML file in the config file and then do the setting with the region and the repo name project ID you have. And let's just uh go ahead and go to our terminal and go ahead to deploy the whole process. You know, if you want to track the process of the you can also go to Cloud's um Google Cloud Console in the Cloud Build to see the process. So now you can see that um we just finished the environment setup and now we're waiting for the build to complete. It's like 100 in the interval. So it's basically following the uh process like trying to first build the image with Docker. Yeah, and keep in mind uh this Cloud Build uh process will take um some time. So uh this step may take anywhere from 15 to 20 minutes depending on how long things take. So um feel free to take a coffee break and uh go get some coffee. Um but uh yeah, we'll we'll be back once our uh build job completes. Yeah, it finished finally and as you can see that we can go to Google Cloud Console to also see the process on like you can search cloud build. We can see the process. We just had earlier. And Yeah, this is a building process. We just having because we have two building process. One is for dungeon, another one is for this deployment. You can also see this image is this rate deployed to cloud run. So let's just go to cloud run making sure that on the top left it's selecting the project and go to the Gemma Olam big service and here we go. We have this. We have this URL that later we can connect to our agent. Okay, so now you have this URL, right? We have this URL and we want to verify this is working by sending it the post request. So here basically what you want to do is this step, you know, this Olam URL this step is to trying to grab that URL. You can also grab this URL just by copy this. But here this is a command for you to grab this URL. And once you have this URL, you're trying to test this URL by sending this prompt. As a guardian of A to words, what is my primary duty? Or you can send anything else and then what you expect to do is expected behind the scene is you want this Gemma for model and like and doing this question. And let's see what is the response. Just going to copy this and paste it to the terminal. Time to verify if everything is working. Yes. So and now all it's doing it's resetting the environment variables because we're running that set env.sh [clears throat] file each time just to make sure that in case you lose context because the build process takes a long time. So throughout this lab we're setting that setting E multiple times at the very top. So, that's the reason why it takes a little bit longer for each step. Uh, but it's just important to run just in case you lose any context. And then, yeah, it's going to print out the Cloud Run URL, and then we're going to call the URL and see if we can get a response from the from the LLM. Yeah, so you have this URL, and then you can send a post request. We are expected to see a response look like this. But, maybe you will not see exactly same response here because, you know, AI cannot generate the same result every single time. But, oh, let's see. This is the result. Oh, wow. Okay, here. Oh, yeah, the context. Oh, here's the result. This is the response. As a guardian of the Aetherius, the primary duty is this. Safeguard because of the Aetherius representing reality. Yeah. Great. So, that means the Gemma 4 uh on Cloud Run with Olama is working. All right. So, I hope you know that we are using Olama, and it has it's really easy to get started, and it's really easy to use, and we have this model baking so that we have really fast cold start. You do not have to like start waiting for it to load in the model and to have the initial cold start problem. However, you have some disadvantages, right? So, I will um what are some disadvantages with this Olama use case? Oh, yeah. So, since we're baking in the model into the image, as you mentioned, it leads to very fast cold starts because the fact that the image is already deployed, essentially if you want to create a new uh version um of a or new instance of that machine for the service, it's really easy to spin up. Uh, in our particular use case, we're only limited to one, but if we were, it would be really easy to spin up more instances for the service. But, the downside to that is because we're baking it into the container image, if you want to change the model version or the parameters of the model, so you want to go from 2B um to a a larger version of the same model to a different model, you have to rebuild the image and then redeploy. So, it's a it's a it's a long process and as you just saw, building these images can take some time. So, as I mentioned before, through the Git for development use cases, oftentimes when you're running these models locally, you may want to use Ollama, but if you're baking it into the container image, it can be pretty inflexible when it comes to kind of rapid changes. But, of course, when you're using something like Cloud Build, you can automate the changes to an extent, but yeah, it's just one thing to keep in mind. Yeah, also another another thing worth mentioning is if your production use case serve huge traffic and you want to improve the performance, that is where you should really consider using V R app. And different from Ollama, Ollama has a model baked in, but here you want to grab the model from Hugging Face and save that to cloud storage. And then, you know, this process is a little bit similar. We try to, you know, we we have this model the CI pipeline is kind of similar. You want to build the image, but this image it do not does not contain the model. It only contain the V R app code. And then, once you uh save that, push that to the image repository and then deploy them to Cloud Run. And now you can see that the container image is tiny because it only have V V R app code. And then, the Gemma is going to store in a cloud storage and then we using cloud storage fuse to mount the bucket as a local folder. So, anything you change in the local folder will reflect it. I okay explain more about cloud storage fuse. Yeah, so cloud storage fuse is a great way of representing cloud storage folders and paths as local paths for your machine for the Cloud Run service. And that's what makes it so efficient. So, as far as any you know, package or uh that's running on the Cloud Run service, it's concerned it could actually access uh GCS files the same way that it would access local files, temp files that are running directly on the machine. Uh so, Cloud Storage FUSE is essentially what mediates that connection between the Cloud Storage bucket and it being mounted as a local folder for access uh directly on the on the Cloud Run machine. Uh yeah. So, anything we're changing in the local folder will also reflect here, right? Exactly, yeah. So, and what makes it really powerful is that not only could you read from that local folder that's been mounted uh the same way that you would read from like a GCS bucket, any files that you write to that local folder is almost as if you're writing to the GCS bucket as well for persistence uh down the line. So, it's bidirectional sync. Yeah, bidirectional, yeah. Yeah, pretty cool. And let's take into how we're going to save the model to Cloud Storage and how to go through all the CI pipeline and set up that uh Cloud uh and set up that Cloud Storage FUSE. Let's get to step step five. All right, let's get started by first download this model from Hugging Face and then save that to Cloud Storage. So, let's get started. And what you need to do is first you need to go to step five. And when you're scrolling down and you can see that we have instructions here. If you don't have a Hugging Face account, you can just create a account over here. Or if you can create account over here. If you already have it, you can just uh click uh login. And you now you're in this page and to login. And what's next is you need to click in this link to generate uh a token. So, now I'm at this page. I'm going to click generate a new token. So, uh you can use AgentVerse workshop token as a name or any name you want. I'm just copy here for the AgentVerse workshop name. When it comes to the permission, uh you can just use the read role. So, we can choose read for this use case and create a token. And now you have this token ready. I just copy this to to save it. So, yeah, next thing we want to do is we want to run this uh command so that it will save the token to uh Secret Manager. And first, we just copy this command and then going going back to our terminal, I will paste it. So, first we want to enable this script uh and then we want to run the script. Essentially, what we have to say is we want to save the token to Secret Manager. And uh I don't know what is Secret Manager. Might explain to everyone what is Secret Manager. Great question. So, Secret Manager is a essentially um it's um it's it's eponymous. It's the speaks for its name. It's um our best practice way for managing secrets, API keys, uh in in a Google Cloud context. So, if you're working in a Google Cloud environment, um rather than you storing your API keys as environmental variables or um you know, sometimes really not good practice is to embed it directly into the file. What we can do with something like Secret Manager is um you can call the Google Cloud uh Secret Manager API or there's like Python SDK, and then you can actually pull the secret um during your actual runtime into your application. Um so, it's a way of preventing you from having to actually kind of make visible your secrets in uh like uh directly in the code or having to store it in an environmental file. That adds more data governance issues cuz now you have to start managing the environmental files and make sure that they don't get pushed um to uh you know, uh your code versioning repo and stuff like that. Um so, overall, Secret Manager is the best practice way for storing and managing your application secrets. All right. So, now if you're coming back, it's asked you to paste your Hugging Face token. Uh so, what we want to do is we just want to copy the token here and and paste it over here. You don't see anything. Uh so, if you happen to paste it wrong, you can go to uh Secret Manager here and delete the token and redo it or you can update the token. And once it finished, you will see the uh token saved in secret manager in Google Cloud console over here. All right, it says uh success and we can double-check by refresh it. And yeah, we have this token created. And we have this one version. And if you want to see the value, you can also see view uh value. Just verify this value is correct. It's a correct uh token. You just copy. So, once it's verified, all right, so we just download the model from hugging face and we have this token saved to secret manager. What's next is you want to save this model to cloud storage. So, how to do that? First, uh you need to create this uh GCloud cloud uh cloud storage bucket using this command to create a cloud storage. And next is you making sure the permission is ready. So, I'm just uh uh copy this command to prepare my cloud storage so that later I can save my model to cloud storage. Yeah. Now, it finished. So, uh we can verify it by go to Google Cloud storage. So, cloud storage. You can search cloud storage here and making sure we select uh agent verse. And here in the buckets, uh as you can see that we have two cloud storage already. So, this is the latest one cuz we have the previous one for uh our We have the previous one for our dungeon deployment. And yeah, we have this uh Guardians uh storage ready so that later we can push uh and then we can push our model here. Oh, you can see we have cloud built YAML file again. What does this file do? Yeah, a great question. So, what we're doing here is essentially we're not going to go through the cloud build process. Specifying the model that we want to download. Um checking if the bucket exists. So, that's step number one. We want to make sure that we've actually created the bucket and that set is already an environmental variable. That's all going to be true. Um two, we actually now want to download it actually from hugging face. Uh so, we're specifying the model that we want to download based off the ID. We're picking installing the hugging face uh uh hub uh library. Um we're authenticating with the token that we downloaded um and saving this as part of secret manager. And then we're downloading the model and we're specifying that it should save to um our local directory. Again, we're using GCS fuse here. Remember earlier on in the setup we did the warm up uh for the GCS fuse cache. Um and we're specifying the model ID and then ultimately just copying that model to the GCS bucket. Um and yeah. Great. So, let's go ahead and create this cloud build download YAML file so that we can continue. Let's continue. And first it will create this cloud build uh cloud build download YAML file for us to follow. This will take about uh anywhere from two to three minutes. It's a pretty quick process here. Yeah. So, here we're just creating this cloud build download file uh with our command. And next is let's using GCloud build command uh G GCloud build command to run this cloud build download YAML file so that it will actually do all the process we wrote in the blueprint. Let's go ahead and copy this. Yeah, so while we're waiting we can just the mapping each of the step uh to our diagram over here. Uh as I I mentioned earlier that first step is making sure my our model bucket exists. So, this is like a pre-check. And once we have we want to log in to Hugging Face and using the token. So, basically, this is a step we want to like make sure we get access to Hugging Face. And then we copy the model and download the model to GCS bucket. So, this This is the next step is we want to like Once we have the token, we download the model and then from the GCS bucket. And once we uh have this, we want to make sure the secret available for the split environment. Yeah, it finished. So, what we done so far is, you know, we are have this token so that we can install this model and copy the model and save everything to the Google Cloud Storage. So, this is what we done so far with this um with this Cloud Build download YAML file. So, this is basically the downloading processing and then we can verify everything in the bucket. I'm going to refresh the page. You can see, yeah, we have Java 4. Everything downloaded in our cloud storage. Great. What's next is Oh, you can also verify cloud uh GCS bucket with the command over here. You can just copy this with the GCloud uh storage LS command to recursively verify things in the GCloud bucket. It's another verify. All the weights downloaded, the optimizers, all of that. Um so, yeah, it's just listing out the files. Yeah. So, now we verified it. The next thing is we need to So, the next thing is we need to finish this uh pipeline. So, I don't know what is uh when we When we talk about network subnet, what is the subnet and why we need to uh compute this network subnet here? Yeah, great uh question. So, private Google access is a great um kind of uh tool and feature that you can use uh when you have different services communicating with each other. In our use case what we have is our model weights being stored in cloud storage, and then we're going to have a deployed model on Cloud Run. We want to be able to pull these weights without these weights having without the data having to traverse over the public internet. To kind of accomplish that we're going to enable private Google access. So this will allow our Cloud Run service to communicate with our GCS files without actually having to pull the data over the public internet. So it's going to traverse to our private subnet. So it's a really smart way of kind of keeping data secure when you have again connections between different services on Google Cloud. In our use case we have Cloud Run communicating with Cloud Storage to pull the model weights. So that's the benefit of that. Yeah, so you can see we have this line over here. Cloud Run pull from Cloud Storage. All right, so let's just copy this and continue. All right, so we should continue once we download this model and we set up the VPN or VPC. And next we want to configure our CI pipeline. So IOL, why we want to like create a Docker file for this VLM and everything? Like what what are what are they why we why why we need to do all those process here with Docker file? Yeah, great question. So what we're doing at the very first step is since Gemma 4 just was released just last week. There's actually a version of the VLM you know image that's kind of specialized for Gemma 4. So essentially what we're pulling is the Gemma 4 version of VLM. Just so it's optimized for you know inference with Gemma 4. And then what we now want to do is make sure that we're using a Transformers library that's compatible with the newest version of Gemma 4. So what we're doing is making sure that we are setting the right version of the Transformers library to a version that's compatible with Gemma 4. And then all we're doing afterwards is cleaning up any default models that were pulled and uh setting the environment um as part of the, you know, uh development process. And then we're setting the entry point for the VLM server when it's being deployed on Cloud Run, and that's it. So, pretty much all we're doing again is we're uh specifying the VLM uh default container image that we want to use when that's specialized for Gemma 4. We're making sure that our Transformers library is um compatible from a version standpoint, and then we're just doing some cleanup afterwards, and then uh specifying the entry point for the Cloud Run service. Yeah, how do we know the right Transformer for the version for this Gemma 4 VLM, or how do we change it for different model? Yeah, great question. So, there's multiple ways you can do this. Sometimes uh if you just want to kind of uh be a little bit more general, you can just say uh if pip install Transformers uh dash dash upgrade, and that'll kind of make sure that you're installing the very latest version. Uh other times you can look at the documentation and see what uh lot version of the library was used for a particular model. So, you take a look at the Gemma 4 documentation for VLM, you'll notice that it's using 5.50. So, it's just a good way to be consistent. Um but uh yeah. Got it. Thank you. So, let's configure it. I coding this Dockerfile. Okay, so now we want to create this Cloud Build pipeline. So, to create this pipeline, basically is like we want to define a blueprint for this whole process. And the first is, as always, we want to build the image with uh Docker. So, we want to build this image. So, have this image with VLM. And then next is you want to push this to uh artifact registry, like this. And then what's next step is you want to deploy everything, you know, the service and VLM fuse service to Cloud Run. So, why I'm deploying uh I'm going to copy this. I'm going to create this Cloud Build YAML file. Uh I do we have different setting here for VL compared to Alama setting Great question. It's pretty similar. There are some slight differences um especially because we're taking a mounted approach with VL. As I mentioned before, we have the model weights in GCS and we're mounting it um uh on the cloud run machine. So, there's a few extra parameters for that mounting. But beyond that, when you actually look at the machine sizing, the memory, the CPU, it's uh it's the same and reason why is because we're pretty much using the same exact model with the same exact amount of parameters. So, the same memory constraints are the same across Alama and VL. But as I mentioned before, we have a few other parameters to kind of one address the private Google access that we enabled. So, to make sure that we can kind of communicate with cloud storage over private network. And then two, to kind of actually mount the the cloud storage uh you know, a bucket uh as a local path. So, you can kind of see there's some volume mounts that have been added as parameters. But beyond that, it's very similar. Mhm. Great. So, once we have this cloud build Oh, yeah. So, this is the cloud storage fuse that we mentioned earlier that can mount Google cloud storage bucket uh with local file. So, this is like earlier we mentioned. And so, yeah, we have this cloud storage fuse explanation before and you can check this facts over here by configure the volume for enable cloud storage fuse. Uh next, we have this cloud build YAML file configured and next is let's to run this Gcloud build submit file as always to follow this whole instruction so that you will actually so that you will actually um follow this CI pipeline. Yeah, build uh uh push and then deploy [clears throat] to cloud run. Yeah. Yeah. And it probably take like 20 minutes or so, right? Yeah, 20 to 30 minutes. Yeah. All right. Looks like it's finished. Yeah. Yeah. So, what's next is let's take a look at this Google Cloud build history. So, if you click if you click this link, you will take a look at this whole building summary, the building process. And also, you can see the Cloud Run service page by go to Cloud Run service. And making sure I pick this project. And you can see that we have Olama that deployed from the previous one and VOM is this one. And yeah, you can see that we have this URL here. This is like the link you can connect to Gemma for model. And just to Now, we want to verify everything is working. So, again, we want to like send the post request with this URL link. And this is the endpoint with a chat completion. And we want to ask the same question. Like as a guardian of Ethereus, what is my primary duty? And we want to verify if this whole deployed version of Gemma Gemma for wholesale Corolla, is that working or not? So, let's give it a try. I copy this. So, I'm going to come into the terminal and I'm going to paste it here. So, let's take a look at what is the result. Yeah, as you can see that because the first time initial load, we have some cold start. Yeah. So, let's look. So, we see the response as a guardian of the Ethereus, your primary duty is multifaceted and so on. If you recall the response from Olama before, as Annie mentioned, these models are stochastic. So, you can call the same model the same multiple times to get a different response. But overall, very similar response to the same same question as before. And overall, it everything looks like it's working. Yeah. Great. All right. So, just have a quick overview of what we done so far with VOM. So, first we download this image of we download the model from hugging face to Google Cloud Storage and then we have this pipeline that we build the image of your VRM code and then and then deploy the whole thing to Cloud Run. And then we set up the CGCS fuse so that it will sync to the it will mount the Cloud Storage with the local file. And let's talk about the advantages or disadvantages of this approach. Cuz compared to Ollama, that Ollama is is very easy very easy to start and try anything do the local development. VRM is a good option for production use case. And it's very flexible to update the model as well. You can just simply swap the file in the bucket and restart. However, it has some initial cold start because it doesn't have the baking the model baking in as Ollama does. Anything to add for the trade-off is VRM? No, that's that's that's a great summary where essentially with VRM with the approach that we took is actually you can use the GCS fuse to dynamically update the model weights if you need to. Macan essentially there's a longer cold start. But one caveat with that is that's really meant for initial invocation or initial user. If you have multiple concurrent users, a thousand users calling the same endpoint at a time, that warm instance actually won't you know obviously have that cold start for future users. So it's really meant that cold start you tend to experience really meant for the first call to that you know newly created machine. But yeah. Yeah, so here is the overview of you know so far we have Ollama and VRM. So those are the advantages of Ollama for local development and prototyping and also has multiple GPU support. However, if you use the production use case like a VRM has the page attention for product efficiency and parallel parallelism for like maximize GPU usage. So yeah, it's really good for production use case. So let's just coming back for this high-level architecture we have. And for today's episode, we learn how do you host in drama for model with Olama V 1 to Collab. So, this is this is this is the piece we cover for today. We cover like how do you using um how do you pull model from Olama and host on Collab. And we also know how to download the model from Hugging Face and then host it with V 1. And what's next episode what's next I will what we going to cover next? Yeah, great question. We're going to learn uh lots of things. So, we're going to learn how can you serve and scale these models at a global scale using something like a load balancer. How can you secure these models and protect against things like model uh prompt injection and jailbreaking? How can you determine if your model is actually leaking sensitive data? The answer for that is also something like model armor. How can you use these models as the brain behind a deployed agent? We talked a little bit about Light LLM uh earlier today. We're actually going to put this in practice tomorrow and use Light LLM to kind of connect with our self-deployed models. So, we can use them as the brain behind these agents. And lastly, we're going to learn how can you actually observe uh the metrics being generated by these models? So, you know, we talked about cost earlier and how can you actually monitor that um from like a how many tokens are being generated um by these models? You can use something like a Prometheus sidecar associated with the Collab run instance to actually be extracting metrics such as how many tokens are being generated, what are the what is the GPU utilization, and so on. So, we're going to be learning a lot tomorrow. I hope you guys all stay tuned. Um and That's a really good summary. So, by the end of this two episode, you're going to learn everything about this end-to-end gigantic system management. Yeah. Yeah, yeah. So, I'll stay tuned. Um but until next time, I'm I will. I'm Annie. Bye. See you. >> [music] [music] >> Mhm.