LLMOPS 09: CI/CD Deployment for LLMOps using GitHub Action on AWS EKS | Deploy LLMOPS Project

Key Takeaways

So hello everyone. I hope you guys are doing absolutely fine. Welcome to the eighth video of our LM op series. Uh previously we have did many videos in our series. Like you can see here uh we have done total nine videos and you can get a complete uh knowledge of how to uh deploy your rag applications in an end toend manner so that use you can use them in a production environment and like current video like today's video what will be about uh today's video video will be about uh deploying our application uh with the help of AWS EKS like we are we are going to use the elastic kubernetes services of our AWS to deploy our application uh previously uh we have seen like deployment but we have seen in ECS plus target that is a serless way. Okay. Uh so today we are going to do that and also we are going to create the CI/CD pipeline using genkins. Okay. Uh so we are going to do that. So firstly let's quickly move on to the sessions agenda. So the sessions agenda today is like firstly we will understand that what is the difference between ECS or EKS. Why we are going to do that in the first place? What is the major difference there? Okay we are going to get that. Then uh like you heard the word like Kubernetes. I said like EKS in it there is Kubernetes in it. So we are going to getting a overview like why what is Kubernetes what is the role of it we are going to uh we are not going to go into the depth of it but we are going to see that why we are using this Kubernetes here and uh the third is like understanding the complete deployment flow like how the deployment is going around. I have created a complete uh file for you which you guys can also follow along with me or after watching this video and you can deploy it easily. In the file you will see all the commands and all the like all the instructions in a very sequential manner. So that will give you a lot of clarity while you are implementing. Okay. And then we are going to uh use the understanding we got in the deployment flow to implement the deployment. Okay. And then uh after that we are going to uh run the deployment pipeline with the help of uh genkins. Uh so this is the sessions agenda. I hope you guys are uh interested in learning that. And also let's move on to the agenda which is understanding KS means understanding Kubernetes role. So firstly you can see that I I have written Kubernetes at KS. So why I have written that? So if you go to the like Kubernetes spelling uh like Kubernetes spelling. So you can see here like it start with a K and it end with a S and between there there is eight characters. So that is why it's called K at S. So uh I will uh so for the correct mapping in your mind also I will use Kubernetes as K8 S in further in the session also. Okay. So now let's understand what is Kubernetes. So for that I have created a flow for you guys. Uh yeah here. So like you can see you are familiar with the docker concept. If you are not please go into the playlist and watch this video u CI/CD deployment and all or you can watch this genkins video with docker also. Here I also explain the docker concept. Uh so in the docker what we have in the docker is used to uh like dockers have images for the projects like your project have some images which can run in a isolated containers and why we uh do that like we do uh it for a proper like solving the problem like the code is working in my machine. What is the problem? For example, you created some project and the project is working fine in your machine in your machine's environment. But whenever you uh send the code to your friend's uh computer friend system or uh the user system uh the code crashes. So this is a very big problem with docker solve with the help of this containerization. Okay. Uh so containerization is says that if a image uh if a image inside a container runs in one system it will uh run in uh any other system. Okay. So dockers ensure it. So that's why we use docker concept. Okay. So what kubernetes do here? So Kubernetes so uh think of like uh let's imagine a scenario for example you have multiple containers you have four containers container one container two container three and container four and in this container you are using using some kind of images okay but but for example uh this four containers are suitable for uh the traffic inside your like project or application with four lakh people okay four lakh people okay or four lakh user four lakh users can uh uh easily use your application if these four containers are deployed. But for example, if the five people like one extra lag amount of people are now trying to access your application. So now uh there should be a extra container, right? There should be extra container uh which should be deployed. Okay, which should be deployed inside this environment so that uh your application uh will be able to serve to the extra one lakh people added. So you have to do it manually. Okay, but how are you going to do it? So you are going to do it you are going to handle it with the help of AS okay with the help of Kubernetes. So Kubernetes is going to do that for you like Kubernetes is going to create this new container okay and whenever your traffic drop for example your traffic dropped to three lakh people so it will just disable this two containers okay so this is done using like the Kubernetes. So the analogy simple analogy of this is like imagine the docker imagine the docker uh environment like docker whole environment as a ship as a ship as you can see the docker logo this veil is like a ship okay and inside the ship there are lots of containers there are lots of containers and and the management of the ship or the direction of the ship is controlled with the help of ks or kubernetes okay so this kubernetes is maneuvering this docker named board you can say that okay so that's why their logos are inspired like that like docker is created like a ship and like uh we have this containers on And the Kubernetes is like a ship wheel. You can say that. And it is used to manual this part last large ship so that you can manage your containers inside this docker environment. So here we are doing it we are managing this Kubernetes uh setup and all manually. Okay. But in ECS like in ECS we do it uh the AWS handles it but in EKS we handle the Kubernetes. Okay. But this container management is done by with the help of KS. Okay. So that's the thing. So I hope you got a clear point like what is Kubernetes and why we are using it and what is the role of it. Uh so now let's move on to the third agenda which is like understanding our deployment flow. So what we are going to do like we are going to deploy our application with the help of EKS. Okay that's our main motive. Uh so for understanding of your sake we I have created a file for you guys. Uh so let's go on that. Yeah here. Yeah here. So you can see uh there are basically three main phases uh inside our deployment flow. The first is like uh configuring our roles like we have to create some roles for proper uh like for running the uh uh we have to create roles for creating the services and managing the services. Okay. Uh so that's why it's a good practice to create a role. So firstly we will create a role. Uh so like the first time for the sake of easy learning I will use the AWS console UI. Uh and after that I will give you the AWS CLI commands so that you can do it fast. But uh if when you understand using the console UI you you got to know that what we are doing. Okay. And then after we are setting up the roles and giving the proper permission to every role uh we are going to do the genkins setup uh which is like which I have shown in details in the past videos. Uh just we are going to do it uh quickly in this video. And in this uh genkins setup uh genkins will be deployed inside the docker container which will be uh going to run inside my local machine and I am going to show you how to run genkins inside a docker container in in AWS like in cloud also in the next upcoming video. Okay. Uh so be prepared for that one and in the fa like we have done with the phase zero we have done with the phase one like we have the roles ready inside our AWS we have our genkins setup ready. Now what is the thing to do? We need to create like uh this uh pipelines. Okay. So here we need to do two things. Okay EKS deployment we need to do two things. What are the two things? These two things are uh let me go inside this loop and let me tell you. So these two things are like infrastructure we have to infrastructure. Okay. And then the second thing is like deployment. So this is the two things. So what is infrastructure? Infrastructure is like setting up the AWS services and all the things we need for running our deployment. So this infrastructure is like setting up that uh like for example you are like for example you are building a car. Uh so infrastructure could be like all the uh tools and all the materials and all the like uh knowledge you want to create that car and deployment is like assembling all that thing so that a car can be easily run. Okay. So that is a two thing. So similarly we are doing that. So we have one flow we have one flow for uh setting up the infrastructure and we have another flow for deploying that particular project using the infrastructure we have created. Okay. So I hope that is clear for you. So we have this two things like firstly we are doing the infrastructure provisioning means creating the infrastructure or setting up the infrastructure and then the uh last thing is like deployment of our project using the infrastructure. Okay. Uh so I would say that this infrastructure thing is less important for a LM ops guy. It is more relevant to a DevOps guy and this application deployment part is more relevant for LM ops part. So I will focus more on this and I will focus less on this but I will show you that what exactly is happening but from a overview perspective and we will go details inside this. Okay. So there is four thing like firstly we are setting up the roles inside AWS. We are setting up the genkins and inside and after that we are running two pipelines inside the genkins. The first is for infrastructure setting up and the second is for deployment. Okay. So this is the overview of what we are going to do and now let's uh uh get started and now uh let's get started and implement the thing like move on to the like fourth agenda of our session and like uh do the implementation of our deployment flow. Okay. Uh so we are in the guide file of our deployment. So this file will act as a single source of truth for our deployment guidance. So we will follow it. So as I have told you uh previously that we our deployment will have four phases. So the zero space is like creating the users and giving the attaching the policies and giving them permission. So for that we will use the Amazon AWS console UI approach. So for that we have to open like the uh so I'm in the AWS console right now and we have to go to the IM and then what you have to do let's see in the source. Okay so wait a minute let me organize it well. Yeah so here so what it is telling like sign into the console. I have already signed in. You have to sign in. uh then you got to the uh main dashboard then I have to choose a root user. So this will be done and then what it is telling that create custom EPS policy here. So how we are going to do that we have to go to the IM then we have to go to the IM. So yeah we will go inside IM and then we have to click on policy create policy JSON. Okay. So we have to go to the policies and then we have to create a policy. So we have what we have to do we have to select this JSON and then remove this default JSON and uh in this file also I have given this complete uh code you just have to copy and paste it and let me explain you what it is doing. It is just giving that okay use this version here at this date and allow what what we are allowing we are allowing all the EKS accesses all the EKS related actions we are allowing in this policy. So we are doing that and we are giving this ID like uh giving it uh the name we can say that to uh as EKS full access. Okay. And then what we have to do, we have to just click on the next. So after following the steps mentioned in this uh like click on the next and then just add the policy name as EKS full access. Uh you will uh get this type of list like when you filter it by customer manage you will you should uh get this EKS full access policy listed. Okay. So this is uh this this must this is must like you have to go to customer manage and inside this you must see EKS full access. Okay. Uh so we have created the custom policy and uh like configured it inside the AWS. And now this next job is to create a user. Uh so for creating a user uh what we have to do we have to go to the users dashboard and here we have to go to create user and we have to give a name. So what I what I should do I should better organize it. So for that let me do this like this. So it will be better visible for you guys also. So yeah. So now what it it is mentioning that like we have to create a user and we have to give a username. Uh so I'm going to give a give it like a EKS uh deployment user. Okay. So let it let it be and if it's saying like we have to leave the checkbox as it is. And then about the policy it has written like permission attach policy directly and for now we will not add any policies and we will just uh going to uh do like next here and after that we are going to create the user. So our user is created right? Our user is created. Now what we will do? We will generate the security credentials for that and we will uh store them at a safe place. So then we we can uh like configure our genkins with that credentials. Okay. So we have to store that. So firstly let me uh tell you how to do that. So give uh go to the deployment user which you have created by mistake. Here I have created multiple. So let me delete it. Okay. I have deleted it and I have to ignore it for now. Let let me go inside this the user we have created and then here from here we what we have to do you have to create this access key and choose like with the command line interface way and after that uh click on the like I understand chat box and click on the next and description leave it empty and here you can see your access keys I'm exposing it for uh the sake of the video but I will disable it after that uh so I let me store them okay so I have stored that and hope you guys have also copied and pasted it somewhere or you can download the CSV file also and Now let me create at the done option. Okay. So we have done this like the third step and like what we have to do now we have to attach the policies uh to the user. Okay. Uh so for that let me expand it a bit. Yeah. So for that what we have to uh do we have to go to the users again and then we have to go to the uh correct user here. Let me delete it once more. Why it has not deleted? Does not exist. Okay. It will it will delete. It has deleted. Let me refresh it. So it must be gone. Yeah it is gone. So let's go here and inside that what we have to do? we have to now add uh some permissions like the policies. Okay. So AWS has set of policies by which we can define that what things we need to give access to and what to not. So it is good for as a management perspective. Uh so let me add some like let the policies. Uh so what what we have to do we have to go to the add permissions and inside the add permission we have to attach the policies directly and we have to search this policies here which I have mentioned in the uh text. So the first custom policy which is EKS full access. So if we find it here, we can find it here because we have created it in uh we have created it uh on our own. So the other policies like AWS cloud formation full access. So let me add it also. And then another is I am full access. Let's do that again. Okay. So we are selecting and we can see that three selected and total six policies should be selected. So yeah that is done. Now this also done. So as you can see uh if I can if I see uh the uh selected policies are six uh which we which are this mentioned here and what we have to do next we have to just press on the next and we can see what policies we are giving the permission to the first is like customer customer manage which is a custom policy which we have created and others are the native policies which the AWS gives gives us directly. So we have to just click on the add permissions and all the permissions are granted to this particular user which is EKS deployment user and how we are going to use this deployment users credentials using the access key and the uh secret key we have uh saved earlier. Okay. So now this is done and now what we have to do we have to do this last thing like create EKS service link road. So what is this EKS service link road? It is like when Kubernetes do its thing uh it basically uses the container uses basically nodes of the AWS and uh these nodes are automatically managed by Kubernetes. Uh but sometimes these nodes do uh need to perform some like task inside the AWS. So for performing that particular task we need to create this uh service linked role because uh we are giving uh the permission to this node which is present inside the Kubernetes. Before that what we are doing we are attaching the policies to a user but here what we are doing we are attaching a policy or permission you can say to a uh like a to a particular node which is automatically managed by a service which is kubernetes so that's why that's why it's called service link role so how to create it let us follow it and let us do it so what we have to do we have to go to the IM console and we have to create on the roles and inside the roles we have to create a role basically and like select AWS services and inside AWS services we have to select EKS EKS S okay and EKS and inside that we have to select this cluster so that this EKS cluster uh we have attached and then we have to click on the next and after that we have to attach this policy which is EKS uh cluster policy. So we will do that and here like uh we have to see the role name as AWS service role for Amazon EKS. Okay. And description keep default. Okay. We are doing that. And now that we have to just create the role. So this name already exists. Why is this already exist? Because I have already done the deployment myself at once before the system. So that's why it exist. But in your case it will work. Okay. So this is done. So I will just go back and hopefully your uh service is done. Uh so our first phase is completed like creating the user and attaching the policies. The next phase is like doing the genkins setup. Uh so the details of the genkins setup I have already told you inside this playlist uh like how what is genkins and why we use it and how to deploy it inside the docker container and all in the previous video like the lm of 7 which the core topic of which is Jenkins with docker. So you can see that for now we are just going to follow the instruction and then we are going to focus on the like deployment part. Okay. Uh so what it is telling uh it is telling to like create a docker docker file.jenkins and provide all necessary tools. So we have to like go to our cursor like ID you can say and I will create a new branch for you guys. Uh so you can follow it well. Uh so I will add it like EKS uh tutorial. Okay. EKS tutorial. So this will be our new branch and all the changes I'm making uh you will get inside this. So that will be convenience for you. So let me uh add in this file like we have to add something inside the docker.jenkins because uh in the previous session we have only this docker cla so I we have to remove that because we are using a cla so let's go and let us copy it and let us paste it here. Okay, so let me explain you what is happening here. Like we are using the base genkins file genkins image and on that we are getting granting a user as root and then we are running like installing the python and it dependencies. Then I'm just using this Azure CLI which I will remove for now and then we are installing the AWS CLI because the genkins will use the command line interface to interact with AWS. Okay, doing the pipeline run. So that's why we are installing it inside the genkins docker and then we definitely need docker cla and then we are using this uh cubic uh cubic ctl. Okay. So this will be used to manipulate kubernetes and then like we are creating a symbolic link which I have explained earlier and then here like what we are doing uh we are like creating the default plugins. So what are plugins basically in genkins? Uh you can watch it in the uh previous video like here I have explained it well. So you can get that and then after that we are just allowing the G to access to all directories inside the Jenkins and then we are doing some things for the Mac and all. So that is not that much of importance and now let's we have done with that and now what we have to do we have to just uh use this docker composin and it will what what it will do it will just build the image and it will host it will host it inside the code. So let me explain you what I'm saying. So firstly we need to create this file docker compost.jenkins jenkins.yamel. Uh so for in my case it's already created. Let me see that it is matching with the source. Uh so yeah it is matching. So what we are doing here we are basically defining the version as 3.8 and then we are defining the service as genkins. I want the name as uh gen name of the image as genkins and then we are deploying like a container using the docker file uh which like we are building the image of the docker using this docker file. Zenkins and then we are defining the platform and then we are giving the container name so that the container will get a name which is using this using the image which is built from this docker file. Okay. And then we are exposing the port like we are exposing AT83 to the outside world and the internal application is using AT8. So the internal projects our project is exposing at8 and the docker container and the docker controller and the docker container is linking it like the AT8 port to the AT83 port to the outside world. Okay. So here and this is also required for genkins and then like we have the volumes and all for the configuration of the directories. So this is done and like the compos is done. So we have to run it. So how to run it? We have to do it like that. So I will go here and I will like deactivate it because you we are using so we we are using UV so we don't need this and I will use the docker compost file and in my case the image is already builded and the container is already present. So that's why it started that much earlier. But in your case, the image will start to build and automatically a container will start using the builded image. Okay. So I hope that is clear and the container is started means we can access the genkins at the local host at83 port. Uh so let me go to my web browser and let me open that. Uh so let me write here local host at83 and my genkins should be there. Yeah, it is there. And for uh me it is like uh giving saying me username and password. So the username will be admin and the password will be like u which I have already stored. So the in your case it must be like asking a administrative password. Uh so for getting the access to the administrative password what you have to do uh you have to basically uh execute this command like docker execute genkins local. So let me tell you. So if you execute this command uh then it will give split out the password for your uh initial uh setup. So what it is doing like we are saying that okay docker please execute this command inside this container whose name is Jenkins local. So we are doing that here. So it will spit out the uh password which is present inside this particular uh path. So I have already configured that. So I will just directly paste it. So let me use that only because that is my default password. So let me keep sign in and it will sign me in. And in your case what it will do uh it will just uh start it to install the plugins. So select to install all the plugins for all the initial setup and all uh you can watch this video uh like this video genkins docker and you will get the idea that how to do it. Okay. So here what we have to do uh like go into the guideline and see what we what we need to do. So we need to configure the environment variable. Okay. Like let me explain you uh why we need to set up this environment variables inside the genkins. uh because our genkins is running the pipeline to deploy our application. So for the deployment what we need to do we need to perform some operations like uh like some actions on the AWS. So for performing that actions the genkins will need permissions. So how will the genkins will get the permission it will get it using uh the it will need the AWS key ID AWS secret key and all and for configuration configuration like when we are configuring creating a cluster and all we need to tell that okay use the EC registry as this and all. So that's why for the uh for the two reason like for the communication the required communication with the AWS and for authoriz for authenticating the communication between the AWS and also for uh keeping our data safe like uh uh we are don't expose we don't expose the APS keys directly inside the source code and all so that's why we kept it inside the uh environment variable. So that's why we are doing that. So let's do that. So how to do it inside the ranking? uh go to the uh settings and in this case we are going to the system and here what you will need to do uh you will need to go below this uh uh to this global properties and then check out this environment variables and you will see like uh the list of variables. So how you will you will do like you will just click on add and you will just uh create the variables like mentioned here. Okay. So uh here let me tell you how to add this variable like access key I have updated this is my previous user's access key access key. So let me update it. So I have updated the secrets and all. And then uh this ECR registry how will you know that how will you get this URL? So this URL is constituents like this is your uh the ECR registry URL is like this format like before that you have this AWS account ID. After that you will uh define the region here. So in my case the account ID is this and the region is US west one but it is wrong. My region is US East 2. So I have to update it accordingly. So I have updated by US East 2 and I will confirm that this is my uh ID account ID or not. So where will I found this account ID? You have to go here and uh on the uh top right section of the AWS console you can see the account ID. So you just need to put it here and you will get your ECR registry URL. And after that what we need to do uh like we need to configure all the API keys which I have already done and then I will just let me check let me cross verify uh the access key is done the region is done. The EC registry is done. Uh the ECR repository is done and then let let me change the name to multi-doc EKS EKS. Okay. So let me do it live here. Okay. So we will know that uh name is done EKS cluster. So let me do it also here. Change it will change live here and uh we have the Google API can drop API key. Uh so almost all the things are done. Uh so after that what we need to do we need to just click on apply and save. So the all the credentials are saved inside the genkins and now genkins know that how to uh what genkins have the uh power to access your AWS. Okay. So now what's the next thing? What is the next thing? So then the the next phase is like phase one is now completed and we are going to move to the phase two which is like infrastructure provisioning. So I have already explained you what is infrastructure provisioning. Uh so for infrastructure provisioning uh we are using like this uh yeah here. So infrastructure provisioning is uh used like the pipeline is created inside this genkins.infa infra file. Inside this we will create a pipeline. Uh we will write a code to create a pipeline inside genkins uh which will like set up the AWS infrastructure for our deployment. Okay. And this uh deployment infrastructure code is written inside this EKS with ECR.L file. So what we need to do we need to create a new folder like a new directory here whose name will be infra like in F infra and inside this we will create a new file uh which is EKS like let me EKS with ECR dot ml okay and this file is created inside this file what what inside this file will be gone like inside this file uh you will have code to uh configure AWS in the structure. Okay. So AWS infrastructure okay and how how this code will going to run like what service. So for that we are using the AWS native service which is called cloud formation. Okay. So cloud formation will be used inside this uh pipeline which is written inside the genkins.infra infra and uh this pipeline will uh see the configuration with this EKS with ECAML doyamel file and at the base the cloud formation is doing the work okay uh so let me create a docker file also uh sorry uh genkins file also so genkins okay genkins infra and let me write both the quotes and let me explain you uh so I have populated this genkins file with the code uh so let me give you the overview of this file so A pipeline has three things basically. The first is environments like here we are setting up all the environment key variables which we require. Okay. And then after that like we are using env.WS. So what it will do it will just say that say genkins that please find the credentials present in your environment which we have stored uh previously and it will like patch it here and then we are defining the stack name and all and then here we are defining the template file which is present inside this enfra inside the ease. Ecriaml. So this template file will be used to create the infrastructure using cloud formation. Okay. So this environment definition is done. Now let's move on to the stage. Inside this stage, the first is check out like getting the code source code. Inside here like we are validating that the cloud formation is correct or not. And then here we we are checking that if the uh stack like the stack is already available inside the AWS. If it is available then we are not going to go uh inside and create and if it is not we are going to create it and if it is already there but it is not same as we want. So we will update it. Okay. Uh this is also done. Now we are going to deploy this cloud formation stack uh like using this uh command. So this is done like here we are using AWS CLA for your knowledge and inside we are configuring all the things okay like the max size ECI repository name cluster name node instances and then that. So this is that this will this stage will deploy the cloud formation stack and after that we are waiting for the stack completion like we will wait for some time to uh like creation of the uh infrastructure is done or not. So this takes some time. The creation of the infrastructure takes some time. And this is kind of a one-off process. Like if you have created the infrastructure once uh you don't need to create it for like two to three months even like in our uh in production uh grade companies or the in production uh we don't create uh the infrastructure regularly or we don't update infrastructure regularly. We will do it like in four five months. Okay. Uh so it's kind of a tedious process and we always keep the pipeline different like creating of infrastructure and create deployment. uh because in creation of infrastructure is a tedious process and uh this is basically a role of devops guy more of a role of a devops guy than an L lm ops guy. So for you uh you should like know know the overview that what is happening and you should know the in depth about the deployment process which I will tell you. So this is done like we are here verifying the infrastructure that like we are running the uh again the CLI command AWS CLI command that our cluster is present or not. So accordingly we are just uh guiding the pipeline and after that we are giving this uh stack output like what is created and not and after running of the pipeline completely we are doing some post action like in success we are doing that in failure we are uh logging the uh reports and all and in always like this will always print that pipeline is ended. So I hope you guys understand the pipeline code uh for setting up the infrastructure and now what the main thing is remaining this configuration of the EKS with ECR okay code. So let me write it. Uh so this is the configuration code of like creating the infrastructure which the cloud formation will use to create. So here uh we can see like this is a kind of a configuration file and it has defined all the infrastructure we want like the node instances type we want T3 medium allowed values are this like we are defining the name here we are defining the capacity the repository name like multi-doc chat and all uh so this must be changed so let me change it must be like uh live okay because we have changed it there and here also let me see so make sure guys uh make Make sure that the name of the like uh the cluster name inside cluster name and ECR repository name is exactly same as you have created inside the uh credentials like environmental variables uh inside the genkins. So this must be same in both like the EKS with ECR.l like a configuration file and inside the genkins infra. So you must check that the cluster name and the ECR repository name is exactly same. Okay. So keep it same and if it is different the genkins will not be able to uh run the pipeline properly and it will give you errors. So take care of that and let us uh get the overview. We are getting the overview. Uh so ECR repository name is done like we are provisioning the resources like we are giving the VPC the internet gateway. We attaching a gateway here like public subnet we are giving and this is like uh the AWS uh infrastructure stuff which we don't want to know in that much depth but just we want to know that it is written here and this is done like we are here we are creating the security group which is required here we are creating the IM roles uh for like using the like for running that EKS uh cluster and all so that's why we are doing that here we have the EC repository the information about EKS cluster node group and finally the outputs okay so this is all done So I hope you get the overview that how about what configuration is written here. Uh so let's move on and let us try to what we need to do next. So we have done all this. Firstly we need to create a new pipeline inside our genkins. So we need to go inside the genkins and we need to create a new item. Here we are I'm giving this uh uh name as infrastructure. Okay. And then I'm selecting pipeline and then I'm selecting okay. And here you can see this is uh the configuration which we want to do. So we don't want to do anything else like we have to go here and we have to define the pipeline script from SCM SCM we need to choose. So if you're confused that what is what is it I'm doing and what is this configuration go watch the previous video uh which is like this. Okay go watch this genkins genkins with docker and you will get a clear idea that what I'm doing here to create this pin and what is this SCM and what is this like uh why I'm selecting g and placing this repository and all. Okay. Uh so here what I need to do we need to create a disk and here we need the repository URL. Uh so let me go and let me find this repository URL. So here is our LM of series repository URLs credential is none. If you if you don't have any private if you have any private repository select the credentials to get give access to Jenkins. And here is the main thing like we have we can specify multiple branches here. So for now I'm doing the one only that uh whenever there's a push whenever there is a push on the uh which branch we have created uh let me go on the ID and see EKS tutorial. So please use the name as it is EKS tutorial. Okay so now we are saying that whenever there's a push run this file okay and then we have to define the genkins file script path. So we are uh giving the genkins file use but add a dot in Java and this is done and now we have to click on apply and then save. So the pipeline you can see is created like you can see the pipeline is created. Now what we have to do we have to run this pipeline here. So you will go here and you will build it now. So uh in this time it will fail. Why it will fail? Because it will not get the branch. Okay? Because we have not pushed to it to the cloud. So it should fail. So it must see that okay the uh branch is not found. So what we need to do? We need to push this branch to the cloud. So let me make the commit like added added infra uh pipeline code commit and sync it. It will ask that okay you want to add the branch to the cloud. So I want that. So here you can see uh this build. Now if I click it will automatically trigger after 5 minutes because I have written it inside the SCM schedule. uh but for now for saving the time I'm just building it manually. So like the pipeline is running and you can see the overview from here like pipeline overview and you can get a clear picture that what is happening. Uh so the running of this pipeline like the this pipeline will take 20 to 30 minutes uh approximately it will it will like u provision uh a cure also like before time also like previously in my case it run uh at around 10 minutes and it's done but it will take some time. So please be patient and uh let me meet you when it will get done. Uh so finally after the wait of around 11 minutes and a total of like a lot of weight like I can say that 12 to 30 minutes uh the like uh deployment is like the infrastructure provisioning is successful and we can even confirm it by going into our console and looking like uh here the uh verification is given like we have to we can go to the clusters like inside EKS can go inside EKS elastic kubernetes service and inside that uh we can see that a cluster we can have this multi-doc chat live cluster Okay. So this is created but currently it is not configured for running our project because deployment is not done but this cluster is ready. Okay. So now what we need to do the infrastructure is ready for us. Okay. The infrastructure is ready. Uh and infrastructure ready. The now the next task is deployment. We need to deploy it. So for deployment we will create again another genkins uh pipeline uh which will be uh present inside the genkins files. And inside this this genkins files.eploy we'll use two files. Okay. two configuration file here for infrastructure we have created a configuration file right which is EKS with ECR which is configuring which is giving instruction to cloud formation that okay please create this cluster and all with this name and all so similarly we will create uh the configuration file uh two configuration file the first is deployment and the second is service.ml deployment will contain uh all the uh configuration details related to our deployment like the cluster names and all the secrets and all uh like all the computation resource we want and all that and inside the service.ml we will add the like the traffic management and the port setting flow. So you will understand it much better when I tell you the source code and all. So this will be created inside the directory which will be named as K8. Okay. So I will create a new directory in our flow which will be named as K8 and inside this K8 I will create two files. Okay. The first is deployment deployment. Uh please be uh like be specific and be accurate with the names. uh because the uh if you write the wrong name it will not be fine inside the pipeline. Okay. So we have created these two files and now what we need to do we will uh bring the code of there. So let me bring it. Okay. So I have uh like bring the code of this deployment. ML and similarly I will bring the code for uh service.ml. Okay. So here let me bring it. So here this is the code and uh one more file we need to create which will use this uh two files like the configuration files to deploy our application inside the AWS. So which will be genkins file.eploy. So let me create a new file which is genkins genkins file doeploy. Okay. And this that genkins filed deploy genkins file. It will contain the pipeline code. Okay. Uh so let me explain you that and after that I will tell you about the uh configuration files the deployment and the service one. Okay. So let me bring the code here. Yeah. So let us understand the pipeline and let's see what is happening. So what it has it has uh four basically components in it. The first is agent like this is about uh genkins agent. Uh so I have tell it about a bit inside the previous video. So you just watch if you don't know about genkins watch the previous video. And like here the edge agent can be any and we are setting up the uh environment variables as always which we have done in the infrastructure pipeline also like the AWS region access key secret key cluster name and all and APS keys and all. So here it is done and what is a trigger? Trigger what is a trigger? trigger is like uh use the pole SCM trigger and at every 5 minutes this is our chron statement uh which will be needed to define the uh schedule of like triggering that how in with in which schedule the triggering must be done like we should check for the changes inside the uh GitHub uh okay so that is that so chron statement is also uh I have told it in detail in the previous video so here it is written that please run it every 5 minutes okay please check every 5 minutes for changes inside the github so this trigger is done now we have stages okay the first stage Check out verify prerequisite verify case login and all. So let me explain it to you one by one. So firstly that is check out checkout is very simple. Uh it's basically uh like bringing the code inside the Jenkins pipeline so that we can run it. So that is done like verifying the prerequisite like all the prerequisites done or not like docker is available or not and then we are Amazon AWS CLI is available not kubernetes is available or not like all that credentials are present and all that. So we are checking the prerequisite and we are verifying that EKS cluster is present or not. Why the EK cluster will be present? Because we have done the infrastructure provisioning already. Okay. So this will fail if we don't do the infrastructure provisioning. Okay. So this is done. Now we are logging to the ECR. We say what is ECR? ECR elastic container registry inside AWS. Uh so that we are logging it using the credentials which we have created like the regions and the ECR registry and the credentials are already stored. So we are logging it inside that. Then we are setting up the docker using the buildex. Here we are setting up uh the docker and then we are building the docker image. which docker image will be we will be build we will build this docker file. Okay, this code docker file which will contain the build uh information of our core project. Okay, so that we are doing here. So that will be done here and after that we are pushing it to ECR like pushing this uh build image to the ECR so that we can use it inside the AWS to deploy it and then we are verifying that okay ECR is verified or not. we are using AWS CLI for doing the same and then after that we are setting up the uh like cubit ctl. Uh so for here we are using that like qbct is set up or not and checking that also and then we are like verifying that node IM permissions are available which we have already configured in uh in previous like firstly we have configured that and then like we have if the cleanup like deployment is failed we are doing the cleanup and we are creating and updating the Kubernetes secrets and all. So you can see we are using just this KBCTL to cubectl to do that and after that what we are doing we are updating the deployment manifest. So what is deployment manifest like what is manifest basically in Kubernetes? uh manifest is just a file which I've already told you uh before uh when we are starting the session that manifest is basically a file which in which we tell Kubernetes okay I want this that I want this uh four things inside my uh cluster and I want to optimize it for this uh traffic and all and Kubernetes will manage our containers accordingly so that our manifestation which is written inside the manifest file uh is meet okay so that is a role of that so we are updating like deployment manifest and all here and here like applying Kubernetes the the updated manifest we are applying it inside the uh Kubernetes uh and here we are updating the deployment image and then after that we are verifying the rollout means we are verifying the uh new version of the file okay and then we are uh like get deployment status if it is done uh we will be uh succeeded okay and then we get uh doing the status health check and all that the API which we got like the endpoint which we got is functionable or not and after completing the pipeline we are doing this things like we are printing the uh like the API endpoint and all and after if if we fail we are printing the logs and the always will like cleaning up it will restore the deployment and all so this will be done so this is how this pipeline will work and let us understand the deployment so deployment is again considering uh consist all the like metadata like the name of this deployment will be this so this must be changed so I added the live like the live test there so that uh the pipeline will not fail and And we are just defining all the deployment metadata and then we are giving that use two replicas. So what is this two replicas is saying? We are we are saying that okay please use two nodes two ports and keep two replicas of uh our project running so that if one fails uh the second replica will handle until the second recovers. So that is what we are saying here like create two replicas we are using the strategy as rolling updates. So what rolling update do basically uh for example we have this two parts for example we have this two parts. Uh let me explain you. Yeah. So we have this two called P1 and this P2. Okay, this P2. So what is P1 and P2? You can consider it like the smallest unit of uh computation machine the Kubernetes have. So you can think of like it as a container, Docker container, single Docker container or or something like a computer also you can think of for analogy like not uh that much like same thing. So P1 and P2 we have. So what we are saying that whenever a new version comes like uh this P1 and P2 both are inside uh V1 like the first version of our application. Okay. This is also in the first person. So for example, they both are in the first person. So what rolling updates tell tell us like the rolling update strategy what telling update tells us that whenever a new version is pushed whenever a new version is pushed of a existing project we want that the uh downtime should be zero. So how the Kubernetes ensure that in uh the downtime to be zero inside this rolling update strategy. What it will do? It will uh firstly like keep the old like uh P1 port running with the old version and it will update the P2 port with the new version. Okay. And it will check that if this P2 port is healthy, if this P2 port is healthy. Okay. If it is healthy like if it is healthy then then it will just update it to V2. So by uh by which we don't get any downtime and our deployment is uh also updated. So that is a goal of uh writing here like use the rolling update strategy and then we are just defining the template complete template of the deployment we want. Here we are like uh giving all the details about the secrets and all the container name the image like the image path and all like the container port. Here we are defining the environment variables uh which we want inside our container and then all the things like uh ports and then the like you can see the CPU memory is also allocated here limits also located here. So this is like we are configuring the deployment in the service. Mamel what we are saying we are using this uh like uh belong to like API version belong to v1 and we are defining the service here like the kind service like we are defining a service configuration here. The metadata is named like multidoc chat service. So this will be a service which with the name of multi-doc name service. Then we are defining the label like multidoc chat live. So it will say that okay uh kubernetes uh label this service as like multi-doc live. Okay. And now we are defining the specification of our service. Uh so we are saying that it's a load balancer. It's a load balancer. So what load balancers do it basically uh used to open our project like open our deployment to the external uh traffic. So it will basically automatically create a public IP uh and uh like guide the incoming traffic to the uh to our back end ports uh and it will like use a selector as this app. So whichever pods have their uh label as multi-doc live it will just guide uh the external traffic to this uh pods. Okay. Uh so here we are doing the port configuration. So we are giving it a name as HTTP and like what is this port and this target port. Uh so this port is like uh our service uh will listen uh to the external traffic at the port of 80 and whenever the external trafficics or the external uh or the user will hit the port 80 uh our service will listen and it will redirect it will redirect it to the pods at80 port. Okay. So this will uh which it means and it will use a protocol TCP and after that uh like this u this this will control that how the request from the same users are routed this uh line. Okay. So this is done and let us what what we will do let us now uh post the code firstly like let me add the code like added deployment code and now let me commit and sync here and in the genkins we have to do some setup so let's move on to there and go inside this and we need to create a new pipeline okay so we need to go to the new item and let me search the name of that so let me choose the name and yes it is done And now let us use a pipeline and we will create it. And now here uh we will leave the description empty for now. We will trigger it using whole SCM. The SCM should schedule should be like this. And if you don't do it, it will just be uh get uh like get configured when we use this pipeline SCM and like get and like give the path of our repository. So let me give it wait a minute. Where is the yeah here and then credential will be none and the m uh branch will be uh the branch which we have created. So what was the branch? Let me see. Yeah, here the branch was EKS tutorials. So, EKS PU P tutorial and the branch is uh like specified now. And we have to update the script part to genen files.eploy and let me cross verify the uh file name genkins file. Let me copy it. Let paste it here. Okay. So this is done. Now we have to see apply and save. And now what we need to do, we need to build it. So let us wait. Let us see what will happen. And if it will be failed, we will uh look for why it fail. It's now logged into ECR. It has verified that EKS container is present. Now it has successfully logged into ACR. It has built docker image. Now it's build the docker image also. Now it's pushing the build docker image to ECR so that it will be available to the AWS so that it can be deployed to EKS. Okay. So it will take some time. So okay so the pushing to ECR is done. We have verified that ECR image is verified and present and now we are setting up the uh cubit ctl and now we are verifying that we are having the IM permissions or not inside the uh AWS. So we have already set up that so it will pass and now we are cleaning up any failed deployment if it is there and now we are creating or updating the Kubernetes secrets like the Google API and gro API and we are configuring Kubernetes with that and after that what we will do we will just proceed with the deployment uh step. So as you can see uh the uh like the creator Kubernetes secrets are embedded inside the Kubernetes. The updated deployment is manifest and now we have applied the Kubernetes manifest and now finally we are deploying the image. So it has failed. So now let's see why it's failed. It is failed because uh inside the deployment test there is no multi-doc. So why it is failing? Because somewhere in the code we have not updated the name. So let me check. So let's go inside our code and let me search multi-doc. Okay. So here you can see somewhere like uh checking at this stage there is lots of uh 14 names. So we need to clean that. So let me clean that and come back. Uh so I have added this multi-doc chat live uh everywhere wherever there is multi-doc chat and let me cover another use case also. Uh okay. So now I have updated the multi-doc chat with multi chat live. And now I will deploy it like uh like push the change like correct corrected the name of app and then I will commit and sync it here and sync. And now I will rerun the build. Go to the pipeline overview and it is doing the job pushing to the ECR. It will take some less time now because we have already pushed it before. So it will use the caching more. So it will take less time for that. As you can see the pushing is done. Now I verify the ACR image setting up the ctl. So the deployment is kind of failed. So let me see why this failed again. Uh we are getting that multi-dup chat service list service not found. So somewhere the service is not created. Uh so the service name was basically misconfigured. Uh so I have revert reverted back to multi-doc chat service. Uh so we I have reverted back and let us see that it works or not. So collected the service name. Let us commit and sync it and let's rerun the pipeline. Okay. Build. So how you can check that the uh proper new build has come or not. So by you can go inside this like in the pipeline overview you will also get in the check out uh you will see the uh like name like corrected the service name. You can see the commit message. So it is using the right uh uh repos like like the right uh committed uh file. So let us wait again. Okay. Uh so the finally the rolling back to the previous name of the service has run successfully. So this time the uh cubic ctl get our multidet service. Uh so that is good now. So our deployment status is like good and we can see the service value is present here. So if I uh run this I will see the project. So it will take some time but it will run. Okay. So as you can see our service is running at this uh URL. So let us confirm it uh that it is present in the uh cluster or not. So in EKS uh you can see uh so inside this node group you can see we have this uh cluster node group and we are running it. Okay. We have autoscaling it. The subnet is also installed here and we get the ARN and all. So our deployment is finally done and this is how you deploy the uh uh an LLM project uh using AWS EKS. Uh so this is the complete procedure. Uh so finally congratulations guys uh you deployed your project uh in the in the AWS using the EKS uh uh service. So this is our complete guideline for deploying that and if you got any troubleshoots and all you can also like get this details here in the back of that and like all the learning decision and like the cost estimation you can also got uh but but but if you don't want to uh if you're doing it for learning purposes and if you don't want to bear any cost uh let me tell you how to clean this up. So below the our EKS deployment guide I have created a like cleanup guide so that uh you guys can delete it. So firstly what we have to do we have to go to our AWS console. So let us go to our AWS console and search for cloud form. We have this cloud formation and inside that we have to search for the stack uh which we have configured. So we will find this stack and we just need to see that what it's telling us to do uh like we need to uh like firstly we have find the stack and we have to delete it. Okay. So we just have to select it and we will delete it. Okay. So it will ask for our confirmation. So I will say delete it and the delete initialization is uh delete will initialize and we have to like what we have to do next we have to go to the ECR like ECR search here and we have to delete the image because it will cost you money also. So here like we have this which we have created live. So we will delete it and and and at the last we will go to the EKS. Okay. And let us wait for the uh deletion of the stack. Let us wait and let's see the cloud formation and see the stack is cleared or not inside the stack. Uh we can see the links and we can see if the delete is complete. So the delete is complete of the public route table. So it will take some time and after that it will be deleted completely and after uh if the cloud formation is deleted uh we have to delete the ECR and then we have to delete the uh EKS. Okay. The things mentioned here. Okay. So once that is deleted we have to verify that all the resources are deleted or not. So we have to go inside this EKS cluster EC2 instances and then we have to check the load balancers check the VPCs and if these four things are not present and finally we have to check the cloud formation and we should show that like deleted stack and we should get our stack and after that are all the resources is deleted. So like for now currently inside our stack we should we are not seeing it in deleted we are seeing it in active. Uh so just follow this guide for cleanup and you will uh able to uh like come on the stage that you will not get any bill from the AWS. Okay. And if there is something uh bad happens like if some errors is coming and all while following this firstly delete the EKS cluster because it will cost a very high amount. So firstly delete it manually and uh and then uh delete the ECR repository and then figure out what is happening. But delete the EKS and ECR firstly because it will cost you very high amount if you don't delete that and if you leave it for one leave it on even 1 hour. Okay. So delete that. So uh if you guys enjoyed the session like if you uh guys like the session and uh so finally we have done with all the session agendas all the five session agendas we have seen uh that what is the difference between ECS and EKS where we have understanded what is Kubernetes what is its role we have understand the complete deployment flow and then then we have seen the implementation of the deployment flow and then we have run the deployment pipeline using the genkins so all the things we have seen and the resources like the this uh file this single uh source of truth file you will find inside the GitHub inside the readme just go inside the readme uh at this particular branch EKS tutorial uh go inside the readme and I will link this uh notion documents URL there so you will find that there and uh regarding the like the nodes the you will get inside this flowscal so to open this flows and all you just need to install a plug-in which is called escal just write this here uh escalator let me show Yeah, here this just install this plug-in and uh this file will be opened as a flow for you. Uh so this file will also help you to guide and all the other files uh you can get from the GitHub repo itself. Uh so I hope you guys learned something new and you are able to deploy your applications with all this knowledge and uh you are u you have followed this videos and if you have any any doubt any particular doubts please ask them inside the comment section because uh we take comments seriously and we uh like look for the doubts and we answer them actively me and Siser both so please comment them and if you enjoyed the session and if you learned something new and if the information provided to you was useful uh please hit the like button and please comment and please do the uh subscribe. And uh if you want to like see all the other videos sir has uploaded a lot of like the you can if you want to know about finetuning and all you can see that and even uh like if you want to know about like what are evaluations how to do that because evaluations are very important and what is the basic concept of rag what is rag and all you can even understand uh me from me also uh from my YouTube channel. So you can do that also and soon I'm going to upload a video which is about like how to deploy uh our application like LLM ops application inside the uh Azure Azour cloud. So please stay tuned. So subscribe to my channel also sun channels also.