LLM Context Engineering Series Compilation | Part 3/3

Key Takeaways

If you can confirm, if you can see my screen, and also hear me properly, that'll be great. Okay. Okay, thank you so much. So, today's session is fully about building Today we'll be building a email reply agent from scratch. So, there are a few things that we will cover, which includes different platforms like Versel for deployment, Railway, Google Cloud Console. Fetching emails from Gmail API, etc. So, at the end of the day, what we are going to build is a web application which mimics email and it will fetch data from your actual Gmail application, and then you can reply through it provided there is a context. So, we will set all of that very nicely in the next few minutes. The prerequisites for this we have discussed in detail in the last lecture also. Just get ready with your Cloud Code. You will need either of the API keys, Open AI or Gemini. It doesn't matter whether it's Open AI or Gemini, because you are building this today. So, you can pretty much use any API key that you have available. I have these two API keys available. That's the reason why I have listed it here. And I'll be building using Cloud Code, but it doesn't matter really which coding agent you use. Any coding agent is fine to to be used. So, we can proceed with that. Just a second. I think somehow there is an annotation forming on the screen. Let's stop sharing and I'll restart. All right. So, here is the actual pain point. Okay, so I built this app myself and I have I'm currently using my own app for replying to all my emails. The goal is not to make a AI agent automatically reply or anything. Automatic replies I personally don't prefer. However, uh add a human in the loop, craft the reply using AI itself, so that it is useful to you in 90% of the cases. In 10% of the cases, the email that is crafted by the uh email agent in your draft will not be good, so you will have to modify it. So, this will not cut down your email reply time into zero, but if the time is X, it will probably cut it down to 0.1X. In the last lecture, I had asked you in the chat, "How many emails are you replying to typically uh on a daily basis?" And many people mentioned it's two to five or 10 to 20. Uh I did not see anyone who mentioned like 50 to 100 or something like that. So, the true utility of something like this will come for those who are using either email or Slack or Discord or any other, let's say, platform for communication in your inside your team, where the order of magnitude of number of messages that you have to process before crafting a reply or the number of replies that you have to craft are on the order of few dozens, because that's when you truly feel the pain. To reply to one or two emails, you don't need an agent at all. But, what I'm hoping is that irrespective of how extensive your usage of uh these communication platforms is, eventually you will find something like this useful. Um you might have also seen Gemini when you are using Gmail. Gemini creates an automatic response, right? I'm sure you have seen that, many of you. Uh my question is, have you noticed anything not so great about the Gemini responses? Like, what has been your experience about the auto response crafted by Gemini? If you have ever used it or if you have ever seen it, you can just mention in the chat. So, I will tell you what I experienced. It is the Gemini response is useful like 1/5 of the times, but it does not have the full context. It may have the context from that particular thread, but it's like it's a bit generic response. Um I use it like one out of six times. If if I see Gemini auto response six times, I may use it once. But other than that, I have pretty much found it very useless, right? And the main reason is not because the large language model is not good. Large language model is of course good, but the reason is we are not setting the context very nicely. So, this problem is very real, and in fact, we are also building a tool where companies can make use of something like this where they can connect their Gmail and their own database so that they can have their own email reply agent. Now, what we're going to build is something like this. We will have a website, which will look almost like uh gmail.com, okay? Inside this website, we will fetch the emails that are there in our primary inbox. The reason why I said primary inbox is the rest of the um categories, social, updates, and there is one more, right? You don't reply to those typically. Those are useful links or when whenever somebody shares documents or when there is a one-time password, you use that, but the main inbox where you will spend time replying is the primary inbox. So, we will create something which will look kind of like the Gmail primary inbox. And what we want to do is uh we want to craft a reply to any particular email in the primary inbox if we click on that email. So, if that email is currently selected, um there should be an automatically created reply. And if we approve, that reply should be sent to the person who sent us the email, right? So, we don't have to 100% automate, but we will 90% automate. Human is still there in the loop. In case the reply is not something which you like, in case the reply is completely wrong, you should be you should be having the ability to correct it. In case the email is the reply created by AI is 90% correct, you just need to make 10% change, you should be able to do that. Okay, so let's look at first of all, where do we need to start to do all of this? We will of course start with Cloud Code and we will require some API of any LLM. Um and in that uh you will feed the context of this entire email to that LLM and that LLM is only ultimately who's going to create the response. Now, before we go ahead, I have one question. We are using Cloud Code to build this application, but once the application is built and once it is deployed, is Cloud Code coming into the picture anywhere? So, Cloud Code is not coming into the picture in deployment. If you are using it in localhost, of course Cloud Code can help. But when we are deploying, we will only have access to one brain. That brain will be API or mean LLM any LLM whichever one we are using. We'll use OpenAI or Gemini. So, basically that will be the brain. Cloud Code we are simply using as a coding agent to build this platform. Beyond that, Cloud Code will not be there in the picture. Now, first we need to have a Google Cloud Console account. I will tell you how to create this if you don't have it yet. And this is where we will set the Gmail API key. And whichever email you want to use, um make a Google Cloud Console on that particular Google account. And of course, we will have Gmail as the application from which we are fetching our past emails or current emails. For storing the data, so when I say data, there are a few details which I need to mention. I will come to that, but for storing the data related to whenever our AI agent is replying or when we made a correction to the reply or when we gave the agent some feedback, all of that we will store in Supabase. Supabase is like a PostgreSQL style database, so you will see that when we are building. The front-end part of our application we will deploy in Vercel. If any back-end functions are needed, which I don't think if we are building a simple app, I don't think back-end functions will be needed, but if it is needed, we will use Railway to deploy. And the whole project we will be deploying on a GitHub repo. So the GitHub repo can be connected to Railway or Vercel, so that whenever you are making changes to the GitHub repo, you can auto deploy through continuous integration or continuous deployment to Railway or Vercel. So this is the workflow. So we will be using all these platforms. Now, can you tell me if we need to craft a reply to an email, what all things should we feed into the context of the large language model? Let's say that the database is the emails that are coming to you is based on some knowledge base. So you have some knowledge base document. Let's say you are part of team VisualEra. We have a bunch of courses, bundles, you we have a lot of information about other programs. Everything may be there in some form of data, JSON file or CSV file. Now, this is, let's say, the students are sending you emails or people are sending you emails regarding some of these programs, specific programs, okay? Now tell me, what all things should you pass into the context of the large language model to create the response first of all? >> So, first of all, the thread current thread which we are looking at, that we should obviously pass, right? Because in the current thread, maybe there are already two or three exchanges that happened. Maybe this is a new thread, so no exchange has happened, but we don't know. So, we will have to pass the entire current thread. That is the the least thing we have to pass, okay? Let me just type it. I think there is a little bit of slowness. So, thread is one thing we pass. Then, what else do we pass? We have a knowledge base, right? So, that knowledge base, do we ideally pass it entirely or because let's say you have 200 programs, uh somebody's asking you about some specific programs. Do you have to pass the entire knowledge base? Because it's a bit of waste of context, right? So, use rag uh to fetch relevant item from the knowledge base. From the knowledge base, KB. What else do you need to pass? You have a specific style of response, right? So, maybe you have to pass some email um writing preferences. What else you may need to pass? Imagine you have already built the application. Wouldn't it be it be great if your uh email response agent is is becoming better over time? Wouldn't it be great? Because what if the agent is not so good and you need to give some sort of feedback so that the agent can become better and better over time. So, there should be some feedback system um for your agent. And those feedbacks can be passed. If the number of such feedback is very, very big, maybe we'll have to figure out a way to condense it, but assuming that the app is in a very early stage, then the quantity of feedback will not be that high. So, it will not pollute the context that much, but this feedback will be very good. And what can be feedback? Like, what are different forms in which you can give feedback? Or or what all what all types of feedbacks can can exist when you're using this application? One can be like a star rating. So, let's say AI agent drafted a reply for a current email. I really like the reply, so I directly sent it. Which means there is zero change between um what the agent created and what I finally sent, right? So, one is um marking or storing the change we made to the agent's draft, storing changes to emails. So, this will be useful. Why is it useful? Because that way the LLM can learn that this is the reason why I did not like its reply. I have specifically changed one or two sentences in the reply, and and the LLM can learn from it. Storing changes to emails. Second thing is we can have some star rating, um which will tell the degree with which its response was accepted by us. We can also have something like um keywords or like a one-line comment saying the email was too generic or we can have a comment-based feedback, right? And this feedback needs to be stored somewhere. We can store it in Supabase. And one more thing that might be useful is past emails or recent emails. So, recent emails that were sent. So, you know why recent emails give a lot of value? Because if we have accepted an email and if we have already sent it, it means that that already sent or already accepted email is something like a gold standard, meaning as a user, I accept this email, which is the reason why I decided to send it. So, that will be very useful for my uh you know, judge. Um not judge, my agent to realize that oh, this is actually a good example of an email, right? So, recent emails that were that were sent, this also we should store store in the database. So, if we have this much uh that we are passing as part of the context, we are hoping that the LLM, whichever one we are using, will be able to um craft a very good response. Then, now let's go back go down. So, we will have Oh, one more thing I forgot to mention is what if there is a new data set that is added to a knowledge base? For example, if your knowledge base had information about your programs, and let's say you are now having some new programs. It is a painful thing for you to go to the back end manually and in Supabase add uh I mean I mean manually add the new data into your database. So, it'll be great if on your application itself on the front end, there is something which allows you to add new information into your database. Right? And one more thing that you can do is uh one more thing which we can implement in the app is setting some sort of rules. So, by rules here what I mean is um let's say your email agent, you you notice that an email agent is making some some big mistake in some of its replies, that it is hallucinating about one particular thing. You can create a rule saying this should never be done and that rule will always be there in the context. So, this rule also So, that it'll be great if there is there is a provision in the application to set some rules which irrespective of which emails you're replying to, the agent is taking care of that. Rule is select Yeah, some sort of guardrail because if something is always present in the context, it is it will act to a great extent as a guardrail, right? In any case, there is a guardrail the guardrail is human. There is a human in the loop unless you are completely automating the pipeline where you create a confidence score and for those emails where the confidence score is greater than 90, you decide to send the reply automatically. If it is less than 90, a human in the loop is coming. In that case, the guardrails need to be very very strong. Um but yeah. And what about the tone of reply? So, all of that will be in this email writing preferences. So, in the email writing preference, we will uh feed information about what type of reply should it be um a long reply, short reply, etc. Is my voice not very clear? If you can confirm, maybe it's it's just my mic or internet. Okay, okay. Thanks. Um so, this also I mentioned, okay? So, AI would have generated an email reply. So, before versus after. If we modify this AI generated email, the before version is the AI's original version. After version is the version which we created after modifying AI's version. So, that also if we store in Supabase or whatever the database is, it'll be very good. Uh another couple of features that you will need is you will require an authentication system. Authentication system as in just like how you log in to your email uh Gmail, just provide the same login access to this so that only you can access it. Second thing is something that will be nice is if there is a way to easily change the API key. So, if you are using OpenAI or Gemini, if the API key, let's say, expired, it'd be nice if uh API key modification provision is there in a simple way. Then a few more features which I think will be beneficial for the application is if you already made a reply to a particular email in the primary inbox. So by the way, we will only be looking at the primary inbox. The updates, social, forums, those things we don't care, right? We typically don't respond to those emails. We reply primarily in our primary inbox. So from the primary inbox, if we already made a reply, it will be great if we simply archive the email. Archive meaning it will not be deleted, it will still be there, but we will not see it in the front end. That way our front end is very clean. Second is we will ensure that the LLM is never auto replying. This This is easy to implement because Gmail API needs to be called. So unless we click a button, Gmail API will not be called. And to make our life easier, we should not have to give any custom command to start generation. As soon as we select a particular email thread, the reply to that particular thread should be created in one click. And of course, we should have a provision to modify the email and send it if we need to modify it. So then the amount of modifications that we will have to make will be high in the beginning and it will be low hopefully as the app becomes better and better with our usage, okay? So this is the um overall structure. Now we can um open our Visual Studio Code. And I have a folder here. So this is my folder. Okay, I have named it as building email agent live. So you can also create a working directory in your um uh open in your Visual Studio Code and what whatever is your IDE. And here I'm showing you the knowledge base which I had for my earlier um you know, my current actual email agent that I am using. So, I'll show you a folder. So, this is my knowledge base which I will not be using today. I just pasted it here just to show you um how it I how complex it is and how it actually looks like. So, one thing I want to show you is this emails folder. So, this emails folder has I think 10 uh 12 or 13 JSON files. Each of these JSON files correspond to all the email threads happened in a in a month. Uh so, all these are emails from our last 1 year of uh emails, okay? And look at the size of these JSON files. If a JSON file is 34 MB, it's huge, meaning that the number of emails here is huge. Uh so, the total size of these JSON files alone is maybe like 200 to 300 MB, which is very high, okay? So, this is one reason why we cannot pass the data um directly in a raw format to our context. It will easily exceed our context or it will rot our context. Second thing is the knowledge base. So, we have bunch of courses, information regarding those courses, all created in this JSON format um and it's very, very long. So, this is that JSON format document and this also, if we pass it directly to our um context, this will not be good. People might think, okay, we can directly convert this JSON into some format or store this as some uh you know, bulk a data set in Supabase engine, then just uh retrieve it whenever needed and retrieve the full thing whenever needed, but that won't be an ideal thing. So, we have to create much compressed version. So, what I'm going to do is I will start from scratch. Uh let me just delete this knowledge base because this is not what we are going to use. We'll use a separate one. Was the knowledge base JSON created by LLM? No, that was not generated by LLM. Uh, that particular knowledge base was generated using an MCP, which connects to uh, my Firebase Firestore database. So, um, uh, it fetches all the course information, creates JSON files for individual courses. Then I uh, ask LLM to write a Python script. And then for using the Python script, I compiled those individual JSONs into one single JSON. But then later I created a much more condensed version. Now, for today's purpose, I have created a uh, simple CSV file, which I will show you in a moment. So, this is the CSV file. Uh, I think I should just open it. Uh, but I think you can see what what it looks like. So, comma separated means each comma before and after each comma is a separate column. So, first column is course name, course link, course description, starting date, etc. So, this I created using uh, ChatGPT just today morning. It's very fairly easy. I just asked it to create some sample um, sample data set, okay? So, I asked it to uh, mention details about 150 courses with these details. So, it's not a very uh, big data set. It's a kind of small data set. So, I will share the knowledge base with you guys. So, let me first create a GitHub repo for this. That also I can ask um, Cloud Code to do. Create a public GitHub repo for this titled um, Viswara email agent. building. No, I have not connected with GitHub MCP. I'll I'll tell you how this works. Just give me 1 second. My computer is slowing down a little bit. Let me just close if there are any background processes. Uh you can actually do in parallel. Uh but if you are stuck somewhere, you can of course refer to the recording. Uh the goal today is not just for you to watch the lecture, it's to build it. And there are so many steps, so I think you can catch up. Okay, now about connecting with GitHub. Um so, I'll tell you how I connected with my GitHub. So, just go to github.com or search GitHub access token on uh Google. And then here you will see an option I mean on Google itself you will see an option called managing GitHub personal access tokens. And where is the link? Yeah, yeah, GitHub auth login should also work. Yeah. I created access token so that each time I don't need whenever I'm switching my workspace uh to my uh either AWS EC2 instance, I can just you know, I can just provide the access token. But yeah, you can you can easily use GitHub auth also. Anything is fine. But anyways, let me just find this first. Uh Satyajit, if I go any slower, this will take you guys 6 hours before you leave. So, um there are a lot to There is a lot to cover. So, the whole reason why I'm recording this is because um I'm pretty sure that because each of you have different systems, some of for some of you Python might be an issue, for some of you uh you might get stuck at some other place. So, I'm sure that 100% of you will not be follow it fully during the live lecture, but you can follow it afterwards. Um but let me just find the If any of you just find the the direct link to get the access token, that'll be great. Otherwise, I'll just show you how to access using GitHub CLI. Um log in to GitHub using CLI. or auth So, once I create the GitHub, I'll share the link with all of you. You can just uh get this dummy data from there. Oh, thanks, Vishnu. So, So, yeah, this is where you can create the uh personal access token. I have already created one uh for my AWS, but what you can do is uh this link which Vishnu has shared, um click generate token, and it will ask you for your pass key for logging in for sure. And you name the token and give it appropriate um permissions. Like, if it needs to access all repositories, okay? So, let's say you click all repository, and then here is an option to add permissions. So, if you add all permissions, basically this access token will act like a complete admin, an admin who has complete access to your GitHub, which is not recommended. If it needs to get access to only some specific repositories, you should only only give that level of access, right? So, in the permissions, you can give appropriate permissions. Okay? But, the idea is if you have the access token, keep it private, and you can use it whenever you need to connect to GitHub. So, it'll be it'll make your life very easy. Uh yeah, the process is basically this link which Vishnu has shared. Just go to that link, create an access token. In the access token, give whatever um you know, just give whatever uh permissions you need to give. Don't give permissions for all, just give repository level permissions. Uh but, that and all, you know, actually GitHub permission is like a lazy thing to do. You don't You don't really need it if you are using GitHub desktop or something on your computer, you can manually push the changes to GitHub. You don't need AI agent to push the changes. But, just because I just I'm just continuing everything here on GitHub uh uh on on CCO, I'm just doing it. But, this is the repo, okay? So, what you can do is Okay, now let me ask it to push everything to the repo. So far to the repo. So, this is the repo. Just open it. Clone it. And you should find I mean, the once this is pushed, you should find the um knowledge base there. So, this is the CSV file with roughly 150 dummy course data set, okay? So, just take a clone of this. Uh next is let's create a basic markdown documents, something like um, initial .md um, in which we will mention all of our details, what we want to build. So, I want to an email reply agent for Gmail. Then, use Gmail API to fetch everything from primary inbox. The agent should craft the reply um, using OpenAI or Gemini API. Then, there should be a Supabase database where the knowledge is stored. Basically, we have um, courses or programs regarding which we will receive emails. So, when the reply is crafted, it should refer to the document mentioning the program info. Um Then what else do we need? We need I need the ability to modify the email drafted by AI before sending. If required. And then the next thing is in the Superbase the original email drafted by AI. Uh I usually use a Whisper flow for speech-to-text, but right now I'm not using it because I realized my computer has become slow, maybe because I have not updated Zoom or something like that. It's becoming a bit slow, so that's the reason why I'm just directly typing. Uh in the Superbase original drafted email drafted by AI and the one I sent should be stored. The front end should be um deployed on Vercel. If back end functions are needed you can use Railway. >> All right, so I just gave this as the command and there are many more things which here we have discussed in the in the mirror notes which we are missing right now. Um Yeah, authentication system. We'll come to that. Um Then I also text stack I think it will ask it will we can say ask for our preferences what type of front end what what stack do we need to use in the back end. Yeah, but there are so many instructions, right? First we should give the instructions and then uh let's let's divide this whole building into different phases. Um okay, then we have to fetch some emails from our uh inbox. So, that I will do a bit separately. I will first ask um Cloud Code to create a separate function for fetching the email. Because the current the today the email that I'm going to use for this demonstration purpose is a different email of mine. Uh so, this is the email which I don't use that frequently. Uh I don't I rarely reply to anyone on this. So, you can see that many of the inbox is like some useless junk, but we can use this. So, here there is really no data which uh somebody can utilize to teach our um agent. But, what I will do is like I will create my own writing preferences separately. But, I will also show you if you need to scrape data from your own Gmail, how can you do that? Um And then you should never send an email automatically. The user should approve with one button click. Implement authentication so that only the owner of the email has access. The login can be via Google login. Um then the other feature that we need is Yes. For every email reply there should be a star rating and um textual feedback option that is stored on Supabase. Uh this we will do later. This provision for adding new data we can add a bit later. Um the next existing knowledge base. Yes. The existing knowledge base in the CSV file in the um working directory should be converted to a vector database. and stored in um Superbase. You will have to perform a rag to fetch the relevant info from the vector database. There are a lot of spelling mistakes, but I'm just ignoring it. Um Okay, I think this much initial uh information is enough for the time being. Um We can add the rest of the things later. The implementation should happen in phases. So, you should plan first, ask my preferences, and then execute in phases. All right. Uh maybe instead of initial.md, I'll just rename this as idea.md or something. Okay, and let me just push this. Actually, you know what? This is annoying. So, what I'll do is I'll show you how the easier way to um use GitHub. So, this is the GitHub repo, right? So, I have GitHub Desktop. If you install GitHub Desktop, you can just say open with GitHub Desktop. Do we need to log the completion metrics? Uh Ajit, what is the completion metrics? Like completion of what? You mean the phase implementation or something else? Uh log the completion metrics. So, when you say metrics, you mean uh for each phase if it did this uh so and so, should we log it somewhere? Um it's not needed because whether a phase has been implemented or not, we are anyway in this particular case it's not needed because we will anyway see it in our front end uh whether something is implemented. Let me not open GitHub Desktop, guys, because again there is a little bit of slowness. Um by the way, I have I can ask it to open it in my GitHub Desktop. Open this in GitHub Desktop. Push latest changes to GitHub. Yeah, yeah, I know I know Cloud can do it. It's just that I just want to show you where I skip the Cloud altogether. Like whenever a change is made, I can just with one click one mouse click from my GitHub Desktop we can just uh commit the commit and push the changes. So, so that I don't need to keep on using, you know, um Cloud Code for it. But anyways, uh it's pushed to the GitHub. So, if you need my exact idea.md file, you can take it from GitHub. Uh now I'll ask Let me increase the font size a little bit. I'm not sure if it's fully readable for you. Uh create a detailed um cloud.md file based on the instructions provided in idea.md. And again, here cloud.md will be only used during the building of this application. Once it is built, once it is deployed, we are not using um anymore the um we are not using anymore the cloud code environment at all. It's a production environment where the only LLM that is coming into picture is whatever LLM we have access to, right? So, this cloud.md will not be used once the app is working. So, meanwhile, just let me know where are you guys at. Um Okay. So, cloud.md is ready. Let's just look at the structure. Uh we can just open it as a proper markdown file. So, I have installed like a markdown preview uh plugin. So, if I just press command shift V, whatever markdown file I have opened, I can see it in a nice way. So, this is the same cloud.md, but I'm viewing it using this preview uh plugin. AI-powered email reply agents, which helps user respond, and so the agent um fetches emails from primary inbox, drafts context-aware replies using rag. Let's say user review, edit, approve. No email is sent automatically. Very good. So, front-end next JS deployed on Vercel. So, here it did not ask my preferences, which is okay. I can I could have said, "Okay, make sure that for front end and back end user preferences are also asked." Uh but maybe it will ask. Let's see. System flow. This This is fine. Core requirements, Gmail integration. AI reply with rag. Convert this into Markdown extension name is called, I think, MD preview or something. Let me just check. Markdown. Uh this one. I think it's this this Markdown preview with 8.8 million downloads. You can just search Markdown preview in extensions. You will see that. Um then Yeah, so email reply. So, once you have the Gmail integration done, this we have not done yet. We will We are yet to do it. Email reply drafting using rag. So, first the CSV file has to be converted into vector database. I will show you exactly how the vector looks like in Supabase. Then human in the loop, for sure. Authentication, so that only people who are having access to the original Gmail should get access to this app. Then Supabase database schema. Uh there is a table for storing emails. There is a table for storing replies. I don't know if there should be two separate tables like that, but anyways, that we can modify. Feedback, there should be one table. Course embeddings, there should be one table. This is also where the vector embedding will show up. So, you will see inside this table the vector embedding corresponding to the course. Then knowledge base {slash} rag pipeline. Uh feedback system with star rating. Implementation phases. First is the foundation. Knowledge base, Gmail integration, email reply generation, feedback and polish, then deployment. Okay, let's go with the first phase. So So all of these instructions are in natural language. You can simply say um you know, um in natural language just to go ahead with uh phase one. So let us start implementing in phases. Now So I'm simplifying this as much as I can. The reason is uh the amount of time it required to for me to build my actual Gmail app uh email app is like humongous because there are a lot more details which I added and I have probably spent two or three weeks on it. This is like a much more distilled version of that. Okay. Before we start phase one, I have a few preference questions. AI provider, OpenAI, Gemini, or for both? So both. OpenAI and Gemini options should be there. Then Superbase, do you already have Superbase project or I have Superbase account, but no project. If you don't have account, you can just log in to Superbase using Google or GitHub. Google Cloud project, do you have one set up with Gmail ID? I don't have Google Cloud project. Then package manager, NPM NPM is fine. UI framework, uh UI, I'm I'm fine with pretty much anything. Tailwind is fine. Uh you can take the call here. I'll just copy this and I I'm not sure, okay. By the way, if I paste this command uh in the chat, you probably won't get the exact same questions, but regardless. So the first command was based on idea.md, create detailed claw.md. Once claw.md is created, it will show different phases. Of course, the phases will not be exactly same for each of you. It will be It will be completely different. But more or less, it will be the same because our idea.md document is the same. So, and then once these phases are ready in the in the cloud.md, just ask cloud to start the execution. Then it will start the execution in phases. And since in the idea.md we had asked it to ask our preferences at different steps, it will ask our preferences. Okay, so manual setup needed first of Supabase, then manual setup of cloud. This also we will do. What else? Okay, so now cloud code has asked us to set up Supabase and set up Google Cloud Console. So, this all of us can do together because all of us will need this. The first thing is Supabase. So, just click supabase.com/dashboard. If you don't have a Supabase account, just go to Supabase and login using your GitHub. Anything should work. And I have bunch of projects here. So, I will You know what this urine smell tracker is? Our earlier office had a lot of issue with the bathroom, so I created a agent to automatically send them an email. Please uh clean the bathroom. Then visualr.us org. So, here I will name the project as um What should I name? Email reply agent V2 demo. Um so, just go to Supabase, login using your username. I mean, um if you have username password login you using that or Gmail I think Google also will work or GitHub. Once you are here, just let me know. You have to create a new project. Um you will have to create one organization. I think this there will be one organization available for you to be created for by free. Then database password, I can just put some strong password. Uh Okay. And enable data API. Enable This I don't think we need to enable by default. So, just create a project. And I don't need this password, so I'm just saying never. So, this is the Supabase uh project and we have to set up a few things. So, first thing is new project name is something like this. Choose a strong database password. Select a region. All of this is done. So, now what we need to provide to Cloud Code is the project URL, anonymous public key, um service role key. So, where you can find is on the left-hand side of the Supabase dashboard you you can see a bunch of things. Um So, here in the project settings if you scroll down or here itself you can see where is that? Um API keys. Before that, let me copy the project URL. I forgot to copy. So, I will just paste this in a TXT file for the time being, but obviously you should not paste these things in TXT file. Um I'm just creating this as a temporary thing, okay? Ideally, it should be in a .env file. I will ask Laura to do that. So, uh keys.txt. So, this is the Supabase URL. Then, I will need anonymous uh anon publish key. And I need a There was one more thing which was um service role key. So, that you can find, I think, in JWT keys. Let me Let me just find. I'm not sure where exactly to find the service role key. Let me just Yeah, I think it's over here. In the API key section, in the second tab. Uh public key I got from this API key section uh here in the publishable and secret API key. This publishable key is the public public key. Um the URL of the Supabase project you can find from the project overview page. And this uh service role key is over here. service Okay, so that is also done. So you need three things from Supabase. One is the project URL, Supabase URL, public key, and the Supabase service role key. You can find it from the dashboard. Second thing is we need um Oh, you can use search function in the dashboard to get these details. Amazing. Thanks for sharing, Satyajit. Second thing you will need is uh the Gmail the API key for accessing Google Gmail account. So for that we need to create a Google Cloud account. Many of you might already have a Google Cloud account. I'm not sure how many of you actually um have tried this before, but you can go you can just search Google Cloud and go to this and by default it will ask you to create an account, okay? By default it will uh if you go to this console option it will ask you to create an account, but I today morning created a free trial account. I already have bunch of other accounts, but I don't want to use that for this demo project. But here in Google Cloud console it will ask you you can create a free account which will last for 90 days. Um and it's free you will have free $300 credits uh two-step verification Turn on two-step verification, okay. >> Okay, so this is done. Two-step verification enabled. Almost. Uh I don't maybe it will ask you for Google Cloud account in the next step for you. Uh but in any case, whether it is asking or not, you should do this step because uh you will need it at the later stage. Cuz you need to connect your Gmail uh to your application, right? So, Gmail API is needed and this Google Cloud is required for that. So, the way things work in Google Cloud is you have to create projects and different API or different apps are part of the projects. Uh this I have already done. Let me just refresh. >> Let me just quickly check my internet speed. It's a bit Okay. Yeah, so basically what we need is the the JSON file and the token correct. So this is our first project and here we need to create Gmail API. So search in the search bar search for Gmail API. Then you will find this Gmail API option. And for this project, so I my project name is called my first project. We need to enable Gmail API. So just click enable. And let's go back to cloud. Cloud will need API and services or auth consent screen. Choose external user type, fill in the app name, add scopes. So This is the OAuth consent screen. Option. You just need to follow these steps. So let me just take a screenshot of this and put on your on the chat so that everyone can follow this. So I just shared the steps on the chat. Just follow that. And get started. App name Vishwara email dummy. Yeah, I have steps for Superbase also. I will share that also in the chat. Uh audience internal external, I think this can be kept external. Then email address, here also I'll give my own email address. I agree, continue, create. Now, we need to create an OAuth client, I believe. So, first external user type add scopes. Uh then in API and services go to credentials. So Have Has all of you created the uh Superbase yet? Just let me know where you are at right now. >> So all odd client I need to create. My application is a web application. So application type is web application. Name I can put anything. Web client one is fine. Then authorized JavaScript origins. So authorized origins I think I can just put localhost 3000 assuming we'll be local hosting this on this particular in the beginning we'll be local hosting it here. Uh name of the app can be anything. Doesn't matter. Uh you can put whatever name you want. It is just for reference. And for authorized redirect, whatever this is followed by uh {slash} API {slash} odd {slash} callback {slash} Google. It's there in the uh instruction the um screenshot which I shared. So now once you are deploying the app you will have a separate um And it'll ask you for a JSON download. So this JSON will have all the information regarding this app uh which you will need for connecting your Gmail to uh uh your web app. So just download JSON. I'm not sure if it got downloaded or not. Okay. So this JSON you can drag and put it inside your working directory. Uh I think that's all we need. I download / copy the client ID and client secret. So, that should be there in the JSON file. Uh now what we can say is I have pasted Supabase details in the keys .txt and the JSON from the Google Cloud is added to the working directory. So, now automatically this will realize that Cloud Code will realize that we have to add this into the .env file. Um so, it should hopefully create a .env file now. Yeah, so keys.txt file has three things. Um the Supabase URL, then public key and Supabase service role key. Uh so, those are the three things you need. Uh and in the JSON file, which is directly downloaded from um from the Google Cloud, it will have the the client ID and client secret. Now, let me scaffold phase one. I'll create Next.js project and set everything up. So, it's now building the Next.js app, and hopefully we'll see uh I'm not sure if authentication Google login authentication is being set in phase one directly or not yet. We'll see. Um where are you guys at? Uh have you done at least the two things? Uh Supabase setup and um uh the Google Cloud setup. Have you done those two things? And by the way, if this same thing you want to apply for Slack, Slack has an MCP, uh much easier that way. Or if it is Discord, Discord also allows you to connect through MCP, I believe. Yeah, so Google Cloud steps um the steps are shared in this particular screenshot. If you scroll a little bit above, um how do I tag it? I just need to pin it somewhere. Yeah, I've just pinned it. So, in Google Cloud, basically, first you need to create a OAuth consent screen. The app name does not matter. That you can put whatever name, doesn't matter. Uh but what you need to put correctly is the localhost. So, put localhost 3000 for the 3000 or any port is fine. Um and then the redirect URL, I'll just show you that you guys might need. So, this you can keep as the authorized JavaScript URL. And obviously, the 3000 can be replaced with with anything, doesn't matter, but by default you can keep it like this. Uh agents.md not necessarily. Uh we can create it. Um please zoom in. Okay, in terminal let me just I'll zoom in one more step. How about now? Is this better? Oh, this is too much. I can't read anything now. Yeah, this is better, I hope. I will also say once done push the changes to GitHub. Also, you will notice for those of you are familiar with this, .gitignore is the file which will not be pushed into GitHub. Okay? So, in .gitignore we have to mention which files from this will not be pushed. So, you see this .env file is included in .gitignore, meaning it will be ignored while the commit and push is being created. Node modules will have bunch of uh files with respect to the npm packages. That also you don't typically push into GitHub. That you produce locally. Um yeah, so you will not find the env file on your side once it is created. >> Uh yeah, that is a good idea. Yeah, I will show you once where the service role key can be found. In any case, these I will be deleting. I know that this is not the right way to uh keep using the key, but um so regarding service role key, this is where you have to go. Um you see this section, this tab called publishable and secret keys? That is where you can find the uh publishable uh key and then this legacy anonymous {slash} service role API key. From that, you can find the um this one, service {underscore} role secret. So, you just uh it. Okay, so all auth authentication is actually implemented in phase one itself. Both Google and Superbase can we summarize the steps again? Yeah. So, Superbase is for database, then Google authentication. Uh, for app to be deployed, you will require Vercel for front end or anything is fine, but I'm using Vercel. Sometimes you might need to have some back end Python scripts running in a server. If that is needed, um, you should probably give access to Railway also. But I I cannot recollect for this particular app, probably some back end functions are not needed, but let's come to that. I don't think Railway is needed for the time being. Uh, please say where to enter scopes for Gmail read only. That I did not modify, Roshan. I just kept the default. So, right now I think it will the API will have all the all the access, whatever is default. Um, you can change it later. Uh, this clerk asked me to do for a clerk. I'm not sure what does that mean. Um, I don't know what it means. Asking to do for clerk meaning what is that? I'm blind. I'm not sure. Uh if no back end then front end code will do rag and LLM API calls etc. Yeah, so that's what that's what I'm thinking. So in the current app which I have I there is the railway back end but that is for a very different functionality. I don't think I'm running a Python script or something to do the LLM API call or doing the rag. But we'll see when we come to that because sometimes the cloud code itself will ask for hosting a Python script on railway. Yeah, yeah. Where is what is the shape of your app currently? Like how much is it still being built? Yeah, so if front end ideally front end should not do this. So in if this is a public facing app, right? Let's say you are building something where the same URL public is coming and they are using their individual accounts or something like that. The best practice is not to do anything of anything in the front end. Do everything in a in back end. So create environment variables in railway and store it store the open AI Gemini API keys in railway. Let's come to that in a moment. Last command was to start building. So I asked to start phase one building then it asked me to give super base and cloud code sorry Google Cloud information. I gave it that. And then now it is continuing to build these four things, the basic setup and uh Google Auth authentication. Oh, I mean this is just uh it'll keep uh it's just something that Cloud keeps saying. I mean next step is simply asking Cloud Code to continue building phase one. That's it. It should It will take you See, it has taken me 11 minutes so far, right? Uh so it'll take you some time. Level of front end will be there, like we will be able to see something. Yeah, yeah, of course, Gitifying is that. Uh but sometimes it'll keep saying some weird uh tomfoolery. It'll keep on saying depending on what it's doing. There are like a lot of memes surrounding uh what Cloud says depending on what it does. >> Uh for email polling, which option uh cron job or Gmail push notifications real-time webhook? Um email polling means here in this context, what does it mean by email polling? Is it asking regarding how often to fetch the um latest emails from the API? It can be based on or is it something else? If it is about how often should the Gmail API latest email should be fetched from the API, simply it can be based on refresh. It need not be a cron job. Yeah. Cron job means I think it will keep on doing um something as a function of time. Okay, how to react for incoming emails? Yeah, yeah. So, for for our own purposes, right? Uh the Gmail required a lot of additions. Um one is speed. So, one of the practices that we have is keeping because the same inbox is being looked at by three or four different people, we need to keep the inbox a little bit clean. Now, Gmail has a certain way uh like you can enable um keyboard shortcuts in Gmail. And one of the keyboard shortcuts for archiving an already replied replied email is like E. So, if you press the key E, an email will be archived, which means it will not be there in the inbox. It you can see it when you go to the all mails, that option. So, one problem that I noticed was um in the app that we created, whenever I was trying to archive, once archive functionality is implemented, it was too slow. So, when I have to move from one email to another, the the the time it takes to make that switch is like 3 seconds, 4 seconds, which is like a bit annoying. Then, we were looking at how Gmail how Google implemented the same thing. And Google has something where And first of all, this archiving is also another API call, right? So, we can't do it as fast as Google probably. Uh I don't know internally what they do. But in our case, what we did is the moment I press archive or e button, the process will happen in the background, but in the front end, we will immediately make the email disappear. Meaning, in the front end, we will see as if that email is gone. But in the back end, it may take a few more seconds to actually archive the email. So, we did something like that to improve the user experience so that we can go through the emails a bit faster. Um then, another issue that I noticed was um in the Supabase, some tables were not getting updated. This issue most likely today we'll we'll again see. And then, sometimes what what happens is in Supabase, you'll have to run some custom SQL command yourself by copy-pasting it. So, to create new tables, sometimes uh and sometimes the cloud code the code itself, which was written by cloud, was making a mistake where um my original plan was whenever an email is modified between 5% to 10%, I have to notice that change, and I have to store the original email and the changed email in the database. But if an email has been completely modified, like let's say there is 50% change or 70% change, there should It should not be stored because then there is no point There is nothing for AI to learn. That was my thought. Then, what was happening was the the way the code was written had a logic issue, the flaw in the logic that all the change even minor change, it was because of a slash tab or some extra character coming in the uh in the email. All the emails, irrespective of whether I make 1% change or 90% change, were experiencing huge change. The the the way Cloud Code was the code was taking it was such that the email what I sent even if a minor change is made is actually 90% change. And because of this, no email was getting stored in my Superbase database. So, this problem we did not realize initially. We ended up realizing it when the quality of our response started degrading. Because no email was getting stored, we realized. Then we had to check the Superbase manually for every single interaction to ensure that the way items are stored in Superbase are the way we intend it to be stored. Um then uh what else? So, where are you guys at? Are you guys still in the implementation phase for phase one? Yeah, so context window management Okay, yeah, one thing I forgot to mention is I had created separate page for context window management for my app. So, in my app I was using Gemini API. So, I wanted to know what exactly is going into the Gemini API call into the context each and for each and every email response creation. Um so, then the I created a context visualizer in which the exact uh content from the each file, whatever is being fetched from Superbase, exact content will be displayed in the front end so that I can see what context is being displayed. Such a thing we can also build here. Because if we don't know visually what goes into the context of the LLM, we will never be able to manage it properly. So, for context management, one thing that we are definitely going to do here is rag. Okay, let me know when you are ready for phase two. Okay, something seems to be done. Let's see. Mhm. Migration SQL in Superbase in SQL editor enable Google OAuth in Superbase authentication providers. Add your OpenAI Gemini API keys to environment local. Okay, let's do this. I'm not sure if Superbase allows any other. I think primarily its database is Postgres SQL. I'm not I'm not sure if it has any other database format. Uh Okay, first of all, where is the SQL code which I have to run? Give me the SQL code to run in Superbase SQL editor in a separate file here. Then enable Google OAuth in Superbase authentication provider. Of course, I can do that. I want to show you one more thing. Um Um, let's just look at this. There is something called table editor. Um, and here you can see that there are no tables yet. Meaning in Supabase there is nothing. So, I just went to the Supabase database. On the left hand side you'll see bunch of icons on the table like icon is there. That is table editor. Or even in project overview, you will see something called um, or not in project overview. Here you see something called database. So, in the database there are no nothing yet. No tables yet. So, to create the table instead of manually if you want to create it automatically, uh, Cloud Code will typically when you deal with Supabase it will give you these SQL uh, functions to run. I'm not sure if Cloud Code can directly run it in Supabase or not. Maybe it can. Uh, maybe we can ask ask it. But in the interest of time since I already have it here, I will show you how to run this manually. So, it will often when you work with Supabase it will automatically ask you um, uh, create uh, Supabase tables using the SQL. Uh, so, just go here, find this SQL editor, and then paste the SQL code which Cloud Code gave you here, and then just command enter to run. And if it is if it is successful, it says success no rows returned. Now, hopefully when we see the database there should be some tables. Let's see if something is there. See? So, some tables are ready. Uh, feedback table, course embedding table, replies table, emails table. So, the tables are ready. Um, and Okay, now we need Gemini or OpenAI API key, so that I have available So ENV file I'm not going to show directly because it's not a good practice. But ENV file should ideally have all the um I mean I have already shown you most of the keys, but in general ENV files let's say you want to keep it separately. You can create a dot ENV dot example file. I can show you a dummy. Um but ideally it should have Superbase uh um links and then it should have the open AI API key and what else? Okay, I have already done already ran the SQL migration. But I have not enabled OAuth in Superbase. I don't know how to do it, so let me ask CloudCode how to um enable Okay, then enable Google OAuth in Superbase. Go through authentication providers Google. Toggle it on, place the client ID and secret. How to Give me link where we can paste in Superbase. >> I'm also not sure why Supabase requires Google OAuth right now. Uh last time the way I implemented was my app required Google OAuth, not not Supabase, but let's see. Authenticate your users through a suite of providers. Okay. Uh also in the Google Cloud Console, add this as an authorized redirect URL. So, that is done. Uh authentication is one time or they have expiry. It's It has expiry, but I think the authentication token will survive for whatever the typical duration is. This also is done. This is also done. So maybe for some of you Superbase and and Google connection, it will not ask. Last time when I built it, there is there was no such thing. Just my web app was getting connected to Google. So I never had to provide Superbase redirect URL to the Google Cloud. Will the script pass the CSV, generate embeddings by OpenAI, and store them? But it will be great if front end is ready for me to see the front end. Can you local host the website so far. Yeah, I will be providing the recording for those of you who are late, for sure. Has anyone reached the point where the phase one is getting closer to completion? I don't think you will everybody will have six phases. In my case, I have six phases. But you may have different number of phases. Yeah, so the callback URL is pasted over here. So wherever you have pasted the callback URL for the local host, right? So you would have pasted something like this in the authorized redirect URL. Below that, just add the URL and then paste it here. >> I'm just manually just local hosting this once. Let's see how it looks like in in dev mode. Okay. So, this is the login screen and here I will have to message is not Okay. So, in the first step I am getting this error that authentication is not working. Terminal prompts as is. So, if I paste So, I have not given too many terminal prompts, by the way. The only terminal prompts that I have given are uh pasting the So, see, terminal prompts will slightly depend on what the Cloud Code is outputting you, right? So, for me, Cloud Code was asking uh to do some things in uh Supabase and in Cloud Console. So, I did both of that, but I think depending on what the structure of the project is, sometimes it will not ask you to connect Supabase and Google directly. So, that's the reason why I don't think it's that useful to have the same terminal commands, but if you ask me which step you are at, I can tell you what to do at that step. Uh but, yeah. If there are any major billing-related commands, I will share those commands directly in the terminal. Okay, phase two is complete. Uh in the login, I am getting this error. So, the reason why I pasted this is This is my localhost, and when I try to log in using Google, um it's not showing me the Google uh login pop-up, rather it's showing throwing me the error. So, we can just fix it. Uh enable Google, go to this, find Google, enter the client ID, enter the secret. I thought I already did this. >> Okay, the problem was I had not flipped this switch to enable the Google login. So now it is enabled. So probably it should work now. Okay, so now it's asking me to log in using Google. Yeah, Google authentication in this S implemented using Okay, now I'm getting a new error. Access blocked. This has not completed Google verification process. All right. Now I am getting this error. Oh, we need to add this user to the test user. Uh, okay. Add your Gmail address, the one that you are trying to log in with. I thought I already did that, but let's see. Uh, thanks for that suggestion, uh, Vishnu, by the way. Any user with Google account? Not yet. Add users. I think this is needed because I had um, I have not published the app. It is in this testing stage. So I need to add the users who are eligible to log in to see the test. Uh, okay. So now hopefully, since my email is here, hopefully it should work. Google hasn't verified the app. Uh, you should only continue if you develop you know the developer. Okay. Sign in. Select all. Okay, so something is working in the app so far. Um, I don't know the I don't know where is the inbox yet. Authentication is working. Okay. So, okay, so so far so good. Uh, inbox has not been set up yet or I don't know if there is a separate link for the inbox. It is working now. Where can I see the inbox? Uh, we did not give the any website information yet because this is still in local host. So, I have not deployed it, right? So, I have not connected it with Vercel. That I will do at the end. Uh, why connect it right now? Uh, now the dashboard just shows a placeholder message. Email integration integration is in phase three. Right now, we haven't built it yet. Okay, so in phase three, we will fetch the emails uh, from primary inbox, display them. Store fetched emails in Superbase. Uh why should we store fetched emails in Superbase? That is not needed. Yeah, I think there is some hallucination here by Cloud Code because the reason why Cloud thought Superbase and Gmail should be connected is because Superbase emails are being stored in Superbase. But that is not needed. We don't need it. The original idea was to keep a record of which emails got AI replies and link them. But we can simplify fetch emails live from Gmail API and only store the email reference when a reply is actually created. Exactly, that is better. That is what we need. So, what Cloud Code was thinking so far is that um it thought um we are going to store every single email in Superbase, but that is not the case. We are only storing those emails to which we make a reply because they are useful emails, but every every single email in the inbox need not be stored in Superbase. That is the reason why Cloud Code asked us to connect Superbase and Gmail Google. But that is normally not needed. So, right now we are simplifying it. Uh so, now hopefully after this phase three simplified update is done, we should see the emails being fetched and our inbox being displayed here. Let's see that. Also, I hate this font. I don't like this style at all. It's really bad. But that we can change in the end. So, the main thing is working now, which is authentication with Google. So, can you tell me can you guys just text in the chat where are you at? Are you still in the phase one building stage or have you ended up getting something up in localhost yet or where are you guys? Okay. Uh for phase one, how many minutes is it showing so far? Has it crossed like 25 minutes and all or is it still uh like 10 minutes, 15 minutes? Oh, you have mail pulls from the Gmail app and shown in the app. Okay, amazing. I saw some Murali Krishna. It's amazing. Do we need this front end Supabase JWT secret? I don't think so. I don't think we need that. Uh we just need three things from Supabase, the URL, uh public key, and the service role key. Okay, so if email fetched email is uh being displayed, it's already amazing. Um so now, ideally, you should be able to reply from there itself. Um reply to someone from there itself. So right now, 90% of the times I use my email app, uh 10% of the times I use um uh the actual Gmail. Uh Satyajit, we don't have to manually um set up the ENV file that um cloud will automatically take care of. You just need to give the API to Cloud or put it in a different file and ask it to create ENV yourself ENV yourself to Cloud code. If anyone can show the screenshot of the UI of the app, you don't have to display any private emails or something, but just um uh if you can just show the UI of the app as I think that'd be great. Anyone who has the app working, you can just you can just take a screenshot without showing the actual email content or something. Just to show the style that just to see the style has that Cloud code has followed. Oh, by the way, one more thing I forgot to mention. So, yeah, 2 days back I was building something with Cloud code, but I accidentally ended up using Sonnet, okay? And Sonnet, the output was so bad. I thought Sonnet was also semi-decent. I was building a web page and the output was so bad that I was like confused like why is it suddenly so bad? Then I thought, "Oh, maybe the model is not Opus 4.6 anymore." So, Opus 4.6 is making a huge difference. My thinking was that the agentic architecture that Cloud has built, that was making all the difference, but it seems to not be the case. So, if any of you ever feel like the output is not so good, just have a look at which model the Cloud code is using, okay? This is great, Murali Krishna. It's a simple Let me just open the screenshot. >> Okay, pro plan is nearly impossible. Uh so in pro plan when you are let's say building an app like this, if you look at the slash usage um have you noticed that it's like so fast in 1 hour if you build for one roughly an hour it it's exceeding the limits is it? Okay. Mhm. It's it's so unfortunate in the sense like at least for you know, UI related tasks I have seen Sonnet is not so good. Opus is really really good. Uh but for tasks where you are primarily dealing with code or just text I think using Sonnet 4.6 is fine. Okay, Vishnu also seems to have a front end ready. Okay, amazing. You can name it something um something which you like. Uh one thing you will notice it this is super slow. Like this meaning not the building process. Whatever apps you have built by default will be a bit slow. Right? So you will have to do additional things to take care of it. Okay, restart the dev server. Uh you should see So now let's see what has become the status of our local host. I am getting this error. Oh, 6:30 p.m. to 12:30 a.m. IST. Interesting. In the back end it has given incorrect category. It should be just inbox. Yeah, yeah, yeah. Um, that's that's exactly what I I got like invalid label category. So, I just copied it and uh asked Claude to fix it. So, Claude is now changing it to inbox. Primarily because uh Gmail API probably does not have a category called category primary. So, uh can I ask you when you use Sonnet 4.6, are you able to manage uh like the full a full building session? Um, just out of your experience, how does how does it work? Cuz I'm using 80% of the times Opus 4.6 and only few number of times. Okay, not for full building. All right. Okay. Yeah, all right. So, now it is working. Although I the UI is very, very bad, uh it is working. I will just quickly show the UI of my original app. Um, although I don't want to spend too much time on it. So, this is how the UI of our current uh actual application looks like, the one which I built. So, uh there's a lot of difference. But anyways, we don't care about UI that much right now. Uh, okay, let's see if I send an email to myself. Can any of you send an email regarding, let's say I have a course. Okay, a rag is not set up, I guess. Let Let's check Superbase first of all what all tables are ready, then we can come back to email. So, in Superbase, go to the section called database. And And then go to tables. Then you will see these tables, okay? So Now, if you want to see the table, so this course embedding should should have a vector embedding somewhere, right? So, let's view in table editor. And these are the courses and one of them should be the embedding. See, this is a vector dimensionality. Let's ask Cloud Code itself, what is the embedding model did you use? For rag which embedding model did you use? What is the dimensionality? And how did you do chunking? And when I deploy the app do you run rag in the front end? Or should we set up back end in Railway? >> Uh reply schema Yeah, reply schema meaning how the email reply should look like. I have not set it up yet. Um so that we are going to do in the next step. So right now the only thing that I have done is created the rags check database uh in which the course details are there. But apart from that, there is nothing set up. Uh how to reply, how short should be the reply, nothing is there. The table it has automatically created, but we have not done anything manual there. Uh but let's come to come to that once I want I want to see how rag was implemented. Text embedding three small model, 1,536 dimensions. Cheaper and fast, good for 150 courses. Then chunking strategy, one chunk per course. Okay, so that makes sense. Each row is converted to a single text block. Uh then drag it at runtime. Rag runs server-side in next JS API route, not in the browser. Uh API route receives the email, embeds the query, Superbase PG vector similarity. Then no railway, separate railway back end needed. Next JS API routes on Vercel handles it. Okay. So this is then how my current uh schema also works. Then railway would be would be needed only if you have a long-running tasks or heavy processes. For this app, Vercel alone is sufficient. Want to proceed with three is four not yet. I just want to see how Superbase looks currently. So Roshan has mentioned three tables are empty. Yeah, for me also three tables are empty. That is why because we have not given any tables any feedback. We have not replied to a single email using AI yet. And this emails I think is the uh I'm not sure what this table is. Let me ask. What is the table called emails doing in Superbase. So, these three will not get populated by default. Uh we have to keep using the app, but we'll come to that. Let's just first make sure that we have the tables which we need. It's only used when a reply is created to store the email references so that we can link it to replies and feedback table. Without it, we have no way connect. Okay. Where are we storing the AI-generated email and the uh version of the email that I sent after editing the AI-generated version. Okay, that is anyway coming in phase four, I guess, so I don't have to jump the gun. Let me just delete this. Okay, let us proceed with phase four. Okay, Vishnu is saying, "For me, there is a separate ingest.py file that loads the data using the key." Uh using which key, Vishnu? Okay, okay. But where is this file being executed right now? Um Is Are you just running it for the time being to create the 150 course rows? Course embeddings is empty. Did you populate it using the run in Superbase code? Yeah. So, I populated it by uh so I think I just got the SQL code, right, to populate it. Uh no, no, no. Wait, wait. Wait a second. I think my cloud code would have already implemented the code for populating it. You will have to ask for it separately. To create the dummy table in the first place, you will have to run to create the tables you'll have to run the SQL code. But if it is empty, you will have to populate it by running the script. For that, you can ask Cloud Code to run the script to populate from CSV file into the Superbase table. Right now in your tables, what are what are all tables are there by the way? I have four tables. Um This is not the table which I wanted. I wanted one for course embeddings for sure, one for one for replies, and one for feedback. But here this emails is like a replies is just a reference to emails. I don't want that. It's okay, but I need one more table for storing the changes happening in emails to track the changes happening in emails. So that hopefully will be created during phase four. If not, I will manually create it. Okay, course embeddings, replies, and feedback. Yeah. Okay, that is that is perfect. You don't need both of these. That's what I'm saying. You don't need emails and replies. Just one of them is enough in my opinion. The reason why my Cloud Code ended up doing this is because initially it started with using Superbase for authentication, which means it thought email all the emails it will have to store. Then it created that replies also it was it created at that time. And now it realized it it need not have that. Uh But yeah, in the local host You know what we can do? We can test this in the local host like this. Maybe um If I open the CSV file >> If anyone else got a decent UI for the app, feel free to share a screenshot. I don't like my UI. It's not not that great. Uh so this is the CSV file. So to test this, you know, uh what we can do is I will share my email with you. Um let's just make sure that everything is uh in proper shape. To that email, if you guys can send an email asking information about some course. So all of you have access to the CSV file through GitHub, right? So pick any random course and um just send an email to my email, my Gmail. This is my email. I'll put it in chat. Sorry for the typo. Can uh all of you send an email to this particular email for information regarding uh any one of these courses. Like I'll just send bunch of course names here in the chat. But otherwise, if you want to refer to the CSV file, you can get it from the GitHub repo. But these are the courses. Just send an email. Hi. When what are the details of this courses? When is it starting or something? Ask some question using which I can craft a response for you. Uh I just want to test because for testing I need some emails in my inbox, so that's why I'm asking. Then we can see if the rag is actually working or not. Okay, Shiva has also shared the snapshot. It looks decent. I mean, these are not the best UIs which mimic what Gmail does, but they are decent. You know when this will become actually useful to you? You should You should have keyboard shortcuts also. Keyboard shortcuts for navigating. So, for my actual app, the up and down arrows are keyboard shortcuts for navigating. E is for archiving. For replying, command plus enter. Uh Claude will automatically include some of these shortcuts that are standard, but it will probably not do it for um it will not keep the archive shortcut. Uh it will not probably have the up and down arrow navigation shortcut. Okay, build to verify phase four. So, we are done with phase four. So, hopefully now the email reply feature should be done. Okay. Murali Krishna has shared a uh So, that is a email crafted response or um let me just download your screenshot. I'm not able to see it right now. >> Okay, it has course details, is it? Let me just open the screenshot. I'll I'll show it to everyone. One thing we have not set yet is the preferences, reply preferences that we should set. And can you guys also ask your agent what is the embedding model that was used for rag, the the embedding for creating the embeddings, and what was the um size of those embeddings? Okay, this is the generated reply um for Murali. If you have last mailer for instance. Okay, there are some issues. I can see there is this markdown formatting that needs to be better. And uh also Murali, can you confirm if there are any URLs provided in the reply? Because I think URL is there in the CSV file. In the reply, is there any URL? Probably not, right? Uh I'm not sure if beyond this if you scroll down. Okay, okay. So, that needs to be set up. So, we need to say that whenever an email is crafted, it's action-oriented. The other person person on the other end should be able to perform some action. For that, we have to provide any a URL or something where which they can click and go to. So, that we have to do. Um but that is easy to do. But, this is already decent. Uh Okay. So, the reply part will be done uh phase, okay? Phase four is built, passed, and pushed uh ready for phase five anyway. Uh did you guys send an email? Can you guys send an email to my inbox so that I have something in my inbox asking any question? This is my email, Sri that okay. Already I have a bunch of questions, so let's look at this. I looked at the course detail, want to understand whether this is recorded query and rack systems in practice. So, then if I click this, let's see what happens. Okay, one issue is uh I have no idea whether the there's an email being craft reply being crafted or not. So, some circular rotating something or something should be there in the UI. So, let's talk about that. I don't uh the AI reply being crafted if it is happening in the background let me know with a rotating circular or processing component. Second thing is if API is not working tell me that. Can you manually check my API key? It might very well be that the API key is expired, right? So, we can just add a processing icon just so that we know something is happening in the background. Otherwise, we don't know whether the button is working or not. Second is um having having an API test. Okay, API seems to be valid. Problem is UI. So, let's fix that. Um one thing you should definitely do to make this app usable. So, we are actually planning to build an application where anyone, let's say you are a company, you want to build an email app, you can come to our website. We'll release a website where there you can simply log in using your Google account, and then with a button click, uh the past X number of emails will be fetched, and those fetched emails will be converted into an email knowledge base. From that, a writing preference will be created. And if you have any other base, that also you can manually add. So, we are creating an application, and you you should realize the value of this for most of the teams. Uh maybe big companies will not like something like this because privacy, what not. Very small companies or very individuals probably won't need it because uh number of emails they are replying to is anyway small. But for any company where number of people are between 5 to 500, something like this is very much useful. Uh it will save them tons of time. Uh okay, make sure to restart after face for changes. Click email. Let's see what has happened now. Okay, now I can see the change here. Generate draft. One thing good about tools which reply on chat like Slack or email or Discord is that latency is not much of an issue. Latency latency would be an issue if this was um like a voice call agent. Let's say you have a speech to text model, then text to text model, and then again text to speech model. It's like a cascade of models to create a speech to speech effect, right? There latency is such a big issue and there you should definitely not have um you know, you should have a much more efficient drag system. Here it took a few seconds, but um this is fine. So, question reaching out to regarding questions about drag systems in practice, courses designed to help, price is so much, starting date, duration, format. Here link is also provided. The only thing is formatting of this email is like markdown format. That's why I think these star and all are coming. So, this should be properly put as txt, but I'll just approve and send. I just want uh I still, can you just verify if you get this or not? I'm just approving it. Once you get this, if you can just share a screenshot of how it looks like on your end, that will be great. Okay, reply sent. And then the replied email will be automatically archived. I don't know if that functionality is implemented yet. Okay, Haraprasad also has a question. Let's SQL for analytics. Yeah, on your end it will be the Gmail rendering, of course. You you should be able to see on your app also, by the way. If this if the email through which you have sent me this question is the same as the one which you connected to your app, you can see it on your app also, but I just want to see on your actual Gmail how it looks like. Whether those star and those things are coming or not. Okay. Okay, star is coming, but the link is rendering properly. Okay, so the formatting needs to be better. And subject Oh, subject is getting printed or is it the subject Is it the actual subject? The subject line is the It's in the response, right? If I'm not wrong. Okay, so some weirdness is there in formatting. Very easy to fix, not an issue. But in principle, a lot of things are working right now in the app. Um Now, few more things are Okay, it's in the response. So, subject should not be there in the response. Um I'll just create Okay, let us continue with phase five phase five. Meanwhile, I will create the you know, the feedback for the current stage. So, continue with phase five. Meanwhile, let's create a feedback document. The email content should Um not show the subject. Oh my god, my Because my computer is slow, when I'm typing and when I'm moving, sometimes it's there is a mismatch uh because Claude is running in the background. There is a lot of slowness right now. Okay, email content should not show the subject. Next is There should not be MD formatting. We can either say HTML formatting or we can just say plain text formatting. Let's go with plain text formatting. Use plain text formatting. And what we need is in the signature Oh my god, I'm These are not my spelling mistakes, guys. My computer is so slow now. Mention my name. Yeah, I will push the current code. Let me just save this and push everything. Okay, anybody else who got at least the emails fetched in the front end? And if any of you, you know, if you got a got any email response crafted by AI, you can just show that also as a screenshot. Or just copy paste the text, anything is fine. I just want to see the format. This is the difference between Gemini and our custom email reply tool because here in the database we can pretty much add anything. Okay, API credit got exhausted. Uh Satiji, did you get that message because you your app set it up in the front end or how did you get to know? Did you get to know about it from the cloud code itself or did you see that in the front end? Uh when I asked a question about something else, this is what it figured out. Let me just download the screenshot. Uh Vishnu, if you don't mind, can you just copy the text of that email and then paste it? Because I'm it's taking my computer a bit of time to download and open the screenshot. Based on the AI replies for all the emails, you're providing creating global feedback that you're feeding to the cloud code. No, improvement is improvement of this app has nothing to do with cloud code in production. Because in production cloud code is not there in the picture, but improvement will come because ideally to the context from the super base these tables should be loaded. Let me just open it. Uh Okay, it seems like you might have reached out to someone else as this is Visual AI online learning. Okay, the reason why Vishnu it says Visual AI is probably because in the course database there is some Visual AI name mentioned, right? Otherwise, other than that nowhere else it is mentioned. Right? So, okay, that makes sense. Let me just refresh this. I just want to see if any reply has been stored here because at least one reply should be stored here right now. Okay, excellent. So, this is Let me just view this. Yeah, so this is the reply which I made to Achille and subject body received at Okay, this is not the actual reply. I need to see the actual reply which was done using AI. Yeah, so AI draft. So this is the AI draft, right? So it is directly stored here. Now one thing that is not yet implemented, I think, is editing the AI draft. So if I edit an AI draft, it should be stored the previous version and the current version should be stored. And feedback, hopefully, will be now implemented in phase five. Let's see if phase five is done. So phase five was If for any of you, if you have ever found email to be painful, you must definitely build this. You will love it. Um the current actual production level app that I have, I'm I cannot live without it these days. It's uh it's really really saving us time because sometimes I'm like, "Oh my god, this knows my mind." Because some emails are exactly like how I would have replied. Almost to the dot, almost. So then I'm like, "Okay, wow, this is truly replacing me. My need is not there." Um Okay, let's see phase five. So phase five is star rating plus text feedback UI and storage, email thread view, error handling and loading state, UI polish and responsive design. Phase six is deployment. Deployment is the last thing. We don't need to do that right now. Uh but I just want to see where have we implemented the um function for storing the edited email and original AI draft. I don't see that. Uh let's let's fix that also. In phase four, it should have been there, but I don't recollect seeing the edit option. >> Okay, and everything so far has been pushed to GitHub. So, you guys should be able to access it from GitHub. Phase five is done. Let's go back to the app. This is another issue with the UI. This is fully blank space, and to see the AI reply part, I have to scroll all the way down. That is not a good thing. Then I have to make two clicks. Here, I have to say draft AI reply, then I have to scroll all the way down, and then here again, I have to say generate AI draft. So, that also is not a good thing. I just want to see if the functionality for editing has been implemented. Okay. So, if I let's say I delete these three these five lines. And if I send this. So, what I expect Okay, and then this is good. How was the draft quality? Let's say three out of two out of five. Uh poorly formatted Then uh too verbose. Did not ask any No, this is wrong. If I say if I give wrong feedback just the same just for the sake of it, the reply might become worse over time. But yeah, submit feedback. So now hopefully this should be stored in Supabase and then it should be loaded into the context. So feedback is zero, but after I refresh it should be one now. Okay, so feedback is now one and star rating poorly formatted. Okay, this is good. But what is not good is I don't see the part where in Supabase I don't see a table for storing original AI draft and my edited final draft. So this should be there. And meanwhile, I need to check what is my model by the way. I have a feeling like this is not Opus 4.6 because of the bad UI. Well, let's see. Uh how do you see the feedback pushed into the context? Yeah, yeah, for that we need to have some sort of context visualization. We'll we'll have a look at that. Uh replies table exists and is empty, no replies yet. What do you mean no replies? Of course there are replies. Uh, replies. There are two replies. Okay. So, replies table exists and is empty. Look for replies table. It has columns ID, email ID, user ID, AI draft, send body. Okay. So, AI draft is also there. Send Send body is also there. Once you send a reply to that, both the original draft and your edited version will be stored there. Excellent. Let me just check for my model. Okay, it is Opus 4.6. Um, now what do we need? Okay, before we edit the UI, we need some sort of context visualizer. I need a page for visualizing the full context or not a page. In the same page where AI draft is generated, I want to see the full context that is passed in to the LLM for generating the reply in text format. So, if we visualize this, we can actually see whether things that are loaded into the context are relevant or not. You're adding this requirement in the terminal prompt. How will it be added to cloud.md? Um, so this this I'm not asking to be added as a This is just a feature which I'm currently building. I don't need to be I don't need this as a It's in It's in the I don't need this in the cloud.md file because uh, whatever is there in the cloud.md file, once it is implemented, right? Because this is an app. It's we are not building technically a coding agent or something like that. It's just an app with bunch of features. So, once the feature is built, even if that thing is not mentioned in the cloud.md, it is fine. However, it would not be fine if you want to create a master app which should act as a So, let's say you want to sell an application whose only job is to create email agents for anyone. So, I want to create an email agent. I will come to this master application and it should build something for me. In that case, you should have an MD file which details all these things. If that is not the case, if you want to do it just once, the moment the feature is built, probably you it's okay if things are not directly written into cloud.md file. The best practice is to create diff newer version of cloud.md file and make sure that uh in the system prompt, it is always there. If if building the application is the activity that you're going to do most frequently, in this case, once the build is done, I'm just going to use the app. So, my focus is purely on what is the going to be there in Supabase and for each instance of AI reply generation, what is going to be loaded into the LLM's context. In this case, the LLM is OpenAI, so that is what my concern is. But when once I build a master application where any customer can buy our app and build their own email agent with just front end some mouse clicks, then I will be concerned about this. Uh Yeah, so feedback you guys saw, right? I hope you also should be able to see a feedback once if phase four or five is implemented, you should be able to see a feedback. In the replies, I just want to see the two versions, original AI draft and the draft which I sent. Okay, so email ID, user ID. So, this is the AI draft. And this is the sent body. Okay, sent body, by the way, it should not have subject, so that is a mistake, but I can see the AI draft and the sent email. So, this is the air draft with subject. This is the sent body with subject. So, here I had deleted three lines. So, those deleted lines will not come in here. So, this is good. Uh have you guys checked your Superbase? Do you see uh four or five tables depending on how you have built it? Do you see the tables with the um you know, with the field entries or is it still empty tables? Just feel free to let me know. And one thing you might notice is the slowness of the app uh to fetch API, it's very slow. Um you will notice that sometimes everything is like bunch of emails are fetched in bulk. You probably don't want to do that. You have to install uh implement pagination so that in one page only so many emails are being fetched. Uh so, there are bunch of extra things which you can do to make the app super fast. Uh one thing I have noticed is that my archive function now is faster than in than Gmail. Uh that is one thing which I'm I really like in my app. Uh but my email fetching is obviously slower than Gmail. Um my once the reply is sent, this speed with which it is auto archive, that is a bit slow. So, auto archive is slow but manual archive using E, that is very fast. So, in some cases, I like my app better. In some cases, Gmail is better. Then later, I had to implement a few more features like right now forwarded emails are not there. If somebody had already forwarded an email to um uh someone else, I don't have to necessarily reply to it directly, right? So, but where do I see whether this email is already forwarded or not? In Gmail, you can see that, but in this app, unless you explicitly build it, that feature will not be there. How the feedback will be used to improve reply? So, feedback will be used to improve reply in the case where imagine the case where you have lot of details about different courses that with nuances. Okay? And let's say AI hallucinated and said something. And you don't want hallucinate in that direction. To give an example, let's say we have sometimes two URLs for the same course. One URL on our um subdomain and one URL on the main domain. Sometimes we prefer it to send the sub subdomain URL because there a lot more details are there. The main domain URL do not have that many details. But I have noticed that many times AI keeps sending the uh the main domain URL instead of the subdomain URL. Then what I did is I manually deleted in some email reply, I just manually deleted the subdomain main domain URL and replace it with subdomain URL. Then AI after sometimes it it noticed that change between the original AI draft and the draft which I sent, which was added to the context and then it it by default ended up sending now um now it keeps sending the proper URLs. So, feedback will be used to improve the reply because these feedbacks will be loaded into the um uh context along with the past let's say 30 or 40 email response email plus uh AI AI email draft plus final email draft pairs. So, just remember this. This table is right now not an issue. If I load this entire table, it's not an issue. Only two items are there. But what if uh I have 300 such rows now in this table? Then I have to again apply another rag on top of this feedback. So, this feedback look at this. It's only one row in this table, not an issue if I load it into context fully. But if this is 500 rows, then again I need to add one more layer of rag on top of the feedback where each entry in the feedback is converted to a vector. And if from feedback, I need to fetch relevant vectors corresponding to my current email, I should be able to fetch that. If from um course embeddings, I need to fetch some relevant courses, I should have a rag. So, here I already have rag. Here, I don't have rag. Even in my original app, I don't have rag yet for feedback. Because my feedback number is very low. But eventually, if I feel like my context is becoming worse because lot of feedback is being fetched, I will implement rag. Um but yeah, let's see what is the status now. Okay. So, this is the This is the latest version. Uh the change that I wanted to implement was I forgot what was my latest command. It was I think regarding Okay, it was regarding uh displaying the context. So, let me now see if I open one whether the context will be displayed or not. Okay, so now this is the full LLM context system prompt you are a helpful email yeah, you see Let's see now what all things are loaded here when I'm replying to uh sign Elima. So system from your a helpful email assistant for Visual and online education platform you're you draft professional friendly and accurate email replies to course inquiries guidelines be warm and these are generally guidelines. I never said these things. Sign off as team Visual do not make up information. These are these are good things. Then user prompt the following email was received draft a professional reply incoming email subject is the subject body hi Sridhar could you please look at this course and provide details. Then relevant course information from the knowledge base course one statistics for decision making advanced. What was the course that that yeah, this is the course that was asked about statistics for decision making. So the course number one which was first fetched by rag is statistics for decision making advanced. Course number two is maybe the not advanced version of the course. Course three is metrics for product decisions. I don't know why this course is also fetched into the context. Uh maybe product decisions yeah, because here it is about decision it's about decision making maybe because there are things about product decision here metrics for product decision this is also relevant. Course four business metrics and dashboards advanced. And course five data storytelling for leaders. So five courses have been fetched uh from the context to draft the reply. But you know what is not added to the context as I can see the feedback is not added to the context yet. Uh and what else is not added? The previous email uh email response pairs, that is not yet added to the context. Right? So, whatever feedback we have added is not improving the app at the moment. So, that also See, this is the advantage of showing the full LLM context. So, now this whole thing is passed into the uh And also, I would like to know the number of tokens. So, let me just copy this. token counter chat GPT So, this is 920 tokens. Uh so, it's not too bad. The context window of these LLMs are much bigger than this. So, con- context uh what is definitely not happening, but what may be happening is we are not providing feedback or we're not providing previous mistakes AI made in the response, none of that to the to this context. So, maybe we can add that. So, this is a good way you can actually, you know, implement a token number of tokens consumed visualizer here. Um so that you can see that you're not exceeding the context. But, if this number was 20,000 or something instead of 1,000, it's a problem. Right now, it's not a problem. So, let's implement one more thing. I noticed that feedback table is not loaded into the context. Similarly, the replies table is also not loaded to the context. What is the point of having them if we are not using it to draft a new reply? We need a systematic way to load that into the context. Can you guys also implement a context visualizer? Just txt format, no need of fancy context visualization, but just I just I'm just curious to know how long is your context in terms of token. Mine is like roughly 1,000 tokens. Is yours also something like that or is it more like half the number of tokens or double the number of tokens? Just try to build that. But rag is working beautifully here. The order in which the courses are arranged are definitely according to what the user query is. No, in this rag I don't think we have implemented any re-ranker. Actually, re-ranker is not that important because, you know, if you are just fetching five items and if you are anyway passing all of that into the context, it's fine. But if you are fetching 25 items and then you want to apply re-ranking and then you want to fetch five items from those 25, then re-ranking would be very much helpful. In this case, I don't need a re-ranker. Re-ranker is not a mandatory part of rag. Let me just check your screenshot, Amit. I have to download it first. Okay, this is Amit's response. Let's see. Uh air draft. Okay, so one thing I notice is in all the cases air draft is with the formatting with the star formatting. We can we can kind of ask AI I like your all of your UI actually. It's much better than what UI I have. Um but yeah, the the response is kind of good. Thank you for reaching out. However, we offer a variety courses. Yeah, this is good. Courses reference. Also, this is a good addition, Amit. This courses referenced being displayed here. Did you explicitly ask AI to add this something like this? Or was it added by default? Okay, you explicitly mentioned it, is it? Okay, good to know. Yeah, yeah, because this this is good to know because um uh I mean, you can see that in the reply, but this is also good to know what rag is actually fetching. All right. If you guys can implement this context visualization in txt format, that will be great. We can actually see whether the context is relevant or not. Meanwhile, I think now we can just go ahead and just deploy this just for the sake of completion. And of course there are hundreds of things which we can make better here but let's let's once this is done let's go ahead and deploy in front end because that will uh finish the loop. One thing we should definitely check is um we should have additional functions uh deployed if we need more functionality like uh so I have added one functionality where if a new program is released uh if I don't want auto scraping or something I will uh simply provide a link or URL to that program and then from the URL manually uh you know whenever I provide the URL uh fetching scraping data will happen. That scraped data will be converted into a format uh which is uh suitable for my uh Superbase database and then it will be added to Superbase. So here I'm in the for that I needed a uh Railway back end also but in this particular application we don't need Railway. It's at fire uh Vercel itself is handling that when we deploy. So let's first finish this and then let's deploy. Uh now deploy the application on Vercel. Uh how to calculate delta in such cases? Uh delta meaning Mayank? Changes like price updated or timings like that in drag knowledge base. Yeah yeah yeah very good very good question. Uh so I have added two functionalities. One functionality is to add new information and other is to visualize the database uh the rows in the table like cells cells as in if if there is one one row it will be displayed display display with an edit option. So in the edit option I can directly make an edit. So I have added that functionality also. So yes, there could be changes like changes in timings or changes in some information about the course about duration something like that for in the existing knowledge base. So I need an edit option and each time if you imagine if you have to edit you have to go to super base manually and edit that it's painful. So you should be able to make that edit from the front end itself. And ideally with natural language. I mean just just speak and make those changes. That'll that'll be the even more um ideal situation. What is the app if I may ask? On what app for your work or for your college on what app do you spend most of your time in terms of communication? Is it Slack? Is it Discord? Is it Gmail? What is that is the most painful thing for you? Can you just mention in the chat? I'm just curious to know. Okay, Teams. WhatsApp do you guys use for what company communicate like for work or for like personal stuff? Okay. So for those of you who mentioned that the number of emails you send per day is like one to five. Is that number more like 20 30-ish when it comes to this chat like Teams or Slack or Google Chat or whichever platforms you use. What is the order of magnitude of per day communication? The number of messages you have to read and number of messages you have to reply on a daily basis. Oh wow, greater than 50. Okay, per day it's that much, right? So it's it's truly a pain then for all of you. I think in that case will benefit from having a having some sort of agent which can handle at least basic things. I mean, of course depends on company policy and stuff like that, but um but it will help you with having some agent who can who has same knowledge as you to handle at least 80% of these conversations. Uh in my case, most of my spend time spent in email. Um that's the reason why I went with email. But yeah, of course this can set up in uh uh can we set up Teams co-pilot for this? Not sure. Uh Teams should have similar to Gmail, Teams should have a provision to have API access, right? Uh how do you have this conversation or chat of Teams in vector database? Uh like how to convert you mean uh to vector database? Yeah. Yeah, great question. So, in email, what I did was I showed the I showed the JSON files earlier, right? Last 1 year worth of email alone itself was around 350 MB or something like that. That too emails were there is some reply in the thread. The sheer volume of raw emails will be much bigger than that. So, what I did was I first got all the raw data from the email and I realized that it's a waste to store all of this in vector database. So, then I ran another loop uh where I spent some money on LLM where I asked LLM to significantly summarize all of this into one or two JSON files for knowledge base. And then those JSON files uh this the individual uh objects in those JSON I converted to vector. So, now the number of rows in my vector database is like 250 or something, not more than that. So, it should not be like each chat is converted to a vector. That will be that will be too much. Um I think the vector database in if it is like a team chat or something should be something like a semantic uh the chunking should be semantic based. Uh in my case, the chunking was made more based on structure as you might saw as you might have seen earlier. It was based on the structure of the email. Each email was one chunk. But if it is like if it had so happens that each reply or each item becomes a chunk, then some item might be high. How are you? This this So it's all some useless chunks, right? So your chunk should chunking should be different. Uh how did you export emails from Gmail into multiple JSONs? Great question. So Mosam, for that what I did was I had written a separate function uh using I mean using cloud code, but it's a Python script that Python script executed and it took me around 45 minutes to have that much volume of email. In this case, each email is a vector. Yeah, that's correct. No, here I did not make any change. Each email is a vector. Um I mean sorry sorry, not each email. Each course is a vector. Here I'm not storing email as vectors. But in my original application, each email is a vector. Each course is a vector. And then both of those vector databases are separately referred to. In this particular application which we are building right now, there is no email vector. We only vectorize the knowledge base for the courses. But uh if you were to build something like this for Slack or Teams or something, the chunking will completely matter because uh if you chunk such that each message is a vector, it'll be like a meaningful vector meaningless vector. But if you chunk based on semantics, maybe group of ideas may belong to one vector. And then whenever somebody's asking a question regarding that idea, you can find the vector that is more more relevant to that. But in my experience, for those of you are spending maybe 1 and 1/2 to 2 hours, because I assume to reply to 50 plus messages on a day, it'll take at least 1 hour of your time if not more. Uh if not uh 1 and 1/2 to 2 hours, right? So if that is the case, I think it's totally worth it to um build something like this. And maybe you can discuss in your team itself like uh whether they they would appreciate something like this. Anyways, now the app is deployed. Let's see production URL is this. Uh Okay, I don't need the authentication. I don't remove the need for Vercel authentication for this app. But meanwhile, one more thing that we should do, which you should not forget, is this redirect URLs. So give me all redirect URLs in a separate document so that I can easily copy. So if we deploy on Vercel, obviously, you will have to go to again Cloud Console. And here, you will have to add that URL. Um so whatever your app URL is, you will have to add that. Plus, you'll have to publish it if this needs to be publicly available app. But otherwise, no need to have that. Um so go to clients. Web client one. So here, in this place, we should add a URL for production. >> Uh and to connect Vercel, okay, you can just ask uh Cloud Code to login via mm Vercel OAuth. So, that should be very easy. Then, it will ask you to log in to Vercel. Uh authorize redirect URL Superbase authentication. Okay, so once this redirect URL is set, go back to Superbase. If this is how your structure is, in my case, I had enabled Superbase Google authentication. That's the reason why I'm again going here. >> Okay, so that is also set and So now hopefully this should work uh in deployment. Okay, Vercel authentication so this we should on or off. Deployment protection you can do that. So usually I don't go to Vercel these days directly. I force Cloud Code to do it itself. I can just go here and just Vercel authentication I can switch off but Cloud Code can do it very easily. >> Uh you guys are you guys in the process of deployment? Just let me know once you deploy. Whether it's working or what's the status. Uh so here one thing you should note is here I did not connect GitHub with Vercel, right? Uh the best workflow is doing that. Connect GitHub and Vercel and then whenever you push changes to GitHub, it will be auto deployed on Vercel. Here I'm directly deploying to Vercel because my cloud code has access to Vercel, right? Um so just just understand the nuances here. Okay. Yeah, we can deploy to Vercel directly from GitHub repo. So you can go to Vercel and connect it to any GitHub repo. And then once it is connected, whenever you push changes to GitHub, it will be auto deployed to Vercel. So that is for most of the projects, that's my workflow. Okay, so here my deployed app is working very nicely. Uh it will only work for me because it's configured with only me as the user. Uh then if I can just reply draft AI reply >> And I can see the full LLM context here. I don't think the issue of fetching things from the feedback is yet uh user edit patterns. Okay, now it is fixed. Past reply history and feedback user feedback on I hope you can see this. I'll just increase the font size. So, issue one two out two out of five as was a star rating request to send. So, this was This is one of the feedback. Only one feedback is there, but then user reply patterns user edit patterns the user modified these AI drafts before sending learn from their preferred style. Then I had it I had removed something from the draft user sent instead subject course detail messages. Oh, somehow I'm not sure if something is getting truncated here. Draft reply to this email if the email is asking about Okay, maybe what might be happening is if this is the full LLM context either what is displayed here is truncated or what is stored in Supabase is truncated. Let me just go to Supabase and check if something is truncated or not in the replies. So, AI draft and sent body. Okay, here nothing is truncated. So, in the context when some but something is being fetched from the uh user edit patterns, things are truncated. So, this we can fix. Uh but I hope you get the general direction. So, basically what we implemented here is we have rag just for the courses. Later we can have rag for if there are large number of emails uh from our past uh to be referenced to. If there are large number of feedbacks to be referenced to, we can have a rag for that also. So, in general the bigger the database becomes, the better to store it in vector format. At least one column should be vector and that column should be referred to for rag. Um Uh I'm just curious to see what is the status of your application right now. Um has anyone managed to deploy it yet? Uh how did you have these conversation or chat of teams in vector data not that where is that? Yeah, we can deploy to Yeah, I think most of the questions um are covered. But I I'm just curious to see did you guys manage to deploy the application? Or is there any hiccup? Uh one assignment that I will give you based on this is I will create a a different knowledge base, okay? And I'll share it with all of you. Um and then what I want you to do is use the final version of I mean make sure that your app is picture perfect and polished. The knowledge base I will make in such a way that it's very nuanced. Uh like the app cannot make general assumptions and respond. I'll make it very very nuanced. So unless the rag is working perfectly, the reply will not be good. Then what I will do is I will set up an agent myself so that if you provide me the email, I will make my agent send one question based on the knowledge base to your email. And then your email from your email, you should use your agent to reply. And once I receive the reply, I'll look at the reply to see how good it is. So uh yes, push the knowledge base. So this knowledge base is already in the GitHub. You should see that uh the CSV file. But the latest changes push to GitHub. So the way I can truly evaluate you will be based on that. Also, you might want to check out our platform called hire.visualera.ai. Just check this out. So, the reason why we built this is because nowadays when we are hiring someone, it doesn't matter what their coding skill is. We really need to know if they can truly work with AI tools to build, right? So, here there are a bunch of assignments. The free trial is for few number of days, so you can try with a free trial. Uh and you can create an assessment for testing skills for a particular domain. It can be rag or something else. So, just um just try this platform once. Uh it's a it's a platform which now some companies have started using after purchasing it from us to test when they are making a hire. So, here we have built an agent who will evaluate how good you are at prompting and building using AI. So, there is cloud code inbuilt into a into Amazon AWS. And then this platform will give you temporary access to that as a candidate or as a hiring manager. And you can use it. So, just try this out. Basically, your AI skills will be truly put to test. The new knowledge base I will be Yes, I will be pushing that to GitHub. I don't have it yet because I want to make it a bit more complex. It should not be very simple. So, I will do that. And one more update is the PDF which I said I will share with the engineer plan and industry professional plan guys. I will finish compiling it sometime end of the day tomorrow or Monday. Then I will be putting it on the course dashboard and I will Once it is done, I will notify all of you. Uh but please I will create one assignment based on today's building exercise. The assignment will be very simple. You need to local host or show the deployed version of your email tool. Okay? I'm sure you have two emails. So, from your second email send an email to your email where the tool is. And then show me the this record using loom.com one three or four minute video showing the reply and and reading through your contacts like display your full contacts in DST format. And then just go show that. But you will have to spend a little bit more time to perfect the UI because right now I'm sure if you got the UI in one shot it is not good. So just make sure that you are spending few more hours to make the UI better. I will give you another three or four days of time for that. So you can work during this weekend as well as few more days during the week. So assignment deadline will be around March 25th-ish. So you can submit that and then the dates for build session with Dr. Rajan Rajit is finalized. I have updated the details on the website. You can use the same link to join that also. Um Yeah, yeah. So assignment I will I will post it. I will post it on the platform. So we can wind up this session today guys here. I'm I'm hoping to see some good submissions. So whatever you have built so far, I know that it will probably be incomplete because you will need more time to build. But I want you to build it and as part of the assignment I want you to submit it. Um so please take good care in making sure that the appearance is really nice. The the format of reply is actually like an email. It should not sound like an AI generated reply. So you have to take care of adding sufficient email preferences. If you need to add that in Superbase, you can do that. If you need to add more in Superbase, you can do that. For email tool, latency is not at all a not at all an issue. The only issue is the reply quality. So even if the time consumption is a bit high, it's okay. The other issue is to minimize your LLM cost. You should minimize your length of token. I mean the context window size. Whatever number of tokens you are sending. But then at the same time it should not degrade your quality. Right? So um So two assignments maybe one short assignment will be slightly making the app better and local hosting and showing it through a video format. Second assignment is um just uh I will submit a new I will give you a new knowledge base based on that uh redesign your application. That knowledge base will be slightly more difficult and then just submit your email. All of this will be mentioned in the assignment. Uh submit your email, then I will set up an agent to send emails to your inbox and then using your agent reply to that inbox and I will be evaluating it based on the reply which my agent receives. So, um uh thank you so much everyone. I hope you know, you guys have built and something something is there in shape at the moment. If it is not if in these 3 hours nothing happened, don't worry. You can um um work with Cloud Code a little bit more and in some more attempts you can get it to working. But, the final output should be a deployed product, okay? A deployed thing where there is proper authentication so that only you are able to get access. Uh unless you reach that stage you cannot say it's a final product, okay? In the next two build sessions, the topic I'm not 100% sure. I can ask uh Rajendra Jha and finalize it with you guys, but at the end of the day you will be building um something proper using Cloud Code. So, something very similar structure as that of today, but uh different uh for different purpose. So, thank you so much everyone. Um Thanks for staying so late. Um and uh I hope you really enjoyed the course so far and uh sessions. I will be sub sending you across a feedback form. I hope you can give some nice feedback of what you enjoyed about the course and stuff like that. It has taken me a lot of literal sleepless nights to build this course and even to go through the course because the content it is a little bit bit vast and to compress it into a format which fits few hours, it's very very difficult. So, that part was the most challenging to decide what to include and what not to include. Um one more thing that was challenging was to uh create the the lecture notes. The lecture notes on Miro took a lot of iterations to get into that shape. Uh I use Paper Banana for many uh visualizations, but even the prompt which you give to Paper Banana and required certain level of detail. So, that that's the part which took a lot of time. And uh you might have noticed that now um Cloud has partnered I mean they are partnering with companies to give uh Anthropic builder certificate or stuff like that to individuals. So, if something like that we start, we will let you know. Um yeah, in the upcoming projects with Dr. Rajendra Rajat, it will be similar, but uh mostly we are agents or sub-agents will be utilized. Um I will I will ask them. I don't know what have they finalized yet. But, I'll really really appreciate it you guys if you can give me some feedback. I'll send a feedback form link. Um Um first of all, thank you all for staying so late. I know that many of you from US time zones, it's like 2:00 a.m., 1:30 a.m., 3:00 a.m. It's like weird times. Uh yeah, so I I really really appreciate that. Um I know that the timing is not the most ideal for everyone, but because we have people from almost all time zones, it's very very difficult to get it correct. But, all the best everyone. I'll hope you to see in upcoming boot camps or workshops. Um I have one session coming which is starting next weekend on vision language action for autonomous driving. So, there I'll be teaching about uh how to build VLA models completely from scratch with general general in general for robotics, but specifically for autonomous driving. If anyone is interested in that space, you can check it out. It's basically on this link. vla.visual-ai. So, just uh check out this link. Uh, if this is something which interests you, uh, because VLA, you might have noticed in 2026, a lot of companies are getting funded for robotics. And, um, I also strongly believe that embodied AI is the future of, uh, lot of things that AI has to show because so far AI has seen amazing things it can do in the digital world, but in the physical world still, uh, there are a lot of tasks, which basic tasks which human can do, but AI cannot do. So cognitive tasks AI can already do a lot of things which humans can do, but physical tasks which simple tasks which humans can do, AI cannot do yet. So, uh, you will learn that from, uh, you know, from that boot camp, some parts of it, but the rest of the parts Rajit is also covering as part of VLA and world models for robotics. And we have something called minor in robotics. I don't know how many of you are aware of it. Uh, I will just share a link. So this is mostly for those of you who think, uh, embodied AI is one of the big things which future AI can offer. Uh, but thank you so much everyone. I will let you guys go and sleep or have lunch. I'm also very hungry. Uh, see you in the Now you will see Dr. Raj in the next, uh, ninth session of this boot camp, and in the 10th session you will see Dr. Rajit. So thank you so much. For any doubts, just email me. This is my email. Uh, sometimes I'll be slow to respond, obviously because of the sheer volume of emails, but I'll try my best. Uh, so all the best everyone. See you soon in any other future boot camps. for a very long time. So what we are going to do is that we are going to build an end-to-end project. So let me just write it down. There are still some people entering the waiting room. So, I'll just admit them. So, what we are going to do in today's lecture is we are going to And instead of calling it a project, let me call it a product. We are going to build a product. We are going to build a product in this class using the practices of modern software development. Using the practices of modern software develop modern software engineering and also context engineering. What this essentially means is that earlier 5 years back or maybe even 10 years back if I were to conduct such a lecture, in one lecture I would probably cover only uh 1/10 of what I'm planning to cover today. But, now we have coding agents, right? We have Claude code at our disposal and not just that it has tremendously accelerated the speed of software development. So, today I have taken a problem statement which is can actually be converted to a product. So, many of you in this call, if you follow along very well, by the end of this lecture you will actually have a working product which you can uh improve upon it a bit further. You can even sell it to um you can even sell it to potential customers. Okay, so this is going to be a very industrial grade lecture. And in this lecture, I'm also going to talk a lot about how I usually interact with Claude code or coding agents to build a product. It's not a random practice. It's a very systematic workflow. So, I'm going to show all of that to you on a live project. Which means that I'm also going to build this project today with you um during this live lecture. So, some requirements for you to follow along is that all of you need to attend on a laptop and not on a mobile phone. And secondly, you need to have uh, Cloud Code installed and connected to VS Code. How many of you have Cloud Code installed and connected to VS Code? Could you mention in the chat if Cloud Code is installed and you have it connected to the to VS Code? Good. You can use CodeX also, that is not an issue. You have Max subscription. Okay, that's amazing. So, today as you will see, many people are thinking, right, that what's the role of software developers? What's What's even the use of a computer science degree if everything can be done with coding agents? Today, hopefully, I can demonstrate to you that humans are also important in this loop. And uh, that's the first thing and second, the speed of building products can accelerate tremendously. So, in this workshop, my goal is to convince you of two things. First is that the speed of product development has become now 10x. The second thing is there is a systematic way you need to deal with coding agents. And uh, this field is actually called agentic engineering. So, I'm going to also teach you some practices of agentic engineering along the way. And third thing, which is the most important, is that it's a four-letter word, which is called care. So, I personally think that in this age of coding agents, what distinguishes good developers from poor developers is how much care they uh, put into developing their product. And today, I'm going to quantify what care actually means. Uh, there are people who blindly give prompts to Claude code and just wait for it to give some output. That's not care. As we'll see today, care is actually thinking from scratch, thinking whether the product will actually benefit your customers or consumers. So, in today's class, I want all of you to think like startup founders or builders of the future. So, don't think about building toy products in today's class. This is not that class in which we'll build a toy project and add it to the resume. In this class, I actually want you to build something which you can sell later. When I say I want you to build something which you can sell later, I mean we want to build something which actually creates value and you can tap into that value by earning revenue. That's the difference between the modern software engineering and traditional software engineering, right? Earlier, I would have taught you a toy project in two year two in two hours or three hours, but now my goal is to build a product with you which you can sell and you can earn revenue. And for that, care is very important. If you don't put care into interacting with your agents, your product will never be used by consumers. Okay, so let's get started with today's lecture. First of all, can everyone tell me if they have Claude code and VS code and they're ready to get started with today's lecture? You can put yes in the chat if all of us are on the same page. Okay, let's get started. So, I'll first introduce the problem statement to all of you and this is this is the first thing which is very important when you're building a product, right? You need to have a You need to think of a problem which actually solves some pain point. So, now I'm thinking from perspective of a product developer or a startup founder, right? So, the first So, we are looking at the CARE workflow now. How to put CARE into building products, right? So, the first aspect of this CARE workflow is identify a problem. Identify a problem which people care about. Or identify a problem which is actually a problem. If you build something random, no one is going to purchase it, right? So, I have stumbled upon a problem which I think and all of you can confirm or deny. But, based on the students which we have been educating, there is one problem which I have identified and that is let's say if you're giving an interview, right? If you're giving a machine learning interview or an AI interview, uh there are many students who want to prepare for this interview. There are many students who want to prepare for this interview before the actual interview. Like, I want to build an actual uh I want to build a system. I want to build a system which facilitates a mock interview, a mock ML interview for a student. And when I say facilitate, I don't mean some random list of questions. I want to build an actual um I want to build an actual system in which which interacts with the student like a human. So, my system interacts with the student like a human. So, if you were to give a name to this project or product what we are going to build today is an AI-powered mock interview agent from scratch. And this is not going to be something which is um like a toy project, right? I actually want people to pay for this tool. Why will people pay for this tool if it adds value to their lives? And why would it add value to their lives? What will it do for them? It will conduct an interview for them. And it will conduct an interview as if a human is talking. So, one very important aspect of interviews is also the stress, right? There are many people who can answer questions, but when a human is sitting in front of them, there is anxiety and stress involved. So, I also want to have a voice platform in which my AI interacts like a real human. So, this can actually simulate the conditions of uh this can simulate the conditions of anxiety. So, these non-tangible aspects are also very important in an interview process as all of us might know, right? What we practice in our house is not the same as when someone is sitting in front of you. So, now this is possible to do. Why is it possible to do? Because we have uh um text-to-speech converters which almost behave like real humans. So, what I want to do is that I want this system which simulates a real human and which interacts with the student like a real human and gives questions to them. So, at the end of this at the end of this journey, a report will be generated. A report will be generated which grades the student on interview readiness. So, what we are going to do today is that we are going to use a modern software development workflow modern software development workflow to build this production grade AI application. So, before we actually get started with the building process let me see if there are any some doubts or questions. Yeah, recording will be available for sure. Now, many of you who are in this call first you can tell me that whether you understood this problem statement or not. Many of you are students, right? Can you think of whether this is a pain point for you? Is there any system right now? Is there any product? There might be competitors for sure, but is there any other competitor which gives you such kind of an experience? I want to speak to a human and simulate for me. I want to get a sense that whether I'm actually ready or not. And many of the students after they take our courses, they ask for interviews, right? So, we don't currently prepare students for interviews. So, then I thought about uh building this tool from scratch. And earlier if I were to think about building this, it was difficult because I would have needed to hire software developers. But now it's not difficult because as I'll show you, we can build this entire product using uh um Cloud Code. Okay, there is a question in please copy paste the problem statement. Yeah. So, I'm just putting the problem statement in the chat. Later, all of us will interact with the coding agent ourselves. So, uh no need to worry about you losing some information. I'm also going to share this after this lecture is over. So, first of all, can everyone tell me, have you resonated with this problem statement? Have you understood what we are trying to build here? We're trying to build a human interviewer human interviewer to simulate uh an interview situation for a student. So, this is the first thing which you need to do when you want to build a product. And that is CARE workflow one. Instead of building something random, you need to get a user feedback on whether something which you build will be actually useful or not. And then comes the CARE workflow two, which I I believe is the most important and which separates good agentic engineers from poor agentic engineers, is um we need to do independent thinking. So, CARE workflow one is actually problem statement, and CARE workflow two is independent thinking. So, what I want to illustrate in this point, which is very important, uh please pay very careful attention here. This same problem statement, if I give to 10 people, and in fact, this is how we conduct interviews in our company now. This same problem statement, if I give to 10 people, right? And all of them are using Claude code. So, it's not as if one person has access to a different tool than the other person, but what I've seen is that the output which different people produce is usually different from each other. So, there is a spectrum of the output being produced. Um in fact, if you look at the quality of output being produced, there are only a few people who produce a tool which is incredible. But a huge number of people produce something which is quite mediocre. And the reason for this mediocre output is after this problem statement, what students tend to do is that they they just go to Claude code and put this. So, I can go to Claude code and say that build this for me. So, as you can see, I'm going to Claude code right now and saying build this for me. And Claude code will start building this for you. So, it's not as if Claude code is going to tell you that this is bad practice. See, Claude code or any coding agent tells you that this is a great project. Let me kick this off, etc. But please remember that this is this is very bad practice. This is very very bad practice because here we do not go to Care Overflow number two. And this is independent thinking. I strongly think that as humans, we need to spend at least 30 minutes to 1 hour thinking about what we want this project to be. Thinking about what we want the final product to be. And I'm going to now illustrate my thinking process. How I do this independent thinking phase for all the projects. First, I actually visualize what the input is going to look like for the uh student, right? For for the consumer. So, what I plan to do initially is that I plan to have a system where students first upload their resume. Because I need to know some information about the student, right? Students upload their resume. Let's say the resume has different sections. Once the students upload their resume, my I need to build a tool which understand this resume. So now imagine an actual interview. In an actual interview the interviewer already has your resume. Then the interviewer looks at your resume and ask questions, right? So after the resume, then the Q&A voice session will begin. And in many things here uh things get very interesting, right? Because if you want to put in care, you will think of this voice session in a lot of detail. So let's try to think about it, right? Let's say this is a resume of a particular student. You can use your own resume, by the way. Uh the sections in a resume look very similar to each other. Um as as you can see. So here this is I would say a resume whose sections will be very similar to other resumes, right? There will be an education section, there will be an experience section, then there will be a project section, then there will be a publications if there are research publications, then some people might have extracurricular activities, etc., right? So if you if you actually Let me take a screenshot of this a resume. So if you bring this resume here, now let's say this is a sample resume. Now let's think from a human perspective, right? If you are the interviewer, what are the questions which you will ask to the student first? Without even going to cloud code, what what questions will you ask to the to the student? Let's say the student enters the interview room. So this is my interview room. So the student enters and my interviewer is uh is sitting like an actual interview, right? And you have this uh the interviewer has knowledge of this PDF. So, what are the questions which you will ask? Think from a very realistic situation. Yeah? So, almost the first So, let's divide it into phases, right? Phase one is background. Or tell me about yourself. Okay? This is fairly fairly standard. And let's say the interviewer says, "Please tell me about yourself." Then I say uh I am from this background. I have completed my college year, etc. etc. etc. Then what does the interviewer ask next? After phase one, what comes next? Now, here is where humans have to do independent thinking. If you want your interview agent to behave like an actual human, you first need to think about what does an actual human do. Projects, right? Exactly. The next phase phase two is actually projects. So, what an interviewer might do is look at the projects which the student has done. Let's say pick any project here. Let's say this project, this project, or this project. And then ask questions about Could you tell me Could you tell me more about this project? Could you tell me more about this project? And then the student starts answering. Then what happens? As the student starts answering that I did this, I did this, etc. Then what happens? Does the interviewer immediately stop and move to another project? So, this is the independent thinking and this is not an easy part at all. This will eventually distinguish your project from thousands of other projects because you are putting in care, thinking about all this, right? They drill down, right? Exactly. They drill down. So, no one stops here actually. Even as an interviewer, my goal is to use the project as a proxy. I will use the project as a proxy to actually check the student's understanding. How can I use the student project as a proxy? Because when the students explain about the project, I will immediately catch on to something which I want them to explore more. So, this I'm calling it as a drill down phase. So, I ask them a question. I ask them a follow-up question and I kind of take it to the point. So, it's like a It's like I call this the Russian doll experiment, right? So, I don't know how many of you know about the this. So, the reason I'm calling it this is that I actually try to see how many layers deep can they answer. So, if they tell me about the if they tell me about the project, I ask them to I ask them one deeper question. Then I go deeper. Then I go further deep and then the final layer could be explaining something to me on the whiteboard completely from scratch. So, I try to drill down to the point where they don't know the answer. And using this I kind of judge where does the student stop. There are some students who immediately cannot even go one layer deep. There are students who go one layer deep. There are students who go two layers deep. Where are the Whereas there are students who go till the last layer. These are kind of the best students who can think independently on their own till the last layer is done. So, I use the project as a proxy to do this Russian doll experiment where I where I'll drill down until uh until they say that I don't know, which is which is perfectly fine. I just want to know till how many layers they can go. Um Okay, and these questions can be technical questions. What technology you used? Why did you use this? What if you use something else? Uh let's say they used a small language model. Why did you use a small language model? Then they say something. Then I say, "Can you build a small language model from scratch?" So, I go down. I go down. I go down and I grill them until the point where they say, "I don't know." Which is fine. I just want to see I just want to see how deep they can go. So, I do this. So, let's say one project is done and the person says, "I don't know." What happens next in phase three? Uh hints. Yeah, so let's say that's a very good point. So, let's say here itself I give hints to the student. I give hints to the student and I say that uh Okay, this is the layer at which you have stopped. Let me give you a hint and push you further. So, I give a hint to the student and see how far how far they can go if supported. Because in an actual job, this same thing might happen, right? They don't know something. They come to me and I give them some hint, but then I want to see how proactive they are. Even after giving hints, actually some people some people use it immediately as a click and uh answer and take it till the end, but for some people it's still hard, which is fine. Now, after this is done, what what happens in phase three? Actually, this may be subjective, but I'm going to go ahead and say that in phase three I might drill them on another project. So, let's say in phase number two I I drill them on one project, which is uh project one, and I do this Russian doll experiment where I peel the layers. And then in phase three I'm going to go to another project. Let's say in phase one I I use this project. In phase two I'm going to go with some different project, and I'm going to drill down on the concepts. So, once the projects are done, what what will I do in phase four? There is still one aspect which is missing. And this is covered in almost all the interviews. After the projects, Yeah, I think Nagashekar has answered this particularly well because there is one thing which is missing, and that is actually factual questions, right? Like actual factual questions, which may be not related to a project, but like what is the softmax algorithm? Or let's say what is flash attention? Or when we go from the attention scores, uh to the attention weights, why do we divide by the square root of the keys dimension? Or if it's a system design interview, as someone has pointed out in the chat, uh you will ask some system design questions. So, these are questions which are factual questions, which have one correct answer. And here I want to know whether they will be able to answer these technical questions or not. And then finally once phase phase one is done, phase two is done, phase three is done, phase four is done, and when phase five is done, what is phase five actually? There's one more phase towards the end. Yeah, the wrap-up phase, right? What's there in the wrap-up phase? In the wrap-up phase I ask them a few questions related to uh human to human interaction like uh where do you see yourself in 5 years? Or how do you work in a team? What are the challenges which you have faced and how you have overcome them? Then what's your biggest, let's say, asset or what's your biggest liability which you bring to the table? So I'm going to ask them all of these questions. Maybe salary So in this call I don't want to have salary negotiations because this In this tool I just want to have if they they're good technically or not. Like if a if a student student just needs to know whether they can crack the interview on a technical and behavioral side. Salary negotiation is another thing. We we can build a separate agent for that. Uh there is a question about job description. Very, very good. So many people have pointed out that well, resume is one thing, but a job description is also another thing, right? So I am particularly looking at ML engineer right now. ML engineer position. So when I'm going to ask all of these questions in their projects, I'm going to ask them questions which are relevant to the job description which I have put out. So when I'm in phase two, I'm going to ask them questions which are relevant to the job description I have put out. >> Let's see if there are any questions in the chat. So, Sampath Singh, I do this this way: self-introduction, projects, knowledge, technology depth, system design thinking, behavioral fit. Exactly. So, Sampath, the approach which I have followed is very similar here. First is the project, then first is the background, then I drill down into the projects with this Russian doll approach. Here, I go in a lot of depth, actually. Then, the second project, then factual or system design questions, and I wrap it up with a behavioral question. Salary discussion, I don't do in this technical round, actually. It's a bit later, after they're fit. Shivam is asking, "Whatever answers user is going to give, we need to ask what, how, and why." Exactly. This is very important, and that differentiates good products from bad, right? Some people who are blindly building this interview tool might just So, I gave the same project to many students for an interview round, right? What many people had built was you ask a question, student gives an answer. You ask another question, student gives an answer. This is very bad, right? Usually, an interviewer asks follow-up questions. Very few people who built the product followed this Russian doll approach, where follow-up questions were asked. That's difficult to build. You don't just randomly go on asking new questions each time. So, these are the five phases in which, ideally, I would want to build this interview tool. So, the thing is, now think about it, right? If you have randomly built this, and if your manager or anyone in your team asks you, "Why have you built this this way?" You have no thinking, you have no rationale. And that's very poor. That's poor planning, that's not a good software engineer at all. A modern software developer knows exactly what they are building. And if someone asks them the question that, "Why have you built it this way?" you can tell them that because I have thought about this project in five phases because this is how I would conduct an interview. And I want my AI system to behave as close as possible to a real human. So, I'm mimicking my thought and I'm drilling it down into my AI agent. That is how you put care into projects, right? So, recently we are talking with the CTO of a company and they were telling that uh projects are being built fast in the company, but when they ask their software developers what's going on or why have you built this, no one is able to answer. I believe that such kind of care practices will help this. Is there anything else which we are missing out here? Let's think about the agent later, but first from a human perspective. Is there anything else we are missing out in this interview tool? Vishnu is saying validation of system. We will come to the agentic system Vishnu later, but from a human perspective, let's say you are the interviewer. What else am I missing out here? There is one thing which I'm missing out actually. Two things now that I've Yeah, many people have pointed this out, right? Very, very important thing which is missing out and that is Here I'm going to say care workflow three. And that's empathy. Now, this is completely missing from AI products because AI wants to optimize the path from point A to point B, right? Does not have empathy. This is what is missing. If you have a human in front of you, they might know the answer. So, let's say if you are in phase two, they might know the answer, but if they are anxious, you can see it on their face, right? So, you offer them a cup of water and you tell them that, "Hey, it's fine. You can relax. Uh Just take a deep breath and then maybe you can switch to an easier question." This is completely missing from our workflow currently. How can we judge whether uh How can we judge whether the person in front of us is anxious or if the person in front of us is anxious or stressed? This also needs to be integrated into our product somehow. I want my product to have empathy. It should not be just asking bunch of questions. If a student is anxious or stressed out, then one more thing which is missing here is probably phase six. Is do you have any questions for me? So this I had forgotten, but many people pointed it out on the chat. So phase six is do you have any questions for me? Uh Speed of speaking. So we'll we'll come to how do we get this? This is another interesting thing. How do we know if a student is anxious, right? There are many ways. You can enable the video camera and you can judge from the face or the speed of speaking nervous. There are people when they become nervous, they try to speak very fast. So this happens a lot in interviews which we conduct, right? People become nervous, they speak very fast. They would be knowing the answer, but they want to skip to the end. This is what you can detect actually. Yeah, Raghu is saying checking the attitude of the person. Again, this is very very important. Whether they are going to be a cultural fit. Because they might be knowing all the technical skills, but if the attitude is not right, for example, in our company, we don't hire people who are not good listeners. Uh although they are technically very sound, it it might depend from company to company, right? Uh Mega is saying, "Check whether they are confidently wrong." Okay, now we have these six phases, and we have thought about So, we have implemented care workflow number one, which is identifying a problem statement which would actually solve a pain point. We have thought about the care workflow two, which is independent thinking, and we have thought about the care workflow three, which is empathy. And all of us have come up with a lot of points which we think need to be implemented in this uh interview system, okay? Um Okay, Rahul is saying, "At the end, there should be key metrics which where a company can configure it." So, we'll come to when we build the agentic system, there should be some metrics based on how we are judging, right? So, each of these phases I'm going to assign a score. Um Okay, more or less is everyone fine with this approach currently which we have taken? Have we covered, let's say, 80% or 90% of the points which would come up in an actual interview? So, only if all of you are fine with this, we should go to the next thing. There might be some more things like uh as someone is saying, "Feedback loop." Uh fake check, etc. But let's start with this as a base structure right now. Yeah, we can So, the thing is, if we are building this tool for a student, right? I'm assuming students who are coming here are serious. If they just check everything from the internet, then it's worse for them. So, maybe we can even build in We can have anti-cheating things. Uh Let's see, but I'm currently not looking at features, right? Features will come later. I'm currently looking at the workflow which I want. So, can everyone confirm if you are ready to move to the next stage? The first stage of this workshop was this care workflow. If If all of you can confirm that you're ready to move to the next stage, I'll proceed with the next. Okay, now Again, here there are two types of people. The first type of people is who have thought of the care workflow and who directly then go to Cloud and put all of this there. But, this is again not correct. Because for a modern software developer, you need to have a structure in which you are executing projects. And the structure in which you are executing projects needs to have a proper workflow. Uh how many of you in this class right now are from the modern software developer workshop? Could you say yes in the chat if you have attended the modern software developer workshop? Okay, many of you here are from there, right? So, in that workshop, we have learned about three things. We first need to build a project roadmap document. We need to execute the tasks. And then later, I will develop a walk-through to see what all what all I have built so far. So, we have a system of interacting with Cloud code. So, those of you who are from the context engineering workshop, I'll just explain this system again. So, we don't directly put in a prompt to Cloud code directly. We do it step-wise, right? We first build a So, I I always call it project roadmap. So, it's actually product requirement document. So, we build this product requirement document and we first verify it. Uh second thing what we do is that we When we build the product requirement document, there will be a list of tasks. Then we will execute those tasks and then I'll just see a walk-through of what all has been built so far. And then we'll do the building in versions. So, first we'll build the version one of the product, then we'll build the version two of the product, and then we'll build the version three of the product. So, those of you who are in the who were in the modern software workshop, you would already have the skills which we have created with Claude code. You would have this {slash} PRD skill. You would have this {slash} dev skill. And you would have this {slash} walk-through skill. For those of you who don't have these skills, I'm going to do this from scratch right now with you. So, we don't have to create these skills in this workshop. Those of you who have these skills, you can directly invoke these skills, okay? But for those who don't have these skills, I'm going to interact with Claude as if skills are not there in a sequential manner. Okay? So, now what we are going to do is that we are going to go to Claude and we are going to put an initial prompt. Okay? So, let's go ahead right now. But I'll first write this in Microsoft Word. So, I'm going to share um share this document right now with all of you. So, you can copy whatever I'm writing here. Okay. So, you can say that I am planning to build an industrial-grade I'm planning to build an industrial-grade Now, see English is the new programming language, right? As I always keep telling in the modern software developer workshop. So, whatever I'm writing right now is the code. Earlier, when I would be typing code, all of you would think that this is a coding session. But right now, whatever we are typing is actually the You can think of it as a code. So, it's very important that we write everything here correctly, okay? So, I'm going to say I'm planning to build an industrial grade product today, which does the following. Um It's a full AI-powered mock interview agent from scratch. It takes in the resume from a student. It takes in Okay, so I have Whisper flow right now. So, I'm just going to speak so that it will be converted. It takes in the resume of a student. It parses the resume. And then we build Then then we are going to build a chat system as if a human is conducting an interview of the student based on their resume. Okay, it did not work here. Uh Let me try again. So, what I want to build is it will take the resume PDF as an input. The resume PDF will be parsed. And then we will have a chat system which simulates how a human would conduct an actual interview of a student. >> So, I have thought a lot about how I want this system to be. Let me explain what I have thought about. I want to I want to build this system in phases. I want to build this system in multiple phases. So, now let us actually see the phases which we want over here. Phase one is where the interviewer asks questions about the background. Phase one is where the interviewer asks questions about the background of the student based on the background section of the resume. Such as, uh, could you please tell me about yourself, etc. Phase two is where the real technical interview actually starts. In phase two, the interviewer will identify the most important project which the student has done. That might be the first project in their experience section or the first project in the project section. The interviewer will follow a Russian doll approach here where they'll keep on drilling down until they reach a stage where the student cannot answer the question. This is very important for me. You can think of it as a Socrates method where I will keep on asking them questions about what they have answered so that I go down in a lot of detail. Remember that the job description is for machine learning engineer. So, when you drill down and ask questions to the student, they will be related to machine learning engineer. The question should be such that let me give you an example here. Now, see this is very important. The more examples you give, it will be better. Let me give you an example of what I mean by drill down. If a student says they have built a project based on retrieval augmented generation, the first question I will ask them is what exactly did you build? The second question I will ask them is that could you tell me how this works? Then I will ask them what is retrieval augmented generation or rag? Then I will ask them what are the different chunking strategies? Could you explain the trade-offs between chunking strategies? Then I will ask them what is vector in indexing? Could you take Could you tell me the difference between HNSW versus IVF flat? Could you tell me the difference between HNSW or IVF flat? Then I will drill down further and explain could you tell me about cosine similarity? What's the formula for cosine similarity? Then you can Then I will ask them what's the disadvantage of rag? Why did you not use fine-tuning? Then I will ask them Then I will ask them about different indexing strategies like HNSW or IVF flat. Then I will ask them why did you use rag and not fine-tuning? Then I will ask them what are the last advantages of rag? This is how I want to drill down on every single project. You need to smartly construct the follow-up questions based on what they have answered before and I want to test their ML engineering knowledge overall by using their project as a proxy. So, this is what putting care into the prompt looks like, right? For a people who would randomly gives an inputs, uh they will just put something very quick over here, but I'm not putting something quick. I have actually thought about what I want to be in different phases. Phase three is where I will ask them questions about another project which they have done or maybe a different type of a project such as a research internship. But, here also I want to have this Russian doll or a Socrates method of questioning. Uh let's see what are the different phases. Yeah. Phase four. Phase four is where I will have factual questions related to uh machine learning engineering. Now, let us think about how should we have these factual questions? Now, can you all tell me how should I ask these factual questions? Do you Can you think of how can we build this agentic system so that it can ask good factual questions based on the student resume? Uh so, this question I'm not asking to Claude but to all of you. Saying "Choosing phase two side heading is missing." Oh. So, it's fine. Claude will understand that. Uh how do I choose the factual questions? I want the factual questions related to ML engineer, but where can I find it? Where can I get good factual questions? Let's say I do not uh I do not trust Claude code to give me good factual questions. Yeah, so let's actually use a tool, right? I want you to use an external tool for this. Um so if I go to machine learning machine learning engineering interview questions and then I go to GitHub. Yeah, so let's say I take this. I want you to use your web search tool, go to the above GitHub link, and look at all the questions here, okay? I want you to download these questions into a markdown file. Now based on the resume of the student, first based on the based on the resume of the student, you first need to identify their field, if it's natural language processing or computer vision. Or any other field. Then you can do a simple similarity search with the markdown file, and then you can retrieve questions based on that field. You see what we are doing here is that we are not just randomly asking Claude to generate questions for us. We are actually thinking a lot here as a human. All of these are human human inputs, right? So when people say humans are useless in this agentic engineering age, I'm not sure that is the case because the more care you put in, the better it will be. Then you can do a simple similarity search, you can retrieve questions. If you cannot find questions in the above GitHub link, you can create a question yourself, but make sure to ask at least five questions in slightly different fields. The question should have factual correct answers, so that we can verify them correctly. If you if If you can't find questions in this document, create these questions yourself. And the question should have a factually correct answer. You can get the answers from the above link as well. At least ask four to five questions so that we can grade appropriately. Uh similarity saying, "Where are we creating embedding for the similarity search?" So, Cloud coding agent will create embedding itself. So, I'm going to tell it that you can use a simple 384 dimensional embedding model. Saurav is saying, "If you mention ML, it will only focus on ML." So, you can make it more general. I'm just creating this right now for an ML engineer interview position. So, this is my phase four right now, but we are still not done because in phase five phase five phase five is where I will ask them behavioral questions such as, "Where do you see yourself in 5 years? What are the in most important challenges you have faced? How do you work in a team? Do you have any questions for me?" So, let's say I'm only looking at these five phases for now. And then what I'm going to say is I want to evaluate the candidate based on their responses in each of these five phases. So I want you to create a robust evaluation criteria for each of the five phases. Now let me talk a bit about this evaluation criteria with you. Agent evals is also one of the most important fields. So now when we write this prompt, we also have to define evaluation criteria. Um So let's see. Phase one, it does not make sense to have evals that much actually because it's just about the student whether they can tell about themselves. It does not make sense to have evals for phase one. In phase two, I want to come I want you to come up with the metric called as the Socrates metric or the Russian doll metric, which tells how deep the student was able to answer the questions, till how much layer they were able to answer. In phase two, I want to you to create a metric such as the Russian doll metric or the Socrates metric. I want you to create a metric such as a Russian doll metric or the Socrates metric. This tells me how deep the student was able to go and still answer questions. Phase three will be similar to phase two. Phase four will be the most easier to judge because we have the correct answers and I just want you to evaluate how many answers the student got correct. PS5 is also important to evaluate. I want you to see if the student is proactive, if they are visionary, if they are realistic and grounded, and whether they can be a good team player. I also want you to give negative points to the student if they don't ask me any follow-up questions. Uh Let's see, I'm using Whisper flow actually. This. So, that's why sometimes my voice is getting detected, sometimes it's not. Okay, is there anything which we are missing here? So, this is what we are going to give to Claude as a prompt, but is there anything I've missed? Empathy, voice to text. Okay, so let's talk about empathy a bit. Very good. There is one more thing which is missing here. That is empathy. So, I don't want to build a I don't want to build a video feature here since it will be too complex, but I want you to recognize the voice of the student. And if the voice is too fast or stuttering, then it means the student might be anxious. At that moment, we need to take a break. This will become a heading, I think. >> For phase five evaluation, I want you to evaluate whether the student is visionary, are they grounded and realistic, and whether they will be a team player. We need to take a break, pause for a moment, and then tell the student to just relax themselves, and then continue the chain of questions. Um there is a There are a lot of comments in the chat how to validate the right or wrong answer. So, that is why we have the evaluation framework here, right? For the fourth for the phase four, we are passing an external tool where correct answers are already given here in this link itself. So, the answers which will be given by the student will be compared with these correct answers, and then a score will be obtained. Video feature would be great, right? So, let's let's let me say. Okay, maybe in the initial versions, I don't want video but we can consider it in the later versions. Because from the video of the student, we can easily judge whether they are anxious or tensed. What is one more thing which is missing over here? Shivam is saying we are not tracking the expertise of the student. So, the expertise of the student, I think is tracked in phase two, phase three, and phase four, right? Because in phase two, we are going to actually judge how deeply they can answer questions. In phase four, we are going to judge whether they are able to answer the factual questions or not. Giving hints, correct. So, let me say say it here. One thing which I want to mention in phase two and three is hints. If the student is stuck somewhere, uh and if they are if they don't answer properly, you can give them a nudge or a hint. In the evaluation, I also want you to evaluate whether they are moving to the correct answer based on the hint which is given to them. Now, actually, uh let us There is one more thing which is missing here, which is the tone of the interviewer. Usually, when you build a agentic AI system, right? The AI is very over enthusiastic. Has all of you noticed this? It mentions things like fantastic. Uh it mentions things like uh it's a pleasure having you in this interview, and I don't want all of that. So, I'll say that I I want you to be very professional. I don't want you to be over enthusiastic and eager. Remember that you are an interviewer who should not show too many happy emotions or too over too much over enthusiasm. The words which you use The words which you use should be concise and to the point. So, add this to your prompt template. Never use statements like incredible, great answer, let's move to the next. Whenever student gives an answer, never agree with them too much because this never happens in an actual interview. You process the answer, you ask the next question. So, this is very, very important. Again, this comes to the fact of putting in care, right? Because if you randomly give it, then the agent will actually I have built the version one of this system way back and then it gives it agrees with everything. Um Yeah, so what Anindya is saying, I've written the same thing right now in the chat. Is there anything else? Now, let's talk a bit about the features, right? I have not talked a bit about I have not talked too much about the features yet. So, this is the cool part. Let's talk about features right now. So, what are the features which we need? What is the main main technology or what what is the main thing which will power this application? Voice also text, right? So, we need a large language model actually. A large language model powers this entire application. I want you to use GPT latest model, the reasoning model. So, let's see. So, if you if you have open AI API key, you should use that. But, I'm going to use GPT 5.4. Here is an example of how to work with GPT 5.4. Now, for the voice part, let us do something interesting. We will use the open AI Whisper model, which is text to speech. I think that's the best. So, here is an important thing, right? If you have Gemini or any other API, you can feel free to use that. But, I'm going to do a very interesting thing here, which all of you can do if you're maybe after this class. So, I'm going to go to 11 Labs. And I'm going to show you one very cool thing. So, I want the interviewer to speak in my voice. Uh how will I do that? If I want the interviewer to speak in my voice, how will I do that? I want the interviewer to speak in the voice of Dr. Raj Dandekar. How how can I do this? Voice clone, yeah. So, I have already created a clone of my voice. So, I've created this already using 11 Labs. Uh actually, let me try to find this. Yeah, see, this is my voice. So, if I Let Let me use my voice here. In a quiet village where the sky brushes the fields in hues of not my voice. Let me Yeah, this is my voice. gold young No, not this. So, let me generate speech. In a quiet village where the sky brushes the fields in hues of gold, young Mia discovered a map leading to forgotten treasures. Little did she know her cat whiskers had a secret. He was the guardian of the map tasked with guiding Mia to not only the treasure, but also to her destiny. It's not very bad. So, uh let's see. I want to use So, first I need to provide my 11 Labs API key. Uh So, I'll say you can ask me for my 11 Labs API key and I will provide it to you. You can ask me for my OpenAI API key and I will provide it to you as well. So, never expose these keys anywhere. If when we are pushing to GitHub, we'll store this in Git ignore. So, one thing which is how will you get this voice, right? So, every voice I think has a voice ID associated with it. The above is the voice ID which I want you to use. So if you if you don't have a clone or if you can just go to 11 Labs right now and use any random voice here. Uh and you can just copy its voice ID. So if you give the 11 Labs API key and the voice ID then the interview will interviewer will have your own uh voice. There is a good question by Karun that 11 Labs is expensive. Yeah, there is actually there are open-source alternatives which are very good. For example, Coqui TTS. This is incredible, fully open-source, runs locally, and it's only 1.6 billion parameters, I think. I have cloned my voice using this model also. It it works amazingly well. Okay, what else is remaining? Now voice is done, chat is done. Uh Okay, one more thing. One more thing, when the user uploads the PDF, I want you to parse the PDF using OpenAI itself. Maybe don't use a Python library like uh PyMuPDF, etc. That's not very reliable. When you parse the PDF, you will need to store different sections. You will need to uh take the resume, convert the it into different sections, and store them in a database. What do we do about the database here? How many of you have used Supabase? >> Let us use Supabase as database. We can use Oracle also, but Supabase is just uh let's let's use Supabase and uh I'll show you how to use this. So, you can go to Supabase here right now. If you don't have uh access to this, you'll need to sign in here. And you can create your first project for free also. So, you can go to the dashboard. And uh what you can actually do is that you can click on account preferences. And then click on access token and click on generate new token here. So, I'm going to type in Supabase access for interview agent. So, let me actually store all my keys in some separate place here. So, this is my Supabase key and my 11 Labs. I'll need to check where to find the 11 Labs key, but I think it will be in the developer section. So, I'm going to say interview agent. And I'm not going to restrict key for now. So, this is my 11 Labs key. And I'm going to say that I will give you the access key to Supabase. You can use that access key to create a database on Supabase and store the respective sections. I will give you the access key to Supabase. You can use that access key to store the relevant sections on Supabase database. This not getting detected. So, I'm saying I will give you Supabase access key. You can use it to create a Supabase project and database tables. Let me see if there are any doubts or any questions in the chat. Okay, good. So, now database is created. So, when the resume is extracted, uh first the resume is extracted. So, I'm saying that to extract the resume, use the OpenAI PDF parser itself. Don't use a Python library. It's not very reliable. Uh when you pass the PDF, you need to store different sections. You'll store them in the Supabase database. Okay, that's good. What about the report? I want you to create a final report for the student once their interview is finished. This report should should indicate their score on all the different phases. I want to develop an anti-cheating system also, but not maybe right now. I'll do this in subsequent versions. I also want to build a full CI/CD pipeline which pushes to GitHub and then links to my AWS, but maybe in subsequent versions. These will be in subsequent versions. So, in this workshop, let's build the first version first, and then you can go ahead and build the next versions. So, I'm going to say CICD pipeline with GitHub actions and AWS deploy. Then, we are going to also look at an anti-cheating system, but maybe later. Okay. So, let's see that there are any questions in the chat. Are you not specifying the agentic framework? So, now we have moved past that, right? This was maybe 6 months back where we would have specified an agentic framework, but now Claude code will by default use its own agentic framework to spin up these agents. Now, so all this is done, right? Then, how are we going to implement this uh I mentioned about these three things, right? Problem the PRD the project roadmap document, tasks, and walkthrough. So, let's Let me write it here. I don't want you to start building right away. First, you can create a project roadmap document for me. Along with this project roadmap document, you can create a list of tasks which you will finish. Once I approve the project roadmap document, you can start executing the tasks step-by-step and mark it as done when that task is finished. I have the PRD and the dev skills which you can invoke. If you don't have these skills, no problem. You can directly start building. So, for those of you who don't have these skills, it's it's fine. You can you can directly start building. So now what we have to do is that we have to give this prompt to Claude code and we'll start interacting on VS code. Uh first of all, can everyone tell me if you have understood what all we have built so far and why this approach is very different than randomly putting prompts to Claude code? We have spent almost one one hour and 15 minutes in first designing this system ourselves and putting care into the system. We have not even had an interaction with Claude code until now. Because that is how human should first put care into the system. You organize the phases in your mind and then you interact with Claude code and then you will get uh answers much better. Uh Gautam is saying, "Can you please tell me how to add skills.md?" So you can just say create a skill for building this project roadmap document for me and then the skill.md file will be created for you. Sanjay is saying, "Could you please explain what are skills?" So skills essentially are when you create a project roadmap document skill, uh you don't have to say each time that create a roadmap for me. You just can invoke the slash PRD skill and then the every time you're building a new project, it will automatically invoke those skills. Have you used Whisper flow? Yeah, I've used Whisper flow to write this doc. Okay, so now can everyone confirm if you are on the same page? We are now going to go to VS code. So just mention yes on the chat if you have understood what all has been done so far and you are ready to move to VS code now. Okay, good. Now we basically have all the fuel which we need to build this project, okay? But first what you have to do is a couple of things. Go to uh desktop and then you can uh create a new folder here and you can say agentic engineering interview system. And then what you can do is go to VS code and then you can click on add folder to workshop or add folder to workspace. Uh so I'm going to add this. Then what I'm going to do this sample PDF, right? You all can use your own resume actually right now. I'm going to use this sample PDF. And I'm going to say here that use this resume as reference. So you can paste your API keys here if you want. Uh please provide the resume. So what you can do is that let me actually provide this or let's say ML Yeah, just take this resume. Now when you go to VS code, Cloud Code when you interact with Cloud Code it will ask you permissions each time, right? So what you can do is that you can go to code. Actually you can go to system settings. Uh not system settings. Where is it? Yeah, you can go to settings. And then you can type Cloud. And then you can click on allow dangerously skip permissions over here. And then initial permissions mode you can select bypass permissions. So that you don't have to, you know, uh answer the questions each time. Sita is saying, "How do you go here? Can you please repeat?" So first have you installed Cloud Code? That's the first thing. Then what you will need to do is that you will need to go to extensions. And then you will need to install Cloud Code for VS Code extension. There is this Cloud Code for VS Code which you will need to install. And then to disable the permissions all you have to do is go to code, go to settings, extensions. Not extensions. Go to code, settings. Then type Cloud here. And then you can allow dangerously skip permissions here and initial permission mode is bypass permissions. So what we are going to do now is that I want you to build in this folder. Copy path and I'm going to paste it here. And then, what we are going to do is that we are going to copy paste this entire prompt into Claude code. How many of you have done this? Could you mention yes in the chat? How many of you have copied the prompt and put it to the coding agent? So, this is where the project actually begins. So, that is what I wanted to illustrate in this workshop primarily that uh agentic engineering is not about directly interacting with Claude code. It is about much more than that. You have to put care into the system. You have to put empathy into the system. You first have to think about the workflow yourself, and then follow this PRD tasks and walk-through approach to interact with the agentic engineering system. Let's see if there are any doubts or questions. Not able to see the code folder in VS code. So, at the top left, can you see this code? So, the code of file edit selection will be all over here. So, you can see what's going on here right now. Uh Claude code is also going through this GitHub repository to see this list of questions which we have. It's going through the resume, the secrets.md file where all the API keys are mentioned. Anindya is saying that is the PRD shareable. Uh Uh yeah, so once it creates the PRD, can maybe share my PRD with all of you right now. So that maybe you can use the same one. So as you can see, you are on Mac. If since you are on Windows, I think that might be in file. Can you try going to file, settings, or some preferences, something like that? May I sing what all secrets should be on secrets.md? So I've currently added three secrets over here. Uh if you can see. I've added the Supa I've added the open open AI secret. I've added the 11 labs, and I've added the Supabase secret over there. Those are the three secrets which I've added. Um can you repeat the steps again? So what I've done is that I have copied this document. And I've interacted with Claude code. So I put questions over here, right? Or I put this entire entire prompt over here right now. And I've asked it to create a project roadmap document for me. At the same time, I have in the secrets.md file, I have actually added three secrets, which is my open AI API key, my Supabase key, and my um 11 labs. So you don't need to use the 11 labs API key. I think it's not absolutely needed. Because if you don't use it, by default, Claude will use some manual robotic voice. But later after this class is done, maybe you can add a new voice from uh you can add a new voice from 11 Labs, maybe. But right now, any voice can actually work. But I've given a couple of things in the prompt. I've mentioned it to stay in this folder, and I've given a sample resume. So, what I've mentioned for the resume is the best test is if you can do it on your own resume. Uh so, I have given some resume here, but if you want a resume, you can directly give any publicly available email resume on the internet. Uh what about the skills which you are talking about? So, many of the people from the modern software developer workshop have these three skills. But those of you who do not, right now, there is no need to create skills. You can do this after this workshop is done. Right now, you We have just told Claude that don't directly build something. First, create a project roadmap document with me. And then, I will approve it. So, see, for me, this PRD skill is being invoked right now. Uh what you can do is after this workshop, you can ask Claude to create the PRD skill. So, it's saying that I have a few targeted questions before I write the PRD and the task breakdown. Front-end preference, you mentioned React. Streamlit would be faster to ship. So, I'll say I want React. So, I'll say V1 should support voice. >> You So, I'm saying you can create Superbase project yourself. So, I'm saying this along with voice looks good for version one. I'm demonstrating this in a lecture right now. So, maybe keep the number of tasks to less number of tasks, maybe eight to 10 tasks. So, whatever I mentioned here, I'm uh copying this and adding it to the doc. Let me see if there are any questions in the chat. Sanjeev is saying wondering whether it's still necessary. I don't think it's necessary, Sanjeev, anymore. So, Cloud Code inbuilt agents already provide these capabilities which libraries such as LangGraph or Crew AI offer. Uh Mausam is saying simply saying create a PRD skill. So, we have to write a lot more detail of what that skill should entail. So, currently what we can do is that if you want to create the skill, I can provide a sample PRD document to all of you, which we actually covered in the uh modern software development class. So, this is the So, this is our project roadmap document actually should look like. So, I'll in this same document, I'll show you it a bit below. So, if anyone of you wants to create the skill, you can give this as a reference and say that I want to create a skill out of it. Uh let me see if there are any questions in the chat. Rakesh is saying, "Did we create the cloud.md file?" So, right now, we have not created a cloud.md file because right now, I just want to show you the PRD driven workflow. We can create a cloud.md file after the uh project roadmap document is done, for sure. Kapil is saying that, "What plan of 11 Labs is relevant for us?" So, you can start with the free plan right now because you can even clone a There is one voice clone option which 11 Labs offers you for free. So, that you can already use. Okay, so you can see here that the task breakdown for me has been created now. React plus fast API, GPT-4, Whisper, 11 Labs, Supabase. Okay, great. This is exactly what I want. Key decision key design decision Socrates depth score. So, here you can see for me this PRD.md file is created because I have the project roadmap document skill. And a tasks.md file has also been created. So, as different tasks in the PRD are executed, we will make a tick over here and that task will be finished. So, then I can say now go ahead and start executing. So, one more skill which we had created for this is {slash} dev. Those of you in the modern software workshop will know this. There is a question, what was the Supabase value? So, for the Supabase, what you will need to do is that once you create the Supabase project, right? You will need to go to account preferences. You will need to go to access token and then you will need to click on generate new token. You provide a name for the token and then copy it. And put it in your secrets.md file. So, see this is how tasks are being executed right now for me. While this is ongoing, maybe I can show you all how skills can be created. So, we create I talked about this PRD skill. I talked about the dev skill, I talked about the walkthrough skill, right? So, if you want to create a skill, you can simply say that every time I work with Claude on a project, I want to follow this approach. Create PRD. Run tasks. Generate a walkthrough file. And I want to create version based prints. I want to create three skills. Uh one is {slash} PRD. The second is {slash} dev. So as you can see I already have these skills, but for those of you who do not, you can use this prompt to create these skills. A sample PRD looks something like this. Um and then you copy paste that sample PRD which I mentioned over here. So this is how you can create these three skills. Uh ideally more details need to be given for the skill creation, but here I'm just uh saying simply. So here itself I'm saying prompt for skill creation. All right, and then um maybe I'll show this in a different color right now or this color, yeah. Mhm. guys think we need to remove this line since we are not using your voice. Yeah, you can remove this line or you can use uh any other voice actually from 11 labs. No need to stick to my voice. You can go to text-to-speech here and use any other voice over here which is available on the free plan actually. Uh modern software development uh recordings are not available. How many of you uh are actually running this right now as we speak? Could you mention in the chat? If you are running these tasks, for how many of you these tasks are being are being run right now? Yes, okay, very good. Nilima and Karun running this task. Sumit, Aritra, Tanvir. Yes, okay, very good. Um but no API keys. So, API key that educate you can even use Gemini API key because they have free tire API key. No need to stick to open AI. Uh Sarthak is saying, "Should I share my PRD?" Yeah, Sarthak, you can please share your PRD on the chat. I'll probably open it and show it to everyone also. So, Karun is right now on task number four. That's very good. It's going ahead very fast. Uh for me, as you can see, I am uh I am on task two right now. So, one more thing which people from the modern software developer workshop and even in the context engineering workshop will recognize is that when you run the different tasks, the ideal way to go about it is to attach every task with tests, which we are not doing right now for the sake of simplicity. Ideally, what we do is that when you create this the skills, right? When you create the skill for /dev what you should also say is that attach um for the /dev skill attach tests along with every task. And mark task as completed only after all the tests pass. Um Karun, for Karun, this is fine. Um But for other people who are who have not from modern software developer workshop, all of you have the skills correctly /prd/dev/walkthrough. We don't need to change anything. But for those who don't have these skills um this is how you can create these skills in the first place. Because as we have seen in the MSD workshop if in the the task should not be marked as completed unless the tests for that task have been finished. Uh Satyajit Singh, if you allow skip permissions after the execution starts, I think you will need to restart the session once Satyajit, but you can check this. Every time I have allowed skip permissions, I have restarted the session. Uh Sarthak has shared his uh project roadmap document. Let me just open it. >> Yeah. So, I think it's actually very similar to what mine is. Supabase has the database. We have Whisper for uh uh text-to-speech. Speech-to-text is Whisper, actually. Text-to-speech, we have 11 Labs. Resume parsing, we have OpenAI GPT 5.4. Database is Supabase PostgreSQL. Embedding model is See, now this decision has been taken by Claude itself, which is a very good decision. I think this is a very famous model from Hugging Face. For sentence transformers embedding, it has a huge number of downloads, also. Prakash is saying, "With this process, can we So, we have a rag component in this process, right? If you uh notice where that rag component is, this system is already based on a retrieval augmented generation pipeline. Where is the rag component?" The rag component is over here in this factual factual questions. Because in factual questions, what we are doing is based on the resume of the student, we are retrieving the questions from the GitHub repository appropriately. So, now when you say rag, by default, it means agentic rag. Why? Because the rag is a tool which the agent can decide whether it has to use or not. Earlier rag systems were by default, like you have documents, you have embeddings, and the query, and the relevant document is retrieved. Um but now it's agentic rag, right? By default, rag has become agentic rag where along with all the other tools, the rag is also a tool. One more thing which you can do which people in the context engineering class, how many of you here are from the context engineering class? In that class, you might have heard of uh sub agents, right? Uh in there is one more approach to work on this project, and that's through creation of sub agents. So, here I I've got the message that you've used 92% of your session limits. Resets in 3 hours, so we can see view usage here. I think the best thing for me to do right now would be to switch uh to a different account which probably has more usage. Let's see. So, every industrial project on Claude actually takes limits like it takes uh tokens, which means that you definitely need to be on Claude Max for sure. So, let me see how much In between, we can switch to any other account as you can see over here. So, I've got this message that I've hit your limit. So, I'll probably switch all of you can go ahead and continue using whatever you have. What I will do is that I will uh maybe I will upgrade or switch to another account. Uh just give me a moment. How many of you are on the uh Cloud Max plan? Probably the Max plan is the best to uh use because it it's heavily subsidized. >> So right now what I'm doing is I'm upgrading to the max plan. So many people are saying I hit my limit, can't proceed now. Usually for these big projects, you will need to be on the best plans because these projects consume a lot of tokens. Right now we're building an industry-based project, right? So it's not not an easy project. >> There is a question in the chat that um in between we can switch to any other account. Yeah, that's you can I have hit my limit. How many people have hit your limit actually and how many have not? Could you mention? That is there anyone who has not hit their limit right now? So, I have now actually restarted to the max plan. So, uh let me check it. Maybe I'll need to log out and log in again. Okay, now let me share my screen again. Uh yeah, so my sharing my screen is shared. Can you all see my screen right now? Uh okay, so as you can see I have uh hit the rate limit right now. So, what I did is there are multiple options you can do after this moment. This is a very common problem which you will face while building projects. Uh I have upgraded to Claude Max now, this Max plan. So, as you can see uh if you go to settings So, I'm on the Max plan right now, which is 5x more usage than Pro. It's a bit expensive, so there are multiple things I have right now. You can switch to the Max plan or you can wait for your uh wait for your uh limit or wait for your session to again restart. That might take around 3 hours. Or what you can also do is that you can use a locally downloaded Ollama model. Um For example, there is way to connect the models such as Qwen with Claude code, which you can locally download. So, I'm just sharing this with all of you. But it's just that for industry-based projects for industry-based projects in my opinion, for those of you How many of you in this call are industry professionals, by the way. Uh could you mention in the chat? For those of you are industry professionals, it really does not make sense to work with Ollama. All of you should definitely definitely upgrade to Claude Max. Because it's completely worth it. For those of you who are students, you can definitely go ahead with Ollama integration with Claude code. But those of you who are industry professionals, uh there is really no comparison between the Max plans and the uh the Claude code or Quen. In amazing, I lost my job. In this case, actually it's better to use Ollama with Quen. It stays The workflow stays the same. It's just that for the model, uh Opus 4.6 is probably the best coding model right now. It's It's way better than the open-source models. There's There's a huge difference, actually. Uh how many of you have finished many of these tasks? Could you mention in the chat? Karun is finished. Karun, now can you do local testing with this, whether it's actually working or not? Uh there is a question. Is there a chance I could get full recording right now? So, I'll probably upload the lecture recording after this lecture is done. Once the recording is processed, Ripinder Singh is one of your sites you said open source is Open-source models are definitely good. I mean, the engineering on open-source models and advanced so much, but in terms of building industrial grade projects which you want to deploy to millions of customers, uh in that case, Opus is definitely very good. Prakash is saying I'm working on a project where word documents are submitted which follow a standard template. Main task is to pass the document and based on the content. So, this is exactly what we are doing right now, right, Prakash? The resume document which is uploaded by the student uh is getting passed. The different sections in that document, they are getting stored in the Superbase database, and then questions are being asked one after the other after that. Uh So, Pinelis is saying we have developed this agentic code. Do we have any security or end-to-end testing of the product after we develop it? How can we Yeah, that's a very good question, So Pinelis. So, there are actually three ways of testing. And it's called as the testing pyramid. We We covered this in the modern software development bootcamp, actually. So, after a product is built, you should ideally do three types of tests. Unit tests, which test individual functions, integration tests, where it's like API endpoint testing, and then E2E tests. So, it's called the testing pyramid. These E2E tests are beautiful because what Claude code can do for you is that it can open the browser, and then it can click on the product workflow for you as if it's an actual human tester. Because these language models are multimodal, right? So, they can actually see what's on the screen. So, you can open a project or Claude code itself can open a project in the browser and do an end-to-end testing. That's called E2E E2E test with Playwright. So, usually after a project is built, you can do all these three tests. The unit tests, the integration tests, and then after that the E2E test. Um we're facing that how do we create cloud code environment where we need two different GitHub repositories? Then I would like to update. So this same question you can ask cloud code, right? So that I have created a environment with three or four GitHub repositories. So the best way is to ask cloud code that hey, I want to work with four GitHub repos. How do I go about it? So you can see for me right now this whole project has been at least the spring to version one has been completed. And now it will have errors. Our job is to check whether it has errors or not. So let's go ahead. So I've opened this localhost and the first is that localhost itself is not opening. Uh I tried opening localhost, but it's not opening. Can you open it yourself? Do end-to-end tests and then tell me whether everything is working. Only then I will start uh my testing. The localhost is not working for me. Can you try opening it, upload the PDF, and check whether everything is working end-to-end, and then you can report back to me only after your testing is done. So here maybe let me try use Playwright MCP for browser testing and share test screenshots with me. Give me the local host link to check after you have tested the workflow into end after you have simulated some some conversations yourself. So, think of yourself as a human tester. So, see this is an experimental prompt which I have given Cloud code. Um where I was Cloud code to do the testing itself without first without giving it to me for testing. You play right MCP is not available. It's fine. You can just install Playwright and use it. Um how many of you have finished these tasks? Could you let me know what you are doing right now? For how many of you the tasks are finished and is anyone able to run it locally actually? >> Not yet. Okay, what about others? How many of you have finished the task and are seeing some output locally right now? Karunisingh testing E2E not yet successful. So, I'm also at the same stage right now where um what we are doing is that we are writing a comprehensive end-to-end testing script that acts as a human tester. Vishnu and Shalini are saying still on task four. So, there is this there is no way to speed it up unfortunately because the more you speed up, the more there are chances of uh things breaking actually. So, right now one simple thing what we could have done is directly go to local uh local host and test it ourselves. And every time we do the tests and every time something breaks, we go back to Claude and ask it that, "Hey, this has broken." But, that's not the best way to go about it actually because that's like very back and forth approach. Instead, what's best is that right now it has written this E2E test, right? So, this is the E2E test.py. Where what it's saying is that it's an end-to-end Playwright test for AI mock interview system, acts as a human tester, uploads PDF, goes through interview phases, checks report, takes screenshots at every step. Now, you can make this testing procedure as robust as you want. So, you can even test the database. Right now, I'm not testing the database really, right? Uh but you can even say that in this testing procedure I want to test whether the Supabase database tables are getting created or not. So, let's see if there is something in the screenshots right now. So, already something is being uploaded in the screenshots. And here you see resume resuming parsing timed out. Let me check what happened. Likely the upload API call failed. So, see what's going on right now is that Claude itself is checking whether uh everything is working by itself going to the uh website and it's giving screenshots to me. So, this is what is possible only with the coding LLM agent right now. This was not earlier possible. Just opening a browser and doing an end-to-end test yourself. The multi-modality is what makes it makes it possible. So, now see the immediate issue which is happening is that OpenAI call format for PDF parsing is not correct. Uh so, this is being changed right now. OpenAI OpenAI response is API format is wrong. And Claude code itself has understood that I have to fix the resume parser. So, now these end-to-end tests will be done by Claude code itself. And during this time, ideally, what I do in my workflow is that instead of waiting, what I do is that I actually start a new Claude code session and I uh work on a new project. So, usually at in any given point of time I'm working on multiple projects at the same time parallelly. On a good day I'm working on four to five projects parallelly. Because the testing I ask the agents itself to do the browser testing for me. Um and while let's say one project is in the testing phase, there might be another project which is in some different stage. Um So it's like now it's possible to do four to five projects at one time, which was actually earlier not possible. Let's see if there are any questions in the chat. So Kapil has actually finished building this system. Kapil, is it working actually? Prakash is saying in is Supabase used So Supabase has PostgreSQL actually. And the language is PG vector itself. Task to verify Supabase tables, okay. Swapnil is saying how about scalability? Is the system able to handle high throughput? So usually for scalability, Swapnil, what we can do is that we can use Terraform and we can deploy to AWS once the CI/CD pipeline is set up with GitHub. So once your testing is done locally, you can set up the GitHub CI/CD pipeline and then deploy on AWS. And if you use Terraform, which is infrastructure as a code, Terraform itself can uh deploy the servers for you appropriately based on how much load uh you're expecting. Oh, actually for Karun, the whole pipeline is done. Everything is working, resume parsing, Supabase storage. Wow, that's amazing. For me, uh I'm still stuck at one earlier stage, so let's see where I am at right now. Uh Mayank is saying what are your So Satyajit is saying please talk about parallelism a bit more. So what I do is that for parallelism, you first need to make sure that every project is running a separate folder on your laptop. So right now I'm in this Agentic Engineering Interview System, right? Uh if you want to work on a new project, you just have to create a new folder over here. And then ask Claude to live in that folder. That way you make sure that the context is not shared across different projects. Every project should have its own separate context. There should be no context spillover. And then you apply the same framework which I've told right now, {slash} PRD {slash} dev and {slash} walkthrough to finish a project and then have a version-based approach. So, right now we are at version one, right? After this lecture is done, you can build a version two, version three and make it as complex as you want. You can add video, you can add anti-testing or anti-cheating features. And then already deploy it. So, right now if you can see here my uh the Claude code is actually interacting by itself. So, see on the right-hand side if you take a look at these screenshots what Claude code is doing is that it's acting like a human tester. Um and see it's drilling down based on the So, this is in the drill-down phase, the Russian doll phase which we started with, right? And I'm not doing this testing at all. I'm waiting for Claude. So, see all of this is Claude code testing. So, what it's doing right now is that phase two is starting now, phase three is s