A Deep Dive into Retrieval Augmented Generation - AI PM Community Session #37

Key Takeaways

[Music] okay I'm I'm just going to start by giving a short introduction so I started my career in computer vision and over the past few months I've been working on a lot of llm projects I cannot uh talk about those projects due to confident confidentiality but I thought I will give you a quick introduction to rxs and it's a very big topic the ecosystem is big but I did my best to cover as much as I could and like ban said if we get an opportunity we will get into the implementation details in another session so I'm just going to share my screen can you see my screen yes okay good okay so uh we know that we have we have we have been Building Systems with llms we have chpt and a lot of other tools and uh the agenda I thought I would first start by talking about the limitations of large land W models and how it has led to uh retrieval augmented generation techniques then I go into a bit deeper into why we need Rags the architecture of building such systems uh then some of the challenges I have personally encountered while building such systems and I will briefly talk about some Advanced Techniques and uh evaluating such systems in the end of course evaluating such systems is an open research topic so I don't get into the depth of evaluation but I think this should give you a very good understanding of uh building production grade rack systems so one of the limitations of uh last landage models is uh at least before gp4 that we had an issue with the context length that we could send to open AI and there is of course the issue of data bias and fairness and the information was limited up to a certain point of time but more that I I didn't cover on the slide but most important is the data security and privacy issues if you are working on an Enterprise application it's not possible always to send uh to send to train your model with open AI or some of these large public models so that got us to retrieval augmented generation so what is what it does is it enhances the performance of large landage models by giving contextual information uh and getting the answers related to the Enterprise data that you have so just imagine you're working in a company you're building a chatbot and you have lot of uh data collected over the years or onboarding tool for your company and this is very private and sensitive data and you don't want to train your model on such a data so what you could do you could enhance the performance of LMS by using this contextual information sending it to the generator or the large language models and getting a response you could of course uh there is this discussion between using open source models uh or a service like large langage models and that's up up for discussion but this is in essence what is uh retrieval augmented generation so uh let's now dive a bit deeper into the key components of RX at a very high level uh rxs consists of a retriever and a generator so retriever is something like a vector database through which I can retrieve semantically relevant information and Generator will be an llm like gpt3 or an open source model on hugging phase Etc but there are of course other components as part of building such a system we have the interaction between the Retriever and the generator and we need to constantly get feedback from both these systems uh if if you are working in a very narrow Niche area for example legal or medical you would want to train your old llms and then combine it with uh Rags then you also have to convert your textual audio or any any data that you have in whichever format you have you have to convert it into a Laden space in an embedding space and that embedding space is then used to retrieve the information and uh we we also need attention mechanisms that's going to be used by both the Retriever and the generator so why do we need Rags first and foremost it improves the performance of uh large langage models it provides more accurate and precise answers it reduces development effort it's not possible always is to train your own model you can use uh different prompt engineering plus rack techniques uh to build build your application it increases scalability and it of course reduces hallucination uh think of think of that you always need a uh you need to site some information if you have an Enterprise data set and you can use racks to get uh the explainability of your of of the answer that you for the query that you have asked so how are we looking with the time Vian do you have any questions that I could already answer yeah sure so we got a quick question on on saati and then I think there are more questions I'll kind of talk about those as well but how do rags increase scalability maybe you can kind of go a little bit deeper on that that'd be great yeah so think of uh imagine you have a lot of uh documents and uh you just want to provide you have a query and you want to retrieve just the relevant information related to your query as so instead of fine-tuning the model you can just get the uh get the relevant information or relevant chunk of data and then send it to llms that's what I meant by scalability that it's not always possible to train your own model especially in an Enterprise setup or very regulated industry so any further questions you can always interrupt me because it's going to that makes sense thank you yeah okay perfect so and there's one more question uh by the way for those of you that um are also interested in digging more here's the link to the cohort program um I think a couple of people also asked me here's the link in case you're interested um but uh the next question that is asked is by harshaw it says when do we say when we say reduces hallucinations how do we measure it and monitor and if there are any benchmarks or test data dat dat um data sets that help you measure hallucinations yeah that this is an active area of research this is an open question how do we bench Benchmark our results and it depends from it's not possible to have a universal Benchmark data set it depends on the domain you are operating in and it could be possible that you would have to create your own Benchmark data set to evaluate the performance of the [Music] model so back in the days when I was doing computer vision we had open source Benchmark data says it becomes a bit difficult with such systems uh especially if it's in a very narrow Niche got it okay so and I think uh we have some talks on the top topic of um performance benchmarks but um I think there's there seems to be a lot of requests there I've seen that before too so uh note taken we'll try to organize a session that kind of digs more into um the basically evaluation metrix I think that's really what it comes down to right um okay so I can see there's a couple more questions popping in says can you give an example of like how rag improves performance I think you talked about it with the chatbot example exle um but um maybe you want to kind of go a little bit deeper so let's look at a system when you ask a query from open AI especially if you working in an Enterprise applic you're trying to build an onboarding tool for new employees and you have lot of uh private data of course llms would not know about this data so you need to provide uh some kind of context to the query so the prompt plus query becomes comes uh the new information and then the llm is used to generate a response based on the context and that kind of improves the performance so instead of just relying on llms giving because it's trained on uh General data it's not trained on any narrow specific narrow data or a niche data set and combining both will improve the performance of the response got it thanks so much for that um I thought it might be helpful um if I also quickly show um one quick screenshot that present um how it's done I just like searched online I thought maybe that's helpful let me would it be okay if I quickly like share my screen sure yeah is that okay so let me quickly do that give me one second I think maybe you can kind of talk through this should I stop sharing uh no I think um yeah you could and I think it automatically basically there you go so maybe you can kind of talk about um this one right here can you see the screen yes like I feel like kind of this is like a good summary of like what happens and you send a request it goes to the specific knowledge base comes back goes to the ML llm and continues um curious to know if there's anything you kind of share by looking at this I I have the architecture in my next slides so oh do you okay got it all right then I'm gonna stop sharing that perfect okay it's going to become uh uh this topic is a bit technical so let's see so before we understand about uh the individual components that is the Retriever and the generator we should understand what are vector databases so back in the days we had keyword-based search algorithms and now what we are trying to do is we are trying to find semantically relevant information so when I write something when when I have a query it's in natural English sentence and I have I have a database with uh all the documents all the chunks that I have stored and I want to find the relevant uh document based on the query that could answer my question and for this we are using Vector databases uh why is it relevant to rxs because rxs sorry okay so why is it relevant to rxs it's relevant because we want to find uh we want to find precise information and we want to find semantically relevant information so it's not just a keyword-based search algorithm it's something more complex and Vector databases provide us certain advantages it's a scale level it can handle massive data sets uh it's the retrieval time is fast and there uh it provides of course different similarity measures like coign similarity ukle and you could also customize it according to your own use case uh but uh there are also challenges with uh choosing a vector databases uh or designing Vector databases that you have to you have to ensure that indexes in your databases uh databases updated so when you get new information or when the information has changed you have to ensure that it is updated in the de and there is this entire space of vector dat Vector database vendors in the market and choosing which database you choosing is a more strategic decision but today I wanted to just cover the technicalities and not a question whether you should have you should build your own one or you should buy one that's a more strategic question for product managers I think uh it's also covered in the course so how do we enhance the retrieval with Vector databases we use uh near nearest neighbor algorithms and usually if you choose a vendor it it comes inbuilt uh with that database uh if you have growing document corpora you want to uh maintaining consistent retrieval performance uh regardless of data data set size is important and it should support different uh similarity metrics for example cosine jakad Etc uh of course you don't as product managers you don't need to know so much the technicalities of these algorithms but uh having a big picture view is important I think and uh uh so like I said there is the retriever part and the generator part and the retriever has a maybe retriever has a different algorithm and a generator has a different algorithm and we have to ensure that both are both can be integrated together any any questions ban I could answer so far yeah so there's a few questions um so Hara is asking what kind of vector database platforms are typically recommended like if you kind of share like I guess what's a criteria of a good Vector database that would be qualified here anything you can share there that's very that's a very big discussion uh so there are popular tools like quadrant uh pine cone depends on a lot of factors the cost uh the data set that you have the scale that you want to use it for uh difficult for me to answer right now which one to choose it's a it's a more strategic decision that you have to sit down with the team and take got it um I can see there's one more question maybe I'll go to the person that has raised their hand voo and then I'll continue reading Hara and s's questions go ahead V yeah hi um this might be a nervous question um the difficulty I'm having in my head is I understand the concepts as you're going through it but I'm trying to sort of map how a solution would work uh when you're using Rags right um so I understand secures your Enterprise data and provides a means to query them and retrieve answers that are specific to your business but what is the relationship between um your uh Vector database that has your proprietary information and an llm so I'm asking a question um the vector database most probably figures out the answer what's the role played by the llm after that I guess so the vector database does not figure out the answer it just gives you the most relevant information that would help to answer the question so it reduces the it gives you the top top 10 or top K relevant in uh chunks that would be used in The Prompt as a context and then the llms will answer that question so it is just for search and information retrieval and not for answering the question so I could be using my preparator Vector database with like an openi or llm yes exactly yes thank is it kind of like taking a shot like you know you used to write kist algorithms or whatever you need to write code or You' use craft databases and then write that to retrieve it so Vector database is kind of solving for that simplifying that right yes but uh it's just now it's not just keyword based search it's uh it's more uh Bas on based on the semantics okay yeah so all right so um got it got it got it that was key okay got it thanks can I ask a question can I ask a question one last question sorry sorry guys guys guys one thing um please raise your hand uh just so that we can control um the flow of the session because a lot of people attending or type them here and then we'll go one by one uh and I apologize for kind of asking to insist on this role just because I want to make sure that we can make the best use of the time um so um go ahead VJ and then was it VJ was it you and then after you or whoever was speaking and then after you we're going to answer um two questions from the chat and then we'll continue so does does rag some use uh use something called Data loss prevention or something to avoid data being fed to LM so is that why rxs this contest is used to kind of as a security mechanism I'm still trying to find figure out uh what's the RO of a rag with respect to context uh reducing the context sample space so when GPD 3 was released there was a limit on the prompt uh that you could send to llms so it doesn't reduce data loss that is the job of the vector database it will replicate your datab base if there is a server if the server goes down for example what Vector what Rags does is it will give you the relevant information to answer the question to reduce the context length and then then you could send this reduced context lent and the relevant one to the llm to generate a response so that is the primary advantage of uh using this system otherwise let's say you had 100 100 documents and the context length would be too much it would also incur you a lot of cost to because for every API call you would have to pay money and you want to just send the relevant one that will help in answering the question okay so reduces the context size yes okay okay um there are a couple other um good question that I think is worth maybe talking about um quickly from chaali she says are you using Vector databases only with llms or also for knowledge graphs uh you don't Vector databases has many applications it became popular because of llms but you could if if you if your uh use case is just simple search semantic search without generating a response you could just simply use a vector database you wouldn't want to use it with the [Music] llm depending on the use case if it's a search or a answer question answer system that you are trying to build okay and then there's one more question um so I think um sati's question maybe two more okay one is from suti can you share um why generator and retriever might use different algorithms uh I didn't get the question do you want to explain it um maybe a miss heard something be because I think at the beginning somewhere you know you're talking about Retriever and I think was Generator you said might use different algorithms so I was curious as to what is the intuition behind that yeah so uh there are different embedding algorithms so how you convert your text into a uh how do you convert it into an embedding layer and how the llm was trained it could be different uh and we wanted to so uh let's say you created a you created some embeddings in your vector database and now you are using a different embedding for your query and there has to be they both have to integrate with each other that's what I meant that if you if you used ABC uh embedding and for the prompt and another embedding for that stored in the database there could be some mismatch and we want to avoid that okay thank you still not 100% but I think that's enough for now thank you okay thank you I'm going to ask one last question um and then we can continue right so the last question is from Hara asking what is the architectural decision point where one decides that we've reached um the limits of prompt engineering so we need to kind of look into Rags as a value addition like or how is easy is it like basically I think at a very high level what Hara is asking is like how do you decide um like what are the criteria for deciding okay now I need to use drags versus not using rags and just like kind of including it in the prompt um yeah that's that's the high level question curious to know if you have any any Frameworks in mind on like how do you decide okay now we need to actually use drags versus we don't need it now so what is a because uh uh the response quality of llms is not good enough and you want to provide some context to get that response and that context could be uh in to increase the relevancy of the information that you are extracting that's when you would so instead of sending all the 100 let's say you have 100 documents instead of sending the entire Corpus to llms you would want to just choose choose the top 10 results that would help answering the question that would be one use case the other other one would be privacy and security and you might not want to use open air you you might want to use your own private model and that's when you would use it with rags to uh provide the more to increase the recall to recall of your of your system so I'm sure in the course you have learned about recall and precision and increasing the recall out here would be important in that case would want to use rxs yeah that's coming up actually um in this current week so um it's going to be on one I think people haven't so I I think some of these Concepts would be too bit difficult to Gras but I just wanted to give a short introduction and uh if there is an opportunity next time I could dive a bit more deeper because I'm trying to cover as much as I could to yeah help you understand the ecosystem all right so let's continue I think given I would love to answer all the questions there a lot of questions coming in and Rags are very important topics are we're going to have more sessions on this but for now let's continue yeah so this is a simple uh architecture so in this architecture I'm not covering topics like llm Ops uh and experiment tracking Etc so you look at the architecture you have your textual data audio data images multimodel data and then uh there would be a embedding layer that's going to convert all this in a vector representation stored it in a vector database and now you are the person sitting in this blue box you have a query your query goes to the embedding layer gets the relevant information or chunks out of this embedding layer and then the prompt plus the context is sent to the llm and then you get the response back at a higher level this is how a rack system would look like but there are many other moving paths like how do you monitor the rack systems how do you evaluate the rack systems Etc which is not part of this diagram but it's an important role as a product manager that you have to uh keep into account so the first step of course once you start building such a system is uh it would be pre so for this presentation I'm just talking about textual data but it be something similar for audio or for video or for images so the first step would be processing the PDF documents you have PDF you want to convert these documents into text and you have different Tools in Python to do this second would be you want to handle multiple Pages it's not just a single page you would have a corpus of documents split into different pages and you want to extract the different sections and paragraphs you want to extract the metadata which will help in answering the your query or filtering the results out and uh so in the third Point garbage in is garbage out you want to clean the data you want to remove I not mentioned here but you also want to remove any bias in your data because that would impact the final result if there is bias in your PDF files and uh especially in regulated Industries that's important and you also want to detect the land of the documents so I have uh I'm based in Europe and I work with lot of multilingual documents so this also becomes important it's not just uh monolingual document you have to tag your reserves with the language Al so tag the documents with the language the next step is of course ingesting this data in your pipeline so this is like any other data engineering pipeline if you have worked on data engineering applications uh I think it's something similar it's along those lines so first step is splitting your text into smaller chunks and there are different types of uh Splitters there is a character splitter and a token splitter so character uh so character the issue with character splitter is that it could increase the token length but it gives you very granular results and if you use a token splitter you would you already know the maximum length of tokens that you would send to llms and you could just split it based on the tokens but the disadvantage with token splitter is you would miss out on some information and so you have to take into consideration both of these topics should I split it based on the characters or should I split it based on tokens so if it's something very easy not so complex you could use a character but it's something more complex to use a token splitter uh so the factors to consider is the cost the latest iny and speed and uh the computational resources now the next part is the generator so once you have the retrieve documents you are going to send the prompt plus the context to the generator or to the LM to the llm so you embed the users's query you convert the query into an embedding and that embedding plus the context uh that's retrieved from the vector database will be uh part of the prompt you pass the query embeddings to the vector database to find and retrieve documents that best match the query context you create the prompt based on your use case use call the llm function and then you get the response and you gener and you show it to the user now as product managers I think what's what would be most important for you guys would be how do I how do I decide which retrieval model should I choose because there are several retrieval models in the industry some are some are classical machine learning models and some are new deep learning models or Transformer based models uh soan I'm not sure have they covered the Transformers so far or they're just getting into it um okay I think that's part of the week one so um some people have started some people haven't um but I think yeah you can just like kind of mention that it's one part yes technology enables things like GPT for now and then they can dig into it more later so there is there are algorithms like DF IDF which gives you the importance of your importance of a term in a document and you can use that if it's something simple but if you have very complicated documents especially legal documents you would use a or a medical document with hundreds of pages you would use a birlik model to convert a text into an embedding layer and the key considerations would be uh the domain I think this is very important because different domains would require different types of retrieval models you you wouldn't want to choose a general model for let's say a legal use case or for a healthcare use case you would want to have a healthcare model that's maybe that's available or you would have to build your own one and uh you want to choose a model that can grow with your the apps volume and user base so any questions I could answer [Music] so all right so um there's one question that I thought was interesting uh maybe worth kind of quickly talking about and a lot of questions are coming in was um does open AI AP open a apis let us Implement rag implementations through assistance API um or yeah I think maybe you can kind of talk about what's available in the open AI context with with with regards to RADS if these two are dependent not dependent open open a has released its own Vector database system so you could use it but uh I think the thing you have to consider is the cost the cost of using open a system and choosing and the scalability of open air system I think uh it would be very expensive if if you're if the number of document or the volume increases to a substantial level it would become very expensive to use that so that would be a key consideration got it okay let's continue and then uh towards the end I'll I'll cover more questions actually one more question from chali that's interesting she says do you use open- Source German llms or are you creating custom L llms for your German customers uh I'm sorry I can't talk about about that okay perfect that's fine that makes sense all right let's continue okay so the next uh as I already mentioned about splitting your text into chunks and the two methods are character level splitting and token level splitting the advantages of character is it maintains highest level of detail capturing the capturing the small nuances challenges it could lead to Long sequence length and exceed the token limits uh the the advantage of token level is it stays within the limits of the token limit provided by open AI but it might miss out some on some information and there is of course hybrid approaches you could for some use cases you could use both character and token level splitting uh and this is of course a strategic decision you have to do lot of experiments figure out when should I use what uh now the once you have uh clean your data set you have convert converted your PDF files to text and you removed all the noise from the data set uh you have to split this document into smaller chunks and then these chunks are going to be converted into an embedding layer and that's going to be stored in the vector datway so the important question is how do you choose uh the chunk size and there is no uh right answer for this there is no one answer for this it depends uh in your it depends on your domain uh for some you if also depends on the use SC if if you just processing emails it would you could just uh it wouldn't matter what is your chunk size but if it's something more complex you want to provide granular information in your context and then it becomes very important uh so large chunks reduce the number of items to process improving efficiency but possibly diminishing the relevance uh so it depends the optimal chunk size depends on the source text characteristics desired retrieval performance and system resources and in my experience it you have to really you have to experiment with a you have to experiment a couple of times to figure out the optimal chunk size so this is uh I have struggled a lot on my use cases but uh the only answer I would say is set up a experimentation phase where you go through where you experiment and figure out what would be the ideal it's it's like a it's like hyperparameter tuning uh in the context of machine learning so are we good on time I have a lot more to cover I'm not sure if I can finish everything oh that's okay I mean if we run out of time then we could cover it in another session if needed so um I think it's okay to go kind of deep and what you're going over and yeah cover so I wanted to cover the the challenges and uh I think I will skip a few slides so initially I spoke about getting uh you you're converting the text to chunks but one more way to improve the performance of racks is using metadata so metadata could be title author keywords language date Etc so when you get a query it might you might want to filter out the results and then uh trigger the Retriever and usually most Spectre databases have have support for have support for storing the metadata and then you use this metadata to reduce the information that you want to look for and then run the retriever to get even fine-tuned results and then the same process follows that you get the information uh you convert it into you send send the relevant information to llms and get the response uh now let's talk about the challenges of racks first is like I mentioned if you have noisy data it's going to affect the performance of your system and you have to spend some time cleaning the data set figure out depending on your domain you want to uh set up a process where you say okay what are the rules uh According to which the data set would not be considered valid is there a bias in the data are there any misalignment or are there some cultural issues that you want to avoid Etc uh I think we covered the model compatibility uh the next one is scalability you want to keep your index updated all the time you want to manage computational resources for large scale model deployment then I spoke about evaluation you want to it's difficult to evaluate rack systems because uh there is no Universal benchmarking data set it depends on the use case and and the domain uh you want to integrate industry specific terminology in the systems and you want to secure your data for Effective fine tuning uh now I want to talk about one of the challenges I have faced building system and it's called it's a very uh there's also a research paper on this topic it's called Lost in the Meddle so let's say you retrieve a big text and now the system usually suffers is that uh the it has been found through experimentation that the most relevant information is usually at the top top of the text or at the bottom of the text and not in the middle so anything that's in the middle gets lost so one of the solutions is to reorder reorder the document that you place the most the least the least similar one in the middle that's not going to help you answer the question and the most relevant and the least relevant would be in at the top and the bottom I'm not sure if I'm going to fast or no or it's the concepts are a bit too advanced so you could let me know what do people think let us know what do you guys think um I think one thing that people are asking is um if we can share on the slides with them after word so I can coordinate all that stuff with you all um but uh yeah go ahead Kristen I can see Kristen's Kristen's hand is up maybe we can let one or two questions hi um thanks for doing this there's a lot of valuable information in here um I I do feel a bit lost with all of the content and I wish we could have a couple more hours to go over all this but one IDE Vian is I know you have a lot of these Community sessions um maybe you could think about for every week that we go through the content um which sessions that have already occurred that we maybe should watch once we have that more of a baseline understanding which ones are more relevant to the content yeah IDE we just had this session next weekend after we review everything for a week one it would have more impact got it got it thanks so much for that feedback that's very helpful okay yeah we can do that so what I can do from my side is um those of you in the cohort you guys have like a notion page that includes all the previous Community sessions what we can do is kind of break them down into Community sessions that are relevant to the particular week um but to be honest the community sessions are ongoing uh will always happen so there will be every time we have like new cohort members joining in yeah the first two weeks it feels just a little bit uncomfortable for them because they're kind of like jumping into the lake and learning how to swim uh but uh point taken and I'm going to try to address it as much as I can uh using the existing content that we got sorry go ahead I see chali has a question uh yeah I think uh one question what I wanted to uh ask you is uh regarding the rhlf right where you are employing human in the loop and also the reinforcement learning algorithms the policy kind of algorithms across I have not personally used R rlf uh not be easy for me to answer that question okay and also regarding the pricing of this uh Vector databases yes uh I could send some links on that how to choose the right vendor and uh the price of different vendors there is a nice website but I don't have it with me right now so I could send it to ban later on sure thank you so I spoke about lost in the middle uh problem and how do we improve this there is uh one technique is to reorder the text that you put the most relevant information at the top and the bottom and the least relevant one at the middle and that could already help the llm answer your query in a better way and if you know of of FL chain it's a popular library for llm use cases they have an inbuilt long context reorder function that you can use to so you don't have to build things from scratch there are a lot of libraries providing functions out of the box there is another labrary called Hast stack and it provides two popular functions that I have personally used diversity ranker and lost in the middle ranker so what it does it it ranks your document based on the relevancy of the information that will help in answering the question and now uh this is going to be actually uh I just want to tell you that I am just covering 1% of the entire ecosystem so whatever you have seen so far it's just a tiny uh introduction to this topic but I tried to cover as much as I could and I hope I have not confused you all but in the last section I want to just cover some Advanced rack techniques that I have been using and uh like we said we could all again revisit these topics once uh you get a high level picture of of this so there are like as I told you there are two ways to retrieve information there is a semantic retrieval information and the previous this one was keyword based so now the question is is it possible to use both of them or should we only rely on uh semantic relevant uh semantic relevant retrievers and the answer is depends on the use case so I have used both in some use case use cases I had to use keyword base searches when there is some when the application is a bit simple or it's a very domain specific uh knowledge base where you have certain key terms and it would make sense to use those specific key terms to search for information and if it's something very very complex you have hundreds of pages then it wouldn't make sense to use keyword based and in some use cases you might want to use both you might want to combine keyword-based search algorithms plus uh semantic based search algorithms and there are different techniques to do this I am just covering four popular techniques the first one is concatenation of the results from the keyword-based Retriever and the semantic search retriever that's popularly found in Vector databases then there is the rank Fusion algorithm and there is a popular paper on this topic so you uh rerank so you get the results from both and then you rank uh uh to prioritize which which one you should use uh then the third one is merging by scoring so you get the relevant information from both and then you rank and you give a score that will help in answering the question and based on the score you can uh choose the one that's giving the highest score so what are some advantages of hybrid retrieval it enhances relevance it provides more coverage so you get more relev you get a diversity of a diverse set of information it's more resource efficient so keyword-based would of course be less expensive uh you wouldn't have to use any neural network models to implement a keyword-based filtering approach and depending on use cases it's adaptable so combining both in some scenarios gives better results than just relying on one uh yeah this is another topic so I could take some questions if there are all right are there any questions so there's one question um that has been asked is um are there and maybe you can kind of provide this after the session they're asking if there any additional um reading material or links that you would highly recommend everybody to take a look like is there any sort of white paper for example you think is great by the way a lot of people are also messaging me saying that the session is great so looks like what you're sharing is resonating with the audience so I don't I am I I could write a book I have plans to write a book on these topics but I have just been experimenting and learning so I have not referred to a book but uh I could find something yeah it's all my experience in the slides okay you should write a book n sorry write a book yeah we'll offer it for free to the cohort community members and F exercises will come for it please do it Mirage we would love to have you publish the book um I think there are a couple questions are coming in um this is an interesting one and maybe you talked briefly about this um what are there any unique success metrics that you have to pay attention to when it comes to rags or is it fair to say the success metrics or more or less the same as um you know your model if you're not using rags I'm trying to make my H's question a little bit more generic that's mostly recall and precision so you you would if you have the capability to create uh F the labels of factual information and then comparing the response of llms with the factual information then you can calculate the accuracy of the system but this is just one metric there are some more advanced metrics and I'm also researching on this topic so I'm not an expert I'm just a curious learner yeah awesome that's great thank you uh let me go to the LinkedIn page your qu because I think there are a couple questions that are also posted there um maybe I can answer them so um to be honest this part never nav feel free to add more clarity I'm not super clear what you mean you're asking the vctor database and embeddings are language agnostic and so is open Ai and llms in general then why does the language detection needed and at what stage don't quite follow the question but in case you did neash go ahead and answer it otherwise we can move on so you so like as spoke about the metadata and uh you have a query and let's say and and your Corpus is made of both German and English documents so if you have the language you can filter out the results to look for information only on those German documents because that that's stored in the metadata and then you find the relevant information in set of German documents that's uh why we need the landage identification but if if it's just English documents you wouldn't need it that's right okay perfect uh chalia I think your question is more generic outside rack so if that's okay I'm going to kind of hold off on it um okay F I'm going to answer your question offline um TR to scroll up a little bit quickly see if there's anything else that we need to cover um is there ranking problem so okay har says so there is a ranking problem in the rag architecture too um okay I it looks like we've covered most of the key questions that are posted um please let me know um if anybody thinks that we haven't covered any particular questions but um we'll probably have another session at some point we're going to do a lab on um Rags we just have to kind of make sure that it's simple enough for everybody to be able to go through it um but uh yeah so we'll take care of that um offline and um I'll keep you guys in the loop um but other than that is there anything else that uh we need to cover near or I can have like a couple quick announcements on our side yeah yeah there are I wanted to cover some Advanced uh techniques but I think it's going to be too complicated for now yeah so we can stop out here I think okay perfect uh so what we could do is we could uh share the slides afterwards um and uh quick update first of all thank you thank you so much NJ for coming in this was really amazing this was very productive and um really speaks to how fortunate we are to have you at our community for everything that you've done I we really appreciate it um for those of you who are interested um next Saturday we're going to have one of the AI product man ex AI product managers at Google who's now working at source graph coming in and talking about um what they're doing with regards to um with regards to coding assistants coding co-pilots uh with their company I think the platform the the tool that they have the AI assistant is called Cody and that's pretty popular um theyve raised a lot of money they have a pretty strong user base uh so that's going to be very interesting session um and uh one more kind of quick uh shout out if anybody is think about um signing up uh to the next cohort we've already started accepting applications I we're not replying yet we're just like currently very focused on the current cohort but uh if you're interested just use the link that we submitted um in the content comment section you can use it to apply for the next cohort and we'll get back to you shortly um other than that uh for those of you that are going through the Saturday session of the cohort U we'll see you guys in a few minutes um and we'll wrap it up pretty quickly thank you so much really appreciate you coming yaj this was very productive very helpful um and uh have a great day everybody