LlamaIndex Webinar: Active Retrieval Augmented Generation

Key Takeaways

all right uh welcome everyone um this is uh one of the first webinar series on kind of exploring uh retrieval augmented generation from the research side of things and today I have the pleasure uh of uh being joined by Jon Bell Frank and Louie uh co-authors of the paper actual retrieval augmented generation uh otherwise known as Flair and we're super excited to host them and I'm super excited to talk about you know both how kind of flair helps Advance this uh world of long-form retrieval augmented generation as well as talk about some of the current limitations uh and uh future forward-looking directions of you know Rag and llm based retrieval in general so pleasure to meet you guys yeah yeah nice to meet you awesome um cool so maybe let's uh to kick it off um maybe like you guys could just give a quick overview of uh kind of like the context as well as like the the overview of the paper itself around active referral augmented generation and I'm not sure if you guys have uh slides if not that's totally fine but if you guys do maybe that could help uh you could guys could help share the presentation as well yeah I do have a slice so maybe I can share my screen sweet well I will let you guys take it away and feel free to give an introduction as well yeah so thank you for for inviting us to join this online session and I'm very happy to to have the opportunity to talk talk about our recent words and also talk about retrievable new generation and also long form generation in general and I do have a slide so to give a brief intro to myself I'm a pH student from CMU working with ram and I'm currently working on mainly working on retrieval augmented models and also knowledge intensive tasks something like question answering and also you know long form generation and uh yeah Luis is also working on similar topics maybe he can also give us a quick introduction yeah so uh hello my name is I'm also from Carnegie Maryland doing also doing my PhD and I welcome retrieval and also some basically I work on the intersection between retrieval and language model generation and in general to basically improve one with the other and things like that yeah sweet yeah so I'm Frank uh I'm also a PhD student in CMU and my research interest is basically uh like machine learning language models for uh software for code like code generation and kind of stuff and I want to use a retrieval augmented methods to improve current uh ml for code models yeah cool so yeah so I think maybe I can take a few minutes to uh briefly go through our recent paper and yeah that sounds great yeah let me share my screen and then for those of you in the audience just a quick note um we'll leave some time at the end uh for questions uh if there are any uh otherwise I'm also happy to ask a bunch of follow-up questions uh and so just feel free to leave it in the chat and then if we don't get to all the questions happy to account like uh forward some of these offline as well foreign yeah yeah did you say it by the way did you say you whipped this up in the the past few hours that's very impressive sorry did you say you you put these together in the past few hours because yeah yeah that's nice yeah uh sorry for the short notice but great uh feel free to take it away yeah yes it's okay it's uh just like a few slides it's relatively easy to put them together sweet yeah so this is how recent work uh so it's a collaboration between CMU meta Ai and also CA lab so the title of the paper of our work is active retrieval augmented generation and before diving into our message I would like to First briefly go through uh retrieval augmenting generation the standard way of doing you know this type of retrieve document generation for a large number model so here is a a figure I I took from uh to from a blog post so given the initial input question which is who is the president of the US usually the pipeline is that you will send the query to a retriever and the retriever will return a bunch of documents and based on the based on both oppression and the return documents the large number model we generally generate the final answer so you perform retrieval at the very beginning retrieving all of the information you need and then by conditioning on both the questions and the retrieve documents you generate the answer in a single forward in a single forward what so this is the standard way of you know doing retrievable generation and this is pretty much the I would say the it is both standard in Academia and also standard I guess in also in industry so those type of retrieval argument generation has has been demonstrated to be super effective for uh question answering but most of them are most of the methods are evaluated on short form question answering for example in this case the final answer is just a single entity and which is true button so it's relatively easy for the model to you know retrieve relevant information and generate the final entity however when it comes to long form generation things are quite different for dump hole if we want to uh want to model the electric model to answer complex information seeking questions for example summarize the latest research work from CMU or meta research or if you want to push it further to ask the log model to generate that IC or a paper or even a book uh for example recognizing about global warming things are really complicated because for those questions or the information needs are really complex it usually requires you know gathering information from multiple sources and reasoning through those documents and generate a coherent long story uh using the line for model so it's pretty hard to to gather all of the relevant or necessary information at the beginning of the generation so this is our major motivation to propose active retrieval augmented generation so the idea the core ID is very simple we just want to perform with people during generation whenever necessary so imagine for example generating an IC about global warming Maybe by first retrieving a bunch of documents using the input question as the query recognizing about global warming you might end up with several documents you know talking about global warming and you might be able to generate the first section of the SC which is the introduction section of the SE based on those documents however when you want to generate details about this topic for example the cause of global warming or the consequences of global warming you might want to refer to other documents that are not other information that are not included in the initial retrieval results so this is our motivation to do this you know to perform retrieval augmenting generation actively so this is essentially uh basically creating this uh for Loop between Retriever and length model a launcher model will generate the output and when the electron model lacks certain knowledge about the upcoming you know generation the length model can issue some queries to the Retriever and the retriever can return with a bunch of documents which hopefully will help the local model to generate filter content so this is essentially you know creating this Loop between retriever like models so that the retrieval and the generation can interlude with each other to help generating long-form content to to make this possible we have several challenges and I summarize them in this slide so the first challenge is when to retrieve during generation this is this is this might this is actually another trivial problem because uh imagine generating an IC you know containing many paragraphs or sections is is we need a method to tell us when to trigger the table and when we want to perform a table what we believe is another thing we need to figure out what information is really needed given the current generation contacts what is you know missing from the current generation so this is another question we need to figure out before generating connections next uh next uh text and the last question is given the retrieve documents how to integrate those retrieve documents to help further generation so those are the three major challenges we want to focus on in this uh in this paper so given those challenges we propose our method which is uh which is called flyer forward-looking active retrieval augmented generation So based on this gift you can get a pretty much a good sense of what we are doing in this method so given the initial question which is generally the summary about Joe Biden uh the model will generate the output sentence messages so in our method we use sentences as the basic unit uh you know to to determine when to uh to determine the generation uh procedure so if the next test generated by the longer model doesn't contain any low probability low probability tokens it basically means that the length model is pretty confident about the generation so we can basically move forward if the next sentence contains uh low probability tokens it it's basically indicating that the length model is you know lacking knowledge so we probably should trigger retrieval to retrieve something to help a large model to get a better you know generation so this is how we decide when to reprieve we basically decide this by based on the probability of the generality tokens in this uh example in this case for the second sentence uh it's talking about you know the education history of Joe Biden and the model is uncertain about the University of the Joe Biden so we stop here we use this sentence as query uh to send it to a retriever which returns a bunch of documents and then we regenerate the sentence by conditioning all those documents and then we move forward to the next sentence and we do this iteratively until reaching the end of the generation so it's essentially uh you know iterative generation and retrieval pipeline where we use the launcher model probability to determine where to perform reproval and use the upcoming sentence as the query to generate to to perform a variable [Music] do that easy so most specifically uh regarding the three challenges uh I mentioned personally uh when to retrieve we use the upcoming upcoming sentence uh whether the upcoming sentence contains low probability tokens to decide when to perform a retrieval and in terms in terms of what you retrieve we have two methods the simplest one is directly use the overcoming sentence uh to your performance level and before doing this we mask out low property tokens from uh the Gen from the upcoming sentence and send it to the retriever to retrieve something another method is uh explicitly call a large longer model to generate the query that could potentially lead to the upcoming incentives for example because the in the second sentence the launch model is uncertain about the graduation the education history of Joe Biden so we can prompt the launch model to generate the question which could be something like you know what is the education uh which college did Joe Biden go to something like that to generate a question like that and then send it to the retriever to return some relevant documents and given those return documents we simply uh we always prepend those documents at the beginning to to to give the launch Model A Better contact to generate future content so the whole whole pipeline is relatively easy to implement is relatively uh intuitive we do this you know iterative generation and retrieval uh to make make it better at long form generation so we tested we tested this model on multiple benchmarks including multiple multi-hub QA Common Sense reasoning long form keyway and also open domain summarization and we compare our methods with several uh baselines including single time retrieval which only triggers retrieval at the beginning of the generation and also another laser model called previous window reviewable which is essentially retreating sorry I think I'm gonna mute you sorry about that yeah so another based on this previous window approval which is essentially trigger retrieval for fixed window for fixed interval for example every sentence or every for example 16 tokens and use the privileged contacts as priority so comparing to both baselines uh single time retrieval and retrieving uh information using the previous window rms's consistently outperform baselines on across all of the benchmarks could you maybe uh sorry I have a bunch of four questions but like for this maybe could you describe a little bit about like what the nature of these benchmarks could be just like uh kind of what is the form of the kind of like context the questions and and how long are like the expected answers Yeah so basically for those uh for all of the numbers I included in this figure we use automatic Matrix because automatic automatic metrics are relatively easier to you know uh to to to use so for multiple QA we basically evaluate whether the generated answer the final answer is correct or not we have the final answer uh final code answer and we also have a long uh long output generated by the model and we extract the final answer from the long output so uh to give you an example of uh model QA the question could be you know who is the who is the daughter of the uh president of the US so in order to generate the financer you need to first you know find out who is the president and then find out who is the daughter so we generate the whole reasoning process uh you know using something like chain of salt and then we extract the final answer from the output and compare it with the goal dancer so this is what we did for multi of QA for common sense uh reasoning it is similar uh so the data set we use here is called strategy QA which is uh all of the questions included in this country in this Benchmark are actually uh ESR no questions so the final answer is always yes or no we generated for this DSS similar to the multiplication sets we generate the Chain of Thought reasoning process you know the the internal thinking process of the language model and we extract the final yes or no answer generated by the model and compare it with the gold reference and quick question about the multi-hop QA when you say like evaluate the generator answer versus the actual answer how do you how do you do that so I basically compare is a stream based matching okay so you don't like because you know there's like a line of work that uses like a language model to compare the two but you just do some sort of like uh string based matching yeah I I think the way because the final answer is only a single entity so stream stream based matching is actually not that uh not that hard to do however for long form generation for example for as QA and Wiki uh Wiki uh aspect so those data size uh are basically generating longer content for for sqa is uh it's a data set uh where the input question is ambiguous so for example who is the uh most popular you know what is the most popular uh soccer player so you might have multiple entities you know you have the most popular you know uh women soccer player or you know man soccer players so you need to consider multiple uh interpretations of the uh ambiguous question and to generate a coherent you know a long answer and for Wiki SP it is a summer edition data says all of the questions are similar to the questions I have in the previous figure so you know General summary about a particular entity including you know several aspects so for both sqa and Wiki SP uh the outputs are usually much longer so for those uh data sets we use um several existing metrics to measured performance including root which is a standard metric for you know for for comparing uh long generating outputs with co-douncer and we also use other metrics like uh entity F1 which which basically extract entity from the output and compare it with the code answer yeah and we we plan to also evaluate our methods using uh human evaluation or you know the method you just mentioned evaluating using uh the most advanced launch models like GT4 I think uh most of the recent papers uh to you know a line language model is to use uh gp4 model to compare Generations so we are also planning to do that but we didn't do this in this paper because those emotions are relatively expensive we need to hire human human labelers or we need to use it before which is quite decisive for us so uh but yeah we do plan to do that in the future yeah have the budget yeah so this is a quick summary of our experiment experimental results and I have a few other directions in mind that could be uh could be done in the future for example how to develop Advanced methods to control or trigger retrieval given generation and how to make this whole pipeline division currently the implementation is uh adjustable vanilla implementation which is uh slower than only performing table at the beginning so how to make this whole pipeline they're interleading retrieval a generation more efficient is another another direction that is uh particularly important in industry and the last one is a line launch models or Phantom number models to better collaborate with retrievers or tools this is definitely a very you know Big Challenge and uh uh I would say it's still challenging for for for the most a state-of-the-art uh lunch models yeah this is a quick summary and here's some resource if you are interested in the details of the paper feel free to read the paper and you know check out the code if you have any questions I'm uh I'm happy to take any questions great this is an awesome presentation I think um yeah I honestly the it was the diagrams are super clear the examples I like the kind of like animations and the visuals too so uh thanks for doing this um maybe a very first question is so you have Flair which is like active retrieval augmented generation for a long form uh Generation Um what were kind of like the baselines before this like uh so you have like single shot retrieval and then generation uh and then if I remember correctly was there also just like a version where you just operate at like a fixed Cadence like you know instead of like looking at low probability tokens you always just do retrieval for every sentence or something yeah exactly so before this there are two baselines the first one is single retrieval only retrieving at the beginning and the second type of Baseline is what you just mentioned contributing for every sentence or for every 16 tokens or I research two tokens I think a notable paper is a cop is actually from the mind the the name of the model is called retro what they did is that they will trigger retrieval for every 32 tokens if I remember correctly so they will do this to table uh you know at a fixed interval and Trigger retrieval for you know at a fixed pace so those are the previous methods we we compare with got it makes sense and then um in terms of like the form or retrieval like what what is the input query and then how do you do retrieval is it like through some sort of like embedding based lookup uh what website the actual formulation of this yeah so uh most approach they use either the previous sentence the previous uh sentence uh to do as a query or to use the previous window uh let's say 16 tokens or search two tokens as the query and I think there are two types of retrievers either using those you know lexical uh tokens and rely on a sparse retrieval like pm25 or they use the embedding of those tokens so I think in the case of retro what they did is that they have another retriever which is actually based on Birch they use the Birch model to encode the previous window or the previous intense to generate embedding and use that embedding to look up relevant information from the data store which is also built based on vectors of documents so yeah I think in our paper we we only use bm25 uh but I think potentially our methods could be it also depends on the you know depends on the quality of the you know dance retriever if the dance retriever is uh is well trained on a large data set if the denser retriever is robust RMS can could also potentially benefit from you know using density trailers uh to relevant information makes a lot of sense I mean I guess the retrieval part is just like some abstraction that you call right throughout this like iterative process yeah exactly makes sense um moving along to you mentioned like there's two modes for Flair and I just want to kind of like dig into that a little bit more there's both like the low probability tokens as well as like having the language model generate um like an explicit query like a like basically in the output say like hey please search for something uh if I'm understanding that correctly um what are some of the trade-offs between both and what's kind of like the performance differences that that um you can observe yeah this is a really good question so I think that this is basically which one you prefer to use mainly depends on the trade-off between large language model and retriever so if the large number model is super strong and the retriever is relatively weak I would suggest using the explicit query one where you generate a query uh containing several keywords or you know a single keyword or several keywords using the launch model can issue those keywords to the retriever because the retriever is relatively weak it can it might only take you know it might only be able to understand keyword based queries however if the retrievers is strong for example if you use dance retriever which is trained on large collection of syntheses maybe you can get rid of you know the the this intermediate step of General inquiries you can directly use the next sentence uh and encode the sentence using a dance retriever to get embedding hopefully the embedding might you know could cover potential information needs that are required to generate an action test so you can just rely on the embedding to retrieve relevant information so I think this is there's a trade-off you know which one you trust more if you trust launch model more user generation you know just use a radically weak or you know relatively off-the-shelf retriever to do the table however if you can deliver to make it stronger at encoding sentences you know with potentially misleading information then you can get rid of this step and just let me players do the do the that's a good point I like the um I I like the whole uh kind of distinction between having like a strong language model uh and a weak retriever versus like you know kind of assuming the retrieval is good enough to kind of like insert a more complex query into it as well um this also might be a bit of a detail but uh did you notice a difference uh with like for instance the implicit query it seems like you're kind of hallucinating a new sentence versus like uh when you generate a new query it seems like that's more of a question format so did you notice a difference when you like try to enter something that's like more of a question versus like hallucinating like a an answer if that makes sense and did that impact retrieval performance at all yeah so I was expecting the two masters to have different performance and me as a human when I look at those you know generated queries from longer model and also the original upcoming sentence uh you know with potentially misleading information I prefer the second one which is you know generating the queries using model which because it is it seems to be more clear to me you know what the link model is looking for and we can just use those queries to to retrieve random information however in my experiments both methods perform equally good so I think uh I think it really depends on the data set if the data set is relatively uh simple maybe you know Advanced you know generating more queries using the launcher model might not be necessary but I guess uh because all of the benchmarks we use in the paper um the longest Benchmark usually contains I would say five to ten sentences uh in total for for the for the generated output which is definitely longer than you know a short form QA which usually generates a single entity or in a few entity names but it's not that long compared to further about generating assets or you know generating a whole paragraph you know summarizing uh you know a relatively complex information so I guess uh my my assumption is that if we move further to data sets where output is even longer and you know the information need is more complex using this query generation approach might give us better performance but I but currently I think this is also something that I want to mentioned in Academia long form generation data set is really lacking so it's like so we currently don't have really good long-form generation benchmarks or eventual Matrix to help us push the limit even further gotcha and and that actually is a perfect segue into my next question and I'm curious to get your guys's thoughts uh on this is well what are some of the limitations of flair you know like it seems like it's tackling a a pretty underexport problem of like long-term uh long-form generation uh what are some like the limitations that you're excited about like potentially exploring and solving as future work yeah so I would say the biggest limitation would be efficiency because this is uh basically in order to implement this you need to stop the inter generation whenever you need to retrieve something so this is I would say for uh for for if you want to treat for example in the worst case or for our method we need to trigger retrieval for everything tests if you can if the output contains 10 sentence you need to retrieve you know information uh 10 times which is uh quite a computation overhead so that is something I think is really important if we want to you know make it into production we really need to make this model efficient I think there are potentially two ways to make it more efficient the first one is uh reduce the frequency of the retrieval so there's a actually also a trade-off between the frequency of the retrieval and the number of documents you want to retrieve if you retrieve information with a lower frequency you probably need to retrieve more documents for each time you call the retriever because you need more information to generate a relatively long you know output long paragraph however if you trigger the table relatively frequently probably you don't need to return that money document so there's a trade-off between oh I see the it's something like you know Reds and depth search so you you call to retriever how many documents you want to you want the model to return you want to return to return and how many the frequency of the retriever so yeah I think this is uh there's a trade-off uh potentially we can improve the efficiency of our method flyer method by you know reducing the frequency and also limit limiting the number of documents we retrieved each uh for for each call yeah that seems very practical like for instance like if you retrieve uh a lot of documents and you might be able to generate like a lot more stuff at once uh but and then it might be a little faster but you might lose out on some like more granular information versus if you were true just like every half sentence or you know every sentence then you might be able to limit retrieval a little bit makes sense um that's super cool um uh moving kind of like a little bit more broadly and this might like talk a little bit about like General kind of retrieval augmented generation and also agent related stuff and this is something that I'm very interested in understanding is yeah so like you talked a little bit about train of thought what is the relationship between your paper uh and this whole like line of work around like agents like react Chain of Thought all these things yeah I I think there are definitely highly related in terms of uh we want to augment language model in terms of the the ability they are lacking so retrieval is mainly helping electron model in terms of knowledge because like models do not have up-to-date information about the world and other tools you know for example calculator or you know uh passing code interpreters those are helping language model with other abilities something like Mass abilities and you know reasoning abilities so I think they are basically in the same line of you know helping augmenting models with with respect to all kinds of abilities and I I think that the they are also similar in terms of uh the the way the the way the two interact with launch model so for example in Chain of Thought or in in react uh they also uh do this you know iterative uh iterative generation and two usage so current model usually take the input question and generate uh some reasoning process and based on the reasoning process the model will also generally trigger certain actions from from the two library to to help the launch model to you know determine for example what is the next what is the next object that the electron needs to generate so from this perspective perspective our message is also you know relevant to to privilege approach uh I would say actually I think in the in the future uh I think there are basically two types of uh opinions regarding this one is that uh just consider retrievable as a specific Tool uh similar to other tools and you know we can uh call whatever to use uh whenever needed and another sub another opinion is is that retrieval is different from the other tools for example if you check GT4 if you have access to gp4 and uh charging the plugin you can see that there actually they have two interface one for uh retrieval augmented gp4 which is something which is called something like uh the D4 with obviously with browsing with browser or something like that yeah we have another interface for the other tools so I think the reason why open AI did this is because uh retrieval is retriever can definitely be you know regarded as a tool but it's it's quite different from the other tools for example calendar or you know or you know something that Google or Google place those tools can be I would say integrated into language models in a relatively simple simple using relatively simple method however if we want to combine retrieval with Cloud models is usually like more complicated and also the frequency of uh using retrieval is definitely higher than the other tools I would say so I think there's also a paper called two former and they generate a data where they use uh additional tools to help launch model improving the generation quality and if I remember correctly in their data sets more than 80 percent of the of the calls to to usage are actually using retrievers so the remaining all right the remaining 20 are using other tools like you know uh calendar or you know code interpreter so got it I think you you kind of like you touched upon this point but I think it's a very interesting broad point that you know we're also thinking about a lot it's this idea of like so you said there's two types of opinions and the first opinion is like retrieval is this like a tool similar to other tools and like I assume some sort of like agent Loop right and this agent Loop can be relatively General so like a react agent Auto gbt you know any sort of like General agent Loop out there and this is almost like a kind of General AGI type interface that can take in any task and uh supposedly fulfill like an eagle and one of them could include like long form generation like hypothetically this thing should be able to do that um but then the other approach which is kind of like I guess more in the spirit of this paper is like yeah you maybe want some sort of like specific more constrained flow for some sort of like long-term generation that allows you to like generate uh like long pieces of content very well because that's like the main focus of this type of technique um and and uh what would you say that's kind of like a rough like categorization of this or they're like how other relationships between the two yeah I think uh just like exactly like what you said I think there are basically two types of you know opinions on this and um honest with you I'm not sure which one is better so I'm you know currently on the fence uh the which one is uh which one is the future or which one is you know a more practical or more you know efficient or more effective so I think that the first opinion treating retriever as a tool just like other tools is uh is is uh is better in terms of the general generalization ability you just you know uh develop this you know a general concept of you know collaboration between agents and between agents and tools so and this one can can be generalized to potentially any tool so when you have a potentially have a new retriever you know or or a new data source you can just plug them in so this is definitely easier to use uh at a larger scale with more tools however it might like you know some you might like a call quality with respect to some you know nuances for example if you want to generate uh if if the link model is lacking certain knowledge and that the model might benefit more from you know this uh finer control between Retriever and generation so yeah I think it's also a trade-off between you know how much generation ability you want to get from the from the from the you know two augmented models versus you know how much uh how much uh you know how deeper you want the model to collaborate with the with the two or the retriever sweet um and then last question before I turn it over to answering some of the questions in the chat um how like you mentioned there's some challenges in getting like a data set uh for like long-form Generation Um how would you think about getting that and how would you potentially do evaluation on on long-form generation yeah this is a very good question and also challenging so which is potentially out of my you know expertise but I I think collecting data is one thing and evaluating the the you know the output is another thing so I think in terms of collecting data sets we are there are tons of you know user interactions on the internet you know between users and activities so we can take all of those you know and potentially filter those questions uh to to develop a benchmark that is what both realistic and challenging but I I will say the the harder thing would could be evaluation so how to evaluate the the generation you know because all of the automatic metrics are currently you know fading so I think how to develop metrics is definitely a super uh important direction for the future also if you generate like a block right like how do you evaluate how good that book is but I don't know just like I'm just thinking about like hypotheticals like like you like at a certain point it might not be about like content matching but also about the quality of the writing like the creativity like all these different things too yeah um cool um I think yeah I I you know it's obviously there's it seems like there's a lot of problems in the space and this is uh great work and great step uh towards kind of like uh solving that and I'm just very interested to see how this uh how this evolves um turning it over to some questions in the chat um I think Kevin asked um so is the token probability of the generated token the only metric for kind of determining if it's reliable or not yeah so that's a good question so currently that's the only magic we use to determine you know whether to trigger retrieve or not and uh we made this decision mainly because there's there have been several papers showing that log models are relatively well calibrated so that the probability of the model can basically uh it's a basically a reliable indicator of the confidence of the lab model so we basically use this metric as a single metric to determine when to perform retrieval however I I think uh in the future there could be other methods to do this or in a more advanced methods to do to do this and I think one potential way to do this is basically if the launch model can generate search queries proactively then whenever the model generally required we can stop the generation and you know use the query to perform retrieval so that is isn't that kind of like your explicit query but you're just saying kind of more on a token mobile yeah it's similar to that but uh in our approach what we did is that we first started the next sentence and we take the sentence and ask the language model as potential it could be another language model you know to to to to generate several queries to that could be potentially helpful to refine the sentence I see I think if we can if we have that because we use a text ability model uh the the model from open I in our paper which is you know which we do not have access to their weights we cannot modify the model so we cannot fine tune the model to make it stronger at generation requirements however if you have access for example if you use open source models like Llama Or you know whatever models you have you can potentially fine tune the model to generate queries you know proactively and use those queries to determine whenever the model generated query you stop the generation and Trigger retrieval so that could be another alternative to to do this makes sense um next question is uh I guess the rewarded a bit what are some kind of like industry use cases of a type of work like this I think uh this is potentially useful in uh all sorts of long form generation tasks I think as far as I know uh both uh being chat and perplexity.ai both of them are you know using replayable to augment generation uh and based on my observation both of them only trigger retrieval at the beginning of the generation so if you check their demo uh usually they will perform for example they will trigger something like they would generally something like a three coverage at the beginning and use those requirements to retrieve a bunch of documents and then summarize those documents into you know either a paragraph or you know a server sentences I think if we want to even make the generation longer our method could be helpful to you know for example for each section or for each paragraph you can trigger the repeal methods the trigger the retriever to generate to retrieve some documents and generate the next next section or next next paragraph um I think I'll ask what's the LM used for the Generation stuff for the last generation stuff so we use uh text DaVinci 003 which is uh uh an instructory P model from open AI that is aligned using you know human feedback so that's the model we use and potentially our methods can be also applied to chargeability which is uh called GP turbo in in the you know in the open AI uh model Hub so the reason why we didn't use strategp is because first of all chargeability uh didn't return probability of tokens so he only Returns the generated tax he didn't he didn't return the associated toggle properties however our methods relies on the token property so we were not able to you know Inspire experiment our method on strategy but uh if you have access to the Token but I guess in the future charity uh as long as chubby uh released uh can can return the probability of the tokens we can also test our muscles on our charity model sounds um and then I think Andrew asks what retrieval method did you use uh and experiments I think you guys mentioned it was like the M25 right yeah so like what you said uh we mainly focus on this you know introdu iterative uh retrieval you know interacting between one generation so we just treat retrieval as of the Shelf model so in our paper pair we just use BMW file over Wikipedia documents as the as the as the search engine we also we actually have two coppers one is a Wikipedia Corpus and the other one is uh using uh being search API which will potentially retrieve documents from the open web so but for both of them I didn't use flavors we we use a spouse retriever makes sense um so now as we probably have time to get through the last three questions uh how can we integrate this with our own um search results that may include filtering or faceting uh uh I mean maybe something like if you have like personal data sets or personal virtual coppers um yeah I'm actually not sure uh maybe if you'd be able to yeah okay yeah I think uh if you have a personal data set uh that you want to augment like model you can just you know take the message and replace the the index the retrievable index with your own index and all the other things should should remain the same you potentially you do need to modify the other components and you should work out of the box however I'm not sure about the performance but the whole pipeline can be one you know by just replacing the index with your own index sweet um Clint asks how do you determine the threshold of low probability tokens I think that's a good question yeah that's a good question so this is indeed this indeed requires uh some you know hyper parameter tuning so my observation is that first it really depends on the data set for some data sets uh retrieving with a lower threshold uh it's better so it depends on the retrieval frequency for some data sets you know retrieving more frequently is helpful so for those students as you might want to set a relatively you know uh a relatively higher threshold to trigger retrieval more frequently however for the other cases where retrieval you you do not necessarily need to perform retrieval for every sentence you can set up set a relatively low uh threshold uh to to make the launch model retrieve less frankly so in our case for for most of the model we we use uh we either use a 0.4 as the threshold or a 0.8 as a threshold so that's like two two different settings two different trade-offs yeah I feel like once you have a top K you might want to just like hyper parameter out of tuning on like a training set or something just see how it was more value yeah yeah if you have uh data set during the parameters you can definitely do that but we we didn't we we didn't have the that amount of uh because we use exactly model which is quite expensive so you know uh exhaustively you know uh uh iterating or all of The Branding sessions settings might be too expensive for us so we just I ended up using a few settings that looks good for us sweet and then last but not least a question from Kevin how does the retriever handle cases where there are multiple sources of Truth so if does it pick a single source is there a mechanism for that yeah that's a good question I didn't manually inspect those reviewable results I think in in most of the data sets we used we mainly retrieved something like three to three to five documents for each for each query so I guess for some cases the document model might take information uh mainly from a single document for the other cases the the launch model might combine information from multiple documents so I think both are possible but I to be honest I didn't check you know those uh those return documents I think also depends on the task you know if the task requires uh combining information you know summarize information from multiple sources to model might summarizing you know information from all of the review documents however for the other cases the model might just take the uh take the answer or take the you know useful information from a single document yeah and the burden is probably partly on the retriever too right I mean like if you have a bad retriever it's just not going to give you the right results but if you have something that can execute and retrieve like Dynamic numbers of documents and then that's great for you yeah um sweet okay well I think that's it and um honestly yeah thanks so much for for a great presentation um I personally learned a lot I hope this is helpful to the audience as well and I'm super excited to you know have this reporting uh be shared and and um yeah uh thanks so much again uh for for coming yeah thanks everyone Thanks for for having us yes thanks for having us see you thanks everyone bye