Benchmarking Methods for Semi-Structured RAG

Key Takeaways

hey this is Lance from Lang chain uh I want to share some results that we've uh recently working on related to uh rag over documents that contain a kind of a mixture of texts and tables um the motivation here is that tables are present in a wide range of documents um Financial reports earnings reports things along those lines academic papers uh white papers um technical manuals so tables kind of present all over and the question is if I want to do retrieval or reasoning over tables how best can actually send these into the context window of an llm and there's a few approaches that you can imagine that have been also reported so one option is well we have some uh llms that have very large context Windows um up to 100,000 tokens or up to hundreds of pages we can imagine just taking documents that contain tables and St stuffing them in together so the text uh as well as tables kind of just just are fed in directly um there's also a set of methods that try to perform some kind of targeted table extraction we'll talk a little bit more about that later on but the idea here is you actually look at the document you try to identify table elements you extract them you pass those into the llm in kind of a more targeted way and the third which is very common for rag in general is you take documents you split them based upon some kind of uh chunk size and you pass those chunks into first the vector store with embedding in order to basically make them easily retrievable and you build an application that can very easily retrieve based on semetic similarity to your question documents and then you document chunks and you send those into the llm context window so option one just take full documents with tables stuff them into the context window of a large llm option two try to extract the table specifically option three try to take documents split them up and then kind of do chunk based retrieval on chunks that also contain tables um so to evaluate these different options we built a public Langs Smith Benchmark so Langs Smith is our platform uh that allows for uh tracing evaluation and other kinds of like observability monitoring and evaluation and what he did here is um we took four earnings reports um from data dog Microsoft uh Google Amazon from Q3 2023 so they're kind of recent they're kind of of of some interest um and two random white papers the white papers just contained a mixture of tables and text and we can see that in particular the earnings reports contain lots of inting tables and they're typically pretty dense you can see some examples here of like kind of segment information as well as um Revenue operating income from Google on the left here um we built 30 questions from these documents and lsmith allows us to basically automate evaluation so we have all these question answer pairs that we've built 30 of them and we have our different approaches for reasoning over these do documents be it chunk based rag long context LMS um and what we can do is for every question we can get the kind of chain generated answer we can compare that to the ground truth answer you can kind of see this happens at stage three um and then lmth is very nice cap abilities to compare those results across many runs debug what's going on and kind of root cause um so that's kind of the approach build a small Benchmark 30 questions um and we're going to take our different approaches for rag across tables and run an evaluation on this data set um so let's look at stuffing documents into long context llms first um what we see here is the uh y- AIS shows you the percentage of questions that are correct and the x-axis shows you different approaches so we tested Claud 2.1 and gp4 128k now these documents combined were not that big they're around 36,000 tokens so they don't go to the full capacity of these models which are around 100,000 128,000 tokens but it's still reasonably large it's not something you could just stuff into like for example Baseline GPD 4 um depending on the model you're using 8K 32k kind of exceeds those typical bounds a bit so you do need these longer context LMS for the for this concatenated set of documents that contain all of our tables and we can see is you can get decent performance now the nice thing is this is very simple you do almost no work and you can get for example with GPT gp4 72% correct it's not bad especially given these are kind of very dense Financial tables now there's been some interesting work this is from Greg camand um that has shown that the ability to factually retrieve information from context of these models depends upon the context length as well as the fact placement depth um and what Greg showed is that for example as this is with gp4 128k as the the context length gets greater so like as your your passing a much longer context into these llms um there are errors um and he did this by placing a single fact in the sea of context at different locations um and then asking can you retrieve that so you can see that there are these issues with retrieval quality with respect to both the length and the placement depth in documents so that's kind of a thing to flag with these long context approaches you can see degradation particularly as your context window gets longer um as your as the the fact placement uh is more Central in the document so here's our results looking at our documents so again our context is around 36,000 tokens so not huge um but again not particularly small not easily fit into like into more some of the common llms that folks use and we do see kind of a cluster of failures kind of in the middle uh of the document and then this this is a heat map I don't show the color bar here but this dark indicates there's three failures kind of in this region uh this is one particular table pertaining to Amazon uh where we continually get a number of questions wrong a few others here so it it would be hard to make strong claims about the impact of placement depth on these errors more would have to be kind of studied there to really be certain about that but it is worth noting that if you take documents along these lines which have very dense tables you send them to a long context L you can get these retrieval failures that may or may not be easy to understand they may relate to what Greg showed kind of in depth here it may depend on the context length that you're passing in as well as like kind of where they are in the document we saw that there's a bunch of saor kind of in the middle here and maybe that's kind of a result that is consistent with some other reports showing you can get these kind of lost in the middle type effects where retrieval kind of gets worse as it enters the middle of the document but in any case I think more would have to be really kind of dug into here but it does show that you can't necessarily expect very high performance by just passing in very large context and hoping to get really really high quality question answering over tables so that's kind of observation one um now observation two is you can try targeted table extraction methods and there's a number of them um the main intuition is this is table Transformer from Microsoft um you try to do some kind of table detection looking at the document seeing where the tables are trying to extract the different columns rows and kind of pull these tables out of documents in some kind of structured way um a number of different apis doc offers some things for this unstructured offers things for this they are it's an interesting class of methods we are actually going to follow up in more detail on evaluations here we have done some initial rounds of evaluation and one of the things I can flag currently is that these methods have a high ceiling most likely it's very promising to think about the ability to Tech tables pull them out particularly for really complex tables in practice what we did see is that uh because tables represented in many different ways in different documents consistency of extraction is a challenge so for example one thing we saw in particular is that um this is one of the example tables from the Amazon document we saw that some of these methods actually failed to grab the uh the the header columns and so none of the questions could be answered correctly because the llm did not receive uh um these headers so didn't know how to reason about this information so in any case I think the intuition here is that table extraction methods are absolutely interesting to consider they require like kind of more sophisticated methods they may be trickier and they may suffer from failure modes due to the diversity of um table representations that we actually encounter in real world documents so we're going to have more evaluations on this we're doing some work with unstructured uh as well as dyami um on these topics and so uh we will provide some more evaluations here in the future but I want to just kind of give a flavor that these methods have a pretty high ceiling but there's also some challenges related to the kind of the diversity of of kind of table representation that you might see in real world documents so we kind of wanted to move ahead to chunking now chunking something a lot of people are pretty familiar with um it's been very common in rag to take documents to chunk them up because context windows are limited you can't just send all your documents into an LM at once um in for many cases and so this idea of chunking and then retrieving chunks based on centic similarity to a question is like a very popular appealing approach now here's the results and I'll kind of walk through this in a little bit more detail but we see an interesting Trend here so we start with a pretty small chunk size 50 tokens so you know four characters token maybe 200 characters and roughly here and you can see performance is really bad and as you bump up the chunk size you can see performance gets better and better and better until we get to this page splits which I'll explain later and this Ensemble thing which I'll also explain later but let's kind of zoom in on the this 50 token result and kind of get some intuition here so Greg Cameron also recently put out chunk fiz it's a really nice kind of UI just for visualization of of of chunking strategies on different um kind of input documents so I took one of our tables I input it here and then I set the chunk size to 200 characters which corresponds to 50 tokens which is our first you can see this table gets split up entirely by the chunk by the chunking approach which very much explains why you have such poor retrieval results because if you take a table and you break it up in different chunks you're unlikely to retrieve all those chunks and reconstitute them such that the LM can reason about that whole table kind of together and this is kind of the a main intuition for the chunking strategy is we want to try to design chunks such that they retain full tables now of course this is what we were also trying to do previously with our discussion about table extraction table extraction is kind of the the probably the most exact highest performance in way of doing this re actually should try to detect tables and isolate them extract them individually chunking you're trying to kind of guess based on chunk size what is going to capture tables and which which is not and that's obviously hard to do we can see in this extreme case small chunks are very likely to break up nearly all their tables and that's a bad thing of course um but one thing we found which is kind of very obvious in retrospect is that in many cases tables honor the semantic boundaries of pages which people kind of have have used to you know improve readability for documents so for example there absolutely are many cases of of tables that span many pages but of course this does hurt readability for the table and a lot of documents at least we are focused on for this noal study did not contain those kind of multiple page spanning tables and so the approach here is very simple if you just build your splits based upon page boundaries it actually retains your tables pretty well and so then all you're doing is you're taking a page embedding that page of text and then retrieving based upon that embedding um and then sending that that particular page to the LM in practice this got us 93% accuracy so quite strong extremely simple approach uh and I I completely acknowledge that this will break down in cases of complex tables that span many pages and it may also break down for other types of complex tables um that were not captured in this evaluation set but I do want to flag that this is an extremely simple approach we found it to be actually quite performant um and um it requires very little specialized engineering you know unlike for example using some method to extract the tables individually um or even like a longer context llm which is kind of not accessible to many people more expensive this is very simple you take your document you basically embed each page and then you retrieve Pages pass them to the llm for final answer summarization synthesis those pages typically en capsulate tables pretty effectively for many documents in particular the ones that we we covered here um so using these natural page boundaries actually did work really well in practice just for these kind of simple documents that we are working with um now there was a problem though we looked at some of the failures for the you know the the proty showed it got up to 93% but of course there's still some headro for improvement there and we saw this so here was one of the questions that failed and it failed because it did not retrieve uh the page with the table it retrieved a number of other pages that discussed what we were interested in so the question pertained to the capitalized software development cost for data dog and what we found is that particular statement was present lots of the text chunks outside of the page that contained our table and so what happened is when we did retrieval based on semantic similarity a bunch of those text chunks got retrieved and we did not retrieve the table and so therefore we couldn't answer the question so a very simple trick that we kind of find in practice worked really well for this is you basically kind of build two different retrievers and you can Ensemble the results and so what I did here was really simple so I iterate through each page and I built what I call kind of this this it's it's using L change multiv Vector retriever but you can think of this as like a a table retriever so what I do is I iterate through each page for every page of the document I ask an LM to tell me does this page contain a table if so summarize it and if not just kind of say there's no table present and what this multiv Vector triv lets you do is let you index those summaries in a vector store so so you can imagine what we have is a bunch of table summaries um that an llm has generated that link to the corresponding page in the documents so again it's like you have a whole bunch of summaries that you've distilled for every every page that contains a table you have all those summaries index that are retrievable based on symthic similarity and then that raw that full raw page you then can pass the LM for answer synthesis um and then of course I combine this with the original just page split Vector store retriever which we had previously and with this assemble retriever you can actually do something that's kind of nice you can effectively tell it to prioritize any chunks from one of the two retrievers so in short what I ended up doing is I built this table retriever at top that indexed all the different pages that had tables with some table summary and I combined that with a result result of just the general page retrieval and if there ever is um a retrieve document from my table retriever I put at the top of the queue in this ensembling and so what this kind of does is if there's ever a table relevant to my question I try to enforce or ensure that we return that because I want to make sure that information is included in the answer synthesis and when you do this that's how I got up to 97% uh and there still is believe this one question that it's incorrect and uh that actually had to do with the need to retrieve documents across a wide range uh of the different documents and synthesize them um and I'll be following up on that in kind of separate probably separate video or notebook that kind of just addresses how to how to deal with that case um but this was kind of the original set of results that we wanted to share um and kind of in summary you long comex llms are really nice for getting started they do fairly well to be honest um you know 72% is actually pretty good for effectively you know very little effort just taking these six documents and stuffing them in if you have smaller context like a single document with tables that probably works very well um the caveat is there can be these failure modes with respect to both like the the text length so if I try to send in more tokens like our case was 36,000 if you I think as Greg shows in these tables if you try to push up to over 100,000 you can get these this kind of degradation performance and also depending on where the table is in your context you could see degradation there's been reports that in the middle of documents does worse Greg shows actually there seems to be kind of a a lot of failure modes towards the the top of the document so uh in any case this it's worth more investigation but the point is that long long condex llms very easy to get started with have some interesting failure modes with respect to kind of where the table is present in the document so that's kind of point one now point two is that chunking is also quite easy very commonly done the only challenge is you have to ensure that your chunking strategy is not split up tables just kind of very obvious you don't want to break them up because then you can't reason about the information present in the table and you can see it these chunk sizes you do certainly break up many of your tables and you and you reduce your answer quality quite significantly in short we found is just simply splitting on page breaks uh at least retains for ourl set u many of the tables and therefore lets you uh get pretty good performance up to 93% and a small trick of additional embling just to make sure that you always retrieve relevant table chunks if they pertain to the question relative to document chunks um is a nice way to kind of boost the performance slightly further um and ensure that information from tables does get prioritized if it's relevant to your question um and that's really it I think um that was all I had for slides um and we'll be following up a bit more on approaches that involve um targeted table extraction in the future thanks