LlamaIndex Webinar: Graph Databases, Knowledge Graphs, and RAG with Wey (NebulaGraph)

Key Takeaways

all right okay hey everyone uh welcome back to another episode of The Llama index webinar series I'm super excited uh today to help host way from nebula Graf we'll be talking about graph databases nebula graph and how this affects the ongoing retrieval augmented generation stack and so these is you know everybody's using Vector databases and and llms um how do knowledge graphs and graphs kind of play into this world and how can they help you create augmented retrieval uh in synthesis with your QA and tripod application um so without further Ado I'll pass it over to whey thank you Jerry and I will share my screen okay so so we can see it right yeah okay thank you uh thanks a lot uh everyone and I'm super excited to have this chance invited by Jerry to share something around what um we had been doing around the graph related rag uh in Lama index so this um and our slice is actually uh in public this URL so if you want to check back and forth you can yeah you can do it um okay so um we will have uh this schedule um in three part so I will quickly go through the overview of the the graph the background and then I will give a brief introduction on what we had done towards llama index we add some Azure uh abstractions to enable the graph on a language module um you know development and finally uh we will have the QE session or with GRE so we will have some more interesting things in in the third part so stay tuned okay so um what is graph I think most of us had already know I've quickly walked through so the the term of the definition of graph was uh initially triggered and mentioned during a study of a old question called Sandbridge problem so that is a old city in Europe so at that time there were Seven Bridges there across uh the rivers and that makes a second mission of land in in a couple of different parts so the people are questioning that if this is there a traversal pattern that we can walk through all the lands without repeating uh work on each Bridges so um and when we were dealing with this problem we finally will abstract uh you know the the bridges into uh connections and the ends into a DOT or a Vertex or node so this is the ultimate form of this problem in Mass and but um why do we need doing uh such you know studies on the graph actually graph is underneath everywhere in our daily life so and one of the use cases called Knowledge Graph and the the term of Knowledge Graph uh was initially uh invited by Google so they were they want to improve the um the search readout and for example if we search MCU nebula and we can see there are cars here so this information are actually some form of knowledge not just you know assembled from the pieces from different pages uh directly so it's actually backed by a Knowledge Graph so for example if we uh search like um Karen glance age yes we got the answer of her age so this apparently cannot be done uh with the you know the Sorting on indexing of existing documentation so it's also backed uh by the knowledge of the reasoning capability so yeah but as you know we we have a layer called not a graph database so why do we need such yet another database so um most of the arguments are related to how we want to retrieve data like uh normally in graph database we have some specific DSL to query the you know towards the connections for example we do a two hop queries so we can do something like this rather than you know the the curse of the nested c chord drawings but this is just you know just the inch just to make your hype happier so what makes the difference why we finally need some uh such database I want to you know add this um in appropriate metaphor to you know show um so imagine we were doing this all the ritual games called uh snow brothers and uh we I I want to make this uh we make her think of this is the um sorry so this is the uh traditional tabular database so we want we will control the brothers to you know throw the uh snowballs to the monster and every stage think of every stage as a different table and when we were doing the you know graph query a typical graph query like traversal from one node to another so that's a underlying a join between tables so we have to walk a jump and and run towards the um monster and that's typically what uh a tabular dab is doing but uh in uh so we can actually add some new things different things to you know run faster so like creating different indexes uh do a lot of magical things on the tabular storage layer but uh it's not changing the the the fact that the drawing is expensive so in case I will use in our use case we want to you know jump a lot of uh Hoops a lot of joints so I want to make the matter for that graph database is doing something like this green uh green bottle where we can actually fly um you know so in that case we can fly from one point to another in a very cheap effort and in graph database we can consider this um this drawing actually in almost or one effort so it's an improperative inappropriate metaphor but I hope it makes sense of why we need a graph database so uh another question is why do we need a level graph so this is the project that I was were mainly working on him at daytime so it's an open source project so uh I want to quote uh one of the uh from our team from AWS so um so imagine this problem that we have a shape um in this small River and if we want to make it you move forward we just want have one guy uh you know to push it yet but uh similar question uh similar problems uh comes in different scale can you know have different totally different solutions so think of this shape it's called Evergreen so if you recall it's a you know a big ship uh block the whole um worth uh transportation for a couple of months so the problem can be solved in the uh quite different ways so never graph this project is a you know designed from day one to be distributed to be targeting to solve the you know the hyperscale graph problems and we are open source projects so when I was asking the users why they choose to use snaprograph and most of the answers are related to Performance and the skill yeah so yeah we quickly uh finished the marketing part yeah we come to the second part so um it's about the graph stores it the story started from the graph Source in in Alabama index so when our background is called the the rag the the main topic that we won't cover today so the basic flow of the retrieval argumented generation process is we want to uh ask the language module to help us finish some sort of task based on some existing data that the model was in the learned from so uh the general approach uh here of the rag is to split the data into different chunks because we cannot you know send everything to language module and we create the semantic embeddings per each chunk and index term in memory or in Vector database which will enable us to you know do so this is during the uh build time and when we are doing the query time we want to solve our problems we can make our task to generate the embedding the semantic embedding of the test as well and compare it to you know like top K related chunks of this task and when we get like top two of the chunks of the data close to this question we send them together as a context to the language module and that's what we were doing uh in the most of the cases which worked uh quite well in most cases right so but so as a as a graph guy you know we were playing graphs every day what we are thinking here is our intuition here is uh we are dealing with knowledge but knowledge is actually a knowledge graph is a refined version refined form of the graph and it's also a fine-grained segmentation uh it's a fine green circulation of the you know the data or information and it's also enabled interconnection between the entities the you know the small segmentation of the knowledge so maybe it can make some difference so this is you know hypothesis and uh maybe we can try we can try to bring the knowledge graph in this stack in this uh Paradigm so another thing is uh building such Knowledge Graph was relatively expensive and involves a lot of expertise um so maybe we can you know get help from the language module as well so with this ituation uh we want to make some you know bring something uh in in this workflow and uh if we look into this uh data part and if we extract the knowledge the knowledge graph from those information we can see yes um they are entities or terms per each chunk and uh if so imagine if our test or our question is related to um [Music] one piece or or two pieces uh of uh for example X but if we are doing the you know the traditional Vector split uh uh method so maybe we we cannot cover all the information related to x uh in like top three or top five right and on the other hand uh if we abstract the you know after we abstract the knowledge uh what we have in Knowledge Graph we have the connections between the terms and that will bring the possibility the capability of the Cross chunk context so this is sometimes matter so with this hypothesis we are doing something and and uh we try to bring the um you know Knowledge Graph to the to the picture and the our initial uh verification shows a quite uh relatively promising results and if we look back to this workflow we just get the um um get the knowledge graph the relations between the entities the small pieces of the knowledge as the contact to be sent to the language module yeah and to do that and we collaborate with the the community of llama index so we Define a new uh storage called uh sorry um called grass store and so this is the uh the the the Legacy uh Vector based uh index creation code so for example we we create one of the vector database backed Vector index and similarly we we can do uh you know quite equivalent uh pattern of the coding so we just create a a Knowledge Graph uh storage packed index called a Knowledge Graph index so with the help of of this abstraction we're not just uh you know bring the the knowledge graph to the you know rag we also uh solved another problem that we mentioned before oops here yeah the building of the knowledge graph the extraction uh you know the small information from the the text is also quite expensive so we uh after we introduced the graph store uh this step the the end to end step was all uh included in in the Llama index ecosystem so this line is uh it's just so when we are doing indexing of our data and after that we will have a up and running uh we would not just can do the rag we have a up and running knowledge graph and persist in the the graph store and here I uh we were using knowledge Naval graph of course and then this is the uh during the query time we can just use the high level API of ads query engine uh to to convert the knowledge graph index into uh you know a QA uh endpoint so for example we ask a question here and so this is the very first uh uh twist that goes well about our uh works and we can see uh here so this is the um okay this is the index creation of the graph store so we set our we leverage llama index we extract the you know the The Entity and the the relationship between the entity from documentation and persist them in in apple graph and then we do the query so what uh what graph is a sweet thing here is it bring the transparency the no the traceability and the explanation during the whole thing uh for example we can see from the the log that it abstract entities from the question so there are four of them and then it will query the uh the knowledge graph to get all those uh in maximum depth to um graph queries so then it combine all those uh retrieved graph uh structured uh context to finally Assemble post the answer as so and what's interesting is here we if we uh you know we we do the equivalent query of this entities to get the maximum uh two uh to the two depths graph query and we got a result from the knowledge graph uh yeah we can we can create other uh stuffs based on this knowledge graph of course and one of them is we can visualize them uh quite easily so that's a quite interesting uh set effect of this work too so um yeah as I mentioned we uh we we want we want to improve something around the rag in in llama index with graphs and we also get a side effect to you know actually build the knowledge actually there were some papers released around how we want to Leverage The you know large learning modules to change it how we uh we're doing now to doing this build the setup of the knowledge graph so this is another uh um another uh place that we can also get help from London module not just the extraction so this is a a traditional QA system called knowledge based query system so here knowledge refers to Knowledge Graph so um and the uh previously we what we are doing here is we are doing a lot of hard Works to you know um extract the the semantic information to detect the intent of the question or task and finally uh create a Knowledge Graph queries and make the null equal queries directly uh you know to assemble the final answers so this is the traditional way when we're doing the KB uh QA system and when we have a language module all of those uh you know hard Works can be you know done by just the problem engineerings and this is quite another uh exciting things we can do uh you know with uh with the graph bring in in this picture so uh we are we already have uh uh things called the naval graphqe chain in launching but uh you know to to to to be easier to you know bring everything connected we also have the equivalent thing called the knowledge graph query engine in lava index and we are still working in progress of of this PR so this is the only the you know the draft of what we already achieved so um basically uh we can we can create uh the knowledge graph as we already demoed so first we we fetch the you know the information so this is a a simple version of workflow we were doing the kbqa so we uh leveraging the Lama llama the Llama Hub so it's a part of the Llama index Community we can easily get the information from the Guardians of the Galaxy vol 3 uh film so it's my favorite uh we can easily fetch the information from the the relatively long Wikipedia page and then we create a a knowledge graph with this one line uh with the knowledge graph index and then we um okay and then we um create uh create a a query a query engine called knowledge block query engine to connect to direct connect to this graph store and then we can uh query the question like tell me about Peter Quail so this is uh and the QA question will be translated by the uh this query engine into a graph query and this is the response from the knowledge graph the naval graph in a cipher result and then we uh we assemble the result into uh into the answer of course and we can uh also query them by ourselves so we do the exactly same query and this is the result and we can visualize it so it's quite interesting so it's the so this is even easier for user to understand the result uh you know the the answer of this query actually from my own opinion yes I think um that's all from the first two parts yeah wait this is the this is fantastic um and I think the slides are up so if you want to take a look uh please feel free to go through and I know there's a bunch of stuff in there to go through including links and notebooks that you can play with and so we're super excited about the upcoming integration with graphqa with llama index uh so so this is awesome um let's jump to just like a few general questions and we'll um you know leave some room at the end like five to ten minutes for questions from the audience as well this is a very basic question um but going back to this like overall idea of graph databases and ignoring outlines for now um what are some of the real life practical use cases for graph databases and how how do they compare to for instance like a SQL database object store or like a vector database in kind of like the traditional uh data setting oh yep so uh actually um in the graph Community we we uh we are withdrawing a lot of different new use cases you know from different domain uh Industries they're leveraging the graph capabilities to cool stuff but there are already out there like around 20 I would say a state of art solutions to Leverage The the capability of the graph so for example we uh we can do um edit mapping so if you have a user system so uh you want to identify some of the user are actually controlled by uh you know some of the username are actually controlled by one person and that can be done with graph uh beautifully so for example fraud detection if you want to um uh this hacked you have a user content system of fintech uh e-commerce system you want to detect the pattern of a certain fraud so most of them can can leverage some sort uh from graph for for example and so I won't walk through every slice here but it's example so if we module uh this problem in graph and we can do simply query to know you know this this is a one person or one device related to a bunch of different uh actions so this can be in a higher risk so this is a query based problem solving so another uh pattern would be um for example yeah I can I can demo one of them uh for example we can leverage the graph algorithms like uh we can so we query some of the information so in this this is still a fraud detection problem so we can do uh like a luvin algorithm so this is the algorithm help to detect the you know the clustering of all those components the entities so here after I run this uh luvin algorithms it will so in a visualize sense it will color the things for us but underlying their marking different entities in different labels so these can be a great input on a next hoop of the pipeline or you know as your one of your feature in your machine learning learning more juice because the clustering information makes a lot of Senses in these problems so for example we can further run a page rank algorithms so that's uh you know that's a simple and old algorithm helped us to to find the the nodes and or entities importance information so this is also very uh import uh input so that's uh this is just you know one uh small uh view of all the use cases uh so maybe another one yeah we can also do the graph neural networks towards you know data persisted in graph databases but uh GN graph neural networks not necessarily no you know to be collaborated with graph database but with the graph database something can be done differently let me find that one um yeah so this is the example that I I create a demo project to leverage uh GNN and the Never graph to to do a real-time fraud detection with GN so we trained offline we we trained the module uh in GN the module will help us inference uh inference uh which node is with higher possibility to be risky and in the real time case we uh we have a transaction to our system we insert them in our graph and we in real time we can query the subgraph like uh in in a couple of hundreds of uh main Ms and then we put this like three thousands nodes subgraph to our uh module that we trained before so this can be done in real time so uh so the ground app is in this phase enabled the the real time of this whole whole workflow so uh when we come back to the the question that you know the differentiate between the different type of um of that basis I would say um in some cases they they have the overlap for example we can in theory persist you know the structure of the graph in in tabular database we can do that but if our use cases are related to you know multi-hoops or ambulatory uh connection between two uh entities so it's really hard to be done from with SQL and when our data is you know in hyperscale and you are doing in the in a high uh concurrency fashion so it cannot be fulfilled in a tabular way so um uh another case but when we want to do some analytical analytical thing or machine critical transaction uh tasks we can somehow do in in graph database but uh it's not that perfect uh you we we have the consequence we have the trade-off to you know persist data in a connected way but we are not you know in a in a bad way to do in a tabular query so speak of a similar applied to like um object uh a store and the vector database so Vector database is it excels at you know the the top key embedding searching so the embedding to me even uh let's back to the language module uh landscape so the the embedding based uh the distance searching uh performs much better than the you know the graphic the graph based uh semantic query uh because you know by nature some of most of our you know simple or a few short questions are related to information data are aggregated together so in that case if we sepulate them into you know small uh triplets so that's not the the best fit for that but uh in some cases we can you know maybe we can uh you know can buy them but uh we are still uh you know we are still discussing and uh exploring that yeah so uh awesome that was a really detailed answer and actually following up on that last point about Vector databases and graph databases so now kind of going to the llm use case of retrieval augmented generation where you know by default I think most users these days are probably uh adding a vector database with an LM to build uh for instance like an lmqa system or chatbot what are your thoughts on the pros and cons of graphic databases and Vector stores and could you elaborate on some of the trade-offs a little bit a little bit more um and then the second part of this question is are they mutually exclusive uh could you actually use potentially both of them um to create better retrieval over your results for llms yep um yes um so I think uh in in this domain so the the semantic the the vector-based uh database or a store is excels at the semantic search when it comes to no you know gathered information uh in the in the information spread uh but when we were doing the you know the distance search space searching we are actually uh you know compressed the the complex uh form uh or structure of the information into you know one dimension so that is the you know the closeness of the distance so this will somehow uh in our uh intuition uh lose the information of the structure itself so we consider uh as we know the you know the word is structured in a graph way so some certain uh sort of tasks or questions you especially in QE uh case could benefit from the you know when we Leverage The the graph structure so uh and the second question is are they uh Mutual exclusive and uh can can they be used together yeah and I think you already know the answer the uh they are not and they can be uh put together so um let's uh look look back to the reg process so what if we uh put uh the knowledge graph based retrieval together with the vector based retrieval so um as we already shared in in the Llama index documentation uh we can see so here we we leveraged the beautifully uh abstracted customer retriever um from Lama index so we uh we we finally create a a custom retriever that will uh retrieve both uh the vector-based rag and the knowledge graph based and finally we combine the two and our result is showing that uh for example we are querying one question that we uh in this case we we pass the data no we we which therapy data that's created newly after the GPT training so it's about the the science events in the Wikipedia uh in this year so we are asking about the NASA so tell me the science events about NASA and we can see the vector store Works excellent and it comes with a uh some fruitful and correct answers uh but still here we found uh the knowledge the purely Knowledge Graph based rag can abstract smaller pieces of the information that's not covered by the the top K uh uh searching of the vector base so after we combine two we indeed find the answer is some somehow uh better and in this case the the actual efforts uh with the you know the the graphics are actually introduced it's just like around seven percent of more tokens so this is just a you know not that uh a full evaluation but it brings some you know promising direction to us because you know we didn't fully exploit the the K the K Knowledge Graph right yet we are just doing some keyword stream matching based and it's ours is already showing some results and on the other hand the pure uh the pure kgr AG is also interesting so uh we we think in the future we can somehow um could put the tool in in in one View and to you know help fulfill some uh some long uh or different difficult uh QA tasks yeah awesome um and and the follow-up question to this actually uh which I'm very curious about um do you think there's plans to integrate for instance uh embedding Vector search into a graph database natively uh and then two is do you actually think there's ways to improve the way we embed things uh by leveraging relationships because right now you know we've got in a betting you just take the text trunk and then you get generated based off of that uh you were talking a little bit about like graph based like GNN based models and I'm kind of curious if you think there is uh Advanced representations and vettings that you could generate through through leveraging those relationships yes we are thinking we are thinking about this too because uh we are trying to bring the the graph database as the memory layer at the Precision layer here so after we persist most of data in the graphs uh Central centralized fashion so that will bring us the ease uh to make the graph aware embedding also uh feasible but we didn't explore this for now yet but uh we because we consider you know the rag is the the is the low hanging fruits about graph plus larger than modules but in the future we can explore like the node two vector or even GN based because we have everything connected already so maybe that kind of embedding can benefit some certain cases because the connections were considered in that case yeah awesome and the the last question I'll ask is so what what's next in your mind for uh this world of llm's uh graphs and retrieval augment to generation I'm sure you have a bunch of ideas for future work uh and I'm curious the Curious offline sharing with the audience on some of these ideas yes uh so this is a very this is just the very beginning of what we have been doing uh you know in the community with youth uh Simon Logan so um in our mind we have ambition to you know fully uh explore the knowledge graph in many ways so so this is just some uh some in iterations in our mind so one of them is we can actually we can do some uh because uh you know in ilama index for example we have some sort of uh like flare or uh uh guidance based uh sub question abstraction so we can do different fashion of the uh change of thought approach so you know to break down the the long uh tasks or long queries so um maybe somehow because we can Define some you know the proven approach of those breakdown patterns and that pattern can be you know by Nature fit in a broad fashion so if we put those uh you know proven patents in graph so maybe combine the every uh those existing uh effective patterns so the the breakdown itself can have some hidden connection so maybe we can have some results on on this part so uh another is uh uh we are thinking of uh well when we were talking about our clothes to our uh uh Partners or users or customers so they were talking about some some uh extremely hard uh long tasks like for example you give some contacts you want the language module to help you generate a real real world novel so they were uh talking about and referring some of the papers that we need to somehow persist the the memory structure some in in some way so and this memory structure cannot be in a simple structure so to me that will you know at least in the first step to fit into a graph pattern so maybe we can uh after we introduce the graph store we can do such uh case uh tasks as well so uh the final one is when we were talking about some real world problems in in their reg applications that in real real world problem we cannot fit one single you know query engine or retrieval approach to fit all the the problems so maybe we can introduce different existing knowledge graphs as some domain specific query Target and you know to get work together with some other uh you know maybe some of tasks are related to some chart building or analytical things so that can be where to different table uh rig so uh Etc so uh now we uh we are actually uh drawing sorry uh so this is just a initial draft that we were thinking maybe a quite relatively uh complex uh rag workflow with llama index so for example we can create a alarm index router in the in the beginning to detect whether it's a long question or not so in case yes we can do the uh the breakdown so for example we can use the uh the guidance integration of the Llama index and when it's break down into the the short question so we can define a router to to fit different like your case when uh rule a matching role so but and that can be you know routed into different uh Target uh query engines but finally we can create a fallback row and in this four barrel maybe we can do some more expensive thing and we can put all the uh the approaches we have in in a row that maybe the cheaper one will be in the front and we do attempt attempts of the queries and every step we will revisit if if it can reach a threshold that we can return this this query or we can move to the next one and we can finally cherry pick some of them or combine them so this is the you know just a initial draft that you know leverage the graph is is one one part of it uh the but with the orchestration capability of llama index and so this is the what in in our head now and on the other hand the final one is uh we are thinking our ambition was to bring to break the traditional ground database capabilities we can introduce something more for example the uh uh the vector search in in in abilograph and uh then we can make the connection between the trunk The Entity uh in in one picture and we can search them not by uh the key matching or a full text search also by the semantic embedding based search so that's ambition in our hat and we're trying to make them you know soon to the Llama index Community yeah awesome um this is this is uh great I I just want to make uh some time for for two key audience questions uh so first question is how do you think graph databases fit within conversational AI engines where uh context like short-term memory is important um and so in a conversation where you want to maintain kind of short-term memory I think uh trollio responded uh that uh with an idea that you could add a timestamp to each node and Edge and then somehow like filter by recency when you have actually do graph based queries but I'm also curious to hear your thoughts but yeah that's a uh that's a great question um we didn't explore this yet uh but uh it's a great idea that because uh never ground have have the capability to module the temporal graph so we can do the temporal graph visualization and the queries so yes we if we uh bring the time step into the picture we can do a lot more precise or uh you know more complex tasks that's you know concerned the time Factor yeah that's a great idea cool and then the next one and probably the last question is um uh I think an audience member asks is entry ql um when you when you show that example of uh the llm powered like you know graph based QA is the nvq of sentence generated by the language model and maybe the higher level question here is like can LMS generate uh graph based queries yeah it can uh and as I showed uh in the Llama index query uh engine uh it can help us to generate you know you just dropped a neural language uh uh and you you can get the uh the query uh Cipher query itself for you we didn't choose to you know uh the background here is never graphs of course both ngqa is our Preparatory query language and the open Cipher so open Cipher is like you know more popular open uh query language so we choose to uh generate the open Cipher one yeah for now yeah makes a lot of sense okay well awesome I know we're a little bit at time uh and so I want to thank way for uh taking the time to speak with us and share this fantastic presentation um we'll share this recording on YouTube and also the slides as well uh probably on LinkedIn and Twitter uh and so yeah thank you all and Happy Friday and for those of you in the US happy July 4th weekend thank you Jerry thank you bye