The Best RAG System On YouTube (Steal This!)

Key Takeaways

all right guys super excited to share this new rag system with you today it uses a combination of n8n superbase and postgress for memory and it is the best one that I've shown yet on this channel so it's going to be watching a Google drive folder and then when you upload a file it's automatically going to figure out what type of file it is the respective logic is going to take place for that type of file it's going to put it into your vector database with relevant metadata like file ID the name when it was created when it was last modified stuff like that then we have another workflow that's watching those files for updates if they're updated it's going to delete the original record from superbase it's going to change the version history from V1 to V2 to V3 and then it's going to reupert them into super base with the relevant metadata if you enjoy this type of content please consider giving it a like it really helps me out but without further Ado let's get into the video okay this is the workflow we're going to be getting into today we have three main parts so up here in the yellow we have the rag agent we're not going to spend too much time here if you want to learn how to set this up step by step as far as the postgress and the super base go check out a previous video I'll link it up top right now and that one will walk through that kind of stuff so we've got the agent that we'll be talking to for rag then we have this green section which is watching for files that are created within the folder we have a trigger here it's going to Loop through download those files figure out what type of file they are it'll go through respective logic to extract the text um you know optimally for that type of file and then finally it'll put into superbase and then when we're loading the information we're going to load up all all different types of metadata we'll get into that um when we're walking through an example and then finally we have in the blue the same fold folder but we're watching for files updated rather than created so this is going to obviously it's going to grab that file ID so it can delete the original record that we just uploaded in the previous workflow it's then going to set the version so in the metadata we'll have version one version two version three and so on and so forth and then same exact logic of going through figure out what type of file it is putting it into superbase with our metadata if you guys want to download this workflow go to the free school Community Linked In the description click on YouTube resources and then go to the Post associated with the video you'll be able to download the workflows right away then if you're looking to take your learning A little bit farther check out the paid Community also linked in the description we've got a great classroom here with with deep dive topics I'm doing one about Vector databases right now this resource library is going to be continuously updated every week we've got a really great community in here all your questions are guaranteed to get answered we also are doing about five live calls per week um so I'd love to see you guys in here finally if you're interested in me building out this type of stuff for your business or you're interested in AI Consulting then check out my website link down below and book in a call okay anyways back to the video real quick just wanted to give a shout out to Victor from the free school Community he helped me out with this one so really appreciate that Victor but let's get into this actual build so we have our agent we have our super base um table which is just called documents in this case and real quick we're just going to chat with this guy ask um what projects do we have in the database and this is basically going to say we have nothing so this is kind of just proving that there's nothing in there we're going to upload a project brief and then we will be able to chat with it so there's no projects in there now let's add something so um looking at this first logic as you can see we have two triggers this one's deactivated because when there's two triggers in the same workflow and you hit test workflow they can get confused so we're just deactivating that bottom one for now um okay so I've got project brief for project wheel it's just got some information in here we are going to take project wheel drag it into our folder called projects and that is the one that is being watched for right here as you can see it's called projects we'll hit fetch test event to make sure that we're getting the right project back and we should see the name of it over here right there project brief project wheel okay so let's hit test workflow and watch this stuff happen so it's going to download the file right now and then it's going to figure out okay this is a Google doc from Google doc we turned it into um a text file we did that right here we extracted the text put it into super base so let's walk through what just happened um okay so first thing we're do we're doing is just setting our file ID and our file type so it's easier to reference um for later so file ID is the actual ID of the um Google doc so we see this one is it ends in wol and we come into here and we can see that right here or actually you guys can't see it it's in the URL but this file ends in WL so that's just the file of the Google Doc that we're looking for um and then we're setting the file type which as you can see was basically a Google doc so from there we're feeding in the file ID right here so that the Google download or Google Drive download node can actually download the correct file it's going to be dynamic obviously and here's one thing that we're doing that's kind of important so within here we're setting the options of you know configuring this node and we're basically saying okay if anything comes back here that's a Google doc turn it into a text file if it's a drawing turn it into a JPEG slides turn it into Microsoft PowerPoint and then sheets for CSV and so now instead of getting back that Google doc we're getting a text file so we can view it right here and it's just the plain text and it's being stored as binary so from there we feed it into the switch node which obviously we have different options um we're basically setting the routing rules so if the file ID is this it's going to go that and that way for PDF um and then all these other options of course um and if you want to take a look for this I know I'm going to go kind of fast through these examples but the workflow will be available for download in the free school Community um link for that will be down in the description um so you'll be able to just grab that plug and play and then you will be able to then you'll be able to just look at what's going on within each node so anyways from there it goes down this path because it's a text file we can see that when we're doing a text file all we're going to do is extract from text PDF extract from PDF and then Excel we're doing some aggregation summarizing and then for word docs and sort of Windows docs we are um going to convert them so from there we are uploading that into our Vector database the only thing really that's fancy going on here is that we are um setting the metadata so as you can see we have all this kind of stuff coming through so the first one is the file ID of course this is important so that later we can feed the file ID into superbase so we can delete the doc and then upload something else in its place um we have the version like I said this later will change we have the Creator created at last modified we have the folder path hardcoded in there and then um file name and file extension so that's what's going on within the um metadata so as you can see this output two items so we should have two vectors in our super base if we go into check here we can see there we go they just popped up so we have project brief project wheel we'll click into here and we can see some metadata so what we're looking at is um the Creator obviously the file ID we see the version and then oops and then other stuff that we obviously configured in there right so that's all that's going on in the metadata and now what we're going to do is since we have that information we are going to first of all chat with our agent to make sure that it's getting it right so in Project wheel let's just ask for some key features of this project so what are some key features of project wheel while this is loading off I'll just say um we're going to be going even deeper with this sort of rag system I'm doing a deep dive right now in my paid Community about about Vector databases and rag so but anyways here's some features of project wheel data integration automated syncing customizable dashboards and data cleaning and as you can see it got that all right so that's perfect um one more thing I'll I'll show is that we have our chat history this is the postgress that we're using if we go down here to um this most recent entry we have some key features so this is what the AI responded with let me actually just make this a little bit bigger so here's the key features the AI just responded with and then here's what we just asked which was what are some key features of project wheel so that's sort of how all this memory is being stored in postgress rather than if you're using a Windows chat buffer memory all that all this kind of stuff is going to be stored in naden so that's kind of how the postgress and super base work in conjunction okay so from there let's test out this workflow and let me make this a little bigger so it fully captures everything okay perfect that was going to bother me if not okay so let's deactivate that node and turn this one on and now all we have to do is come into project wheel we will change something real quick so let's just change um let's just change the name keep it real simple project um butter I don't know why butter is the first thing that just came to mind that's probably not good anyways Project Butter um we'll go back into the folder and make sure that it gets modified okay so 312 um that means that we just modified it and it has seen this change so we'll come into the um file updated node we'll hit fetch test event and you know hopefully this one actually registers that that was an updated file so we'll test this workl flow we'll see everything happen um so right now these record IDs were okay so it just got deleted as you can see it got deleted in here we're setting the version so the previous version on this side as you can see was V1 and now coming out is V2 um downloads the file it's still a Google doc does the same thing with text and then we're uploading the metadata over here which as you can see right here the version now is V2 so let's hop into subbase is this going to automatically refresh or do I have to refresh okay I'm going to refresh we will see we have only two so it didn't duplicate them it didn't just delete them it deleted them and then put in the new information which as you can see right here Project Butter perfect and then in here we should see V2 so we have the version um also what's cool to see is that we see that it was originally created at this time um I'm having trouble understanding what's going on here 2202 and then 2212 so as you can see it's changing it know that it originally was created at that time and now we've modified it at this time it's still keeping everything else pretty much consistent but that's how that works so um if I was to come into this guy and now we ask about Project Butter what are some no I'll just say what's our budget for project butter and hopefully that works because um I changed the name of Project Butter but also within this project wheel okay so that's what I was worried about so we only changed it here but other places it's it's known as project wheel and up here it's known as project wheel so maybe it wasn't the best example I probably should have changed some actual information that came through here but that's not really the point of the video as you can see it got updated here so um yeah it worked as it should I just didn't really think through the what I wanted to change because right here it's still known as project wheel so I guess we'll just do one more thing real quick so previously we're saying that this kickoff meeting is December 1st um let's change this to December 15th okay so our kickoff meeting is now December 15th going to our Drive okay this just got updated it knows that we modified it let's go back into our agent and run this again so we will see now it we'll go to version three so we'll hit fetch test event I don't really need to do this I could have just triggered the workflow we'll test the workflow we'll watch it delete um the reason I'm limiting I'll show you guys in a sec also um once we just make sure that this is working so we should have IDs 56 and 57 in here now oh I guess it already did it too quick for me okay yeah it did it too quick for me so V3 as you can see and now we can come back into our agent and ask about the um timeline of project wheel okay what is the timeline for Pro wheel it should understand that and we should be getting back um December 15th is the kickoff like we did so December 15th December 18th and January 12th okay so now that we see how that's working and we see that um you know we're getting the version changed and the modification date changed let us look at what's going on okay this is the execution of the V3 um project wheel that we just did so we're downloading the file um we're grabbing the file ID that we set here of course so we set the file ID the file type and then within sub base we had to do sort of a little filter to make sure that we're deleting the correct item so as you can see the operation is delete we're deleting a row within our table called documents and then we're selecting a string and this is kind of the filter we're doing so it's metadata um and then the dash with two greater than signs kind of just signifying okay within the metadata what are we looking for which is a file ID so back in here this is our metadata right and then within the metadata we have we have right here file ID and then we have the file ID so we're just defining file ID equals like Dot and then within two um asteris we're doing the actual file ID of course and so then once we do that what we get back is two items because there were two vectors right so in this case it was only two vectors if it was a longer document and maybe depends on the way we were splitting it and you know chunking it obviously but it's likely going to be more than just one vector that's associated with a single file so in that case it would return like maybe 15 items and if you did that then it's going to send like 15 items through the rest of this logic so that's why we just limited it to one so no matter what it's going to grab the first one which always is going to have the previous version the previous file ID all that kind of stuff so we're fine with that from there we want to set the version and so my first thought was okay let's do a code node just so every time it's going to iterate through it's going to increase the version by one and when I tried that as you can see on this left hand side version is a string so you can tell obviously right here there's an A so it means it's a string which is just a word um but if you look up here like lines is 40 and one and there's a little hashtag meaning that this is a number so just a different variable type variable type so when I tried to do it with a code node it wasn't updating the version accurately because you can't do math on a word right so I realized okay well maybe I could try to in the code node convert this to a number variable type and then it could do it but I was like okay well we might as well just grab a 40 mini or you could probably even use like um even even a cheaper model 40 mini is pretty cheap but then I just basically said your goal is to take the incoming version number and add one to it because it's going to be able to understand what's coming through so the example if you get V1 you're going to get V2 if you get V4 you're going to give out V4 or V5 and I said always output it in a field called version just so when we're referencing it later for the metadata it's consistent um and then I just gave it the incoming version number down here so I just drag that in there and that's all I did within this node so once we have the version um and we still have the original file ID right so we're getting this kind of stuff but then we can reference the file ID from back here sorry it keeps jumping around when we're in the execute or yeah execution um then we just do the same exact thing of downloading the file we're going through the route of course and then we're inserting it into superbase and then within this superbase um or sorry not superbase within the the document default within the data default loader um we're setting the metadata and everything's the same the only thing we changed obviously is that the the version we wanted to reference um whatever we got from that open AI node that's called set version we're just grabbing the message content called version and so then here's what's getting put into our um super base as you can see the version has been updated to V3 and then that's basically it okay sweet so now real quick let's just activate this one and turn this one back on and we'll just show like a PDF running through so I'm going to go to our Google doc which was Project wheel going to download this guy as a PDF and then we're just going to drag this into our projects so we got that in here we are now going to just test the workflow and we should see that it's being downloaded obviously it's a PDF and as you can see now it went to the router of a PDF we extracted the text from this binary and we got the same exact information back and now that should also be in super base as you can see it just pop through and this is Project Butter and this was a um a version one because it was a new thing that we created right so as you can see it's also the file extension PDF whereas the first time we did Project Butter it was an application Google doc so yeah awesome that's how that works um I didn't want this one to go too long I kind of just wanted to give over give a brief overview of what's going on here um now expanding off of this right so so like I said in my paid Community I'll probably be building off of this and talking about um making things a little more production ready and one thing that I'm starting to play around with is how would you let's say we came in here and we deleted this right we deleted this file we no longer want this project nothing's going to happen it's going to stay within our super base because we haven't set up any logic to handle deleting stuff and right now it's no big deal right because we know these two were the PDFs from this path and we can just delete them but when you get to the point where you have multiple documents and a document has 10 or 15 vectors rather than just two you can't be doing that so um basically the logic that we're going to be going through is we're going to be running one one flow that's going to grab all of the file IDs right it's going to grab all the file IDs that are in your super base and then we're going to have another one that's grabbing all of the file file IDs that are within this drive and then you're basically going to merge those together and check which ones don't have matches and then any of them that don't have have that can't join we will feed into a delete superbas node and we'll feed those file IDs into there and this would be some sort of scheduled trigger it could run once a day it could run every few hours just to check it's basically just comparing right um and yeah so that would be a really cool way to pretty much really make this thing automated but for now that's all I've got um like I said if you guys appreciated this video please give it a like it really helps me out the link to download it will be in the free school Community down below and um that's going to be it for today so really appreciate you guys making it the end of this video um I'll see you guys in the next one thanks