LlamaIndex Webinar: LLM Challenges in Production (w/ Mayo Oshin, AI Jason, Dylan from Starmorph)

Key Takeaways

like um quick quick and all on form I I figured we just go through um do some intros um do like the panel for uh you know 20 30 40-ish minutes and also takes some questions from the audience so if that pops up um let's get started all right hey everyone uh welcome back to another episode of The Llama index webinar series uh today super excited to have some leading AI Educators and content creators join us for our panel discussion on llm challenges in production um featuring uh Mayo ocean uh Jason and also Dylan from Star Wars AI um so all these uh everyone here today has a huge following and has basically inspired a lot of different uh developers to help start building LM applications so they've seen a lot of different use cases so I'm super excited to ask a lot of these questions about understanding some of these use cases and and really understanding what are some of the most like Salient pain points that users are facing today and also discuss uh and bathroom ideas on how to fix some of these issues either today or also long term the tooling that we would need to build to help solve some of this um so without further Ado let's get started and maybe we can start with some some intros and I'll pass it over to you May yeah my name is Mayo um I was an early contributor to Lang chain it's quite quite uh influential and pushing out Lang training the early stages uh in terms of the chat with data movement um so far I've reached 3.5 million people through my content this year um and um yeah so since then I've kind of branched out um to start my own Consulting AI consulting company and uh you know being fortunate to our train do a combination of training development with different Enterprises worked with leaders at BCG PWC our University of Pennsylvania um yeah it's been very interesting to see the challenges uh actual businesses have with integrating AI versus just demos that we see on Twitter so looking forward to this conversation passing it over to you Jason yeah hi guys everyone um my name is Jason and uh my background is actually a product designer and product manager uh so I provide the least technical here but I started getting into AI development and larger language Implement uh or the end of last year uh and also in the meantime star like document about the interesting project of building uh that's how I started the YouTube channel a couple months ago called AI Jason um and uh so in the meantime I also doing some Consulting business at on the side as well where there are a lot of requirements from SMB and also business that they want to do some internal automations through AI as well and recently I also joined a company here called randoms AI where we are building a no code platform to enable people to build AI Automation and agents so directly to discuss those challenges we face with you sweet um Jason's also joining us from Australia I think and it's 2 A.M where you are so really appreciate you taking the time to constipate with us um passing it to you don't hey everyone my name is Dylan my background is in graphic design product design and front-end development mostly and I always like to play with kind of new experimental Technologies and when I saw that ML and AI was kind of weaving into web development I got really excited and just started experimenting made some YouTube videos of different repos and um and yeah it's been really growing into you know a lot of people have a huge level of excitement with all the new capabilities so just been building with a lot of uh small businesses and kind of collaborating and and helping um integrate the AI into the platforms they already have and Jerry thank you so much for having me on I'm really excited to to be here with you guys great um well let's kick it over to the to the questions um so just to start with what are some of the biggest use cases that you're seeing today um I think these days a lot of both developers and also companies are building different types of L1 based applications but we see a variety of different use cases and also a spectrum of complexity from simple to kind of more complex and autonomous so structured data extraction summarization conversational chat Bots autonomous agents um like passing it over to you guys what are some of the biggest use cases that you're seeing and maybe we can start with Dylan sure I would say I would say it's definitely evolving and of course the space is changing really quickly and there's uh so much great content and Frameworks coming out that I think almost every week There's a different thing that people are you know really interested in exploring and kind of pushing the frontier on um but in general I would say that rag is definitely a really common thing that um small businesses are interested in integrating and I think agents people in my experience find very conceptually interesting and I I'm definitely bullish on it long term but I haven't seen it being used as much personally as as raj methods so far uh Jason yeah um so there are a lot of use case and I think the hard part is figure out what is the real deal and versus the ones that people are playing with and might not consistent um so the ones I try to see is what other you so I have different people and team try to build the solutions uh the ones that I see people actually have high retention and continue using it are kind of two so far one is the data classifying and characterization uh for example if you are a support team or research research department for Consumer products they normally have a huge amount of feedback they need to categorize which is normally down manually by people so the electronic model is really great at just helping them categorizing all those all those data at scale very easily and on the outside it's not just for research and support I'm also surprising to see almost a one use case every single venture capitalist tennis is like categorizing startups as well so I was helping a few we say almost every visiting to have received hundreds of startups every movie weekend months where they need to categorize whether they survived the category of stuff for them to focus on because data is often outdated so data classifying and categorizing seem to be one big one but apart from that uh there's there's also a lot of use cases regarding the uh those kind of hyper personalized content generation because uh because of this structural data extraction and summarization capability uh the uh they can actually set up a very great automation flow uh for example uh a recruiting agency they can upload a resume of people extract core information compare with relevant job posting by the very hyper personalized email to the hiring manager why this person is great fit things like that so these two use case I see uh are actually people using it continuously and but apart from that there are like didn't mentioned not not retrievable it's a quite a big use case um but it's also really hard because everyone has simple mindsets as I just want to run GP on my files but it's never that easy so um and also have a question is like does for those use case uh do they actually got end user adoption a lot because I know I totally understand it but does the end user actually use this it's the question I do have um but apart from that one I personally most interested is like AI autonomous agents and as they also mentioned there are a lot of challenges there but it really changed the better population once it actually achieved um the the quality bar uh the two big part I faced with one is like how can I get the agents to actually follow the instruction I give um especially in terms of um Can it ask for human imposement and know that when it should I found that's really hard and also how can I dedicate it to do things that he think he can do without those tools especially like uh knowledge one of them or content generation um yeah right without fixing those two I found it very hard is that a is the Adrian use case a use case that you want to see happen or something you're you're seeing happen across like different companies that you're working with yeah so I actually have you uh people who want to build uh for example for the content generation as example uh for now they do they have kind of linear workflow just to if they put in some contents in and then generate something out but ideally is they will want to have a almost AI marketing intern where I can you can just do the research about what's the training topic for my industry do the research gender content but without uh without the agent actually follow the specific specific rules or or workflow and also without it to actually ask for human input properly the result is just not there so I see the use case but if those two things are fixed I see I see make sense um and pass it over to you Mayo yeah can you still hear me yeah yeah um so I I come across more I'd say traditional Industries kind of reach out to me I'd say um I don't have um and so most of the use cases they're interested in it's optimizing the existing divisions they have so we're talking about customer support uh sales and marketing um HR which he just mentioned that's it that's actually a use case I've come across in terms of just um personalizing the recruitment process um and I'll highlight that word I'd say that's really one of the things they're looking for is personalization um and personalization because of just the fact that it's higher conversion rates because the business doesn't really they don't really care about the technicalities in terms of advice and agent or whatever they just care about the business outcome and so from their point of view um you know I'm seeing basically things that would usually hire people for but that they're looking to automate that with AI so you know for example how do we you know use natural language to extract information from our database right um that's something that would be you know actually I've seen quite high demand for there's obviously the chat with the documents thing it's it's been it's it's still um of interest but I I notice where companies actual smbs and Enterprises um or medium-sized businesses they don't just want to chat with their documents they want to have an integrated solution so they want to not not only extract information from the documents they have but also their database and also maybe they've got a CRM right and and it's it's this kind of um combination that I see that um you know there's a lot of work to do in terms of solutions um I saw um the CEO of uh for sale post recently about how how do we the challenge of combining rag with you know kind of tapping into the database those are two two different approaches and then you know Jerry you've also put our stuff in fine tuning so I think it's the question really from a lot of companies the use cases are um designed to replace people or to optimize the existing processes through tapping into the apis database existing documents um sometimes they have multimedia uh stuff as well um and then crms that's a big thing too a lot of companies have so I think from that lens that's the way they're looking it's how can we you know improve ourselves and marked in from 10 conversion to 20 how do we you know save costs and recruit recruitment um that is going to be a big um uh I think moving forward for for AI to kind of break into society beyond the first wave we kind of need to figure out the marketing angle of how we position things because just saying agents that doesn't that doesn't mean anything to them but when you say this is going to help you you know double your revenue or your bottom line or whatever then they're they're more interested in what we have to yeah no that's a that's a great Point um I think you you bring up a few points one of which is basically this idea that um in the end businesses just care about like improving business outcome or improving efficiency reducing costs and so how do you position the use cases that doesn't matter if it's rag or agents towards actually solving some of those um and in terms of like use cases uh We've mentioned like um you know there's like classification uh to rag to agents um to kind of uh like personalization um so uh it's great to get it back from these use cases uh that you guys have brought up um maybe shifting gears a little bit um now given some of these use cases and given some of the people they've worked with um let's talk about like the core of uh what the webinar is about today uh which is like like challenges right like challenges and developing um these applications uh today um and and both the Prototype experimentation phase uh and also as you try to iterate and try to productionize some of these apps um and so for you personally as well as like some of the uh clients and customers that you've talked to what are some of the biggest pain points that you're seeing in development whether you face them personally or have seen other space and obviously you know there's like a long laundry list of kind of like different categories of pain points um but just to name a few of examples like there's everything that I've heard from hallucination and accuracy like this lack of like principled uh knowledge uh to like difficulty getting human labels the scalability costs latency security uh and and much more um and so maybe we could start with uh with with Jason uh yeah definitely so uh one big problem I actually face is exactly what Mayo mentioned uh is for knowledge people uh they often don't just want retrieve one type of data as multiple types of data uh it's from the if this use case is simple as help DOC website it's fine but they normally have mixed up the helpdoc website to a lot of PDF and inside of PDF it also have a bunch of other different type of data inside PDF like image diagram charts and it just failed at the very beginning in terms of data extraction there's no very clear good solution at that point and then when we move down to the data extraction part they're also quite a bit challenging in terms of retrieval because even though we get those data out properly uh literally different ways of retrieval for different type of data if the CSV data it won't work with the normal Vector database and for certain type of questions Vector search is just not not answer keyword search actually works better so with all those complexities in terms of the end user questions versus what we prepare at the beginning that's quite a huge gap here um as well together to get rag really work well and apart from that um also the yeah so the patient use case and personally have very interested about but I just couldn't it's very hard control the control of the agent Behavior at moment um yeah we'll be keep pretty Keen to hear other people if you guys explore that agent Behavior control if you've got any success in terms of getting it ask people's input as user input that would be uh that's also very big problem I faced with too sweet I know we're talking about what kind of tips and tricks and just a bit um but what's like the um like you mentioned like CSV files like how do you how do you process CSV files maybe just really quickly uh that's like or anything just think about CSV file it also have multiple different scenario like if I want to create a few short prompts let's say the the use case I was helping the other day is uh they want to create a larger base about the best practice in terms of customer support so they want to create the SSV file is this is a list of a customer email I received and this is the best practice answer so for them it's like you need to create this database but the where uh where you have two columns but in terms of the query and the retrieval you actually just want to search for the customer email because there there's quite a big difference in terms of a customer saying I love your product versus a perspad thing I love it product it can be very different scenario so for that you have a specific tab set up but if it's a other type of data like the financial reports then it's uh into setup probably that it needs to know the metadata of this one is is financial report so refresh real methods need to be different like it were do more kind of code interpreter way use a traditional SQL or or you can get it to write python code to do the retrieval um yeah so it's there's no clear answer I thought and that's actually the hardest part um yeah and the the only solution I found so far is you can't have an agent layer there to almost like classify the question and decide what are the what are the database and what's the retrieval message it can use and then and then to retrieval um but the problem there is also in to Define uh make a trade-off between the latency versus quality um got it got it um uh Mayo passing it to you yeah I mean aside from the obvious ones who lose this nation latency um I would say the biggest issue from from a development point of view is just the decision tree of choices is just so much like it's like you know you've got so many knobs you can turn and for each knob it's a different outcome so for example you know um you know what if you're going to go down rag you know what chunk size do you use what you know what how do you split what fact store are you going to use what um uh you know method of retrieve or you're going to use metadata filtering and then you have the whole discussion of well the data structure of the metadata attached to the embeddeds right you know what does that look like ahead of time I've had situations where we've built our solution for the client only to realize that we had the wrong data structure for the match data then we had to go back and do the re the embeddings from scratch so I'd say the whole you know thinking the whole thinking about the the optimal variables or the optimal parameters to get the outcome you're looking for is is definitely a pain point and even prior to that I'd even argue that a lot of a lot of clients themselves just don't have good data or they don't have good data structures to start off with you know they don't have an AI issue they have a data issue and so the question is do you assist them with that or do you kind of suggest to them then look not every use case at this point would given you know you don't have a certain data structure uh maybe feasible for AI and I think that's another discussion how everyone everyone wants to it's a great AI but maybe some companies just don't have the data you know quality required to do that so I think that's a huge pain point because some companies just that data is just all over the place their apis all over the place and is it your job as a AI consultant or developer to go fix that data I'm not sure that's what we're there for so that's interesting I I kind of wonder if you could elaborate on that a little bit because I I figured one of the interesting things about lens is the fact that you the kind of like bar to um improve like data quality is actually maybe a little bit lower because you can just dump like plain text into the prompt um I figured because and have the owl lab figure out what's going on um what what are some of the complexities you're seeing in terms of data quality and then how would you recommend like some of your uh clients or customers actually go about like trying to clean their data for for uh to make it more suitable for LM use cases yeah I mean for example you can have a client that comes and they've got you know a database of uh you know valuable information that they currently have paying customers for and they want to create a chatbot or you know interface for that database the only problem is that their API schema is all over the place right um not only that they're just not following the best practices of apis and how to kind of store stuff in the database and then expect the AI to be this magical solution when in reality you actually have fundamental issues fundamental software engineering issues right so you have you know those kind of use cases you have people who want a uh you know to integrate AI with the the documents but you know as Jason was saying for example maybe you discover oh half of these are OCR right half of these are scant documents so how you know or they've got images or they've got tables you know tables in the document they want you to extract so it's I think it's it's it's yeah it's those kind of um requests where it's it's kind of like it's a combination of data um traditional machine learning data science then they want you to add the AI I I think people just kind of lump everything together and assume you're supposed to do all of that you know I see gotcha yeah if it makes a ton of sense well thanks for sharing and uh passing it to don't I definitely agree with what's been said so far um with yeah with the data and I would say in one word to kind of summarize the a lot of the pain points that I see in development is integration across different fronts and this is something that both Jason and Mayo already touched on integrating with I think we have we have a lot of new experimental technologies that have high potential coming out but they're kind of separate from like you said the traditional machine learning or software engineering systems that we have and we can see the potential of okay if we can integrate these things we can make a really powerful system but there's a little bit of a gap between the the data that's at other companies the um the API the the already coded software that's at the other companies and then the functionality that the AI has or could have if it's correctly integrated with those so I think you know things like uh the Llama Hub connectors there's a service called psychic.dev that integrates with apis there's a lot of kind of almost micro service tools out there to create these to bridge these gaps but 100 agree with um mayo and Jason when they're saying that it there was kind of a wave of excitement of the magic of what these tools can do but it really does take a lot of time to integrate the potential of these systems into already existing systems to make it actually functioning to a point where it can perform reliably to um to service the business's needs because you can you can load in all your documents into an AI and ask a question and it will retrieve something but if you need it to retrieve a very specific set of information um then I think you know what we're getting at earlier with the kind of CSV or the database where you need a more structured approach I think that's still a system that is being explored and um yeah different so for me that kind of ties into another thing which is the retrieval and the structure is a huge part of the integration that's a challenge right now uh for me at least and to answer your question earlier uh Jerry on that I I've kind of been slowly I've always been a JavaScript typescript person but I'm slowly becoming more and more interested in Python because I see a lot of the structure and reliability there and you know things like pandas um and there's some there's some great streamlit uh tools for at least entry level CSV interactions um so I think yeah that the structure and integrating both with the business and also like uh Mayo is saying the the data and creating the the reliability and the structure of the data to parallel the the AI systems to work with that is definitely a huge pain point and then just one separate last point that I'll say is I think another challenge and this is something I've been working on with star morph is figuring out how to turn this challenge into like a strength of the um of the company and I think this is possible for especially small businesses and startups is just how fast the landscape is changing and how many new tools are coming out um that yeah figuring out a strategy to be able to kind of iteratively integrate and and add new repos and add new functionalities and be able to be really agile in this fast changing software development I think is another another challenge that a lot of people are running into got it so you're saying like just because of the rate at which tooling is coming out people have to stay on top of everything and then you have to figure out what's like the right abstraction lateral I use these different tools yeah I mean yeah having I've I've probably I'm just throwing out an estimate here but working with I don't know I've probably spoken to maybe 200 different organizations and worked with um 30 and around 30ish like closely and just even doing the maintenance on you know that many software platforms when new models are coming out um you know like the versel AI SDK came out a few months ago and that's a thing there's big there's big updates coming out that sometimes it's easier to almost just switch to a new platform than to try to integrate all of the new updates coming into a current platform so I think the maintenance and the ongoing development with the abundance of tooling even just keeping up with all of the the news it can be I hear so many people say like it's a lot just keeping up um at some point you have to kind of check out and just focus on a few things because there's so much happening right well yeah thanks for thanks for the thoughts I I want to touch on one thread though pretty much I think all three of you uh roughly touched on um with respect to something around like there's some aspect of data parsing uh structured uh and and there's some aspect to that that's tricky um because it seems like if that data is really messy or you're parsing it the wrong way or you're structuring it the wrong way and you're kind of maybe implementing the bad uh uh sub-optimal retrieval algorithm on this you're not going to be able to get the best results especially if you're trying to build a ll map um so maybe like really diving into that what really kind of um are the key complexities and parameters that you're seeing people struggle to figure out uh when we talk about like data structuring is it really about stuff like uh trunking is it really about stuff like adding metadata to it um and what parts really contribute the most to like sub-optimal uh retrieval um and and maybe we can start with uh you Dylan because I mean you were talking about like data structuring Integrations that type of yeah um I think I to be honest I think Mayo definitely is the expert on this and um in my and I think that I'm still learning and that's a that's a big pain point for me now is having the structure and the ability to really fine-tune the different parts of the system but I think I think all of the parts of the the system matter depending on um the use case whether that's you know the chunking or I would say definitely the retrieval method is maybe one of the key ones in my experience um and just really having the the reliability and figuring out what the exact use cases and how you need um how you need the data to be that's going in and then what you need the output to look like and being able to reliably create the that kind of like we said earlier personalized system for that use case I think that something that's helped me is looking at it in a very modular sense and trying to have kind of portable modules for each part of the process that can be augmented for different projects because maybe you have one use case where you know one llm um is just a little better suited to handle that or you want to chunk things a certain way or maybe you're doing more of a summarization and it can be a little more freestyle and creative and you don't have to worry about the reliability so I think the pieces that need to be augmented depends on the use case and having them all um available in different environments and in different yeah in different kind of portable modules that can be integrated uh quickly and flexibly is kind of how I've been approaching it makes sense um Mayo did you want to show some thoughts yeah so you mean in terms of like the main culprits of of the issues uh yeah yeah basically just like I mean data like yeah uh kind of really diving back into that spread of like data quality and then also parsing yeah yeah this okay so let just hallucination for example in my experience all it takes is for one retrieved chunk to be like not accurate and it can't excuse the results right um I've seen the difference between the performance of the output let's say it's a five out of 10 result and then we increase the uh threshold similarity threshold and because you increase the threshold the retrieve chunks are more relevant it's jumped to like a nine out of ten so what I've discovered is even if the retrieved chunk is semantically similar and even if it does contain some some truth or something related to the question if it just deviates a little bit right especially if you're using turbo um they can skew the results it can actually make the output you know trash so there's definitely I've noticed that um and this was a pain point I I I've had a lot of developers reach out to me about you know a framework like Lang chain because you can't you have to almost strip it down all the ways then you can tweak these variables because out of the box it's just doing the retrieval for you and you don't realize when you start debugging that actually the cause of the issue you're having is that the the relationship to the Back store in the the threshold that you've set to to um generate the contacts um in terms of latency um I'm saying that if you um there's all kinds of weird things with different and this is another issue like every Vex store seems to have their own quirks like in terms of you know what makes it perform well what doesn't make it you know so for example a Pinecone if you have metadata that's not like null you do that you might not realize that's what's happening under the hood but you might say link llama index FICO and you've got met status no it's it's going to break your application right um I've had that experience before it took me three days to figure out that that was the cause um another issue is if you have too much metadata attached to the embedding itself it can slow your retrieval system by another 30 seconds right so this is like the difference between the results coming in immediately and now maybe you're metadata because you're trying to be fair grounded or I know Alarma index has tons of tools for kind of the nodes and and kind of filtering you know and so if you attach too much metadata to your your embeddings that can also slow down retrieval significantly um so you've got that that issue with latency um as well I mean we can talk about all the all the other causes of it but in my experience that's what I've seen so far did you say did you say 30 seconds uh if you add a bunch of metadata I can uh like slow down approval by that much that's actually a surprising number yeah yeah there's one of the projects um we worked on recently and that was what happened um it's uh it's especially with something out of the box like super bases PG vector or any things that are not dedicated Vex stores you you do run into slower retrieval times that way gotcha cool um Jason do you want to show us some thoughts uh yeah uh you mean like uh yeah so part of the two things one is that you mentioned for data structure uh problems specifically uh one is uh like a few products I found with uh it's just for the normal documentation um the there are list of parameters you need to really or leverage you need to pull like the chunking size uh stuff like that or the all the data data storage you choose also have the impact on the uh on the on the actual results and it's very hard to so far I can find best practice to be honest every time just to like start from something and then do like iterated through model times and uh uh but so that's kind of just for a simple one but the other part is that internal metadata I was when I was helping with another clients who want to build some internal knowledge base because for the one topic they might have iterated version of The Truth uh like one person might enter multiple times so you during retrieval they even got like contradict data so I have to add metadata into more like the data of when this blog or article was added so that large energy model has more uh more kind of context but things like that is that thing is also not useful for many other scenarios that that they don't need the admitted metadata which and that just add on context limits for Unnecessary so I haven't figured out a very smart way to handle the step situation like to decide when to pull this metadata in versus not and and a problem that I always think what I already talked about like the all those PDF and CSV file how do we gather just accurate data out properly um but uh yeah so this kind of data structure part but in terms of the actual retrieval uh it's just uh different questions that user ask in terms of the knowledge retrieval system they require different methods uh so how can we uh yeah apart from using agent I so far I haven't figured out a really smart way that it's fast but also can know when whether the question need to be breakdown properly or uh choose the right method to use um and uh yeah and and the main part is also just like there's so many Liberty pool uh it's very hard to know uh what's uh what's uh what's the specific uh thing that need to be fixed especially when things goes wrong uh I remember the one time I spent a lot of time to figure out whether my prompt has problem but only when I actually look at the result in return I realize probably this some setup wrong initially the other score for the for the result is very similar so that's the time I realized oh it's actually not setting up properly at the beginning um so yeah there are a lot of a lot of area can go wrong it's very hard to say it is so it sounds like a lot oh sorry yeah one one major issue popped up in my head I completely forgot to mention was a lot of use cases I'm seeing are popping up recently a kind of very time related so for example you can have someone it's not static so basically the data is constantly being updated and um and so the challenge is how do you continuously update your Vex store or update whatever machine learning or AI system you've put up in place so that it's in sync with the application you've built for the client so I'd say the the constant updates and keeping up with the data changes is is another major uh pain point yeah no that's a that's a great pain Point we've heard that from some users too it's um it's uh definitely very much kind of like a system of the engineering thing but very much like a core part of making these like uh production ready um now yeah thanks for guys thought this is great and it seems like the high level um compilable themes is uh a lot of stuff is pretty tricky and a lot of this is pretty use case specific and so we're kind of like figuring out like tips and tricks for more specific use cases um maybe kind of like if since you know we're on the um uh space of also helping to educate others on maybe what are some like good valid practices to follow and maybe Solutions and tools to look for to try to improve uh these types of systems and try to solve some of these pain points um what maybe are some kind of either General practices or tools that like an AI engineer or skills uh that an AI engineer should have uh within um like their toolkit or skill set in order to start tackling some of these issues um for instance like should they for instance think about how to prop properly evaluate and experiment with things should they think about how to like really you know Trace through like prompts and log like the inputs outputs of the llm I'm sure they maybe have some like benchmarks set up so they can measure metrics whenever they try to do a little bit of prompt engineering or tune the retrieval armor then um yeah that type of stuff and and this is something that I've been kind of thinking about as like General practices as well and maybe we start with Jason uh yeah so um in terms of the practice uh one thing I I found uh that uh left using is probably one is from the uh generation another is from retrieval uh from generation part uh it's just always making sure uh you actually have some practice adding into a prompt making sure like very specific instruction only based on this content uh if if no just return no random results so that you actually know when things go wrong and on the other side uh obviously just ability to log in have the ability uh it's uh it's a very good start to debug because uh at the moment yeah if you have ideally you want a way to have uh observe the whole rack apply so you know uh bear since actually went wrong from chunking to retrieval to the to the actual results and there are a few tools like uh last Miss they can support you to do things like that um and uh um yeah so that's uh the other two things I would say how am I if that's I would say that I think observability is definitely a huge thing um um I'm a big fan of learning by doing so I think that just deploying uh deploying an llm app and you know setting the system prompt and uh just getting something going and modifying just toying with it see and really learning just by seeing the results is a really powerful way to experiment with with modulating these systems and while you're doing that too while you have the systems available then you can get into the observability some tools that I've used that are really helpful um one is called helicon.ai that gives you some background info on what's Happening behind the scenes you can see what chunk is being retrieved from the vector store and brought into context and that can really help understand what's what's happening behind the scenes another tool is highlight.io Axiom and Sentry are all logging tools for JavaScript web applications and just really getting that getting as much visibility into what's happening as possible I think uh is critical for evaluation I have had an engineer help in the past kind of a prompt engineer evaluation uh focused person to you know we I can we can save a lot of the question um answer responses to evaluate them and also you know those can be used for for fine tuning things like that as well so I would say yeah diving in and and like we've been saying kind of experimenting with all of the different pieces whether that's chunking the metadata structure the retrieval the vector store the llm the all of the pieces I think are great to experiment with and and kind of jumping in can get you a good a good part of the way and then it is really helpful also to have you know someone who has that formal machine learning experience and is going to build that that structure to create a more advanced and reliable system to kind of bridge the gap from where just jumping in and experimenting might get to actually getting to a production ready application so I would say I would say definitely just playing under the hood of these systems and um collecting as much information as you can to get a more educated um you know get the data to understand what needs to be augmented to get the best result possible yeah and and that's a nice set of tools that you mentioned um like Helicon century and um Mayo passing it to you yeah I yeah you've guys basically nailed it um I'll just add to that that through my experience I've just quickly learned that the first thing that I want to know quickly is what what exactly you is the client looking for in terms of outcomes right because I can do all these fancy stuff and then find out at the end that the results I think look good it's not what they looks good so The Benchmark for me is crucial um Joe I think you mentioned that already um whatever that looks like could be a simple CSV of sample questions and answers and then potential um retrieved sources that the client says hey this is what a perfect uh retrieval would look like to us and if they can give us 5 10 15 the more they can give us the better then we have a benchmark to start with um and as a matter of fact my dream situation would be just to take that CSV and then some some way to automate the configuration of the rag system so it's like oh done now all of a sudden they ask the question and the answer is exactly what they are but so far that's that's like we're not there yet uh you know software wise but um yeah the benchmarking for me is crucial because that makes sure everyone's on the same page this is what you you said you want it's a question this is this is the answer to a satisfactory answer the question and here at the regime resource documents that you're happy with um but yeah other than that in terms of techniques like I said playing around with the threshold of similarity search you know um and and all the other stuff you've all mentioned um you know playing around with different Vex stores tweet you know maybe you go for a hybrid search instead of just semantic search or do you use keyword search um and then Jerry I know you put out some interesting stuff recently about fine-tuning plus rag so I haven't played out with that but I'd be interested to see how that plays out too sweet um we're talking about like some of our benchmarks how do you get how do you create a benchmark um would you recommend do you recommend your clients just like manually do it like they just go in and Eyeball a bunch of stuff uh like create like eyeball 100 rows of data um yeah how do you how do you do that yeah I know I know there's ways you can automate this for AI right but I'm not I'm not leaving anything to chance right I want them to put it together so there's no no there's no one's going to complain at the end and say Hey you know this is not what we wanted so yeah we definitely definitely there's a lot of manual curation in the site and then once that's done we just get to work so that's what worked but I'm open to hearing successful automated uh Benchmark creation I'm Jason Dylan have you guys created benchmarks or or have like manual or automated ways of doing that uh maybe I can I can go first on that um I I most of the time I actually like uh mayo or I just uh I don't see the needs for those SMB clients to actually set up very popular Benchmark yet uh but there are times I to set up a like list of example questions uh because people are a few clients I realized when I changed the maybe the prompt or the pipeline uh it works for this question but some for other questions don't work again so I normally uh work for those type of situations I would like get a list of those uh database of questions and answers uh and then just surround that uh for all those 10 or 20 questions to see this is not a huge amount but I found just enough I would say that I think this is where some of like my design background comes in that I like to work like very uh collaboratively so a lot of the time I'll give I'll set we'll set up a client with a Helicon and it's kind of an ongoing process of like himea said it earlier it ends up becoming uh my responsibility a lot of the time to figure out you know what data do we need in the first place and um working with the data before even getting into the AI just preparing all of the data I mean sometimes that takes months to to get all the data ready uh to be honest and so I like to kind of build in parallel um with the client and so setting up them up with a helicone where they can kind of help me do the evaluation like I guess I use a reinforcement learning human feedback uh mechanism to do evaluation but on more kind of software levels of that um yeah typically for all the Bots the chatbots I create I'm storing uh the questionnaire the question and answer responses in multiple databases so we have those available to import into different tools for doing evaluation there's some spreadsheets of views that are kind of pre-set up where you can just input um the row Pairs and then get an idea of if that's what you're looking for and then I have one of my good friends is uh really deep into just the details of prompt engineering and stuff so sometimes I'll like pass him that and he'll help me um explore some more advanced like prompting like uh Chain of Thought and tree of thought uh systems and things like that to kind of experiment with what outputs we can get from the llms um but yeah I would say I would say definitely logging and um and having that like you said a CSV of of all of the outputs and I think if yeah having the output that you want to get to is a great idea I love what you said about you know maybe you could make like an agentic pipeline to start to um automate the modulating of these systems to create different uh outputs depending on like what you're looking for I think that's a great idea um yeah I would say that's that's my answer great thanks for our thoughts um as we enter into a last few uh no that's of the webinar um I figured uh I'd ask like one last question or maybe we have time for one uh-ish uh audience questions um what's kind of like maybe the top of mind on your wish list uh for the future uh in in tooling uh I guess uh you know album developments that type of stuff I'm gonna start with uh well I mean Jerry if I say here there could be some people listening gonna steal your ideas so how much do you want me to say behind closed doors our secret stuff yeah yeah we've lost Secrets um man just yeah just a lot of the stuff I said it's just there's so much like guesswork we're still doing in terms of you know you've got you've got new a new uh prospective use case and then it's like you try something you tried before it doesn't work as well and now you're trying to tweak all these variables and it's just yeah it's a lot of work it's still it's just a lot of guesswork and then you've got new models coming out all the time um new even the same models used before they change their behavior with the prompts as well right so it's just I think the dream situation would be not just observability as cool it's it's good it's like Google analytics right but that's not going to help you build a website that's that's really good with SEO right you still need the application so for me the the dream situation would be tooling that not only allows me to observe like for debugging but also provides automated testing or an automated way to optimize the different variables required for the outcome so if if a user provides if a client provides a benchmark of questions and answers it would be amazing if I could just plug it in somewhere and get the optimal prompt the optimal you know rag configurations and then that's it I'm done I don't need to spend months tweaking and I'll find the optimal prompt to find the optimal retrieval system so I think that would be the dream situation is tooling that just automates the based on what the person this person's desired outcome is what are the optimal configurations and even the optimal components because uh Jerry as much as I love your stuff man every time you post I'm just overwhelmed I'm like oh my God another idea another way of doing this right it just doesn't end it's just it's just new ways at the time it's easy to fall behind so there's so many components there's so many ideas and how can we find the optimal approach for a given use case um would be would it be amazing that would be the dream situation yeah so uh I I actually the the one you mentioned I feel quite excited when I think about it it's almost like a way to fine-tune the rack system uh by providing the final output um but I was also thinking about something similar like if the part I'm actually interested about the evaluation system is if we actually imagine we have a large energy model can do the evaluation very well and then have another agent can figure out based on the results figure out the optimization for the final output then I think the assistant can be really interesting uh it's just self iterate uh till the till the most optimal configuration so that's one thing I do I think can be really good but short term if someone can just really handle the data loading I would be really really appreciated uh like handle all those PDF all those like State documents to just clean up automatically that would be really cool uh but on the other hand like I almost feel like if there is a one kind of Harbor Place uh where I can see how other people are setting up the rock this whole rock paper line uh so I can watch and learn as well that would be super useful because I at the moment my main source of the learning is also your tweets as well I sometimes see you post something and then I can look into this new tactic but it's also very hard to see the full picture about uh about how what's the best practice for that tactic as well or something I'm done yeah I agree 100 with what was just said and I would say I would say I think we need I think naturally that you know this is a new booming um ecosystem of tools that we just need like kind of we were saying earlier on in this call the maturity of these systems Bridging the Gap to already ex the already existing software industry and uh so I would say the maturity of these tools integrating with um existing software and then also I think I'd like to see modularity and portability so we can get to you know the point that we're discussing where we can automate and and quickly assemble um a wide variety of these systems depending on the needs that we that we for that project um so yeah I think being able to integrate everything in in a way where it's going to be stable and reliable and you can you can fine tune each part of the system easily and know that okay I don't need to uh this isn't going to break my development environment this is still going to work in a few months and I know that if I want to get this kind of structured output I need to tweak this module of the system if I want to get you know this kind of rij system I need to use this Vector store and just having a kind of bird's eye view all within one mature ecosystem that um yeah can be easily integrated with already existing things it's kind of an abstract answer but I think I think the tools are are all out there's so many amazing tools and potential out there now and I think we just need some time as like a development um Community to take the time to kind of weave them together and once we bridge the gap with everything that's out there we're going to have some amazing amazing stuff absolutely yeah thanks for your thoughts all right last um last question I think this is the audience question so I figured just to wrap it up we'll make it 10 seconds for each person um I I think basically the address of the question is are we at the peak of an inflated expectations in Ai and I figured it would just be something like our lens you know over hyped under hyped uh right now uh what's your what's your take uh starting with Dylan oh that's a that's a that's a 10 seconds 10 seconds uh oh 10 seconds okay I would say that I think there is a wave of excitement that sort of ended with the magic uh in the first few months and I think we're maybe gearing up for a more long-term but more meaningful integration of the real utility of these tools so I think yeah it's definitely critical long term in terms of hype uh it might be going it might be going up and down I'm not I'm not sure all right Jason uh I think now it's a ride mod hype it's like comparison a few months ago people expectations here but what we can do is now here but after a few experiments like people's expectations under like more uh aligned to the reality so I feel like it's right amount to be honest ly uh yeah no the hype's definitely died out I still think expectations are I would say they're high I'll just say they're unrealistic and that's just lack of knowledge right um so I think as an industry we need to do a better job now of speaking to the language of of the vast majority of users you know talking more in business linger or just the language they understand so that they don't expect us to do um they don't expect AI to do a lot of things that it's not designed to do so sweet okay well thank you so much uh mayor Jason and Dylan for uh joining this webinar today I think we had a great discussion um and there's a lot of very interesting points being raised uh everything from the pain points to the need for uh you know kind of like standardization of best practices um to figure out like best solutions for for building rag agents all these types of things um so yeah thanks for taking the time and uh for the audience hope you enjoyed the webinar and we'll have the recording up on YouTube uh in a day or two all right thanks guys thank you so much