LangSmith Highlights: Evaluation

Key Takeaways

okay so in this video we're going to show how you can do automatic evaluation in link Smith we just showed you how you can easily collect data sets in link Smith as a reminder these are inputs and outputs that someone has deemed as a good response so if your agent was given this question and it output output uh a response like this uh this would be considered a golden answer or a true reference data set and so we make it pretty easy for you to create test runs in links Smith this is really the only way that you're going to be able to gain confidence that you can put that application in production you have to observe over dozens of inputs how your agent performs and see if it meets your criteria bar on the quality side or accuracy side so we have a couple evaluators that you can borrow that are off the shelf uh like correctness you can test for things things like helpfulness or insensitivity and all that we're doing here is really building up some evaluators that we can copy into our linkchain code often evaluation though is pretty bespoke to your business and we give you space to create custom evaluation as well maybe it is that your output needs to be proper Json so you might want to test that it runs through a Json D serializer properly or maybe you want to test that it runs through a SQL ler properly but we really let you as your business Define exactly how you want to evaluate these runs and this is where often really hard work happens uh to test uh over you know thousands of examples and build that confidence so we can copy this code into linkchain which we'll do now and in this eval example that I have uh what's happening is we're we're using the same agent that we've used prior and we are going to evaluate it on a few criteria in this case the correctness and sensitivity and helpfulness we're going to use a higher level model in this case GPT 4 to do the grading so a lot of the automatic evaluation can be done with an llm grading the output and so people like to pick uh more cognitive llm to do the grading and gp4 is an excellent choice then you name uh reference uh data set uh in link Smith so this was the planetary successful data set that we've been working with and you're going to give it a name and because I am less creative uh I will just use the autop populated name that uh we are given here and I can run it and I can see I have something running already but I can just pause and rerun from the top okay so we're going to rerun this code and the other thing I wanted to point out is that I've added a tag and this is really important to keep track of what's what being run so in this case I'm using GPT 3.5 under the hood so I've noted that here now my runs my test runs will be tagged with GPT 3.5 often people will also tag the commit sha for the prompt and that way you know what prompt was running uh when you are testing a particular iteration of a prompt or an iteration of a chainer agent you'll have precise information of what you're what you're actually testing against so let's go ahead and and run these and we can see in lsmith there's a new test run that's uh happening now and these usually take a minute to run but we already see uh it's it's started and it's kicked off I'm going to actually use a run that I uh just ran to to walk through that's completed but we can quickly see that um in these runs we have uh some aggregate measures for the correctness so only 38% of these test runs came back correct zero were insensitive and only 63 were insensitive were 63% were helpful so I can click in and and really get much more information on on what's Happening so you can see uh these are the inputs that are in the data set and really this agent is just getting iterated over all of these inputs and uh it's getting compared the output of that agent compared to the reference data set output which is also tagged here was this response that the agent provided correct and we see a lot of times that it was not correct and so this is a comparative evaluation we'll click in and and show what that looks like so we you can see that given this input this agent responded the following but the reference output uh said you know the response was something different and we could see that you know in fact the uh score for correctness here was was Zero it wasn't helpful um but it wasn't insensitive at least but if we wanted to click in and see how these evaluators are actually doing the evaluation we can see it's really just another EV it's another llm run and so if I click into this helpfulness run and I dig into the actual prompt that's being used you can see that it has uh some prompt around you know giving this submission and describing what a helpfulness is is this submission meet your criteria for helpful and it replies why why meaning yes it was helpful and that just gets reported really easily on that particular run so that way we know that this run was was helpful um but this makes adding evaluation uh a lot easier and just to of double click on a few few more runs to show it in action you know not all evaluation criteria is comparative in nature certainly correctness is and in this example example um we did get that the answer was correct and you can see just eyeballing it the output that this agent gave compared to the true answer the reference output both got 10 so it's automatically grated as correct um even though uh you know these are pretty semantically different responses and maybe the um the the distance uh of these two embeddings might not be that close uh you did get the correct response and so this all gets added uh to your test runs and you can get these aggregate measures around helpfulness uh and sensitivity and accuracy because those were the measures that I asked you grade on and one of the things that I really like about the test runs in link Smith is I can iterate and tweak my chainer agent and continue to run it so in this run it's now completed in the background uh remember this tag was GPT 3.5 uh we could see that it ran and it was helpful 88% of the time it was not insensitive and it was accurate 100% of the time by this by this measure and if I compare these two test runs I can even get a much deeper view of you know where did they differ so if I want to filter around a run where uh it was maybe not correct so a contextual accuracy of Zero by removing this one I can see how this agent performed when it was incorrect versus this agent using a different underlying model did get the answer correct and I can just read right across get given this input given this reference output these two versions of my agent as I tweaked the underlying model they had different responses and I can comb through as a reviewer to see exactly how they differed and in some sometimes you'll see that the um you know llm grading wasn't wasn't accurate these are two pretty similar texts but when we scored via the llm we got a in said it was zero here so it wasn't accurate even though it said it was accurate in in this case you're not always going to get um perfect responses by the llms but I'm able to really narrow down the number of uh rows that I need to look at when I'm comparing between two different test Rons uh and and seeing how as I iterate and change my my chainer agent whether it be a different underlying model or a different retrieval strategy or a different prompt I can pairwise see uh how uh the responses change