How we debug AI agents using AI agents (real trace debugging workflows)

Key Takeaways

Hey everyone, I'm Sally-Ann, a product manager at Arize. Hey, I'm Jack, an AI engineer at Arize. We've been working on Alex for about 2 and 1/2 years now, and recently we've been really setting out to build the cursor experience for AI engineers, and uh it works out pretty well because we're AI engineers ourselves, and that gives us the unique opportunity to build the tools that we actually want and need as we're trying to build Alex. I think that it's been really helpful to be able to use Alex to build Alex makes our work process a little bit more uh efficient, but we've been really setting out to build a deep set of tools that are actually useful. We didn't want something that just demo well or was surface-level. We wanted a deep set of tools that were going to be helpful in our day-to-day and anybody else who's building AI agents right now. This is not only about how we literally use Alex to build Alex, although we do a ton of that. It's also about how we've used our experience with Alex to determine what bets to take and what direction to go. And I think overall Alex has helped us drastically reduce our feedback loop, and I think that's the real value. So, I want to take you through one of our most powerful use cases, which is our trace debugger. Uh usually when I'm working on Alex or developing, I have like the Alex UI open on one screen, and then on another screen I have our Arize traces. Um and usually I'm testing, and when things go wrong, I directly go to the trace to try to understand what's happening. Um and now with Alex, rather than needing to try to go through each of the spans uh manually to try to figure out the issue, I can just ask Alex. Um and that is really powerful, not only for us, but um our customers as well, because this trace here, it looks complex, uh but this is actually simple. I have some customers who have hundreds of spans uh that make up their trace, and I knew it was going to be pretty unrealistic that they were going to want to come to the UI, comb through all that data to figure out what's going on, and and that's really where we got the idea of we needed this trace debugger. So, um recently we added the eval builder to our main trace tools, and it worked really well. I was able to create a template, but what it wasn't doing was responding back to the user and telling them exactly what the thought process was, why it chose that eval, or any of the analysis it did. And as a product person, I felt like that wasn't the best user experience. So, what I did is I came to the trace, which we have up right here, and I began to ask Alex why. Why is it that the why is it that you're only responding back with just the eval template and not with any other information. So, we can ask Alex that now. Yep, so I can you can see here I'm saying, you know, the trace only responds with just the eval template. And you know, for a better user experience, you know, you can see that the user asked like come up with categories, tell me what the most critical issue is, and none of that was in the response. So, I'm asking Alex, you know, what do you think was the problem here? So, it'll think, it'll look at some of the you know, example spans and get an idea of what the shape of the data looks like. It looks like Alex is able to figure out is that in our prompt, we actually had explicit instructions that said when the eval builder tool was being used, only you respond with the template, not with any other analysis, which I think in some tools that makes a lot of sense, but in this use of it, it was a little bit of a confusing experience. So, now that we knew exactly where the issue was, it was a prompting issue, we were able to pull this into the playground, use the Alex prompt optimizer to actually fix the issue, and something that would have taken me you know, 30 minutes to an hour to comb through all of our prompts, all of our code to try to figure out why this was happening, took us you know, a matter of minutes to get to the root cause and actually also resolve it. And I think this really starts to set the the of what the future could also look like instead of having to kind of go to the UI back to code. I think we could definitely uh shorten that and make it so that, you know, the IDE has access to this analysis as well. Yeah, and you know, just to give you an example of what it would take, you know, to look manually look at your prompts and try to figure this out yourself, you know, you can see that our system prompt here is pretty pretty long. Um so, you know, this eval template output rule is buried somewhere in that system template, but you know, good luck trying to read every single line, trying to understand what's happening, and then trying to figure out yourself, you know, what is relevant to, you know, the problem that I'm trying to solve. So, you know, Alex is easily able to figure out the exact issue that um was causing this. So, one way that we recently used Alex to debug Alex was we had a um pretty big dogfooding session with almost all of the engineers and um product leaders at our company, and you know, they put a lot of good feedback into Google Docs. Um But, the problem with that is that, you know, sometimes the error messages are very vague that we present to the user. You know, it just says something went wrong. So, you know, to the user, they they don't know where it's coming from. So, what I'm looking at in front of me are the traces from Alex. And um these are all the errors that we've kind of gathered over the last few days. Um most of them from the dogfooding session. You can see there's, you know, 68 traces with errors. So, you know, Sally and I can kind of like look through each one of these and you know, read the error message. But, you know, we want to work a little smarter and you know, this gives us a great opportunity to see how we can use Alex uh in this case. So, you know, all I can all I need to do is very simply say, you know, categorize my errors. And Alex knows, you know, which column to look at. Um it'll kind of semantically read all of those columns. Um and then it'll form the categories that it thinks are most reasonable. So, you can see it's extracting categories. Now, it's going back and assigning those categories to all the relevant spans. Um and, you know, pretty soon it's going to uh come up with uh some categories that, you know, we can kind of click through. Um so, you can see kind of these examples where um the system failed to provide uh a response. Um and you can kind of see, you know, why. Um so, that's uh one way that we've been using Alex to uh debug Alex itself.