Algorithmic Context Engineering for Agents with Optimizers | Jacques Verre | PEC London 2025

I'm going to be talking about content context engineering for agents with a very specific focus on optimizers which we'll talk about something that I'm quite passionate about. So before we jump in, just a quick introduction. I'm Jacques. You can call me lots of things. Jacques, Jack works. Not a huge fan of Jacquis Jacquis, but I've heard all different variations, so just don't worry about it. Currently head of product at Comet where we're building OPIC, which is an LLM evaluation and observability platform. Also allows you to perform agent optimization, which we'll be talking about before today. I also do lots of site projects. So you can find me on GitHub lately. been working on a little project to share your LM chats, but just enjoying playing around with LLMs. So, before we jump in, we're going to be talking about agents today. Lots of people say you can't really define what an agent is. I have my own definition that I like to stick to. You can always tweak this a little bit, but I like to be a little bit opinionated. So, for me, an agent is an LLM that has access to tools. And what's most important is it can decide what to perform. So instead of telling it exactly what it should be doing, you essentially give it a number of capabilities and then it decides what it should be doing next. And that's really what an agent is. If you start to peel to go down it a little bit and be a little bit more specific, one of the things that I always recommend is don't worry too much about frameworks. Look at what information you're passing to LLMs and that's really how you're going to understand what an agent is. So when you look at this, it becomes actually really simple. Building an agent is not that hard. The first thing that you need to do is you need to define a system prompt. And that essentially tells your agent what it is supposed to do. If you're building a coding agent, you tell it you're a coding agent that is going to help the developer or the user build applications. And then you need to give it some capabilities and these are the tools that you're going to be defining. Again, if you're building a coding agent, you'll have a write tool, an edit tool, search tool, and you can define lots of different tools. lots of people. You also now have the whole topic of MCP servers. At the end of the day, an MCP server is just a way to define tools. That's what they are. What's really cool about MCP is not the protocol or anything else. It's just that it's a great way for third party providers to add functionality to an existing app. If you think about cursor, what's really cool about the MCP server is that me, Jacques, as head of product at Comet, I can build an MCP server that gives you a tool for agent optimization and I don't need to get in touch with them. Essentially, it's an integration play. But really, the way to think about MCP servers is they're just more tools. And that's it. That's your agent. One of one thing to note is what do I actually send to the LLM? So you'll send your system prompt. You can't really send tools. What does that mean? What you're going to do is you're going to describe each tool. You're going to say the right tool allows you to edit a file. And you're going to say, okay, if you want to write a file, you need to tell me what the path is and you need to tell me what the content is. And that's essentially what you're going to send to the LLM. You're just going to send a description of those different capabilities and the LLM will come back and tell you what to do next. So what does that look like? So I start off with my LLM. I have my system message. I have the user message. And then I have my tool definition. Not the tools themselves. It's just something where you describe what your tools can do. What will happen is your LLM is going to come back and say, can you please read this particular file? You're going to do this and you're going to send back to your LLM the tool result. Here is the file that you wanted me to read. It's then going to come back and say, can you read this other file? you're going to do the same thing and you're going to send it back. And then finally, the assistant is going to say, "Okay, I've read the two files. Here is the information that you wanted to know." And all of this information is sent over to the LLM. So, the reason why context is so important is as you get here to this tool request, for example, or even the tool result that you'll be sending to the LLM, what you're doing is you're actually sending all of this information below. Now, imagine what happens if you read the first file and it's 10,000 lines long. You're going to be sending those 10,000 lines to the NLM. Your context is going to grow and you're going to have a big problem on your hands. And so, I was talking to somebody a couple of weeks ago and I found that I couldn't share exactly who it is, but I think they really summarized what this is all about. Building agents is about managing your context window. It's not about building the right tools and adding more tools and all of this. It's really how can I make sure that after reading one file, I don't run out of tokens that I can send out to the LLM. And you could always say, I'll just use an LLM that has more context. Sure, but you're going to be limited to about a million anyway. And performance is going to be terrible. I don't know if any of you have ever sent a million tokens to NLM. Yeah, it's going to come back, but you'll have time to have a coffee, a shower, and probably a dinner at the same time. So, you're not going to really want to do that. So, all of this is really about managing your context window. As I mentioned, you get the first response, but then your users are going to keep chatting with your agent and you'll send another message and you'll send the tool definitions again and we'll keep adding over here to to the context. The more the context grows, the more the worse the performance gets. As well, for all of you that are maybe using cursor or any of these tools, you'll notice that quite quickly you start a new window, a new conversation to avoid the context getting too big. So, how can you avoid context explosion? There's a few things that you can do. So, the first thing that I like to do is I like to think about, okay, what information am I sending to the LLM every time I want a response from it? And this is very similar to what we saw before except instead of having it as a conversation, let's think of it about in terms of the tokens that I have available to use. So if I can send my LLM a particular question and I want to and I have 200,000 tokens that I can use. Here's how it breaks down. You have your system prompt. The longer that is, the more token it's going to take. You have your tool definitions. Again, that's just a description of what your tool does. MCP tools. Then you have your actual tool results. As the conversation was going, you kept adding more and more tool results. And then finally, you have your user and your assistant messages. So that's every all of the information that I send to my LLM. Every time I want it to return a response. And as the conversation gets longer and longer, you'll see the system prompt is going to stay the same size. That's fixed. The tool definition is also going to stay the same size. Same with MCPS. But then your tool use and your messages are going to get longer and longer. So what can you do to make sure that you're not sending over too much information? The first thing that you can do and it's probably the simplest is you can just truncate. You don't always need to send all of the information from the entire conversation back to your agent. So this is called message and truncation often. And so this is something that chat GPT did very early in the early days where you just have a sliding window. I'm not going to send back the 150 messages that exist in this conversation. I'm just going to send the 50 most recent ones. That's probably what's most relevant. The downside of that is you lose a lot of information. A file that you might have read at the very beginning of the conversation, the LLM is no longer going to be aware of it, for example, and so you have to go fetch it again. And that can become expensive depending on exactly what your tools are. What you'll notice if you're using cloud code or cursor these days is you can also instead of just removing them, you can summarize them. So you'll summarize the context and that will make it a lot smaller and you can continue with the conversation. But again, you are losing a lot of information and so that has some of its downsides. But that's probably one of the simplest techniques that you can use to manage your context. Another one that you can do is called dynamic tool loading. So, one of the things that you'll notice is sometimes you can have a little bit of an explosion of tools and they do take up a lot of your context due to the way that they're formatted as well. And so, what you can do is depending on the message that the user is sending, maybe I'll detect which tools I should be providing to my LLM. So instead of if we go back here in my coding agent, let's say I'm in cursor and I'm in ask mode. If I'm in ask mode, I'm never going to be able to write anything. I'm never going to be able to edit anything. So let's not send those tool definitions to the LLM if the user is in ask mode. So you're dynamically choosing which tools you will make available to the LLM. And that's going to allow you to reduce the size of the context that you send because again that tool definition is just going to be a lot smaller. It sounds really nice in practice. There are some use cases where this works really well, but it can be really hard to know what tools you should be making available. So the cursor example is really straightforward because it's essentially two different modes of the product. But if you're building a chatbot that has to answer user questions, how are you going to know which tools you should be using or not? That's why you've built the tools in the first place and why you have an agent because you don't know exactly which tools to call and you want the agent to do it. So, this one's a little bit related to the way that your product or your agent is built. One quick side note, if you ever start running into issues with cursor, I do always recommend checking out the different MCP servers that you might have installed. Usually the performance of cursor really goes downhill as because you've added the GitHub MCP server and the JRO1 and each one of those has 30 or 40 tools that you never use. So do take it do keep an eye out on this definitely a problem if you start integrating MCP servers into your application. These are two pretty simple techniques but the one I wanted to focus on a little bit more is agent optimization. What we've talked about so far are deterministic. Right? I just compress my messages, my context. I decide which tools. But what you can also do is you can use agent optimization. So the idea here is and we'll take a simple example. You can also apply this to the rest of the context is I'm going to take my system prompt and I'm going to take my system tool definitions and I'm going to try and compress that as much as possible. So, the idea is I'm going to try and keep the same performance while making my system prompt smaller. Because what usually happens is you'll start with a relatively small system prompt that explains what your agent should do. Then you'll run into a bunch of edge cases that don't that your agent doesn't really perform well in, and then you're going to start adding more and more things to your system prompt, and the next thing you know, your cloud code with 10,000 tokens in your system prompt. So what you can do is you can actually use optimization algorithms. There's a few that are quite popular. Japa is probably one that you might have seen on Twitter that allows you to do some of these things. I'm actually going to talk about another one, a hierarchical reflective optimizer that's a little bit simpler to explain. And we'll see how we can compress the system prompt and the system tool definitions while maintaining the same performance. You can also use this as part of some summarization algorithms that you can use on the tool use and messages, but that's a little bit more advanced. So let's talk about the hierarchical optimizer. It's very similar to what we heard in the keynote around everything starts with evaluation. The first thing I need is I need a data set, a set of user questions or a set of things that I want my agent to be able to do. I define then my agent. I define a metric. How do I know if the agent responded with the right thing? And you run your data set through your agent, evaluate it using the metric, and you get your agent performance. What does that look like in practice? If we take the example of my coding agent, I'm going to add some coding tasks, write this Python file that reads something or like lots of different coding tasks. And my metric might be, does this code run successfully? the end of the day, that's what I need my agent to do. If it runs successfully, h pretty likely that it might actually work. So that could be the metric that you define. I have my data set, my agent, and I just want to make sure I know, okay, my agent can perform about 80% of the task that I have defined. That's the evaluation part. How does the optimization work? What you can do is you can run your evaluation. You can see my agent was able to answer 80% of the tasks. It means that it got 20% of them wrong. What you can do is you can actually use LLM and GPT5 is a great model to do this to do root cause analysis of the different failure modes. So you'll have your agent, you'll see the different tasks that it's not able to perform. That's part of your data set. The optimizer algorithm will then run a root cause analysis on this. You can do it in a hierarchical way because if you have a data set with a lot of items, you're not going to be able to pass all of that to your LLM. As we've talked about, context is a big problem. So you might split that up, compute the root cause analysis on a subset of your data set, and then summarize everything back. And that will tell you, okay, my agent is really good at this, but it got these things wrong, and here's why. And what you can then do is you can make updates to your system prompt or to your tool descriptions based on those root cause, based on the failure modes that it has identified. So here is an example. So here you can see on the side it wasn't exactly a coding task that I was working on but you can see an example of the of the root cause analysis that it performed. So I'm doing different iterations. In this case my data set had 100 different items in it and you can see that it analyzed it and it identified five or six different failure modes. And you can see for example here the task I had was I asked my agent to follow some specific instructions that I had given it. And you can see that it's able to identify that when I as when my data set had instructions that asked the LLM to repeat the prompt exactly, it wasn't able to do this. And so we have this failure mode, we have a description, we have a root cause. We can now update the system prompt so that the model doesn't do this anymore. So that becomes a way of editing your system prompt or your tool definitions using an optimizer. That doesn't actually get us exactly where we wanted initially. What we wanted to do initially was we wanted to compress our system prompt and our system tool definition. We didn't just want to make our agent better because the way that it's going to do that is it's just going to add bunch more instructions in my system prompt, make my tool definitions even longer, and I'm actually going to be growing the context. So optimizers, if you use them out of the box, they have a tendency to just increase the size of your context. What you can do instead is you can actually do multi-objective optimization. This is something we added to the OPIC library a couple of weeks ago. But what you want to do is I'm not just going to optimize for performance. I'm not just going to have my optimizer try and get the best performance. I'm going to define a second metric. Not just does my code code run, but also what is the length of my system prompt or my tool definition. And what you'll do is you'll define a metric that says okay I do I want you to increase performance but not at the cost of making everything longer. So you define this multi-objective optimization and when you run it not only is the LLM going to try and make your agent better, it is also going to try and make the prompts smaller. So what we've seen is we've seen a ton of folks use this multiobjective optimization to keep performance pretty much the same but to increase latency or decrease latency should I say or decrease cost because at the end of the day if you compress your context and the number of tokens you send to the LLM then everything's going to run a lot quicker. So OPEC has an implementation of this that you can try out. DSPY is another very popular framework for doing these types of optimizations. The one challenge with the spy is you do need to define your agent in their framework. So essentially they are an optimization and an agent framework. So you can't use it with langraph, you can't use it with open AAI just code. And so that's one of the downsides of the DSPY library, but there definitely have some really cool algorithms in there as well. So we can then put it all together. So if you put all of this together, you can use your optimizer that will be compressing the system prompt and the system tool definition while maintaining good performance. You could do dynamic tool def tool selection so that you can make your MCP servers and your tool definitions shorter as well. That's going to give you a nice boost in your context window. And you can use summarization or compression to be able to reduce the size of your tool use and results and your messages. So by doing all of that you can typically compress your context often by about half and that really makes a big impact on latency and cost for all of the different convers for yeah for your agent as a whole. The last thing I wanted to mention here as well is one of the things that I do recommend you spend quite a bit of time on and this is where we're taking our optimizers at OPIC is to really be careful about the data that you return from your tools. As I mentioned reading a file that can return a ton of information that ends up being part of your context, but you can tweak your your tool so that you only read 200 lines at a time. That's what cloud code does, for example. And that way you're forcing the LLM to be more specific about what they need. The second thing that makes a big impact is being really careful about failure modes. If you're calling a tool and you're not able to get a you don't get, for example, you're calling the database with a user ID, but it doesn't actually exist, returning a very specific error back to the LLM as part of your tool result will avoid your model going into a little bit of a tool loop. So I don't know if a lot of you have noticed this in cursor but sometimes it starts to read a file and read it again and it's not quite the right format and it tries six or seven times. That can happen in your agent really easily. That adds a ton of context because your tool use and tool results grows and grows. And that is typically because what you're returning from your tool call doesn't actually tell the LLM what went wrong. And therefore the LLM doesn't really know if it should try again, if it should try slightly differently. But if you're very specific in the errors that you return, then you will be able your LLM will just be like, okay, that doesn't exist. I'm not going to try this anymore. I'm just going to return this to the user. So, another quick tip on how you can optimize your tool use and your tool results. So, we've covered kind of three, four different techniques. I do recommend checking out OPIC. We've got lots of different optimizers that we've been working on and also the DSPY library that has loads of really great algorithms. Awesome. And so, feel free to connect on LinkedIn. I'm Jack. I'm going to be here most of the day as well. And I added a QR code here if you want to try out the OPIC platform. >> Awesome. Thank you, Jack. First of all, questions. Any questions? I will have my own. If we don't have any, I'm going scooch to the other side here. Cool. Then I will be able to be selfish and get out of the way, but also ask my own questions. So, when a client comes to you, what's your starting point in terms of managing context around we could say it's a coding agent or whatever. What's like the first two things that are like the biggest mistakes I guess you see when working with new clients? >> Yeah, 100% of the times it's the metric. So before you can do an optimization, at the end of the day, you're trying to make a number go up or go down. You need to have a really good number. And so the metric is what we end up spending most of our time on. Usually teams already have their agent built. Usually they have a data set, a set of examples that they test upon. We spend quite a bit of time defining the metric. And once you have the metric, then everything else falls into place. But really, that's usually the hurdle and where I recommend teams spend most of their time. And the best way to do that is to think when you read the data, what do you do? And that's what you then try to convert into a metric. >> Yeah, it's interesting. A lot of the time when doing these types of evaluations, it's understanding what's going on in your brain and then trying to translate that back to metrics or to instructions. That makes a lot of sense. Is there in terms of managing system context, we know certain models can have a million tokens, two million tokens in the context window. There's varying studies. I'm not sure if you're familiar with context or but generally >> you obviously don't want to use all that. So do you have a rule of thumb of hey this is like where we feel comfortable advising our clients to have this amount of context even if the model can support more do you have is there like a hard mark >> to try and say underneath >> we typically recommend the 200,000 token mark but usually what we end up doing and it's related back to this multioptimization is it really depends if you're doing something in batch and it's an internal process and you're analyzing data and you're triggering different things or if it's really userf facing. If there's a person on the other side, no matter the size of the context window, no one's going to wait that long and then you lose a lot of the benefits more from a product perspective of the agent that you're building. And so for those two reasons, one performance when it starts going above 200 no matter what different providers claim, then you do have that problem of not being able to perform very well. But usually the time to first token in the agent response is what ends up in practice limiting your token like your context window much more than the limits of the model. >> Cool. That makes a lot of sense. Thanks for answering that. And I guess any other Yeah. question in the back. All right. >> Can you It's a team effort. Thank you everybody. I so I'm going to date myself a little bit here, but when I see this magic that you're doing around managing the different elements of the context window, it brings me back to my early days of computers when we were trying to load device drivers for early versions of Windows, right? And this is a problem we spent hours and hours like every time you got a new machine you spent hours trying to get your sound card and your network driver and your hardware your scuzzy driver and all these sort of drivers loaded into memory so that Windows would finally load and then the architecture changed and that problem just went away. So that's kind of my leading question right is principles today on 200k makes sense and that's a great guiding factor today. just kind of looking forward, what are the things that are going to change around LM architectures and what do you think is going to be a time frame before we're going to spend less time managing these contexts or is this something we're going to we're going to really manage for a while longer? That's my question. >> I think we're going to keep managing it for a while. It's just the 200 number is going to change. It's if and it's all about the latency. If models start getting faster when you send a lot of tokens, then that's going to keep happening. And so this is why I believe a lot in the optimizers is that every time there's a new model, it's okay if the token if the context window gets bigger. That means it can I can put in more information. I still need something to optimize it to make sure it's as fast as possible while performing as well. But I think what we've seen in the last two or three years is currently we're at a point where LLMs are good, but they're not the best when you actually try and build like a business or a real product or an application around it. And so as soon as you get an advancement in the model model families, should I say, you're going to make use of it as soon as it's available and then you're going to come back to the same problem. So it's not Yeah. So context windows get bigger, but we just use up more and more of it as soon as it becomes available. And so being able to optimize that for cost, latency while also keeping performance is why we believe quite a lot in these optimizers. You can do all of this manually, which is what a lot of people do like staring at prompts all day. But it becomes tricky because a new model comes along and then models don't behave the same. they call tools more or less and then you're back to the beginning and every month you spend your time learning the specifics of every model that's released. And then there was another hand up somewhere I think.