SWE-Agent | An LLM-based Software Engineering Agent

Key Takeaways

in this video I'm going to do an overview of this new technical report based on this software engineering agent that was released a couple of weeks or months back and what we're going to go through is the actual framework that was used and some insights and also share some of the results that they reported in this paper at a high level what they introduce in this paper is thiswe Agent an autonomous system that uses a language model to interact with a computer to solve software engineering task so the idea here is that you want to build a system that removes basically a human interaction and basically solves the task using you know a language model and a framework that allows for interactions with a terminal or interactions with a system to be able to carry out the software engineering tasks and what they have shown or what they introduced here is this agent computer interface which we are going to get into details in just a few minutes overall the main result was that this agent is able to solve 12.5% of issues compared to previous best of 3.8 achieved with a rack system and we go through the details of what this rack system is and other systems that they compare with the main focus of this paper is this ACI this agent computer interface that they're proposing and the way I understand this is you're essentially creating an interface that allows this llm based agent to efficiently and effectively interact with you know a terminal or a file system to be able to complete the software engineering task in this paper what they do is they use this framework called react and we talk about the react framework we actually teach about it in our courses um and what are the different components that you need and that's basically what they are using here this react framework allows and a language model to be able to take actions right do some thought processing and then also make an obervation in the environment and the environment will return a feedback and that feedback is then considered and it goes you know step by step iterating figure out what the solution was for that particular Ty ask that's what the react work is about and you can read about it we actually have a guide about this as well in our prompting guide and what they're proposing here is that they want to optimize or design an agent computer interface which is what summarized here instead of focusing on optimizing the weights of this language models we're going to focus on creating this interface that allows this smooth interaction between the llm based agent and you know the system that needs to be or the tools right that involves the tools like the terminal and the F system to be able to carry out whatever edits whatever changes need to happen with the code what they do is they build this s agent based on that agent computer interface right and they use a gp4 turbo as the base language model and here they were they're showing kind of the results but what's really interesting about this is that they have actually built an interface that really works well with gp4 turbo and they also reported this here you can see that this particular s agent that they have created also sort of works also using something like Cloud tree oppus right which is a very powerful model with very long context support and that one can solve 10.5% it's not as great as gp4 turbo but I think it's because this one was designed on top of the gp4 turbo mod so let's go through each one of the main components here and some of the interesting bits that this paper proposes right so we know that this language models right act as an agent by interacting with the environment right to an iterative way taking actions and receiving feedback this is basically the react framework being applied here so if you know about the react framework right the model comes up with a thought right and then it comes up with an action as well you can provide it information about what certain actions it has once you have that it combines this kind of thought plus action and then it gets an observation from the environment once it gets that observation then it has that extra knowledge that it will integrate and then it will take an additional thought in action and it goes through this iterative Loop until it figures out you know a solution so in a way the agent receives feedback from the command execution in the environment right and the environment that we're talking about here is you know you can have different environments but here what matters more is this computer environment which will be the terminal and your file system those are really important right because this particular agent needs access to like the file the repository right and needs to be able to also modify that and also it should be able to have access to the file system like if it wants to create a file or edit a file as well one main thing that's highlighted this paper without going into too much details is that this particular agent computer interface serves as an operator that provides the necessary information to the llm agent to carry out the task right that's what we're seeing here at the top so the building or designing of this particular interface is really important so we're going to go through the components that are important for this ACI you know framework there were some observations that were made on what really makes an effective G computer interface there a couple of things that the authors mentioned so action should be simple and easy to understand and this is about making things simpler because again we're not trying to optimize the large language model we're optimizing the interface right and the way we optimize this interface is we want to make it as simple as possible to understand because at the end of the day we need to give this back to the language model which might have you know constraints in terms of its capabilities a wrong reasoning so we need to make sure that information that's coming is relevant right it's simple to understand and so forth and know for this particular task it's important to have you know this concise documentation for example number two says action should be efficient right so again this goes back to the point of Simplicity is you may be doing a lot of operations you may be taking a lot of actions in an environment but it could be that all those actions right will make the system inefficient so what we want to do is we want to help the agent actually make progress towards a goal in a single step so if you do a bad design here right you can have many many many simple actions um and some of some that are not even needed and that will lead usually to you know this particular system not be able to complete the task so it's really important that the actions be efficient have a minimal set set of actions instead of having you know a bunch of actions and some that won't even be needed so this is just saying that you need to put a lot of careful attention to how the action set that you're basically enabling in the system number two here is environment feedback should be informative and again if you are providing the all irrelevant feedback right if you're giving it I don't know debugging logging or logs that are irrelevant then you know that won't be useful to the system right so you want to make that context that you're providing that feedback should be as relevant as possible to that particular task right to that particular step that the model is taking to complete that task I don't think this is specific to software engineering specifically I think this could apply to pretty much any task one interesting bit that more is more specific I would say to software engineering is the fourth one which is guard Wheels mitigate error propagation right so what we want to do here is we want to be able to handle errors and also apply something like a code syntax Checker right like a code lter for instance that can help the model kind of okay understand this is the particular mistake that was made and the agent can recognize and then fix those right so you can be in a loop right this agent can be in the loop trying to fix an error and it's just going to propagate throughout and it's not it's going to be costy and it's just going to take forever to fix the actual problem this section tree is talking about the actual design of the ACI the agent computer interface for software engineering specifically so this might not apply specifically to other types of tasks but this applies mostly to software engineering task right I I also like that in the paper they're very transparent about that and they're going to evaluate this particular system swe bench a benchmark for evaluating LMS on real world software issues collected from kidub the task is basically to give a code base and a natural language request such as a feature request bug report whatever that may be generate a code based revision or a patch that passes unit test which verifies that the issue has been remedied so what's going to be measured here or what's going to be taken as a metric is if the particular solution right provided by this model or revision can pass this unit test and can resolve the problem solving this tasks requires performing a number of software engineering subtasks so like bug localization which is really important program repair and writing test so this is why we need access to the file system we need access to like a terminal because that's where how we're going to be able to kind of complete this task so all of these things are again specific to sof for engineering task but if you are working in a different domain a different problem you might have different subtest that you are considering in your particular framework one important note here for this ACI that's being designed so it says that a well-designed ACI should help the agent understand the state of the repository given the previous changes it should also help the agent recover from mistakes remove unnecessary context from prior observations and suppress any unusual leny program outputs so there's a lot of little design choices that are happening here what should we provide the model right how should the model use certain context how to optimize that context that we're passing right what what's the output what's the feedback that we're passing to this agent so all of that should be taken into consideration this was optimized carefully there's different hyperparameters that were kind of optimized to come up with you know a good solution so this there are certain configurations here that we need to actually pay attention to in this particular page we see the different components of this swe agent interface right so we have search and navigation which is important right to search through files search directories and so on that's kind of typically what you would do if you were to try to solve a task right that's that navigation or exploration component is really key for this agent and that's what is mentioned here and interesting they say that if a search produces more than 50 results they do not report the results and instead suggest to the agent to write a more specific query so again that has to do with providing the relevant context you have the limitation of context length and also understanding longer context as well with these models so it's important to consider these choices when you're building your own agent there's also file viewer file editor context management a file viewer is again trying to figure out uh what file to open and so forth right that's about navigation file editing is obviously there's need to be there needs to be some patching that goes on right if the model finds a mistake what is the patching what's the edit that happens and there's going to be a lot of editing that's happening obviously if you want to fix a certain issue right if you're patching some code or fixing an issue in this code so this is an important one to have access to as well and you what you see here is these different commands scroll up scroll down edit edit and so forth for different component like search directory and so on which this model has access to if you look at the example here at the top see that these are special commands open search there edit edit edit of edit and so forth and here's a more clear example I would say so the file viewer right it shows like uh in what context right these particular lines lie on so it's like there's 400 lines above and 2,684 lines below so that's really important context for or information for the model to have and that's part of the observation and in terms of the response right you have when you're doing editing you can see here how it's also presenting the lines all that contextual information is really key for the model to have access to so you can read more about it I think one of the most important components of this I would say file editor file viewer it's not surprising to me the navigation and search we are we notice that the more we try to optimize these models with some type of search component is basically we're trying to to guide the model better right provide better guidance so I think this is a general type of insight that we're seeing from these mods right the ability to search better for relevant information that's needed as context but I think the most important part of this work is this context management you can have all that information you can provide all you know the file editor these commands can be well explained the model maybe knows what command to take on has a good understanding of this particular domain but if you're not passing the right information you know you won't get the right results with these models right we know that basically you know the idea of prompt engineering is basically that to try to optimize the context as much as possible to your particular task and this is where all of that applies so if you're passing the model in this case with the software engineering task the error messages the history processors right to keep the agent context concise and informative it's about conciseness and making sure it's informative right any addition information you're passing to the model and in this case there can be a lot of irrelevant information right because you are interacting with this external environment right which could be a terminal could be something like that that provides like logs these all these logs that you can even understand and you're just feeling that back to the model you know that that that won't really give you the results that you want and you have to manage the context a little bit better to be able to deal with errors to be able to to fix the solutions fast that's basically what this context management is about here are the experiments and this is the experimental setup so we have the data sets the swe bench data set right which includes 2.2k test instances from 12 different repositories of popular python packages and we're focusing on python here that's something to note as well uh because that shows you this is one is particularly constraint on python but it might not transfer over to a different language like go or Ruby and so forth right so just keep that in mind uh the results May Vary and there models the main model was used is gp4 and Claw tree oppus uh obviously they got better results with gp4 tuba because they were optimizing and making configuration decisions based on that model in particular and other models that were tried were Lama tree and deep SE coder which are also very powerful models and one question I had earlier when Lama tree was released how good is it at actually code Generation Um it turns out that for this particular task um you know not as great and that has to do with the context window limitation but I think it also has to do with the fact that this is a general purpose model that might have not seen uh so much code data or it's not really tuned to do code generation really well uh but it's important right if it's a general purpose model it's important that it's able to reason about this particular task and be able to perform this task uh that's something that you know probably we will see a lot more better results and systems and different ideas proposed to improve you know even the smaller molds as well on this particular kind of atic workflows that we're building so there are two basins that we used here which is this rag system and this shell only system and this shell only system it basically involves an interactive BAS system which asks the llm to resolve the issue by interacting with shell process on Linux and the S so agent involves this agent computer interface that was built with all the different functionalities and access to different tools that are needed to solve the task and the metrics that we use here are percentage result and average cost the main result is here you can see uh with the rack system the shell only agent and then we have an S SW agent right gbd4 turbo and Cloud trios was used this one um you can see how it performs on the different benchmarks so for S Benchmark bench which is the main result here we have 12.47 but also for the benchl we have 18.0 as well using GPT for Turbo and here some results of the different components right and it says you know the importance of all these different components like editor search file view context right and you can see how for this little cowboy hat emoji here that that's the best result right and we talk about different search Solutions in a bit but the editor with the linting on gives you the best result here right and this summarization um search sumarized search gives you the best there's different search that you can use as well you can do alterative search which is shown here an example of it is provided here which you know you use this kind of next previous search result it's like basically trying to mimic like vs code if you have search for something you know you get these pages and then you can go next next next and the model has access to that right you can use that command to go next or to go back or previous and so forth that's kind of iterative but you can imagine if you're doing that you can be going through like a hundred different pages of search results and that could get you know pretty pretty like long to process and so that could be very inefficient but if you're summarizing it right you can potentially even use another language model to summarize and enhance this search um but if you can summarize that give all the results in one single output then I think that's more efficient way to do this and that's what they show with the summarized search U version they they also show shell only ex results here as well how how the performance stps compared to the the main um result here file view as well interesting here like you may think you know the more information you provide the more uh you know the better the results but it turns out that there's kind of like a good amount of information that's good enough for the system and if you give it too much right you're adding a lot of noise this is something I always talk about with language models that you're trying to optimize the context and by doing so you may think providing more and more and more additional context you know will lead to better performance um but that's not always the case and this seems to be the case as well with this particular task that there's kind of a really ideal number of lines that you want to pass says if you're providing full history it the performance Goes Down And if you provide it with demonstrations um you can see so providing relevant context is key here so this one says human user interfaces are not always suitable as agent computer interfaces this is a really important point because we want to make these models work with any environment and interact with any tool but the fact of the matter is that you know those systems that we have built and those tools that we have built we have optimized them for humans right for human usage and so as an example if you were to use you know the shell only environment that interacts with this terminal right you have access to commands like CD LS and CAD you know those will give you some output and the language model can leverage those outputs but it turns out that you know the information that you're getting from these particular commands very inefficient according to this study and you can also use something like GP and find and there might be other clever and more efficient ways on how you interact with the terminal to get access to information then to fix things right and it says that if you do that you know you might get better results right but usually there's a lot of irrelevant information that you're introducing as well so you have that problem of introducing too much noise and less signal so they are hypothesizing that better localization is possible with faster navigation and a more informative search interface so here there's a lot of potential right as I mentioned the idea of using the Shell only search versus iterative search versus summarized search right so we have to kind of differentiate like how we feed information or how the language model should optimally use the information and optimally interact with tools as you know compared to how maybe a human interact with tools and the interfaces that humans um have access to and how they interact with those to this again goes through some examples when you're using I mentioned the idea of using linting right like a code linter also helps when especially with the editing component right because giving the model that signal whether there was an er error you know A Minor error or maybe a major error in the changes right of the code that's really important for the system to have uh because that will determine you know how effective that system is that complete in the particular coding task here's just a deeper analysis into the agent behavior and there are a couple of things that were highlighted here uh you know which are these um which they see as consistent problem solving patterns that emerge when LMS are are equipped with a useful intuitive Asian computer interface so there is idea of reproduction or localization in the first step so localization right so there's what are kind of the patterns that these agents are using to solve or resolve these this this particular task that they were given right software engineering task in terms of there are a couple of interesting patterns here and this table summarize basically the most common patterns right which is like editing editing editing and this has to do with reproduction right because there's obviously going to be reproduction of code that's happening here because it's obviously trying to fix these issues in this code this specific code code base and then there's also like you can see create edit python as well um you know edit python RM edit python edit there's a lot of like in combination you know edit in com combination with create right and that has to do with reprod reproduction and then localization has to do with more like um you know Finding information searching directory searching files and so forth and you can see that also that's very common pattern here um this particular chart is interesting it says that a majority of turns are spent editing right I just mentioned that but after turn five and onwards right remember this is an iterative you know system right after turn five you can see that it's spending most of its time right on two actions which is edit and python so you know just using these edit and python um you know you can also see it here from the table right edit and python is using those a lot to actually fix the problem right so she's trying what I understood from this is trying to like really get to a solution as fast as possible and that's something that is highlighted here as POS as well right and that's that's where the question of you know can the system find a really optimal path or optimal pattern of commands right or actions that he can take to solve the problem ideally that would be something that this model can figure out really quickly to do that wouldn't until potentially right having to tune these models on this particular domain this wasn't done in this study right but that's something I would have want to see as well if there's any kind of improvements that can come from that just giving it more knowledge or training it or tuning it on more knowledge like that I think that would be an interesting experiment too um but I also like that they didn't do that because they're optimizing and getting the most that they can get from this from keeping the model fixed right keeping the language model fixed and they also mention here agent succeed fast and fail slowly um this is an interesting Insight I think um what they're finding out is that um they find that runs submitted relatively early are much more likely to be successful compared to run submitted after a large number of steps or cost and I think this has to do with the fact that I think providing all that unnecessary context that you might not even need right you may think oh you know if this model is given more more context or it does perform more steps that it will you know potentially solve the problem but turns out that it actually it actually fails a lot of the time if it's taking too long and that those are good hints to have because you know that could save you in terms of cost um that could maybe give insights to this model if it had some kind of meta prompt or meta component that's looking at it and it's monitoring Its Behavior and how it's you know dealing with that particular stask and how strain to implement it and how fast it is right it's solv in specific part of the problem so I think that's an interesting insight as well um I think there there's more potential to do deeper analysis here um but that's something that you should definitely take a look at and they report some results and just want to mention that you can take a look at table 10 there's a huge like appendix here they do a lot of different analysis and so forth there's a ton of pages here you know with so many results like how they handle errors right and how they're passing in what's the template that they use as well because this is an agentic workflow and they're using react so all of those components right there's a lot of results here that I won't be able to go through in this video um but do take a look at that my hope is that this was interesting to you and that it can provide you some insights if you're building agentic workflows my takeaways of this paper is that the importance of that context management I think that's really key here like you have all this information from this environment that you getting us feedback you have all this information right from the actual code base as well how you're passing that information uh in an optimal way that's really key for the system the rest of stuff like react you know the file editor the file viewer the search and navigation I think are easy to figure out and yes it will involve some kind of experimentation or maybe additional experiments on what are the type of like actions you want to give them all and so on the react framework is very intuitive to understand I do encourage everyone if they're interesting in building agentic worklow to actually look at the react paper it does provide a lot of good insights into actually building these type of agentic systems and I like that they kept it really simple and they used the react framework right because they could have chosen to use some other more modern version of that which there are so many variants of react today um I think it makes the paper easier to understand and so forth so a lot of potential in this paper you know the metrics also could be different as well uh they could also try to enhance the swe bench test set because you could include different programming languages to see if all of this approach actually generalizes uh what types of problems as well you know what are the software engineering tasks um does something like this actually you know generalizes as well to like other other like Frameworks other types of software engineering tasks right and so forth so all of those are kind of questions that I have in mind and I think from here you know we will see more work around using agents for software engineering task so I'm very very excited about that and hopefully this video gave you like some insight and idea on what's happening with the space what are people interested in and so forth okay that will be it for the video please consider leaving a like that gives me a signal whether you're enjoying these videos and this format is useful and please subscribe also to the channel that also kind of helps the channel and it tells me that you want to see more of this type of content have a good one