Rethinking CI: Actions, AI Agents, and the End of Commit-Fail-Commit | DEM301

Key Takeaways

All right. Hi everyone. Uh my name is uh Salil. I'm a product manager at GitHub. >> Hello everyone. Uh is my audio coming through? My audio is not coming through. Now is my audio coming through? My name is Dennis Sun. I'm also a product manager uh at GitHub uh supporting Salil today. >> And we're going to be talking about rethinking CI, so actions, AI agents, and the end of fail commit or fit commit fail commit. So, actions usage has been accelerating at an unprecedented rate. At the beginning of January 2026, we were running 2.8 million jobs per week. The beginning of this year, we were running 550 million per week. Just a month ago at the beginning of May, we were running 850. That's a 60% increase almost in only a few months. And while a lot of this increased usage is coming from AI and agents, CI is still one of our primary ways that people are using actions. But, we've noticed a lot of uh our users are running into a bit of a challenge when they use actions. And they get into what we call the commit fail commit cycle. So, they commit some changes, they run a workflow, and something looks a little off. Maybe it's a failure, maybe it succeeds, but it looks a little weird. But, the only reason uh way that they have to solve this is they go, they try and make a small change, add print statements, something like that, recommit, rerun, and keep on going until they eventually break out of the loop. Now, how many of you have encountered this before? This sort of fit All right. Bunch of people here. So, I'm going to be talking through two ways that we are working on help giving uh all of you tools to help resolve and break out of this commit fail commit commit loop as quickly as possible. The first is going to be an agent-based solution using agentic workflows to help AI diagnose problems when they they up. The second is going to be a sneak peek of our Actions debugger, a way to actually dig inside the runner itself and help you troubleshoot and debug issues as they come up. >> I'm curious, how many of you guys have heard of Agented Workflows? Can you raise your hand? You? You guys are using it? Pretty familiar, nice. Awesome. Uh but for those who maybe aren't familiar with Agented Workflows, uh they are a way that you can uh create an Actions workflow but infuse agent logic in it. And the way that you're basically building them is you are creating uh markdown files uh and expressing what you want to do in markdown and then we turn it into Actions YAML. So, I'm going to show you an example here of uh a workflow. Uh this is a not This is not an Agented workflow, but I'm going to show you how uh a special Agented workflow that we call the CI Doctor can help you resolve uh and uh diagnose issues more quickly. So, I have a pretty basic workflow here. It's uh running a Python script that's going to read a handful of files that look kind of like this. They're a bunch of just key-value pairs uh and sum them all up and return a value. So, if I go and I try and run this particular workflow. >> So, we're looking at basically any typical Actions workflow right now. >> Actions workflow, but I'm going to show you how an Agented workflow can help you diagnose and debug it quickly. >> Perfect. >> So, if you look, this workflow has failed uh and if I dig into it a little bit, I'm going to see something in my Python script has failed. Something weird has happened. >> It cannot concatenate string not in something. >> Yes. Well, one thing I can do is if I go back to all my workflows, you'll notice right after this CI Failure Doctor workflow trigger. And this workflow is an Agented workflow. It's designed to look and see when workflows fail and when they do, it's going to run an agent that actually does uh that that actually analyzes and figures out what happened and provides root cause analysis and suggested solutions for how I can actually deal with this going forward. >> So, that's the AI failure doctor action that's running. That was triggered automatically? >> triggered automatically. This particular one is set up to trigger anytime that the demo sum key counts from files workflow fails. And what I can do is I can show you what this workflow looks like. What's actually running here. And if you look one special thing you'll see with agentic workflows is normally you just see a workflow file. In this case, we see the agentic workflow file. And what this is, if I pull this up, is you can see there's a little bit of sort of traditional YAML actions information here, but the rest is all markdown. It's saying that you're the CI failure doctor, you are investigating failed GitHub actions to identify root causes and patterns. And there's a lot of data and information in here about how it actually do the analysis, the different steps and components it should be returning, and and how to actually work through this. And this particular agentic workflow is actually one of a number of different pre-built templates that we have that you can find online and very easily add with a wizard. We have the CI doctor, there's a number that help support with issue triage or PR triage, automatic documentation. But in this case, this particular one is designed to help us identify and resolve open issues. >> So, just so I understand, what we're looking at is an MD file that you can create. You're using a template for the CI doctor, but can I go in and just like edit the edit the file as I want and it it would change the behavior? >> You can not only edit the prompt, but one thing that's really cool that you can do is you can actually use Copilot to help you build and edit that prompt. So, for example, I actually made some small customizations to this one to add in this slash command protocol because as I was going through it, I noticed it was producing some odd results, and I actually was able to use Copilot to help me modify this particular agentic workflow to adapt to my needs. So, I could add in a slash command to provide the CI doctor with additional information as I was going through it. >> So, you were able to customize this for your needs. What happens to this MD file? All right, MD files are easy for us to understand. Is this physically then used? What what comes of the MD file? >> So, when you actually put the MD file into your repo, you then use a specific command, which I'll throw up right here. Basically, what you have to do is when you create the the MD file, you run a command GitHub AW compile. AW is an extension for agentic workflows that you can set up and install inside the GitHub CLI. And I just run this GitHub AW compile. You'll see it's thrown up a couple warnings. In this case, it's telling me I should make sure to restrict it just to the main branch. I took that out just for this demo. But, what it'll do is it'll let you it'll basically run it. And in this case, since it's already compiled, it's not going to really add anything. But, what it does is it turns it into an actions YAML. Something that's not really readable for us as humans, but actions knows how to read it. And you basically every when you make changes to your markdown file, you recompile and you add it in. It all gets committed as part of your workflow. >> So, I as a human I start with MD, then the agentic workflows process lets me convert that into an actual workflow file. >> converts it to a dot lock dot YAML. We just put dot lock dot YAML because we wanted something to differentiate between non-agentic workflows because a lot of people are still going to have those. >> Love it. >> So, why don't we jump back and see how the CI failure doctor is going. Uh it looks like it's still running. >> Well, it's still running. Another question. So, per repo, can I have different files then basically that have different jobs? Can I share my agentic workflow files across repos? How does that work? >> Yeah, so what you can do is you can actually take any agentic workflow you've defined, you can share it with anyone else in your you know using your your repos. And just share it the same way you would any other file that's in your repo. And then those teammates can again customize it further or just recompile it and have it added to their their workflows. So, you can do you can basically treat these like any other file or any other workflow that you work with. The other cool thing is at GitHub we are building more and more of these agentic workflows to help solve common problems that we're seeing. So, the ones that I brought up before are ones that we've run into in the past, but we're continuing to add and iterate and all of them better over time. >> Love it. Looks like it's done. >> Looks like it's done. So, what >> This will have done is in this case if I go to my issues, it'll pop up the CI failure investigation. And if I look at this, it will tell me dealt with a type error because this particular file counts 13 contains string values instead of integer values. That's actually correct. I went in and deliberately made that error to make sure that this workflow failed, but if you look it'll say, you know, provide information. These are strings instead of integers, and it will talk about it will also indicate that this function that I have in there it isn't doing any type casting. Now, obviously this is this was some deliberate errors that I introduced, but it it's something where it can show you why things failed, provide you information, suggest findings, and even provide different suggested actions that you can take. So, in this case it suggests fixing the actual file. It recommends adding in an actual casting operation to force these values to be integers even when they're strings, and then making sure to even add a data validation step. Some people part of my actual workflow that I can use to to help this. And one awesome thing I can even do is now that I have this issue come up, I can actually assign it to an agent and an agent will actually go start a session and try and fix all of these different errors. >> So, this is really cool, right? So, at the end of the day what we're what we're seeing is you're able to trigger an automated agentic workflow based off of an event. In your case, it was the CI actually failing. Once the agentic workflow noticed the CI failed, the the CI doctor Agantic workflow, specifically that one, immediately kicks off. >> Yep. >> And then it produces an issue for you to review and understand what happened. And you didn't have to trigger anything on your own. It was kind of all in the background. >> Exactly. I I put it all together and and and it just runs on its own. One thing I could do if I wanted to is I can add in uh you know, if say I it found an issue uh and maybe I didn't realize maybe I I found okay, there's actually some other reason why this isn't working. I have the ability because of that customization that I was able to make, I can then add in uh more information about what's going on and and where the issue is. And and this actually can speed up my process cuz now, once I know the problem, I can just like I said, assign it to an agent and it will take care of it for me. >> I love it. >> Um or I can just assign this to to someone else. I can assign it to Denzil and he can take a crack at the making that change. So, this can work really well in a case where a workflow's failed, something's gone wrong, um and I have very obvious data to diagnose. But, one of the other problems we sometimes uh uh run into is your workflow doesn't fail. It succeeds but does so in a strange way. Something doesn't add up the way you would want it. Uh and in those cases, an Agantic workflow providing that information might not be enough. You might just need to start rooting around at things. And uh what we're going to what I'm going to show you next is uh the uh a sneak preview of our debugger. This is a way that we're giving for users to actually dig into the runner itself and figure out how to troubleshoot problems. So, let's pull this up and so, uh here I have a uh a PR that's open and uh there's a workflow that's going to run on this PR uh that will basically check any markdown files that have been modified uh and it will include them as part of the PR description. This is something that we actually use on our own team because we write a lot of documents in in markdown and it's an easy way for us to get that information and and sort of see in a in a well rendered form what this will look like. And so far I've added in this one file called page two, but I want to continue working on this particular PR, so I'm going to go switch over to VS Code and I'm going to actually take this file. I'm going to copy it and I'm going to make a another version of this which I'm just going to call page three. And I'm going to make a small edit to it as well. >> So basically right now we have a a workflow file that's supposed to merge all of them D files into one MD basically one view that I can look through the actions workflow. >> No, so it's a little different. What it's going to do is it's going to identify all the workflow uh the MD files that have been modified by this particular PR and then put links to them in the actual PR description. So that way I can just looking at the PR see uh what those files are and then move to an easily rendered version of it. >> Perfect. >> So I already had uh page two modified. I'm going to add in page three. And I I forgot to save this. So let's save that. Uh and commit this. And pop it up and if we go back to uh this to uh this particular PR, we're going to see some of these workflows are running. >> So we want to see then page three as another link. >> Yep, we should see So for example, this one page two, the link it points to is this. It's just a better rendered version of it. >> Got you. >> But we should expect it to see page two and page three, but for some reason this workflow is only showing page three. It's a little weird. That's not It's not what I'm expecting. >> So the action didn't work as you wanted it. >> Exactly. The workflow didn't didn't operate the way I expected and normally what I do in this case in the old world is I'd have to just kind of put in a bunch of print statements. I'd have to go in and open my workflow and try and guess what exactly the problem is and it might take me a while. But instead what I'm going to do is I'm going to use the debugger. So, what I'm going to do is I'm going to pop open uh this particular uh workflow and the job and I'm going to do a rerun and you can rerun with debug logging but I'm going to >> And this is a real sneak peek, right? Like >> Yeah, this is this is not out yet. This is we're we're still This is still an internal thing. Um but I'm going to enable this debugger rerun the job and it's going to kick off and when I open this up you see it's it's kicking off and you'll see it says waiting for debugger client to connect. And so what I do is I copy the link to this job. I go back to VS Code and I'm going to do GitHub action debug running job which is part of a slightly modified version of the GitHub Actions extension. I pop in this job and in a second I should keep get connected to the job that I'm running. And the way that we've set this up is that we're using something called the debug adapter protocol or DAP. Uh basically DAP is a protocol that is used by lots of different debuggers um for uh debugging debugging different applications. It's standardized protocol. We've chosen it because it allows you to tightly integrate with debuggers. I happen to be using VS Code right now because it's my application of choice but any uh application which supports the DAP protocol will work. So, the lead engineer on this is a big fan of Neovim. He does all of this in Neovim. Uh and it's something there are lots of different integrations for DAP and it's something that that can be added pretty easily. >> So, I was able to basically connect to the actual actions runner using the URL you had. Can we go back to the the browser really quickly? It seems like we I you even see the debugger is now connected. So, no matter what client I use, basically I'm connecting to this runner's uh computer >> That's right. I'm you're connected to the runner and you're actually operating and you can run commands on there. So, uh for example, I should be able to uh you know, if I were to do something like a run uh ls here. I can see the different files I have >> run commands on the runner itself. >> You can run commands and that's that's the whole point of this. It's being able to run commands as well as if I go to the actual VS Code debugger, I can see in this variable section all the context information on here. So, if I want to see what's in the GitHub context, I can pop it open and look in there and see all the different values. I can see environment variables that are defined. Um I can see secrets that are defined, although I can't actually see the secrets themselves. We obviously want to redact those. Um but you can see kind of all the information and be able to try and debug and work into this. So, let's let's jump in a little to the next step. So, um my first two steps, checking out the repository and setting up Python, probably not likely that either one of those two are the issue. But if I go to this step of collect changed files. So, what this basically is doing is it's creating a temporary directory uh and then it's going to basically be writing uh doing a git diff between uh two commits and then writing that value out to a uh to a a particular value. So, why don't we scroll down and see what those values are? So, my head value is 309c9. Well, if I go back to my PR, 309c9, that looks right. But then if I look at my uh my base sha, that's a07ff, well, hey, that's the commit right before. If I go back to what the head of main is. And look at that. Now, my main The head of main is something totally different. So, it looks like I'm making a the this particular uh the way this workflow is written, I'm looking at the a bad comparison. So, I'm instead comparing the last two commits of my PR. That's not really that helpful. That's why I saw the page three and not page two. So, clearly the two values I'm using here in this the that I'm setting it with these environment variables are wrong. So, I need to figure out what the right answer is. >> debugger is basically you're able to view all the variables that that the debugger that the runner is using using the debugger. And now you've been able to identify, "Hey, my base sha and head sha is what you're validating with the variables." And notice I notice that the base sha is wrong in this case. >> Exactly. So, what I can then do is this the base sha I'm looking for must be in here somewhere. So, let me look in the GitHub context. Let me look at the event GitHub event context. There's a pull request on here. Let me check that out. And if I open that one Oh, there's a base here. Let me check that out. Well, base has a sha. And if I go B49A57, that looks right. So, what I should probably be doing is using that instead. Now, I could stop here, go and make those changes and and and go ahead. But, what if I'm wrong? What if I want to just double-check this? Well, let me actually do this. I can do run and I can basically I can uh just run the actual commit that I want to do. So, I can do a git diff name only. Uh and this one will be uh let's copy this value. >> So, now you're bringing those variables in and you're going to rerun it. >> I'm going to rerun it. Now, because I those this particular uh you know, base one is not set, I'm going to have to do it manually. But, it's not the end of the world. That's pretty pretty manageable. Uh then I'm also going to I can go and I can pull down uh what the value of after of the after was. If I find that here, I can grab that and put that in here. If I run that, now I see both files. Okay, that seems like it was the problem. Um and what I can do is I can actually run through each individual step and go through and actually run everything um and see all those values and see exactly how everything works and and regenerate the files, even try running all the steps at the end. But, I know what the problem is now, so I can always go back and make that change and rerun it and be able to solve my problem. And now the debugger has told me exactly where I was looking for. I didn't have to commit a bunch of stuff, do a whole bunch of print statements, add in extra steps to try and figure stuff out. I jumped in, I did some research, and I was able to find what I need. I can make my change, commit, I'm done. >> So, it is a real debugger basically for our runners, right? I can jump over. I can I can really play around and run run commands and do all that within the context of any DAP accepting client. >> Yeah. And and and we and DAP is is designed for debugging, so it's designed to handle this sort of stepping over, being able to return variables, and and contain that information. >> Love it. >> So, that's those this uh you know, one thing that you're probably all wondering is for both agentic workflows and for the debugger, when can you guys actually play around with these? Uh well, the exciting news with agentic workflows is that is going into public preview next week. Uh you can if you search for our GitHub agentic workflows online, you can see our uh uh the the preview site online. We can actually pull that up now. Um and it'll look like this. Um and this has information about how you can quick start with the CLI, as well as being able to see lots of different examples. So, brought us some of those before, issue management and uh continuous documentation, continuous improvement. We're working all of a lot of multi-repository stuff too to help you coordinate different repositories. Um, but you can find this information there and it will be going uh in public preview next week. Now, with the debugger, this is still an internal project. This really was kind of a sneak peek at what we're working on. Um, it's something that we know is very important for our customers and we want to get it out. Uh, however, given that this is allowing direct access to your runners, we are very concerned about security and abuse, which is why we're spending a lot of extra time to make sure that we've given this the appropriate level of protection so that when you run this uh and when your teams are running this, they are uh getting the maximum value without risking any potential security holes. And that's something that we're working spending a lot of time trying to make sure we have done well. Uh, and so I don't have a uh a date on when it's going to be released, but since security is so paramount, you know, we want to But we also want to show you what we're working on because we think it's pretty cool. >> If there is any feedback, you have any thoughts, any other agentic workflows, debugger, please, please, please reach out on any of the public forums on GitHub. Uh, best way for us to kind of look at that and understand even for stuff like the debugger that we're actively working on still. >> And in particular, agentic workflows, this is something that's being iterated on and like I said, we're it's going in public preview next week, so getting feedback and also learning about the cases where you think you might use agentic workflows is really valuable to us because it means we can help build some of those uh pre-built agentic workflows to help speed up and accelerate your process. >> Fantastic. Lovely. Thanks, everyone. >> Thank you.