Ensuring your code works when AI testing isn’t enough | BRK208

Key Takeaways

Well, good morning, everyone. Um, my name's Simon Wilson. Uh, this is a slightly retitled talk. I think the new title, it's the same talk but a better title. I'm going to be talking about verification patterns for agentic engineering. Um, uh, my clicker just stopped working. Go on. Go on, work. Oh, that's frustrating. Sorry about that. I'll do this instead. Um, >> [clears throat] >> so, um, so, I have been using LLMs to help me write code since 2022, which in this world makes me a grand elder of the space, which is completely absurd. It's been 4 years, but, um, it's such a new space. This is me talking about GPT-3 and how you could get it to write you little SQL queries and explain how code works back in When was this? June of 2022. And I've been exploring them on a constant basis ever since. And there we go. Um, and in further grand elders of the space, this is Andrej Karpathy in February of last year, so again, positively like years ago in agent times, when he introduced the term vibe coding. And he said, "This is, um, when you've fully given to the vibes, embrace exponentials, and forget that the code even exists." Um, no, most people did not scroll to the bottom of that tweet where he said that this is for fun, it's for weekend projects. You wouldn't want to write code seriously. A bunch of people took vibe coding and started applying it to every element of using AI to help write code. I think that's a waste of a term. I think that vibe coding as a sort of irresponsible way of building things is is is much more useful. But, the big question I've been thinking about recently is what should we call vibe coding when we when we're if it's applied by professional software engineers to ship real software? Like, we're not just giving into the vibes. We're actually building software that we care about, that we want people to be able to use. What's the term that makes sense for that? The term that's bubbling to the top at the moment for that is agentic engineering. I think it's good enough. One of the Chinese AI labs was using this in one of their press releases, which was the sign to me that, okay, it's spread far enough that it's okay to to start using. And just to check, of the people in this room, how many people here have shipped code to production without reading that code? That's quite a few people. When I pitched this talk 3 months ago, I think that would have been nobody at all. Like the speed the rate at which this has taken over is kind of shocking. So, on one hand, this is a very new problem, right? Coding agents um And by coding agents, I mean the class of software where the LM both writes the code and then executes that code and can test it and iterate on it and improve it. They only really got good in November. Like November 2025 is when Opus 4.5 came out, uh GPT-5.1, the coding harnesses got really good. And suddenly these things went from a sort of interesting curiosity to something you could use as a daily driver, something that was a reliable partner in getting work done. I've been calling this the November inflection point for these tools. The reason they got so good in November is both OpenAI and Anthropic spent all of 2025 working on code. They were doing reinforcement learning against coding tasks. Turns out that's a really strong pattern when you've got something with an obvious right or wrong answer. And all of those advances in that year came to came to a head in November, just in time for the Christmas holidays when everyone went home and started tinkering with these things and trying to figure out what they could do. But on the one hand, they only just got good. There's something you should never do is talk about code productivity in terms of lines of code written. So, that's exactly what I am going to do right now. Um 2004, Code Complete, like very, very classic software engineering textbook by Steve McConnell, he said back then that um the industry average productivity for were product is about 10 to 50 lines of delivered code per person per day. And I found another thing from 10 years later that said that if you can do 600 lines of code, that's an high outlier in terms of developer performance. I am literally writing 600 lines of tested and documented code on my phone while I'm walking the dog on a regular basis right now. Like the amount of code that we can produce has just gone up by a terrifying magnitude. But is the code that I write with my dog any good? Is kind of the whole question, right? How do we accelerate? How do we work at this much higher velocity while not just producing slop and and rubbish that doesn't work? I call this a new problem. Turns out it's also a really old problem. Think about a company like Microsoft, right? Microsoft, 100,000 plus developers, all varying levels of experience of of of experience and expertise around the different parts of the code base. How do they ship anything at all? How do large companies where you can't have one person review 25 million lines of code? How do you get quality out of those? And I think the big challenge for us is we need to figure out what are the things that work for these giant companies, these giant teams, and how do we scale those down so that they run on our laptops, so we can use those same kind of ideas, but for the code that we're writing ourselves? I actually had a an earlier version of this revelation about 5 years ago when I realized that That's interesting, isn't it? Uh We're back. Um I had an earlier version of this when I realized that a lot of the stuff I'd been doing at a large organization, I used to work for Eventbrite with engineers in like five different time zones, a lot of those processes were really good for personal productivity as well. I started doing all of my all of my software with fully um comprehensive automated tests, like docu- um comprehensive documentation, and I found that rather than slowing me down, it sped me up because I could run 100 projects at once and treat every single one of them like it was a new contributor every time I rebooted on it. And so, that we're now trying to having to apply to the whole field of software validation. Cuz fundamentally, our job is to produce code and verify that that code works. Like it's no good just churning out code now. We need code that is trustworthy, that other people who are reviewing that code feel com- confident that we've we've done the work to make sure that it's all there, code that we can feel proud about, code that we can maintain into the future. But we're doing it at this at this massively accelerated rate. Cuz the the typing code into a computer bottleneck is gone, right? That That one bit of our job where we sit down and tap things on the keyboard, that's been automated. There are so many other bottlenecks that this reveals. I feel like one of the challenges with verification is it used to be that it would take you like it would take you a bunch of time to build a feature because you were constantly twiddling with that feature as you went. So, all of that invisible QA work was just part of our daily part part of the daily grind was you change a line of code, you hit refresh in the browser, you make sure it's all working. You actually do quite extensive manual QA of things as you're building them. Now, I can one I can one shot it with Claude, come back to a thousand lines of code, none of I did none of that work along the way to make sure that it was working. The most interesting example of this I've encountered was back in October again, many ancient ancient times by the standards we're moving. And that was this organization called StrongDM, who were doing something which they called a software factory, but I've also seen called a dark factory. The idea of a dark factory is if you've automated enough of your factory, you don't need human beings anymore. The robots do all of the work. So, you may as well turn the lights off. There's no There's no reason for illumination if the the work is entirely automated. The so- a software dark factory is the same kind of idea. How do we build software in a way that is so automated that that the lights for the human beings are not even required anymore? And so, StrongDM, this team that was just a team of three people um working on these ideas. They had two fundamental rules. The first rule was the code must not be written by humans, which was radical when they started this 9 months ago. Um today, not so much of a surprise. I think quite a few of us um are not typing code by hand anymore. The more exciting rule was rule number two, code must not be reviewed by humans. And that one deserves a big exclamation mark. The um thing I found so interesting about this is they were building security software. They were building software for managing authentication, issuing new credentials to users, and all of that kind of stuff. This is you would think the most security-conscious software you could possibly build. And they used a bunch of techniques for this, which I think are worth studying and borrowing for our own less ambitious work. Um the first one is that they had an army of agents that were testing that software. And they um they reused the term scenario testing for this. The idea is that you have scenarios described like a user must be able to log in, do this thing, buy put something in the shopping cart, and sign out. Those scenarios become the test scripts, and then you have all of these automated agents that are just running through different scenarios trying to find edge cases. Well, it's very much the automated version of a really high-quality QA team. Um so that was step one. And then the other thing they started doing, which I thought was fascinating, is they built something that they called a digital twin universe. Cuz the challenge they had is they're building software that integrates with Slack and Okta and Jira and all of these different enterprise platforms. And all of those platforms have rate limits, and they're occasionally flaky, and they're not something you can run 10,000 tests an hour against. So what StrongDM did is they built clones of those technology partners. Um this is their Slack clone that they built. They put one of their coding agents on the task of rebuilding Slack. And what they actually did is they took the API documentation and the client libraries for those for those APIs and use those as the specification for what to build. Plus, agents will know what Slack is already, so they can take a pretty good guess. And they span up actually as little tiny Go binaries, these complete duplicates of the bits of Slack and Octo and so forth they were integrating with, which meant that they were un- no rate limits. They could they could hammer these things as hard as they needed to. They could even simulate weird bugs that they found in production by having their digital twin spin up a a clone of those bugs. And if we zoom in, we can see that this is one of their testing agents saying, "Hey, I'm Janet Sanchez joining as an employee in general." This would then trigger a whole sequence of um credentials onboarding and all of those kinds of things, entirely simulated. There was one aspect of what they were doing that I did not buy. And that's um their third rule. They said, "If you haven't spent at least a thousand dollars on tokens per human engineer, your software factory has room for improvement." This was token maxing before anyone was think was was calling it token maxing. Um just this morning, I heard that Uber has set a cap of $1,500 per engineer per month on the tools that they're using, which I think is a lot more sensible. So, there are lots of lessons I want to take from the StrongDM thing. Spending $30,000 a month is is not one of them. So, if we're not going to spend $30,000 a month on automated swarms of testing agents, what are the patterns that we should be using? Um so, I'm going to go over some of the patterns that I've been finding to work over the past few months. And the thing about this field is it's all so new. Off like we're discovering patterns all the time. I'm trying to show you the stuff that that seems to be holding, but if you ask me in 6 months, maybe maybe I will have found that there are much better ways of doing this kind of stuff. My favorite one, um for the times that I do review code, and I've got a policy where I think very hard about the stakes at play. If the code is like a little like icon on a web page, I don't really care too much about the details. If it's part of an authentication flow, I'm going to review every line of code very, very carefully. But, reviewing code is kind of miserable. Like, nobody wants to spend their time reading through a thousand lines of a get GitHub pull request. And so that you can kind of glaze over when you're faced with those. Something I started doing that works really well is what I call active refactoring. And this is one of those things that's a complete anti-pattern if you're working with other human beings. Somebody ships you a pull request and you go through and you comment on every single decision that they've made. You tell them to rename this variable and refactor this thing. All of those kinds of things. With human beings, it's kind of rude. And if you've got even like a couple of hours of round time, you can end up with the review being stuck for weeks. Agents, you can't be rude to an agent. If you're rude to your agent and it gets upset, you can reboot it and it forgets everything. Um and the I find that nitpicking the design of these things, actually going through and saying, "Let's call this this thing instead. Refactor this bit. We can remove duplication here. Oh, I don't get how this works. Explain it and add comments." I can It's much more fun for me. It's much less boring than just reviewing the code and clicking accept at the bottom. And I come out the other end of it actually understanding what the code does. So, this this active refactoring has been working really well well for me for those pieces of code that I feel do deserve the efforts. Like, an example prompt some refactor the tests to reduce duplicate code, rename variables for consistency with this other file. I do things like this all the time. It works really, really well. And it makes that reviewing slightly less miserable. Um Cuz really, it's about considering the stakes. It's about thinking about what are the stakes of these bit the bits of code that you're reviewing. And looking for the seams as well. The seams between different pieces of code. A really important I I feel like we're evolving now. We used to mostly write code. Now, we're much more software architects and software designers. And so, really, our job is to make sure that those high-level those high-level aspects and the seams between the different systems are as good as they can possibly be. Um A great example of a way of doing that is I think one of the most important decisions we make is API designs. Right, anything where you've got an API which some other code is going to be called or some other developer is going to be using, that's where you want the design to be flawless. You want the design to be as good as you can possibly make it. The best way to verify an API design is to write lots of code against it. Agents are really, really good at writing throwaway code against your experimental API designs. I do this all the time. I'll say, "Review the last commit." Um this is a great trick for getting an agent to understand what it is you're talking about. You You If you're If you're doing good commits, which you should be, just saying "Look at that last commit" is enough for it to go, "Oh, I understand now. It's this feature in this part of the code base which has been changed in this way. So, review that, then brainstorm prototype three features against that new API." You can tell it to run in the branch, you can have it in a work tree. It will churn away and you will get the equivalent of if you'd asked a bunch of co-workers to to hack away at your API, but you'll get it instantly. The feedback from that is absolutely invaluable. I've been using this I've I've much more confidence in the APIs that I'm designing because every single one of them has been exercised in sometimes weird ways by the agents on my behalf. And that ties into just the general idea of prototyping. Um Something I do Something I've I've always been a prototyper. I've always loved building little interactive prototypes or proof-of-concepts of things that I want to build. It's not just for the interface, it's also to prove that something is possible. Like the big question with so much of the software we build is, "Can we do this?" especially given the tools that we have available. "Is Redis capable of serving up this combined timeline?" All of that kind of stuff. I love using prototypes. That prototypes are effectively free now that we've got the I got agents that can do them for us. So, what I'll do is I will use the agents to build prototypes, and then later on I can use those eight prototypes as input to the agents to get more pieces of work done. This is what I did just the other day. Um, I'm building I'm building my own agent because in 2026, the hello world of programming is build your own agent. I'm building an agent, and I've decided to add the feature from Claude where if you copy and paste a large block of text into the Claude text input, it spots that it's a large amount of text, and it treats it as a file. So, I am prompted God, I can't even remember I think this was probably Claude code. I told it to build a prototype text editing where you can type directly in, but if you paste more than X characters, pick X, I don't care, it treats that as a paste file event. And it spat me out this, and this is a little text editor, and if you copy and paste a big chunk of code in, it shows up as a pasted file at the bottom. This is great because I've now got a verified version of this feature. Like, I've tested this. I I've tried it in a couple of different browsers. I know that it works. And so, later when I come to build the actual feature, I can tell I can tell my agent, "Add a paste file feature based on the prototype in file-paste.html." I am hoarding prototypes of things now. A lot of the most interesting projects I work on right now have spent over a month in the prototyping phase with me just trying out little different different patterns of different things that I think I'm going to need. And eventually you get to the point where you're like, "You know what? I've covered all my bases. I know how to build this thing. I've verified that I have working code for all of these pieces. It's just a case of cobbling it all together and and and building that into production software." Another thing I've been tinkering with quite a lot recently is agentic documentation. And this I never thought I'd get to this point. I've I've always been very firmly of the opinion that um if I'm going to read a piece of text, the person who wrote it needs to have put more effort into that text than I put into reading it. Otherwise, they're a vampire on my time. But I've realized that that's absolutely true for for blog entries and for anything where somebody's trying to convince me or use personal anecdotes or argue for a position. Code documentation has none of those characteristics. The thing that I want from code documentation it has to be accurate, it has to be short and totally uncreative and tell me exactly what the thing does. Agents are very good at uncreative prose. So one of the one habit I've picked up recently is whenever I'm about to land a pull request or land a branch, I will start with this prompt. I'll start with diff against main, again using Git as the input to my agent. Run diff against main and review if any documentation needs to be updated. Sometimes it'll won't find anything at all. If it does find things, it'll offer to update them for me. Sometimes I'll say yes if it's sometimes I'll say no. But this means that I get The main thing this gives me is confidence in my documentation. Something I've learned over time is documentation trust is very difficult to earn and it's you can lose it instantly. The moment somebody consults your docs and it the docs mislead them, they're never going to read that documentation ever again. So having docs that are reliable, that people can trust over time is super important. And we've now got this very powerful mechanism for ensuring that our documentation stays up to date. Oops. Um a very strong rule I have about agent LLM generated text is no opinions and no rationales. Often the agent will throw in a little flowery that says, "This feature exists so that X, Y, and Z." And most of the time that's not quite why I built the feature, which means it's misleading. So I edit that stuff out. I've got a lot of commits. This commit message was just tweaked some overly promotional language. There was a code comment that said, "The whole point is to persist an async response." None of that. Like it's very important we keep those things out of our code bases because people lose trust. They they they won't trust documentation that that feels artificial or overly promotional in that way. Here's a great example of one of those things where there are there are things that every development team wants that are just too much work to put into practice, or at least that's how things were in the pre-agent times. My favorite example of that is continuous deployment, the thing where anytime you push a branch or a pull request or push domain, something trans light and it deploys a version of that to hosting somewhere. I use continuous deployment on my main branch for a few of my projects. Continuous deployment for previews, I think is just a it's a there are no downsides to being able to do this. If you have a robust preview environment with good authentication or stuck on a VPN or whatever it is, if you've got something you can trust, but this this takes the risk of reviewing code by looking at it down to almost zero. And that really is one of the most important ways to verify code. Just reading code will never tell you what you need to know compared to actually putting that code through its paces. Having preview environments, having automatic deployments of the of of of the pieces of code you write gives you so much value there, both for your own testing and for sharing and and and collaborating with other team members. And the great news is that this used to be really difficult. This used to be like an engineering team for a couple of weeks to get this working. You can try Promptly say, "Build a GitHub Actions workflow that deploys new PRs to a fresh fly.io machine or cloud run instance or Vercel or pick your hosting thing of choice." And then you tack on, "Ask clarifying questions first." And if you don't do that, they'll just make stuff up. That will get you get all of the information that that the agent needs out of you and into it, and you'll get a you'll get that preview environment you'll get the sort of dream preview environment you've always wanted without having to spend any time messing around with GitHub Actions at all, which is delightful. Um a bigger topic that I've been exploring recently is the idea of reducing the blast radius for mistakes. So, we're writing code like like um agents write code. We try and make sure that our code is right. Mistakes will slip through. The same is true of human engineers and of coding agents. The coding agents' mistakes will probably be weirder and maybe a bit harder to to identify cuz of the the overconfidence that they that they display. What but there are a whole bunch of techniques in software engineering that you can use when the code that you're writing isn't necessarily completely trustworthy. If you look at Mozilla Firefox, Firefox has a whole bunch of security issues reported at them. Very few of those are significant priority one issues because they run everything in so many layers of defenses that a bug down in this part still can't break out of the browser and read files on your computer or whatever it is. So, three of the things that I've been playing with a lot recently. There's content security policy HTTP headers that can stop web pages from loading code from external domains and restrict what they can what what what they can do if something does go wrong. Sandboxed iframes are absolutely fascinating. It turns out you can have a area of a web page, you can have an iframe on your web page that can run malicious JavaScript which can't do anything bad. It can't steal cookies. It can't break out of there. And these things are trustworthy because they're used for banner ads. Like we've got decades of We have several decades of of of hardening on our on our sandboxed iframes. They are not Finding documentation for how to do this is almost impossible. But if you ask your frontier agent about this, if you ask GPT 5.5 to help you get sandboxed iframes set up, it's pretty good. It's It's getting It's It's taught me a huge amount about this field that I've been trying to figure out for for years. And then the really big one is WebAssembly. Like WebAssembly mainly it was invented as a browser browser technology so you could run like compiled C and C++ code in a browser in a sandbox where it can't do anything bad. More recently, there were web assembly technologies that run on the server. WASI, the web assembly server interface, I think it's system interface. This thing means that we can start using web assembly in our server-side programs to run code in a very robust sandbox. Code that can't read files, it can't make network connections, there's a strict limit on how much memory and CPU can use. I wrote pieces for her that does this for running Python inside of web assembly inside of Python just a few days ago and it seems to be working cuz I put GPT-5.5 in it and said, "You are running in a sandbox I have built. Try to break out." It couldn't. I want to try Claude Mythos when I get access to that. But, this is kind of important. Like, a problem I always have with sandboxes is I don't trust them. If you've put a bunch of agents in a sandbox and have them try every trick that they know to get out of that, that can give you a level of confidence that you didn't have before. Amusingly, a bunch of people have reported that when they try this in Docker, the agents are very good at saying, "Oh, I recognize this. This is Docker. If I connect to this host port and as with this thing, I can start running root commands." So, it's good to test your sandboxes in this way. A sort of related idea is I now have zero tolerance for flaky tests. Like, often, if you've got a test which fails one out of 50 times, you know it's going to be a miserable job to track down what's going on with that. And as a human being, I often look at that and think, "Well, it's going to take me between an hour and several days to get to the bottom of this. That's not worth it. I'm just going to tolerate and put up with this." I now have zero tolerance policy because the agents, I don't care about wasting their time and they're very, very good at simulating the conditions that leads to a flaky test. Just the other day, actually with one of my sandboxing projects, I had a thing that was crashing in GitHub Actions CI against Python 3.14, but working fine on my Mac. And I literally told Codex, um, "You've got Docker. Try and reproduce this thing." And it did. It reproduced enough of the Linux environment that GitHub Actions runs as a Docker container on my Mac to replicate the bug, figured out the the figured out the the the root cause, gave me a very confusing patch that I had to interrogate about its accidents and all what it was doing, but it got me to it got it got me to a solution. This is the kind of thing like I could have done it myself, but it would have taken me a heck of a lot of sort of poking around in Docker. I literally ran this in a background window and forgot about it. And when I checked again 15 minutes later, it had solved the problem. So, we don't have to tolerate flaky tests anymore, which is is such a such a huge relief. So, this extends as well to any other tool that helps engineers that any tool that helps engineers, any tool for human productivity, almost always helps the coding agents as well. And that's all the tools I've talked about already, but things like onboarding documentation, all clawed.md is is it onboarding documentation for your agent. We should have had that before. We just didn't really feel incentivized to do it. Um, things like a continuous integration, code linters are fantastic. I'm running linters against everything now because the agents figure out those gnarly little error messages and warning messages and sort them out for me. Code formatting, debuggers, logs, in some cases entire programming languages. I never really got to grips with Go in the past, um, just because of that learning curve that I I would have to climb to get productive in it. Go is a fantastic um, language for coding agents. Anything that compiles quickly and has really good error messages, the agents can prac- practically brute force their way through through building things. And you get to learn uh, it's a great way to learn a new programming language is to generate overly ambitious projects and then and then sort of pick through them and figure out how they work. So there's an aspect of this that we've seen before where if you think about um like website accessibility, website accessibility it was always a difficult pitch to make to people frustratingly that they'd have to spend the time on this. And then the moment people realized that Google that SEO for Google, Google was just the world's most um prolific site-less user, the moment they realized that, they got all on board with accessibility and semantic headings and so forth. So accessibility was the trick that got um so SEO was the trick that got people to care about accessibility. Agentic engineering, I think it's a trick to teach everyone good engineering practices. So stuff that's been around for literally decades, it was always useful, but now because of the acceleration that we're seeing, because we can produce code so so much faster, because we now all have to work as if we were sort of mega corporations but just on our laptop with our our armies of agents, all of these tricks are are worth learning now. Um I'm actually rereading The Mythical Man-Month from 1978 at the moment and three different chapters feel directly applicable to agentic engineering in 2026, which is kind of cool. So if we're going to build these dark factories and my factory's not really a dark factory, mine's more of a sort of like slightly gray hazy factory. I'm still I'm still reviewing the code when I need to. It means that we the most important thing for us to do is to build these verification machines. Anything that we can build that helps our agents and helps us verify that the code is good and verify that that that verify that it works, verify that it's high quality, anything like that is is really important. I feel like if we're going to use these tools to build worse software faster, we're missing a trick. Like we should be using this stuff to build better software faster. We can have higher quality software that's easier to maintain, that has more features. All of those that's one of those two out of three except that we get to have all three of the things. And I think that's really exciting cuz fundamentally we get to reinvent how software is built. Nobody really asked for this. Like a year ago we didn't think we'd have to reshape the entire like software development process. But that's kind of what we're doing. And nobody really knows what the end state of that looks like. Nobody's entirely confident about it. But it's it's such a thrill to be here for the moment where we get to try these things out and see what works. I write a lot about this stuff. Um mostly on my blog. I'm on simonwillison.net. Um and you can follow me on Mastodon and Blue Sky and Twitter and all of those kinds of places. And I've got quite a bit of time for questions. So what would people like to talk about? There's a question right here. I think we have somebody running around with a microphone. Yes. >> Hello. You talked about uh Mythical Man-Month. Um I really like all this information you're giving. And so uh could you just expand on what you're reading and listening to? >> Yes. Okay. So the Mythical Man-Month stuff in particular is um there's a concept in that of Oh, I've forgotten the the term that they use now. But there's this conceptual integrity, right? Where if you're building a large system, the thing that matters the most is that you have concept that the system has conceptual integrity such that you can think about it and understand what it's doing and explain it to other people. And I found with coding agents stuff, that's a really big risk of that going away. Like I've got projects where every feature that idea I came up with was just another prompt. So I'd prompt it and it would add a new feature and you end up with this thing that's this very weird shape of a product. And if you asked me what it can do the next day, I can't remember cuz I've lost that feeling conceptual integrity about how it all works. And again, that's not different in 1978, but actually it's 100% applicable to what we're what we're doing right now. I think in January of this year, I had my own version of sort of coding agent psychosis, where we had these new tools and I'm like, "Brilliant. I'm going to take on the most ambitious projects I can possibly do." So, I rebuilt JavaScript in Python. I did a I built an application server in Go that could do like lazy loading and proxying and all sorts of stuff like that. I built about four or five giant projects, which I have not touched since February, because they were terrible ideas. Nobody wants a Java a slow buggy JavaScript interpreter written in pure Python. That is not a thing that has any kind of like market fit at all. And I built it and it was fun. I don't regret it, because I learned a bunch from it. But, just because we can build all of these things doesn't mean it's a good idea. That's sort of um that one of the big challenges right now is personal discipline, which I have a enormous problems with. And again, looking back at things like the Mythical Man-Month, it's it's basically their constraints were so different. Like, they were like a thousand machine instructions per year per per engineer was was a good good good score back then. But, the the thoughts about the sort of the shape of the software completely apply today. I'm going to glare at people until I get another question. There's one right there. >> Who was first? >> Okay, so all of this is very interesting and it seems like a lot of homeworks, a lot of things to do. Um so where can I find like good templates, things that apply these principles or or I don't know, repositories where you can start getting things that work and start adapting and adopting them? >> So, it's amazing that you asked that, because sat in the front row we have Jesse Vincent who built Superpowers which is a set of skills for Claude code and Codex and so forth which makes a lot of this stuff in. So, I would encourage you like I think it's that that they're all of these pants are still being evolved. The there are there are some very interesting sort of starting points. Superpowers is a really good one for things like brainstorming ideas and then doing test driven development on them and all of that kind of stuff. But honestly, the most important thing is just to try it out. I think it's a great time to take on new side projects right now. I've been doing game development which I've never tried before in my life and I've spun up about half a dozen games. Some of the I I don't know Rust. I've got a game like a 3D game that I built in Rust. They are terrible. The big thing I've learned is that the hard bit in game development isn't drawing pretty things on the screen. It's coming up with a compelling game loop and the concept of the game that's actually fun to play. I've not cracked that one at all. But it's so much fun and it's such a great way to start exercising the sort of muscles and figuring out what works and what doesn't. And that concept of play just generally like the only the problem with AI, the problem with LLMs is they have this thing that we call the jagged frontier where there are things that they're really good at and things that they're really bad at and it's almost completely unpredictable which things they will be able to do. The only way to learn is to try them out. So, you constantly throw overly ambitious projects at them and see where they completely fail and where they actually go further than you thought they were going to do. A really good trick is if you try something with an LLM that fails, stick that in your pocket and try it again in 6 months because often something that didn't quite work before suddenly now starts working. You you may be the first person in the world to to figure out that an LLM coding agent can now build this kind of thing just cuz nobody else has tried it yet. It's astonishing how many advances in this field come from people who just started playing with it and tried something nobody else had tried before. Uh yes, at the front. >> Hi Simon. You You mentioned you tried things in Go and Rust and and and Python. How do you think about the different languages and how they work in the agentic flow and which ones are you finding have friction points here or there? How do you think about that? >> A year ago, there were very clear winners. Like a year ago, Python and JavaScript, they were fantastic at A year and a half ago, Rust I'd often get them them fail to compile. That has changed entirely just because of the the the new models that came out in November and the coding agents. Like you can get them to write code in a language that you just made up and they'll do it. They will brute force their way through the language. They'll try things out. As long again, if the language has good error messages in particular, it'll just fly through it. So, a problem that I've had recently is I'm running out of tasks that they can't do, which is embarrassing if you're trying to like test a new model. You're like, well, everything I tried works. What what did I even learn for that for it learn from this? But yeah, so language-wise, I'm optimizing for my own readability. The reason I'm doing Go and not Rust is that I can read Go and understand basically all of it and Rust I still get hung up on the the refs and the less than the the the the function the the signatures and the borrow checker I've not quite got into my head yet. One of my tests for new models is can it teach me, a Python programmer, how the borrow checker works in Rust? And they're getting better at it, but I still haven't got it myself yet. So, but yeah, um so I And but it it also opens up new avenues. I've been writing software in Swift UI, like a Mac Mac native applications. I don't know a single line of Swift, but I know that I want a little menu bar icon and when when click it, it'll show me how much um how much my GPU is being used and that kind of stuff works. So yeah, it's it's it's kind of baffling how many different things we how much we can spread our wings now as engineers. If you've got those fundamentals, if you understand concepts like red green test driven development and getting a good test harness set up and all of that kind of stuff, the the the world is kind of your oyster now, I think. >> Hi. I have a question. You have mentioned that you use HTML to prototype and then you use it as an input to your prompt, right? >> Yes. >> So, have you tried spec driven development? Do you think the HTML is better than that? >> topic. Yeah. No, um so spectrum development um the most basic version of that I that I that that I still use is Claude code's planning mode where you talk to it and it generates a plan and often I'll do a version of planning without using the official feature. Like I will brainstorm with model say, "Okay, write that down as a markdown file." And I'll have a markdown file that describes the new feature and then you use that as input to the new models. It works so well. It works so well across all of the models. You can brainstorm with Claude and implement with Codex. All of that kind of stuff works. And it also makes the review so much less painful because reviewing a bunch of code where you have no idea how it's going to work is is a huge amount of mental effort. If you spec out, we're going to use this and this and this. It's going to have this bit and this bit and this bit. You can race through the review cuz you're basically just checking if it did exactly the thing that you'd agreed with it before. So yeah, I'm a huge fan of all of those sort of the the the versions where you're you're building a plan with with the models. I think it works really well. >> Um so I have a question. Um you mentioned there should be zero tolerance for flaky tests. Does that also apply to the UI tests? >> Yes, 100% and this is a great example because browser automation tests have always been notoriously difficult to get right. Like I've worked for companies where we've thrown away the entire test suite because we couldn't get it stop being flaky. A couple of things have changed there. Firstly, Playwright, I think, is such an evolution on the flaky test point because Playwright will automatically do things like, "Oh, the link's not there. I'll wait a second and try again and see if it's appeared." All of that sort of stuff. And secondly, all of the agents are fluent in all of the browser testing technologies. They know Selenium, they know Playwright. Turns out you don't even have to install anything because Google Chrome has a Chrome developer protocol thing that it can start up, and the agents can just talk JSON to it directly, and that works as well. So, yeah, I'm I feel like browser automation is one of those problems which has quietly got solved over the past 18 months. And I use it for everything now. It's great for scraping data as well. If you can get some data to appear in a browser, your agent can fire up that browser engine and start scraping things that way. So, yeah, I absolutely think that um UI tests are so much more valuable, they're so much cheaper than they used to be, and if they're flaky, we can we can solve the flakiness as well. >> So, how how do you describe them? Do you describe your UI tests in terms >> Um I'm not No, my my UI testing is very I mean, I don't have anything against the whole sort of behavior-driven development stuff. I'm just doing very basic, like, open this window, navigate to this link, click on this button. Most of my UI testing, I'm that's code that I don't really review very closely at all. Probably, one of the great things about UI testing is you can watch what it's doing. It'll pop open a window and you can see it click that button, select that thing from the menu, do that thing there. Once you've seen that, I feel like looking at the underlying code doesn't really add any value at all. >> Hi Simon, thank you. Um your slide said building a dark factory or having a dark factory means building a verification machine. Um fully believe in this and for regulated industries >> Ooh. >> SOX compliance, releasing code to production, the conversation with the CISO, board of directors, etc. Trying to move in this direction, what's your advice on getting that workflow into production in those kinds of industries? And in a prior talk about this, there was this Swiss cheese model >> Mhm. >> like Layton [clears throat] space about various layers. So, in my mind, I'm thinking each of those layers produce artifacts of some kind that can meet controls that then satisfy requirements. Have you experienced getting production code through this kind of a model? Do you think about it differently? What should organizations be looking at at with that CISO conversation going forward as we try to reinvent software delivery within our companies? >> I'll be honest, the last time I dealt with these kind of problems was 8 years ago, so long before the the code engine thing. So, I don't have I don't think I've got anything useful I can say about that, unfortunately. The good news is that every regulated industry is facing exactly the same problems right now. Um one pattern that I've I've used in the past that I really like is find peers at other similar companies and organize a cabal with them. Basically, once a month, go out for coffee with the person who does your job at some other organization and swap all of the all of all of the dirty tricks that work. I used to do that as an engineering manager and it was absolutely fantastic. But yeah, I don't I don't [laughter] have anything more useful I can I can advise on that. >> Hey Simon, thanks for this. Um big fan of of this and I know that you think really hard about just the practice of software development. So, maybe a philosophical question is code readability, does it matter anymore? If you're building software that you don't even know the language very well that that it's building in, should we care about that and does it matter if the agents are also going to be the ones that maintain and verify and keep the code. >> parts of that does come down to the it's the blast radius and the the the like the stakes, right? If it's high-stakes code, then yes, I absolutely care. Honestly, a lot of my sort of UI code that I'm just sort of vibing out for a little like little visited web page, I don't really care. Because like you said, if the the code is kind of crap, the agent that next generation agent will be able to clean it up anyway. Um but it's an interesting trenchant because also I feel like code that is good code for humans is good code for agents. Like there's there's always there's a lot to be said for factoring your code well such that the next agent that comes in keeps it in that kind of state. The other thing is agents are really good at consistency. If you've got a good code base, an agent coming into that code base is probably going to keep it good. If your code base is a little bit gritty, there's that risk that over time over iterations will get messier and messier and messier. And given that it's not that expensive to keep it in good shape, I think it's absolutely worth investing in in in readability and making sure that the code makes sense. But yeah, that's a complicated question. I don't think I certainly don't I'm not confident that my answer is correct for that one. >> Hello. Um now with code velocity and code servers being so great, uh code review being so difficult, do you feel like the transcript from the coding agent itself is almost a necessary artifact? >> So much. Yes, this is my my biggest frustration with coding agents is they don't treat the transcripts with respect. Claude code, last time I checked Claude code by default deleted transcripts that were more than 90 days old. They must have fixed that by now. This was like 6 months ago, but there's gold in those, right? The transcripts are most of the work that I do is now in baked into that transcript. And so my favorite coding agent at the moment is Codex Desktop purely because there's a little copy as markdown button that you can click on any of your transcripts. And then I paste it into a GitHub gist and I stick a link to the gist in my commit message, and then it's archived forever. And most of the time I'll never go and look at those, but I see them as the same kind of category as commit messages, right? Commit messages mostly don't matter until you're trying to figure out why on earth did we decide to do this thing that way? So, yeah. No, I'm I'm I I I feel like coding agents should value their transcripts a lot more than they do. I think I can do one last question and then we're at time. I can't see from the lights. >> No. >> No? Well, thank you all so much. There's apparently a little um a little lounge area over there that I'm happy to hang out and talk to people, but thanks very much and enjoy the rest of your conference. >> [applause]