AI Models are changing the way we build AI Agents | Humans Talking Agents Episode 3

Key Takeaways

all you have to do to build an agent is have a single prompt you inject in the transcript and then you give your business logic um I think the logic is still not good enough today model intelligence matters a lot less than than people think where where the real game is is like specialized models models that are really good at really specific tasks right and I think that's where you'll actually have more differentiation it's not about building of a single agent it's about managing a 100 agents as AI models get better it's going to completely change the way that you build AI agents let's talk about it So Daniel the agents the the models behind the agents have been getting exponentially better over the past few years and this is nothing new right from the time of gb4 there's been a plethora of new models coming out but I think it's really started to accelerate re accelerate recently for a long time open aai really had a monopoly on like the high-end models right but I think now you're starting to see models like deep seek you are starting to see uh anthropic release models that uh at the time when they were released like often were better than open AI models there's just a ton of competition now and I think that's only going to continue to increase yeah and I think where like where things are going to start to happen like I find that the quality of models like the output is now just getting marginally better where before it was like much bigger leaps and so because of that a lot of companies are competing on like cost is obviously one thing speed is another but at a certain point like speed is also going to become uh marginal and then the third one which is kind of interesting is like there's there's ux conversation so I think as a builder right the models are always going to continue to get faster better and cheaper and then like as a builder you need to think about how are users interacting with my product and I think that's like one Avenue of discussion and the other one too is like on a Model provider standpoint I find that there's actually a bit of ux baked into models themselves too that lend to being able to build on them differently so for example um like we were using deeps R1 um in an experiment we were running like last week right we wanted to be able to create like kind of a a version of perplexity where it's able to stream all the reasoning what we found is when we were working directly with the Deep seek model um the way they streamed reasoning like was kind of hard to work with it was shorter sentences like one at a time and the reasoning was kind of weird like it wasn't polished um and then we were using perplexity model which is their version of Deep seeks open source they just build on top of it but what they did really well was they added in web search and they also were able to uh really reformat the way that reasoning was streamed to the end end point so the experience that we could build was much stronger um where users could actually ask questions right like how many solar panels does it take to power the us and you're able to see um all of the reasoning in a much like clearer format it was better chunk together and then also included um sources throughout the entire thing right so we were able to build a better user experience because of the model but there's I think there's two separate angles how how does that change the way that you build agents because like you know that's just you know for the purpose of like using a foundation model kind like raw right using as a search engine you know ability to just like get information how does that actually change the way you build agents yeah I think like one I think a big part of of being a builder is like the or the the work that you're doing behind the scenes in terms of like validating data like accounting for like different paths accounting for different logic I think that is going to start to get less complex over time I like models are going to get better to be able to handle a lot more of that complexity um and where it becomes more important is less about the amount of orchestration that you're doing behind the scenes and so you see all these images right of people screenshot like a gigantic flow in voice flow and I think what's actually going to become more important is how your agent is interacting with your front end to create a user experience that is really geared towards solving your user's problem and so going back to the perplexity use case right like I think it's a good example where they've actually built around that where like I use perplexity all the time for web search because I think they've nailed the way that it looks at sources it presents sources it presents other information allows me to dig deeper into my question and so I think that the user experience of the agent that you're building is going to start to become a lot more important than a lot of the orchestration work that you're currently spending a lot of time doing behind I mean it's a bit of a hot take but I think that the canvas on voice flow but I mean you know really a ton of tools goes from being 80% of the you know place where you do the work to like 20% right it becomes like very bespoke little automation flows and most of the work is going to be done on the model side we may reach a place where the models get so good that you can do an entire multi capability agent multilingual agent multi- interface all in a single prompt but it's not there yet yeah I think over time like that's going to be the trend yeah I actually think it's like a little bit more um interesting than that I think that today so I'll give you like a really tangible example right um in voice flow we uh have a front end our web chat and we uh built in this feature that allows you to render like custom apps or custom extensions basically within the web chat and actually allow your web chat widg to interact like with your website or with your app and so like the most interesting projects I'm seeing in voice flow now um are actually really thoughtful about how the outputs that the agent is creating is actually creating a specific UI or a specific experience inside of the web chat right so obviously there's like typical examples like seat Pickers and like forms but I think like Connor had a really like Connor umbrell from the community had a really good example where he had built um basically his own web app right where on the right hand side you've got like your like chat screen but the chat is straight up a straight co-pilot for the app itself so as you're talking to it um it's being able to like pull the information out of what you're doing pull information from the back end and then send uh specific custom signals to the front end to now render things like um employee org trees like calendars Etc so that's the end agent experience I agree that like as the building gets easier yeah you'll see more focus and differentiation being put on the uh on the user experience right it's almost like the floor is lava game we used to play as a kid yeah where gbt 4 and C gbt like when these things first launched just being able to have the thing was like all the value but now it's become so easy to be able to build on top of them that like you're seeing um the basic like single rag chat bot where you upload like a PDF and you can chat with it that has become so commoditized now you can put it together in like five minutes right using like any plethora of app and so now people are demanding in terms of differentiation increased capabilities on top now at some point you know like the app that you're just chatting about um that uh Connor and I believe his agency is called umbrell or his YouTube channel has created that's still using a lot of business logic behind the scenes it's using a lot of like custom actions and extensions and things at some point you know and that still requires a pretty big canvas at some point as the models get better even all that will be subsumed into a into a single prompt right and so suddenly the building is almost entirely prompt based um and then like you're it'll be really interesting to think about how you actually build when the entire agent is a prompt and you're just looping into like a you know trying different iterations of a prompt like the building experience is really going to be all about like prototyping and then prompt iteration right like it's going to completely change from like the flow-based nature of today yeah yeah it'll be kind of interesting because like from a tactical building perspective like initially what you saw was before AI right like the the biggest part about building was like all of the edge cases that you were handling right like you have so many of them and that was like 80% of the design the edge cases the term was the happy path yeah which was like the 20% yeah and then the 80% of the time where people say something unexpected and that that was and then when large Lang models came out that was the thing that that got removed right cu the large language model was now handling all of your like incorrect paths it was like being able to handle people asking like random stuff going off the rails and the knowledge base right like the idea of asking any sort of question um made you go from a project with like you used to have like thousands of intents to handle any question gone um and now a lot of those designs straight up focus on the logic and that's where a lot of the work is today right I'm importing speciic variables I have hard logic that I've got built in and that's going take you down different paths or allow you to do different things I think that the next leap is probably going to be around the logic side um it's going to be how do we um you know maybe provide the business rules uh to your model and allow it to leverage them kind of like they leverage Tools in their decision- making right um there's obviously going to be a level to that where it's super sensitive you might need more control but I think that'll be the next jump and so that's where I think like that part of the design will become commod ized and when it comes to then building it out like really where today right and this goes back to the whole interface conversation now it's less about the agent itself doing the work behind the scenes and more about the interface yeah well you know an interesting term term that I've start to see some companies really adopt is um agent management platform right so we're kind of talking about like the the platform side of things here if it becomes so easy to build a a highly capable agent where where you can spin one up in let's say a day right maybe even less and it's all a single prompt and you add in your your function calling your tools your business logic all that stuff is in a single prompt you have a highly capable agent the work doesn't stop there then you start to get into what else can I automate because I think right now people might spend a week or a month iterating on a single agent right but like these people aren't just going to stop if it becomes if you reduce the time to build an agent by by 10 times right as models get better it's kind of like Airlines it's like as the price of Airlines came down it's not that people flew less they actually just flew more right like the airlines actually like just offered more destinations I think you'll start to see as models get better the number of use cases for automation just goes up right because people are going to start spending their time away from just the core use case which is today lead gen and customer support those are the two big use cases you're going to start to find people say hey like it only takes me an hour to SP up an agent let's go try this use casee let's go try this use case so I think you'll see an explosion of the number of Agents per company as models get better and then what will happen then is that whole term of like agent management platform where it's not about building of a single agent it's about managing a 100 agents analytics insights being able to allow for uh agents to communicate with each other being able to permission agents who can build this what apis can this agent use versus this agent like it becomes almost like a I mean who knows maybe like in five years voil is like an HR platform for AI agents right like you are permissioning what these agents can do and where and it's more about management than it is about creation because today most of the agent building platforms it's all on the creation side it's less management yeah yeah that's super interesting um because I feel like today even like we're seeing that explosion to use cases already right like I know you mentioned like Le gen for support of the two big ones they're the two big ones from a perspective of like where businesses I think are spending the most money but but you're already seeing a lot of people start to experiment with these more internal um agents and I think that's where platforms like NN and make are are doing a really good job because um you know those those internal agents need Integrations to all your tools and they don't necessarily need like complex State Management right which is what you do need if you are talking with a lot of end customers at the same time yeah and that's where like a tool like boy like makes a lot of sense um so yeah I think you are seeing it today you know what's really interesting is on the context window side when a lot of the um models that we all talk about and use today like a you know gbt 3.5 or a 40 when they launch they they often have smaller context Windows than some of the new than the new models right and that actually placed a lot of restrictions on how you build agents when the context windows are smaller you would have to do a lot of uh dialog management and and and variable memory management right because you can't just pass in the entire trans script things would start to get lost it like the model would start to get confused you'd blow up your token costs but as context Windows get larger and to token costs go down I think the way you build agents is you you don't have to worry about memory management as much there's no long-term memory and short-term memory because today that's kind of the way you build you uh take things out of the conversation out of the transcript and you sort in variables in like a dialogue manager and then you inject that memory back in like your short-term memory or that's your long-term memory you inject that back in when you need to but like suddenly if if it's able to handle an in you know what is effectively infinite context window all you have to do to build an agent is have a single prompt you inject in the transcript and then you give you know you give it your business logic like that part of again we're kind of getting back to that whole concept of like the flow-based order going away yeah because you don't even need to do memory management anymore yeah I think that makes sense right I think but the key part there I think that's new that hasn't happened yet is the like the commoditization of like the logic part because today like it's it's not reliable enough for what people think to do right like like context models are already big right you don't need a lot of memory management now because you can just save the entire transcript and pass it back in or try save the entire memory so I don't think that part is future facing I think that's happening today um I think the logic is still not good enough today um to be able to handle like and that's why people use a tool like Voice Low right or write it in code CU like a lot of times you need that logic to be hardcoded you're trying to break up like a single pretty big prompt into like several sub prompts so that it doesn't have to do as much right like if you have a you know we worked with a car rental company where the logic to extend a rental is like you know 15 Steps right and part of that is because of the way that their apis are built part of that is just the way that they handle their business process there's a lot of stuff that has to go into actually doing something like that yeah so it's not like an email capture where you can do that all in a single prompt today right name company email totally fine but it when it's like 15 Steps and the logic is also branching as well that's other thing suddenly like you have to you have to break it up today but as models get better like you shouldn't have to do that but the writing of the prompts would be incredibly complex because inside of the prompt will'll have to have nesting logic yeah I think that's interesting so that's like one that's one aspect to it but again I I I want to go back to the interface side CU I think that's actually where things get um quite interesting and I think even today for Builders right like I think the interface is overlooked a lot and that's the core thing that will differentiate your agent cuz um you know building a good agent today is easier than it was a year ago it's exponentially easier than it was two years ago and so I think everything we're saying is that building agents will become easier in the future do you think that the interface is overlooked because there's still so much work to be done on the content side on the actual like agent logic side that people you know it's like the last 20% we're saying that's going to become 80% of value that interface but people are spending all their effort on the building of it also looks at like where's differentiation right and today like differentiation is on like agency accuracy and reliability yeah and I think that will start to go away as models get better because it'll be easier to make your agent more accurate and more reliable and the differenti will need to come on like the interface side right is this the best UI to solve my problem and I think again like if we just look at the model providers themselves we're already seeing that differentiation right and we talked about this in the interfaces episode but like I go to perplexity to search the web because it has a better UI for my goal of web search I do not go to perplexity to write code because it's not built for that and so your agents are the agents that people will build I think will start to specialize for problems and a lot of the work of the Builder will focus on the um on the UI and I'm already seeing this today from Builders who have basically gotten to a point where their um the build underneath it is good enough for what they need to do um and so they start to spend a lot of their time focusing on the UI because that's where they're seeing differentiation for their customers you know what's interesting um to kind of give like a tangible example to what you're saying you could either have a very complex prompt to do date logic validation yeah or you could have a calendar selector component in the UI and they're going to do the same thing but one is more Frau with errors right or one is more prone to errors which is the The Prompt side right I think my take my like my hot take here is that um you will never see in the future you will never see a standalone agent um agents will be behind the scenes of basically an app right and the web app is or the app itself is a focus and the agent goes away as you're thinking you don't think about talking to an agent anymore it's just the thing that is orchestrating what's happening behind the scenes right and so we stop having this conversation of aent agents and we start going back to having the conversation of apps and apps in their nature are like conversational they're agent focused they're Dynamic they're personalized and so I think that's where the world is going um and that's where when I think about you know people building agents right now it's seen as like an external thing interacting with your app I think it's going to go away I think the first step is that agents will be thought of as like straight up a co-pilot as a way to interact with your app and then um you know as the modes of interaction get better I think we'll stop thinking about them as like a separate thing it's not going to be co-pilot to your app it's just going to be part of your out self so for for the conversation so far we've been talking about model Improvement really in the context of intelligence right but we haven't talked about model improvement from the standpoint of capabilities such as multimodal y right when you start to see and this is like really the big Direction you see opening eyes they're starting to mix modalities more and more it's not just a language model it's uh Visual and its uh text right eventually you'll start to see the video get in there as well how does that how does the introduction of um multimodal agents change the way we think and build agents as agent Builders I think it goes also goes back to like you don't need to use every tool in your toolkit to solve a problem right like you mean by that so for example a lot of the core agent like the model platforms and the buis they built around this like at least from my understand they're all racing towards building this like um like an AI but like an assistant right someone who's like a person basically didn't interact every business doesn't actually need that right like there's aspects to it that are useful right like if you have a bought a website like maybe you want to make sure it has Vision so we can understand like what you're trying to do on the app or on the website and help you do that right so I think it goes back to like what is the goal of your agent like what problem is they trying to solve right is this an agent built into your app to be able to help users get to Value faster and help navigate your app right like within Photoshop or something like that um in which case you have a built-in UI and you need people to understand like where things are going and you understand what they're doing or like rep agent right where it takes screenshots of the app to be like are things in the right place and so that's where I think it like what is your goal or what are you helping someone to do and figuring out like okay with all these new modalities like which ones are actually needed to help accomplish that yeah some cases you know you might need all of them in some case you don't need any or you need just need one of them well I think our industry often has like shiny object syndrome yeah where and we we see this all the time whatever the the latest and greatest thing launches people are instantly like I need this right now right and it's often not driven by customer demand it's often driven by just like we're technologists we want to play with the latest and greatest things which I get I we also instantly as soon as something launches we start playing with it on our on our own but we don't immediately add it to voice flow until we see is there a use case for this right like what is the strong um demand going to be and you know a good example uh I guess this is like a couple months ago if you remember we had that customer that an agency bill for there's a very large company um wanted to run like a national campaign for a consumer goods product they actually decide to use no large language models at all and so what was interesting is like the the toolkit available to bot platforms and agent Builders continues to expand right you know you've got large language models you've got file uploads you've got like image recognition voice video like it the toolkit continues to get bigger but it doesn't mean that a company always wants to use the latest and greatest and I think often again as technologist we have shiny object syndrome where we assume that the customer will always want to use the latest and grest but in this case they were doing 20,000 messages a minute they were like just give us buttons this thing has to be bulletproof we want zero hallucinations zero token costs zero whatever and it totally fit their need it was a huge success right and I think it's a good highlight of sometimes we need to be aware of the tools in the toolkit so we know where the ceiling is but don't assume that the customer always wants the latest and greatest right it's just it's another tool yeah I mean unless they do that's the part right like I think in today's world people are you know they see Ai and they think magic so when they think of a project they think about the project with AI and they think magic and so I think half the battle is also understanding ay what what problem are you actually trying to solve and like what's more important to you is that is it that we can solve this problem effectively or that you have this like AI feel on it so it looks cool well even in bot Builders though like again when you're aware of the tool kit and you know what tool is good for what thing right it's becoming very common in as agent sort of like model orchestration to use different models for different tasks right hey you are doing uh just a rag optimization you just want to optimize an utterance for a rag inference maybe you're going to use like a anthropic Haiku right very fast very cheap model now you're doing uh you know the the final response generation it's got like 15 items of metadata tons of variables it's a very long prompt I'm going to use O3 mini for that right and you're starting to mix and match the tools in the toolkit instead of just using the latest and greatest model in every single position because it's not always needed right as models are getting better and faster and cheaper I think it becomes less important and I think like you're always going to use the lacing gra yeah like I use foral mini for everything like realistically have an unlimited voice Enterprise account no even for my personal projects right like unless I'm using uh but and I think this is where I'm getting to where when it comes to stuff like that um I think models are just going to get they're going to get accurate enough that they can do like conversation pretty well yeah I think that models will also start to get specialized into areas and then those are better for your specific tasks right like I'm I use cursor and I use cursor a lot with like Claud son it 3.5 because it's just been so reliable for code yeah uh when I switched to another model like all of a sudden it didn't it wasn't as good right or it started recommending really different changes to my code base from how I built it um that like when I actually implemented them and took a look I was like these are like suboptimal and so I think that like there's going to be these models that are generally good at everything that will get faster and cheaper and that's the battle on that side okay but at what point is that increased intelligence not even matter you know like we're and that's what that's what I was saying at the beginning is that I think like there's two types of models there's models that are like generally general intelligence models that now in my mind like the marginal gains on intelligence are are more marginal and it's all about just like cost and speed that's it then on the other side I think you've got like where where the real game is is like specialized models models that are really good at really specific tasks right and I think that's where you'll actually have more differentiation between them and so like I go to the point right where like for mini versus another model versus three mini I'm like you know what depending on what I'm putting out here like it's kind of going to give me a decent answer and so at a certain point it's not going to matter anymore and just going to be able what's faster and cheaper but then yeah it's a specialized models that I'm interested in yeah I I think for like agent building today the model intelligence matters a lot less than than people think like when you see some of the demos that are coming out from like you know the foundation model companies they're doing like PhD level work right but then you're building a bot that just needs to pick out an email in a city right from a paragraph so it's it's it's not that level of intelligence but where that level of intelligence is going to be increasingly helpful is in that like business logic embedding into the prompt because it is going to require a lot of um intelligence to understand like the implicit rules in that business logic right if you write a what it might actually do maybe the best way to uh to kind of explain this is like let's say you hire someone to be like a Bria at Starbuck a really intelligent person you would be like hey here's generally how you process a refund right and a really intelligent person should be able to figure that out they'll be able to imply a lot of the unwritten rules versus a lot of the uh less intelligent models you have to be very verbose and so in fact what it might do is allow us just to write prompts that are less complex in fact we might actually see prompts just you know get haved so I think like maybe kind of putting a lot of this conversation to a point right if you're talking to a builder today what are the core skills that they should be working on that are going to be more that are going to be that 80% of the work today are 20% I think agent Persona I think and interface optimization I think these are probably so and and interface building itself so let me let me kind of talk about those three things Asian Persona is the language that a bot uses often as bot Builders today we are so focused on just getting the thing to work and to do the thing we want it to do right I was building an agent last night where that was all I was focused on I wasn't concerned about the language and and all that kind of stuff but suddenly if it's easier to get it to do what you want it to do you're going to spend all your not all your time but a lot more time on like okay but how does it sound how does it feel how does it respond if the user is sad how does it respond if the user is Happy etc etc like that Persona side of it the actual content um so I think that's one bit the other bit is on the actual interface itself and we've talked about this a lot so I'm not going to go to too deep into it like the building of a calendar widget the ability for the agent to select the right interface for the right moment in time the last bit is actually uh it's interface optimization and so I would include uh interface and modality is actually being um kind of synonymous here typically we like to separate them out like voice can be applied to several different channels like voice is a modality and a channel might be Tony or like a web chat with you know with voice but I'm going to make them synonymous here you want to actually start to optimize the content yes across persona but to be able to work appropriately across the modality and channel so for example a lot of chat agents today when people convert them into voice agents they're not optimizing the content to be for uh a TTS model so you actually get like a lot of weird pronunciations just the the way these TTS models work today where like it will use a bullet point list and like a human would never read out a bullet point list to another person say hey Dan iiel here are my three points one do you know two like it sounds weird right and so I think you'll start to see people on the Persona of the content but then also on optimizing the prompt to respond appropriately to the channel modality so I think those are kind the three areas I think people should be spending more time on today the models aren't there yet that shouldn't I mean it'd be great if you had a lot of time to spend on it and you should but often we're just spending over our time making it work versus optimizing it yeah yeah agreed go build front end UI okay well I think with that then chat a lot about how models are going to change agent building hopefully that was insightful we went out a lot of different tangents and uh that's it that's the humans talking agents [Music]