Building LiveKit Agents with Gemini Live API

Key Takeaways

[music] >> Today, we're building a voice agent using the newest addition to the Gemini Live API, the first native audio model built on Gemini 3. Now, this isn't just an incremental update. The new native audio model brings noticeably better instruction following, smarter tool calling, and reduced speaker drift for long multi-turn sessions. And it's all possible because the audio is handled natively, not converted to text first. And we'll get to all of that in just a minute. By the end of this video, you'll have a working voice agent running locally, and you'll understand the patterns you'll need to take it to production. So, let's dig in. So, let's start with a fresh Python project. We'll UV init {hyphen} {hyphen} bare. And then let's UV add LiveKit agents and the Google uh plugin, and then that is at currently version 1.4 on the LiveKit agents. And then python.env as well. Now, you will need to get a LiveKit API key, secret, and URL from cloud.livekit.io, and a a Gemini API key from Google AI Studio. Drop both of those in a .env.local file, and we'll load those up in just a second. So, let's create our agent. We'll create a new file here called uh agent uh {hyphen} py. First, we'll add our imports for .env, LiveKit, and our Google plugin. And then we'll load our environment variable files. Then initiate our agent class. Notice here, this is where we have our instructions. We're going to expand on this in just a bit. And then we'll initialize our server agent. And then our session here. So, in our entry point here, we have our agent session, and this is where we're defining our LLM, which is our Google real-time real-time model. And we're defining that here as the Gemini 3.1 {slash} audio, uh currently EAP. And we can specify our voice here as well. After that, we're going to start our session, and that is basically it. Now, notice here that we are using this Gemini 3.1 {slash} audio model. This is the first Gemini 3-based native audio model. It processes and generates audio directly, rather than going speech-to-text-back-to-speech. And that's what makes it sound so natural. Now, note here that this is the early access model, so please check the official documentation to get the latest model for you to use here. But before we go any further, let's talk system prompts, because this is where most voice agents fall apart. A few things to consider here are be explicit about your persona and your scope. Don't just say you're a helpful assistant, like we have in our code here. Tell the model who it is, what it can and can't do, and how it should handle edge cases. Write for the audio and not for the text. Your agent is speaking, not writing an email. Short sentences, natural pauses, avoid bullet points in the system prompt. They don't translate to speech. And then add guardrails inline. Don't rely on a separate safety layer for basic things. If there are topics your agent should redirect, say so directly in the system prompt. And then shape the voice with director notes. Uh because the model has deep audio understanding, you can instruct it to speak in a specific way, and it will translate that directly into the delivery. Now, if you want it to speak in an Irish accent, for instance, a a slower pace, a higher energy, just say that in the system prompt, and the model will handle the rest. Here is a simple example. I've added here a system prompt, and then added that down into the instructions as well. And so, in this system prompt, we can see that we're giving the model a name and a personality. Uh we're telling it how to sound, defining what it can and can't help with, handing it a guardrail for off-topic questions, and enabling multilingual support. Uh no config files, no extra API calls, just right here in the system prompt. And now for the stuff that is unique for this model. Uh first is improved latency and reliability. This model was built to be faster. Google upgraded the Live API infrastructure specifically for this release, improving numeric precision, function calling, and audio grounding. And then there's reduced speaker drift. One of the biggest complaints with long voice agent sessions is that models' voices gradually shift away from the selected persona over time. Now, this new Gemini model is specifically designed to maintain persona and accent stability across uh long multi-turn conversations. And then it's multilingual. The native audio model supports around 70 languages, and it can switch between them automatically mid-conversation. You don't have to set a language code, it just figures it out. And if you want to restrict which languages it uses, you can just do that in the system prompt. It's so easy. And then, of course, it's multimodal. The model supports more than just audio-to-audio. It also understands text and video, and it can respond with audio. Plus, you can enable output audio transcription to get the text transcript alongside it. Now, tool calling is where you get real flexibility. There are two approaches: server-side function calling and client-side function calling with RPC. Now, server-side is the standard approach that you're used to. You define tools directly in your agent code with the @function_tool decorator, and the model just calls them when needed. And here's what that looks like. First, we'll import function tool. And then we'll define our tool here. So, we're going to say, "Look up a customer's account status by ID." This is where you would actually do the query on the back end. And then return the results. Now, the agent understands what the tool is by the description that you gave it here. When the model decides to call a tool, your agent handles the response and sends it back. You get full control, and it runs on your server. Now, the built-in search tool is a good one to highlight. Add it to your tools config, and the model can query Google Search in real time to answer questions that it doesn't know. And here's what that looks like. So, after our instructions, we'll add our tools, and we'll pass in the Google tools Google Search. No extra API key needed beyond your Gemini key. The model knows when to use it and when not to. And lastly, there's the client-side via RPC. LiveKit's RPC system lets you call functions that run on the client, the the browser or the mobile app, from your agent. This is great for things like controlling the UI, uh triggering device actions, or reading local state. It's the same function calling pattern, but the execution happens on the other end of the WebRTC connection. So, let's test it out and see how it works. We're just going to use the console here to test it. No need to deploy or anything like that. This is all local. And what we're going to do is we're going to look at the latency, the function calling, uh speaker drift, multilingual use, and and just just see how well it has a conversation with us. We're going to run UV run agent.py, and then console to run it right here in our console. Hey, how's it going? Hey, what's up? Doing pretty good myself. Uh what can I do for you today? Yeah, can you check on my account status? Oh, sure thing. I'll need your account ID to pull that up, though. Do you have that handy? Yeah, sure. My account ID is um A1234. Cool. Thanks. Let me check on that. Okay, yeah. Looks like everything's in good shape here. Your account is active, and no issues or anything. Anything specific you were wondering about? No, that's pretty cool. Um yeah, can you tell me use Google Search to tell me what is today's date? Yeah, sure thing. According to my super search skills, today is Monday, March 2nd, 2026. Anything else I can help you look up? That's correct. Um tell me yeah, tell me what the weather is. Use Google Search to tell me what the weather is in Houston, Texas. Okay, let me check on that. So, looks like it's currently around 76 to 80° in Houston right now. Mostly sunny, maybe a little foggy this morning, though. Seems like it's going to get up to about uh low 80s today, so pretty warm. Planning a trip or It's pretty warm. Yes, it is. Uh hablo español? Ah, sí, claro. ¿En qué puedo ayudarte con español? ¿Tienes alguna pregunta, necesitas ayuda con algo, o qué tal? Ja, also uh sprechen Sie Deutsch? Ja, klar spreche ich Deutsch. Wie kann ich dir dann heute helfen? Gibt es etwas, das du wissen möchtest, oder ein Problem, bei dem ich unterstützen kann? Uh um parlez-vous français? Oui, je parle français. Comment puis-je t'aider ? Pas besoin de vérifier l'état de ton compte, poser une question de facturation ou autre ? Par exemple, est-ce que tu aimerais connaître la météo à Lyon pour voir si c'est le même genre de temps qu'à ? Now, that performed very well, in my opinion. But I want you to give it a try. When you try, try these things: latency and function calling. Ask the agent to use a tool, look up an account, trigger an action, whatever you have wired up there. And then, with this new Google Gemini model, function call accuracy should be noticeably improved. Try back-to-back tool calls, and watch how the model handles chaining. And then, speaker drift testing. Have a a long conversation, maybe 10 or 15 minutes long. Switch topics frequently and pay attention to whether the voice stays consistent with the persona that you set. And then the multilingual aspects. Switch languages mid-conversation. Go from English to Spanish, French, Japanese. No configuration needed. Just start speaking in another language and the model should follow. Try switching back and forth and see how it handles that transition. And then the Google search tool, uh you can use that to ask it something that requires current information like recent news events, stock price, or today's weather. Uh with the search tool wired in, the model should reach out to Google in real time rather than guessing or saying that it doesn't know. And lastly, RPC. If you wire up a client-side RPC function, uh maybe a function that updates the UI, tell the agent to trigger it. And this is where the agent stops being just a voice interface and starts actually controlling the app that it's running inside. Now, this is just uh the foundation. You have a working voice agent now backed by the new Gemini Live native audio model covering everything that we walked through, ready to build on. So, from here, you can deploy this to LiveKit Cloud with a single CLI command and it will auto-scale without you touching any of the infrastructure. If you build something with this, drop it in the comments, please. I would love to see what you make with it. >> [music]