Do All Your AI Workloads Actually Require Expensive GPUs?

Key Takeaways

[Music] You're watching the new stack makers, a podcast for people who develop, deploy, and manage at scale software. For more information and articles about at scale technologies, please visit thenewstack.io. Now, enjoy the show. Google Cloud helps organizations build quickly, securely, and cost-effectively with infrastructure that is optimized for the demands of AI, modern enterprise, and distributed workloads. Hey everyone, live from KubeCon. We are here with the Google team. Hey all, how you doing? Doing great. Doing great. >> Good to see Alex. We've been talking for the past few weeks about this uh discussion today about Google Axion, and I would love to first of all though, get some introductions. So, Pranay, why don't we start with you? Sure. Uh I'm Pranay Bakre. I'm a principal solutions engineer at Arm. I'm Andre. I'm part of Google Cloud. I'm a technical solution consultant, and I help our customers adopt our AI technology. Okay. And uh Gary Singh, uh one of the uh product managers for uh Google Kubernetes Engine. Well, excellent. So, we're going to be talking about Google Axion and and and some of the news you have that just came out. I wanted to start by thinking of some context for our discussion today. I like and I always like to think of like, how do we get here? Now, of course today, you know, in the world that we're in, we're hearing so much about AI. But, there's another side of things too, where workloads have increased tremendously over the past 10 years within cloud services. And the demands on workloads has increased significantly. Now, over the past years, we've started to see the arrival of you know, new ways of thinking about you know, models and how you think about their deployment, and how you think about their development, and how you think about their management. So, with that in mind, I'm curious about your news and how it relates to where we are now where we came from. And now so, wondering if Pranay, you could talk a little bit about the Arm story and how how how you came to this point today with Google. Sure. Sure. You want to go first? Sure. Yeah. Okay. So, um Arm uh Neoverse is our server class CPU architecture, which is what Axion is based upon. So, we worked with Google We started Arm Neoverse uh few years ago, where we wanted to establish Arm as the industry leader in server class CPUs, and uh grow the adoption for that. Now, Arm Neoverse architecture is energy efficient by design. So, all the benefits you can get by deploying your workloads on Axion utilize that energy efficiency and power consumption gains. So, for example, developers, now since you talked about AI and workloads increasing so much in the today's world, uh a lot of developers are deploying huge number of apps uh on the on the platform, Google platform. Um and uh Arm architecture supports that by you providing that efficient architecture, uh so to speak. So, so Andre, so Andre, maybe we can talk a little bit about Axion and a little bit about of its historic the historical perspective behind it. And you know, and and why it started, and what is significant about where we are today with the Axion processor? Yeah, absolutely. Thank you. So, Google has always had the history of custom silicon. We are dating back many, many years through our TPU famous TPUs that were specifically designed to handle infrastructure for machine learning. But, we saw a shift in data center needs, and also a shift in public consumption. People wanted to have more performance at a lower cost, at a lower power consumption, so more data centers can have more compute. And as a result of it, we've invested many years ago into our first generation machine, which was Arm-based, and called a T2A as an experiment for people to give it a try. But, now we're seeing that our customers are ready to shift over to Arm, take advantage of better performance, pay less for the same performance, still get great reliability, and effectively get more for less. So, with our introduction on C4As, we launched them in many, many regions, and now N4A is bringing better price to performance ratio than C4A could do before. Tell us about the M4A. Either you Andre or Gary. Uh I mean, I guess you know, the the cool thing from the M4A perspective, you know, from my perspective, especially coming from the kind of Kubernetes side of the house, right? Where uh people are very love to mix and match kind of shapes of you know, your pods that you're deploying out there. I think the great thing about the N4A really is from from we've introduced these things called like custom machine shapes, if you will, or types. Um allowing you to basically mix and match how much memory and RAM you want. So, no longer having necessarily having fixed sizes, right? And I know we'll talk a little bit more probably later about some some finops and things like that. But, that's really um you know, one of the great benefits. And of course, it's just a general purpose, not just, but I you know, for general purpose workloads, right? So, you're getting price performance for you know, your standard microservices type workloads, um which makes it you know, super simple to switch it out for a majority of the applications that people are using today. You know, one of the things that's been noteworthy for us over the years is the rise of platform engineering. And it's still a very popular topic. And I and I think it relates directly to what you're talking about today, doesn't it? Can you tell tell us how this relates to the platform engineering story? Yeah, I think you know, a lot of times, so there's multiple pieces I think to platform engineering, you know, when when it first sort of starts out, people are building like, what's the developer experience? What's the what's the overall kind of platform that we're running on? But, over time as it grows, and if you're successful, uh one of the ultimate things that comes on is, you know, price performance, finops, etc. um becomes one of those key things. Like, you're supposed to be building this centralized platform, optimizing all its usage. What's the cost of this? How are you optimizing price performance? How are you abstracting this from the developers? Um so, as you start to look at that, obviously, if you can have the best price performance compute underneath the covers, um you're going to you know, meet those goals. So, again, Pranay, over the past several years, we've seen the number of workloads increase, right? I mean, it's just scaled you know, beyond what you could have imagined 5, 10 years ago. And with that, we have seen the need for you know, uh ways to think about the cost a lot more a lot differently than you did with x86 chips, for instance. Tell us about that, a little bit about the the shift to Arm, and what has driven it, and where do you see that today? Sure. I think um I'll leave the cost bit to Andre uh about Google Cloud, but I can talk about the the shift from just x86 to multi-architecture. Um so, we are um um promoting that message that you can develop your applications to be multi-architecture compliant. Meaning that it can seamlessly run across both x86 and Arm at the same time. And again, like I said, uh the power efficiency is one major factor. And now, Axion uh builds on that, both the C and the N series. So, C is more designed for higher throughput uh and providing the maximum output that's out there, lower latencies. And N series is more about the general purpose compute, right? So, we kind of covered the breadth of that spectrum. Now, as it comes to when people are deploying more and more towards this multi-architecture deployment, they are realizing the transition going from x86 to Arm, that you get much more compute since it's theoretically you get a one physical core whenever you're deploying it in the cloud uh entirely. So, you get more compute for uh a cheaper cost. But, again, I'll talk to send it over to Andre, but we you can deploy your applications in a much uh cheaper fashion, and we want to promote that message. Great. So, when you're thinking about you know, the how you guys have built you know, the the this the the CPU around that you know, with Axion, what do you think is special about it? What do you think is you know, you talked about the lower the efficiencies and lower cost, but can you tell I let's put ourselves in the in the shoes of a platform engineer, for instance, and the day-to-day problems that they face, right? They're managing multiple workloads, right? Tell tell us about that story, and how you see that being then beneficial to that that individual. Right. So, let's say you're talking about like platform, right? Specific. So, let's say you're building a CI/CD pipeline in your environment. Uh with Axion, you can essentially deploy more uh workloads on top of it. But, again, because again, it gives you more uh cores and flexibility to deploy those, and you can add uh more, I'd say, efficiency around it. Whenever there are instructions that are optimized specifically, certain workloads uh like Java, Nginx, etc., they take advantage of those efficiencies and run run much more faster and in an efficient manner. Because there are a lot of things like I can go into instruction details like SV Yeah, right. >> But, I don't want to go there. You can read that, too. >> Yeah, yeah. Exactly. >> Yeah. But, uh essentially, all of these workloads take advantage of that. That gets uh much more importance whenever you are doing specially AI/ML workloads, like you are doing inferencing and others, uh that comes more into play when you're doing that. That's interesting. So, I'm curious then, Gary, about the M4A. And what did you hear from customers or you know, in the past 12, 18 months? What were they telling you? Or even longer than that, about about what they're that what they were really needing. And how did that come How did that then materialize with the M4A? Yeah, so I I think a couple things. I mean, on the on the overall, you know, arm front, I think, you know, as Andre said, we had, you know, the T2A a while back. We've been hearing for a long time, you know, as as arm moved into the server market, right? That, you know, you're going to have better price performance for these type of these type of workloads. But what customers really were looking for, you know, from us was one, you know, massive scale, right? So we can roll this out kind of everywhere. And then really, can I run just my normal kind of, you know, kind of general purpose workloads on this, right? Can any kind of workload? You know, we've talked about, you know, Engine X, microservices applications, right? Can we run that? But, you know, we had some complaints on well, I also want to make sure that these machines are way more flexible, right? And this is feedback from earlier designs, right? How do we ensure that I can have whatever kind of mix, if you will, of If people know about the cloud, you typically pick a machine shape, right? So on some of our Intel machines, you may pick a, you know, an N216. It's going to have specific this with 16 cores, right? That's what you get and like 4X the memory, I think, right? So it's an N21664. And a lot of times, right, you don't necessarily need like those ratios, right? So how do we make this way more flexible in terms of that offering and the shapes that people can use. I will say also, you know, and then how can you One of the things that we've added in there, and maybe Andre, I don't know if he wants to talk a little bit about this, but was also in the overall design wasn't just the arm part, but also doing network offload, right? So we can focus specifically on processing power just for your applications and offload all the network processing. We even have better price performance than you'll see. If you could elaborate on that, Andre, that'd be great. Yeah, so C4As and N4As are part of our newer generation virtual machines. What it means is they are powered by atranium, which is effectively us taking all the network related tasks and all the storage related tasks away from CPU calculations. So you get more CPU horsepower out of those instances and offloading the network IO and the storage IO back to the dedicated hardware. That allows us to scale our networking independently from the cores. It also allows us to have a lower latency to our storage. Additionally, all the newer instances on Google Cloud are powered by hyperdisk and hyperdisk allows us to scale our IOPS independently from the size of a volume. Additionally, hyperdisk pool allows us to do thin provisioning. So when you look at workloads, specifically that potentially don't use a lot of data or they have repetitive data, the combination of all these innovations allow you to have a most performing core possible, more cycles dedicated to your workloads, and the rest of the stuff is taken care of by Google either infrastructure like Kubernetes engine or platform as a service like DataProc or Dataflow. I'm going to come back to you in a second, but Gary, I want to come back to something that we were just talking about, and that's related to price performance and how that compares to X86. And, you know, we got to be got to be careful about the marketing here, right? Just to make sure that we're talking about, you know, what's real. But there's some interesting data there about price performance for web servers like Engine X. Can you talk to that? Yeah, I mean, we've seen some again, especially with things like, you know, the the M4A, we can talk a little bit about the C4A. But remember, a lot of these applications, right, you know, work well with, you know, per core kind of performance, right? When they're looking at parallelism. So when you look at what you can get out of that, that's obviously going to reduce the price whatever, decrease the the overall pricing and increase your price performance. So we're seeing with like Engine X and typical web servers, I believe, you know, 90% you know, improvements in price performance. Another heavyweight in there is my old friend or or new friend Java, which is also, you know, well known to be a little bit, you know, heavy. Yeah. Right? So great thing of, you know, porting the JVM to run on arm a while ago, and now getting about, I think we said 80, 85% you know, price performance improvements with that. And those applications are a natural fit, right? Because they're generally rely on sort of per core and binding threads to cores. So Andre, one of the interesting topics that I find about all of this is when you think about data analytics, right? And you think about the, you know, the batch job that's associated with it. And I, you know, I was doing some research with you all about ZoomInfo. Can you get use ZoomInfo to illustrate, you know, about that those data analytics and batch processing capabilities? >> they are first reference customer who has gone publicly to say that they saw a 60% improvement in the overall price performance ratio. It's also worth noting while they are one of the customers who is coming forward, generally speaking, data analytics workload ETL pipelines are all CPU heavy driven and they are a perfect fit for an arm architecture. So a lot of our partners in the ecosystem who depend on Kubernetes to run their ETL pipelines have adopted C4As and N4As for improved performance, as well as our own native cloud workloads such as Dataflow, which is our ETL pipeline product platform as a service, as well as DataProc, do take advantage of C4A, and we do see a performance increase as well in that particular space. And again, kind of going back to faster memory, dedicated cores, amazing storage, combination of it all allows you to run things much faster at a much lower cost. So let's go a little Let's Let's dive a little deeper into the C4A cuz we've talked about the M4A. Maybe we could get a perspective on how it does deliver that high performance. What is it about it that makes it really appropriate for large-scale databases and in-memory cache and those kinds of things? Maybe that's a question better for Yeah, I can help me out a little bit about the magic of the memory inside of your uh on It's that of our secret sauce which is shared by you. But again, the magic starts for the databases mostly Most database engine, if you look at the the the commercial ones as well as the non-commercial ones, they don't necessarily benefit a lot from a hyperthread. So a C4A instance is always dedicated high frequency core, dedicated core. The second component of this in C4A do allow you to have high memory instances, which allows you to cache a lot of stuff in RAM, and due to the power of the memory that is the secret sauce of arm, it allows you to also retrieve a little bit faster. Yeah, yeah. And I think, yeah, add on to that, right? So the one point that Andre mentioned about hyperthreading, so with arm, like I said in the beginning that you get like a physical dedicated core for any of your workloads, right? So you essentially get more compute to kind of play around, and it's efficient by design, so you get essentially better performance. So that kind of contributes to that. Again, not going into too many details, but yeah, that's essentially the sum of it. So with C4A then, how do you How is it performing on GKE? Yeah, so we've seen both, you know, the N4A and the C4A, you know, have great great performance for GKE, right? I think it's a mix and match of the you know, types of workloads that we need, right? We have a lot of customers who have started to look at this for the more compute-intensive workloads are using C4A. You know, and again, some of the good price memory CPU to memory ratio there, higher density, are great for computational workloads, reducing a lot of that kind of price performance requirements. And then from a, you know, GKE perspective, I think people love to use Kubernetes. You still got to orchestrate all this magic, right? You're not just running one machine, two machines. How do I just in time, you know, scale these out, right? How do I provision more VMs, more nodes, provision them with the right shapes and sizes. So we're seeing, you know, a real uptick, you know, in that. Yeah, we're seeing a real acceleration in the uses of microservices, for instance, right? This seems to apply to this story. Oh, yeah, for sure. I mean, microservices have been, you know, as you as we talked earlier, you know, the architecture involves a lot of a lot of, you know, containers, smaller containers, and a lot of them have been about horizontally scaling sort of these components, right? Running more instances of them. Clearly, you don't want to go to the old school way of like over-provisioning, right? 15 servers, right? What if I can sort of, you know, just provision when I need them and when I need to scale. >> Yeah. And if we can mix and match the right size node, right? My pod needs one core and two two gigs of memory, right? Only provision that amount. And then when I need to scale, provision more and more, right? And that's one of the things about the sort of N4A. And then on the compute-intensive workloads, we'll see the same thing, right? And again, we can just we can mix and match those, so you can run databases, batch jobs on one set of compute, and you can run the C4A, and you can just as easily run your microservices on the N4A all in the same cluster. So I just Go ahead, yeah. I also add that Kubernetes has the automatic capability of scheduling your workloads. If they are tagged for arm, it will find the node pool for arm and make sure that it uses the most cost effective resource effectively to power your workloads, which will translate to secret dollars appearing in your budget to do more with less. So secret dollars, I like it. Yeah, there we go. So let's let's discuss some AI ML inferencing we were talking about. I was talking a little bit earlier. Um It's come up a few times at the conference already and how inferencing is going to be done by millions and millions of companies at some point. Right, we'll all be kind of thinking about how do you do inferencing. So I'm thinking about how does Axiom fit into the AI ML world? Yeah, absolutely great question. So the latest, as you probably know, is agents. Everybody wants to run an agent. >> Yeah. Everybody wants to know LLM somewhere or some sort of machine learning model. What's not everybody is realizing is part of your agents and your agent orchestration systems, agents will have tools, and those tools will call our things, potentially just classical machine learning models that technically don't always need to have a GPU behind them. Secondarily, from an inferencing perspective, GPU come with VRAM. And sometimes that VRAM is not sufficient for your workload or it's the opposite, too big. And cloud after all is all about efficiencies and gains. So, if you're seeing an underutilized GPU, there's a lot of workloads that can benefit, specifically smaller LLMs that could be inferencing on CPUs. If you don't have a clear need for low latency out of the response and more of a job that relies on batch, you will potentially see that a C4A instance will be a much cheaper alternative versus you renting a GPU. Similar fashion, if you are running tools or you're running a rag or if you're running embeddings, a CPU is just as efficient to be able to do the pre-work, the pre-queuing before it sends data to the LLM. And again, you're kind of decoupling your tools and processes and only using the right things when it matters. And not necessarily just throwing everything out to the GPU or suffering the performance downgrades of a CPU-based inference. So, smaller jobs, experiment, and you'll find a clear path. >> Did you like to add anything to that? >> Yeah, I think you covered it pretty pretty perfectly. But I think one more thing is that with inferencing, and especially that's the message we want to land, that whenever you hear AI, you don't need to run a 3 billion parameter model on a GPU. You can do that well very well on a CPU. And you can go beyond that. So, inferencing is well supported. So, one more thing in addition to that other than LLMs is about any traditional use cases or MLPerf, for example, recommendation engines like DLRM V2. They are highly optimized for Arm. So, you get like much lower latency with Axion, and you get better performance in there. So, again, I'm going back to that performance and efficiency story, but the entire landscape of AI/ML kind of fits well in that in that story. So, I just want to follow up with a question that like relates to you know edge AI. And is this is this is this applicable to the edge AI or is this more mostly for cloud-based data centers? Or or are we thinking about this this capability useful in you know deployments such as you know edge devices? Or is it more applicable to the data center? So, it depends. So, by edge do you mean like running your models on the edge on Arm? Yeah, I mean if you're like a like let's say you're like I'm I'm in an industry, you know, I let's say I'm a you know, I'm in some in the auto industry and I want to be able to use you know, you know, I want to be able to apply this to my factory for instance inside and then actually have the instance there at the factory. Is this applicable to that? I mean, yeah. I mean, Arm architecture is present at the edge as well. Yeah, yeah. So, you can in theory, you can run it. But again, we have to look at the form factor, what kind of models you're running. You need to quantize the model to be able to run it in as such a smaller factor. And we have so Arm has libraries, Cloudera AI, that runs underneath. So, you can take advantage of that and run it a very efficient fashion even at the edge. Yeah, I just wanted to add also Google um has been famous for the use case that you're describing. We you believe I believe everybody knows we own Android. Right. >> Android has been making strides in running local LLMs on our cell phones. >> Right. >> you look at the Pixel 9 and now 10, they will run a local version of >> Right. um our model paired with a CPU that's Arm-based. >> Yes. And most importantly, we also have MediaPipe, which will allow you to adopt uh the same technology whether it lands on top of an Android or an iOS. And we do see quite a few customers asking for this use case. Google's belief in that particular space um is multi-tiered, I would say. So, there's certain things that would be great on top of a cell phone. Then more powerful devices might be needed from um our partners. Google also has a distributed cloud, which will leverage a combination of CPU and GPU to help with your needs and bridge the gap where horsepower is needed at the lower latencies. All right. Well, let's move on. So, um so, I want to ask you, Gary, about this concept of custom machine types and why they're important. What makes them interesting uh for using them in the N4A? Yeah, and we were I think we alluded to this a little bit earlier, but um you know, again, we've talked a lot about what people really want to be able to do is be able to right-size the compute for their kind of workloads, right? And and workloads come in all kinds of different sizes and shapes, right? Um and you also, as you mentioned, we want to be able to provision, you know, more on demand, right? And not have to pre-provision, right? So, with custom machine types, um we're able to basically kind of mix and match that sizes, right? You're no longer limited to fixed shapes. Uh this makes it really easy for us for like from a Kubernetes perspective to pick a node that's going to perfectly match whatever pods we needed scheduled, um which is going to of course we already talked about the price performance in general, but now we're reducing any extra spend, uh you know, extra cores or over-provisioning. Uh and then we're able to provision more and more of those. Um and the other nice thing, I guess to chop up the Kubernetes side on there, is that one of the big things that we have been working on on GKE for a long time is what we call node auto-provisioning, right? And very much like it used to be in the old days where people picked you buy 15 servers from somebody that are this shape and you just provision them out there. That's kind of how people looked at things from a Kubernetes perspective. I'm going to use this node and it's going to have, you know, 32 Arm cores and 64 gigs of memory even if I only deploy a pod with, you know, one core or whatever, and I've got 63 extra cores left. So, we've always been the ability to say, "Hey, based on your workload, we will find and pick the right machine shape um for the for those workloads are in the backlog. Now, we have the ability to pick to customize that shape even more, um which is going to greatly reduce your cost. You know, one of the interesting aspects of Arm is how it, you know, is questioned, I think, maybe a little bit about how difficult it is to use at times, maybe in the in in the data center environment. But you have simplified multi-architecture images in in quite a significant way. Like for instance, how how difficult is it to to build a a multi-architecture manifest, you know, in Docker, for instance? It's single-line change. Can I say that? No. Just wondering. No, so but it's it's very simple. I think the the notion comes from it where people have people who are on the legacy architectures have been in the past, they they feel that resistance moving on to a new architecture, that it's going to be a much more heavy lift compared to a straightforward answer. It is in majority of the cases is a straightforward thing. If you are building your images in a multi-architecture fashion that can go to both x86 and Arm64, you can deploy it seamlessly. If you are cross-compiling your applications from one architecture to other, there are let's say a Golang-based application or a Python-based application, you just have to recompile it, make sure that there is an Arm binary that can exist and can be deployed. It's not a heavy lift. We did a workshop yesterday, which essentially what you were talking about, that migrating to Arm or Axion, it's not that hard. You can just you just need to start and then just go for it. And we have so many resources. We can help with that transition, too. And I'm sure the Google guys can. Yeah, I just wanted to also note that we're absolutely agreeing with everything you're saying. Google has been on this journey to transform to Arm as well. A lot of our services that we picked to battle test Arm for our native services have gone through the journey. We're seeing tremendous results. So, it's not only you consuming it in Kubernetes or other services in the cloud, Google eats its own dog food. We've been eating it for a while. We see a tremendous amount of performance increase in our number one products like BigQuery, Spanner, and the rest. And not the ones that you see in the cloud, the stuff that actually powers everything under the cover for our cloud native for our normal Google services. I guess in conclusion, we're running out of time. So, so Gary, how can developers and platform engineers here, you know, learn more about um N4A on GKE on GKE? Well, I mean, the easiest way we have I you know, just go to our site. Uh we've got we've got great, you know, great blog on how to do it. You know, sort of great tutorial. Um and you know, follow through on sort of getting started, but it's just available for people to easily get started. We have examples of how to build Arm-based images um or even cross-compile them. Uh simple way to configure this with a thing we call compute classes, just annotate their pod spec, and then the magic will happen. We'll automatically provision those nodes for them cuz they're the best with the cuz we look for the nodes that match with Arm and best price performance. And if you're not Gary and you don't know how to use Google Cloud, there's always the cloud assist that will be there to help you gentle your way into our ecosystem. And don't forget our cloud assist as well that could hypothetically help you at least merge to 80% gap to convert to something that's other than x86. Well, thank you guys for for for chatting a little bit about Axion and and its direction. Thank you very much. Yeah. Thank you. Thanks. Thank you for having us. Thanks, Alex. If you like this video, please give us a thumbs up. And if you'd like to see more videos like this, you can always subscribe to our YouTube channel. We're on all the major social media platforms. You can always find us at thenewstack.io. We hope to see you soon. [Music]