Open source large language models with Sonatype | Amazon Web Services

Key Takeaways

Hi folks, my name is Dylan. I'm with Amazon. I am a solutions architect. I'm here with Tyler Warden. >> Hey y'all, I'm Tyler. I look after product at Sonotype. >> Today we're talking a little bit about AI. Who's not talking about AI right now? Open source and AI. >> The the landscape has changed a lot in the last few years. We're seeing open source models really starting to to really compete, right? like they're not, you know, when when it first came out, when when chat GPT was first released in 2022, there wasn't this landscape of model selection. Now we're seeing hundreds of models that from various sources and providers, you know, um DeepSeek came out last I think it was this year maybe even. Yeah. >> Uh you know, and it immediately shook everyone, right? We have Llama with Meta that's also competing, right? Um I'd like to hear a little bit about what Sonotype's doing with open source models and >> yeah look we we have uh customers that are coming to us and they frame the challenge this way is out there in the web you have hugging face and on hugging face I think we're close to two million models now large language machine learning models visual all sorts of stuff and these models are for all intents and purposes open source right And where organizations have said to us is, "Hey, look, you help us manage things from npm and you help us manage things for Maven and Pi and Rust and all sorts of other areas." But we're seeing our development teams grabbing hugging face models and pulling them in to their SDLC. So in the dev and in the build and the deploy side these models are working their way into uh data science teams and into application teams. How can you help us manage uh that problem space these AI models? How can you help us govern and manage and uh secure those as they're getting worked into their their products? So quick question, why are customers moving towards open-source models on hugging face? >> Yeah, really for the same reason that the open- source components have grown and exploded to help drive innovation over the past what is it 20 years. Why do you use open source? It's free. Well, you don't have to pay for it up front. Maybe it's not free to use pay for it up front and you get to take advantage of the work of others, right? So I don't need to write um a database driver or a database access layer myself. I can just go grab one that's already written, right? And there's this uh philosophy that if it's open potentially could be more secure, people can look and iterate on it and innovate with it. Well, much the same reason with hugging face. If I'm going to build and deploy my own software, ship my own software, use it, it's a much different kind of commercial model if I have to pay an OEM and bring it in versus if I can just take a open-source model and embed it in my software or use it in my own uh my own tooling. >> I gota Yeah. There's probably another thing with performance and the size and you know the variety, right? Um a lot of models are doing strong things in certain areas. The data it's trained on it can be different as well. >> Yeah, it makes a lot of sense. So I see dev build deploy. I see hugging face. I see npm. If you've watched any of our other videos, we're seeing a pattern here. I have a feeling zone type is going to be plugged in along here somewhere. >> Yeah. Let's let's let's talk about how we help solve uh these problems. First is we have the same story you might have heard if you watch these other videos around Nexus repository where these models can be uh proxied or cached. Right? This is about I get a version of of the model down and if I ask for that version via dev or build or deploy or any other stage you may have in your PDLC or SDLC I'm able to go get that version of the model I know it's that I know it's that one that I want. So you've got better build times more of of the build. But then you have multiple constituents, right? You have your application security teams. You have your um engineering and developer teams. You have your compliance and legal teams that want to be sure that the software that that they're writing is uh compliant. It has very little tech that can move fast and it's secure. Right? So you have these different constituents. That's what needed to ship software in the real commercial world. Right? And so what we have done is extended our SCA uh capabilities. So we call that our life cycle product whereby these at every step of the way these uh leaders can use the policy and the scanning capability to go and identify the models that are there and that's the first part of governance. Hey, what are are you using? We can scan those models for vulnerabilities, legal risk, compliance. And that scanning results are powered by our sonotype intelligence that we have extended to now contain data information, behavior analytics, all sorts of stuff from hugging face. You can now set policies that look with the scanning um to determine what models are being used and do you uh allow them. And then maybe you have models you don't want to allow in at all. Maybe you don't like their country of of origin. You don't like the data they're trained on. We're not here to tell you what your policy should be, but we're here to tell you you should probably have one. And so our our repository firewall can scan those models that are coming in using that same intelligence and could actually block any models that you may not want entered in uh that don't adhere to whatever policy you have. So just like open source uh components, open source models can be kind of governed and empowered in very much the same way. >> And so the various leaders in the organization so policy decision makers, the head of engineering, IT folks, compliance folks, they all contribute to this this decision-m framework and you know correct maybe the devs are making a request over here to the engineering leadership. They go talk to their compliance team. Then you add that to the repository and then it's >> and that's very much about scale as the policy can do that for you. If it's compliant, use it. manage by exception is a lot faster than manually reviewing everything. So look, this is how we're um helping organizations govern and manage their open-source models. Um when it comes to their software supply chain, how is uh AWS if I wanted to use open source models? Um how would AWS help me kind of manage, run, execute? >> Yeah, great great question. I'd say the the main the main uh the main thing we're focused on obviously at AWS is you know your infrastructure, right? So there are a couple ways that we do open source models AWS starting with everyone's favorite service is you know Amazon EC2 I think everyone's pretty familiar with EC2 um nearly 20 years later maybe it might be 18 years we're probably approaching 20 years where folks are running their own models directly on EC2 right so you >> deploy it you lo you SSH in but realistically you have your pipeline deployed to their EC2 instance your favorite way of uh you know running your model. Uh the next way when folks want a little bit more offloaded to Amazon, right? So for folks if aren't familiar, we have the shared responsibility model that branches to security that response that branches to operational excellence. Uh in this scenario, Amazon EC2, we provide you the the instance you have to control the security, the the ports being open or closed, for example, exposure to internet, etc. Most folks are pretty familiar with this. Um, when we move down a little step here towards SageMaker, that's the next that's the next way we see folks using open source models. This is a kind of a functional managed EC2 instance. So, think about, you know, deploying your model, but with a little bit more control than you would on a fully managed service. You're not just hanging an API endpoint, right? You are still controlling the instance with a little bit of guardrails around the edge and policies you can set. Um maybe if you want to do a bit more of like maybe training your own model like a parent uh teacher model for example, SageMaker is pretty powerful in that regard. The next way we're seeing folks if they really want it uh to be managed by AWS and again from from top to bottom I would say is unmanaged versus managed would be Amazon Bedrock. And so there's two ways we are seeing folks use Amazon Bedrock for this use case. So the first one is that AWS has partnered with a lot of the open source model providers like Meta um DeepSeek to actually bring these to be fully managed so that you just select them in the drop down or you make it you put it in your SDK uh and you call that model you don't think about it. Bedrock handles it. The other way folks are doing it so I I'll label it just to be super clear. It's like you know almost like a marketplace of models. The other way folks are doing it is with a custom model import. So for the more maybe the little bit more obscure models that are kind of more niche where maybe you're getting this from hugging face, you're using the custom model import feature to bring those models into bedrock where we're going to handle the infrastructure for you but maybe you know we're not validating that model and the source origin etc as much as we would be with you know the marketplace model uh approach. That's >> great. Well for Dylan and myself uh thank you very much for watching. >> Cheers.