Local GenAI on Jetson: OSS models using different inferencing frameworks: Ollama, llama.cpp, & vLLM

NVIDIA Developer · Intermediate ·🏭 MLOps & LLMOps ·1mo ago

Key Takeaways

Runs OSS models like Gemma and Qwen on Jetson using Ollama, llama.cpp, and vLLM for local inferencing

Original Description

This opening session builds the foundation for running popular OSS models such as Gemma, Qwen directly on Jetson — no cloud required. We cover when to use Ollama for rapid local prototyping versus vLLM for higher-throughput serving, show how the same workflow applies to both power different OSS models, and walk through the real decisions behind model choice, containers, quantization, and performance tuning on edge hardware. We close with a teaser of OpenClaw and a bonus take-home challenge to kick off community building.
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Related Reads

📰
9 Best MCP Gateways for Model Context Protocol Deployments
Learn about the top 9 MCP gateways for efficient Model Context Protocol deployments and improve your MLOps workflow
Dev.to AI
📰
Building a self-healing MLOps pipeline on AWS: from raw data to a model that fixes itself
Learn to build a self-healing MLOps pipeline on AWS that automates model fixing, increasing model reliability and reducing downtime
Medium · Machine Learning
📰
Building a self-healing MLOps pipeline on AWS: from raw data to a model that fixes itself
Learn to build a self-healing MLOps pipeline on AWS that automates model fixes, increasing model reliability and reducing downtime
Medium · DevOps
📰
qModel Open-Source Platform v1.2.0 Released: Streamlined Python Model Integration & Execution Pipeline
Learn how to streamline Python model integration and execution with qModel Open-Source Platform v1.2.0, a tool for MLOps and AI development
Dev.to AI
Up next
Pole Pruner How A Rope Lever Shears High Branches
Innoforge Studio
Watch →