Local GenAI on Jetson: OSS models using different inferencing frameworks: Ollama, llama.cpp, & vLLM
Skills:
LLM Engineering80%
Key Takeaways
Runs OSS models like Gemma and Qwen on Jetson using Ollama, llama.cpp, and vLLM for local inferencing
Original Description
This opening session builds the foundation for running popular OSS models such as Gemma, Qwen directly on Jetson — no cloud required. We cover when to use Ollama for rapid local prototyping versus vLLM for higher-throughput serving, show how the same workflow applies to both power different OSS models, and walk through the real decisions behind model choice, containers, quantization, and performance tuning on edge hardware. We close with a teaser of OpenClaw and a bonus take-home challenge to kick off community building.
Watch on YouTube ↗
(saves to browser)
Sign in to unlock AI tutor explanation · ⚡30
More on: LLM Engineering
View skill →Related Reads
📰
📰
📰
📰
9 Best MCP Gateways for Model Context Protocol Deployments
Dev.to AI
Building a self-healing MLOps pipeline on AWS: from raw data to a model that fixes itself
Medium · Machine Learning
Building a self-healing MLOps pipeline on AWS: from raw data to a model that fixes itself
Medium · DevOps
qModel Open-Source Platform v1.2.0 Released: Streamlined Python Model Integration & Execution Pipeline
Dev.to AI
🎓
Tutor Explanation
DeepCamp AI