▶ Videos →

📰 Dev.to · Randy AP

3 articles · Updated every 3 hours · View all reads

All Articles 110,584 Blog Posts 120,942 Tech Tutorials 28,193 Research Papers 22,443 News 16,572 ⚡ AI Lessons

Running Mixtral 8x7B at 21+ TPS on Pure CPU via io_uring and Predictive Caching

Dev.to · Randy AP 1mo ago

Running Mixtral 8x7B at 21+ TPS on Pure CPU via io_uring and Predictive Caching

The current consensus in AI infrastructure is unyielding: if you want to run frontier Mixture of...

I streamed Mixtral 8x7B from NVMe on a $0.40/hour VM and got 3.32 tps, here's how

Dev.to · Randy AP 🏭 MLOps & LLMOps ⚡ AI Lesson 1mo ago

I streamed Mixtral 8x7B from NVMe on a $0.40/hour VM and got 3.32 tps, here's how

I streamed Mixtral 8x7B from NVMe on a $0.40/hour VM and got 3.32 tps — here's how Most...

I built a Rust inference engine that streams MoE expert weights from NVMe SSDs, no GPU required

Dev.to · Randy AP 📐 ML Fundamentals ⚡ AI Lesson 1mo ago

I built a Rust inference engine that streams MoE expert weights from NVMe SSDs, no GPU required

Most people trying to run Mixtral or DeepSeek-V3 locally hit the same wall: they don't have 80GB of...