Designing Machine Learning Systems | Chapter 7: Model Deployment & Prediction Service

onepagecode · Beginner ·🏭 MLOps & LLMOps ·2w ago
Skills: ML Pipelines53%

About this lesson

Read the detailed version on: https://onepagecode.substack.com/ Use this url: https://onepagecode.substack.com/p/large-language-models-architectures In Chapter 7 of "Designing Machine Learning Systems" by Chip Huyen, we explore one of the most critical and often underestimated parts of the ML lifecycle: model deployment and building prediction services. Get the book on amazon here: https://amzn.to/3T5SBTV This chapter starts by debunking common myths about ML deployment — such as “you only deploy one or two models,” “models don’t degrade over time,” and “most teams don’t need to worry about scale.” We then compare the two primary ways models serve predictions in production: batch prediction (asynchronous) and online prediction (synchronous), including the emerging pattern of streaming prediction that combines batch features with real-time streaming features. We discuss the challenges of maintaining separate batch and streaming pipelines and the industry trend toward unifying them. The chapter also covers model compression techniques — including low-rank factorization, knowledge distillation, pruning, and quantization — which are essential for reducing model size and inference latency, especially when deploying to edge devices. Finally, we explore the trade-offs between running ML on the cloud versus on the edge, how compilers and intermediate representations (IRs) help optimize models for different hardware backends, and the growing potential of running ML models directly in browsers using WebAssembly. What you’ll learn in this chapter: • Common myths and realities of deploying ML models in production • Batch prediction vs online prediction vs streaming prediction • Unifying batch and streaming pipelines • Model compression techniques (quantization, pruning, distillation) • Cloud vs edge deployment trade-offs • How compilers optimize ML models for hardware • Running ML models in browsers with WebAssembly This chapter is essential for anyone moving models fr

Original Description

Read the detailed version on: https://onepagecode.substack.com/ Use this url: https://onepagecode.substack.com/p/large-language-models-architectures In Chapter 7 of "Designing Machine Learning Systems" by Chip Huyen, we explore one of the most critical and often underestimated parts of the ML lifecycle: model deployment and building prediction services. Get the book on amazon here: https://amzn.to/3T5SBTV This chapter starts by debunking common myths about ML deployment — such as “you only deploy one or two models,” “models don’t degrade over time,” and “most teams don’t need to worry about scale.” We then compare the two primary ways models serve predictions in production: batch prediction (asynchronous) and online prediction (synchronous), including the emerging pattern of streaming prediction that combines batch features with real-time streaming features. We discuss the challenges of maintaining separate batch and streaming pipelines and the industry trend toward unifying them. The chapter also covers model compression techniques — including low-rank factorization, knowledge distillation, pruning, and quantization — which are essential for reducing model size and inference latency, especially when deploying to edge devices. Finally, we explore the trade-offs between running ML on the cloud versus on the edge, how compilers and intermediate representations (IRs) help optimize models for different hardware backends, and the growing potential of running ML models directly in browsers using WebAssembly. What you’ll learn in this chapter: • Common myths and realities of deploying ML models in production • Batch prediction vs online prediction vs streaming prediction • Unifying batch and streaming pipelines • Model compression techniques (quantization, pruning, distillation) • Cloud vs edge deployment trade-offs • How compilers optimize ML models for hardware • Running ML models in browsers with WebAssembly This chapter is essential for anyone moving models fr
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Related Reads

📰
Building a Self-Updating ML System: CI/CD, Deployment, and Everything That Broke Along the Way
Learn to build a self-updating ML system with CI/CD, deployment, and troubleshooting
Medium · Machine Learning
 📰
Building a Self-Updating ML System: CI/CD, Deployment, and Everything That Broke Along the Way
Learn to build a self-updating ML system with CI/CD and deployment using a real-world example from an MLOps portfolio
Medium · Deep Learning
 📰
The model alone won’t make the cut
A well-performing model is not enough for a successful product, emphasizing the importance of MLOps and software engineering in machine learning development
Medium · Machine Learning
 📰
Your AI App Works in Demo but Fails in Production — Here Are the 7 Missing Pieces
Learn the 7 missing pieces to fix your AI app that works in demo but fails in production, ensuring a smooth deployment and operation
Medium · DevOps
Up next
Pole Pruner How A Rope Lever Shears High Branches
Innoforge Studio
Watch →