How Does the Kubernetes Scheduler Pick a Node?

So, today we are going deep into one component, the kube-scheduler. This is the thing that decides which node your pod actually runs on. Sounds simple, pick a node, done. But, it is not simple at all, not even close. Let me explain. So, picture this. You apply a pod, kube-control sends it to the API server, the API server writes it to ETCD. And at that moment, your pod has no node. The field spec.nodeName is empty. Just empty. So, somebody has to pick one. That somebody is the scheduler, and it has to be fast. Really fast. On a busy cluster, hundreds of pods per second, each one needs a node. Each one has constraints, resources, taints, affinity, topology, spread, a lot of things to think about. So, how does it do it? Since Kubernetes 1.19, the scheduler is built around something called the scheduling framework. And the idea is beautiful. The scheduler core is thin, really thin. All the actual logic lives in plugins. Plugins like node resources fit, taint toleration, image locality, pod topology spread. Each one does one job and does it well. You can disable them. You can add your own. You can even run multiple profiles side by side, and pods pick a profile using spec.schedulerName. Pretty flexible. So, the framework defines these extension points. Think of them as hooks. Stages in the pod's journey where plugins get to run. And there are a lot of them. PreEnqueue, QueueSort, PreFilter, Filter, PostFilter. That is where preemption lives, by the way. Then, PreScore, Score, NormalizeScore, Reserve, Permit, pre-bind, bind, post-bind. 13 stages, 13. Each one a chance for a plug-in to say something. And every single pod walks through all of them. Step by step, step by step until it lands on a node. So, here is what we are going to do. We are going to take one pod, a simple engine X pod, and we are going to watch it go through every single stage. From the moment the scheduler sees it to the moment it is bound to a node, queue, filter, score, reserve, bind, the whole thing. Let us go. So, how does the scheduler even see a new pod? It does not poll. That would be very inefficient. Every kube scheduler uses something called a shared index informer. It opens one long HTTP/2 stream to the API server. Just one stays open. And then what happens? A pod lands, writes to etcd, spec.node name is empty. The API server fans the event out to every subscriber. The scheduler's add func handler fires. Event-driven, no polling, no delay. So, the pod is in, but it is not scheduled yet. It waits. It waits in one of three queues. First is active queue. This is a priority heap. Highest priority on top. Then, there is back-off queue. That one is time-ordered. If your pod failed recently, it sits here for exponential back-off. And then there is unschedulable queue. These are pods that just would not fit anywhere, but cluster state changes, right? New node shows up. Someone deletes a pod, and those pods get moved back to active queue to try again. So, the pod gets popped. First stop is pre-filter. Pre-filter runs once, just once per pod. This is really important to understand plugins like node resources fit, node affinity, pod topology spread, inter-pod affinity, and volume binding. They compute what this pod actually wants. It's resource requests, it's affinity terms, it's topology spread, and they stash all of that in cycle state. So, later when filter and score run per node, they just read the cache. Compute once, read many times. Smart, right? So, now filter six candidate nodes on the table. Each one gets checked in parallel. One go routine per node. Node resources fit runs and says node A2 is full, no room, eliminated, gone. Then taint toleration hits node A3, GPU taint, out. One unschedulable verdict, just one, and the node is out. Binary, no second chances. And on a really huge cluster, thousands of nodes, percentage of nodes to score caps the scan. You do not need to check every single node. That would be way too slow. So, what if every node fails, every single one? Zero survivors. Now what? That is when post-filter kicks in. Default preemption plugin, and it asks a question, if I evicted some lower priority pods, could this one fit? If yes, it picks the victims, sets nominated node name, deletes those victims gracefully, and the pod goes back to active queue. Tries again next cycle. Expensive, though. Really expensive. 50 milliseconds, sometimes seconds. But most pods, 99%, they never hit this path. So, filter survived. Four nodes left. Before we score them, pre-score. Same trick as pre-filter. Pre-compute once. For plugins that do heavy per node work during scoring, pre-score runs once and caches the result. Things like inter-pod affinity and pod topology spread, they aggregate topology counts, build affinity maps, stash it all in cycle state. So, later when each node gets scored, every call is basically O of 1. Just a look up, really fast. So, now score the leader board. Every plugin rates every node 0 to 100 in parallel, of course. But, here is what makes it interesting. Each plugin has a weight. Most plugins are weight one, standard. But, taint toleration is three. That is a big deal. And inter-pod affinity, pod topology spread, node affinity, all two. So, higher weight means bigger say in the final winner. That is how the defaults express what matters. So, quick zoom on one plugin, image locality. I love this one. It asks a super simple question. Node, do you have the image layers cached? That is it. And node B1 has engine X cached. Score. 100. Maximum. Node B2. Nothing. Zero. Why does this matter so much? Because on a cold node, kubelet has to pull the image over the network. That could be seconds, sometimes tens of seconds, but a cached node, pod starts in milliseconds. Huge difference. So, all plugins have scored. Time to pick a winner. Normalize score rescales every plugin's output to 0 through 100. Standard range, then for each node, multiply every plugin score by its weight, add it all up. Sum, and node B1 wins. 847, highest. What if there is a tie? Random. Really. Goes rand. dot int Why random? Because deterministic ties would hotspot the same node every pod every time same place, not great. So, winner picked. But the API server does not know yet. First reserve. The scheduler takes Node B1's in memory snapshot and subtracts whatever the pod requested. If your pod asked for 500 millicores and 512 megabytes, that is what comes off. No resource requests on the pod at all. The scheduler does not make up defaults. That is limit ranges job back at admission time. Here, it just reserves zero requests only never limits. So, the next pod in this same cycle sees Node B1 as already loaded. Makes sense, right? Anything fails after unreserve rolls it all back. Then permit. A hook, a plugin here can approve, wait, or reject. Stock cluster, no op. But gang scheduling, queue, volcano, co-scheduling, they wait here until all pods in the group are ready. So, finally bind. The commit. One API call. Just one. HTTP/2 post to {slash} API {slash} V1 {slash} namespaces {slash} default {slash} pods {slash} engine x {slash} binding. That {slash} binding part is a sub resource. A special endpoint, the API server receives it, sets spec.nodeName to Node B1, one etcd write. 201 created, done. Pod is bound. So, that is the whole cycle. Watch event to bound pod. Roughly 8 milliseconds on a 500 node cluster. 300 pods per second on a single scheduler binary. Pretty fast. And the scheduler already gone. already on the next pod. Meanwhile, node B1's kubelet has its own informer. It sees the binding event, picks up our pod, and then the whole sandbox CNI image pull downs kicks off. But, that that is a different video. So, enough theory. Let us actually see this happen on a real cluster, live, no tricks. So, here is my little cluster. One control plane, three workers already, nothing fancy. Now, keep an eye on worker three. We will come back to it in a second. So, worker three is not just any node. It has a taint. Workload equals GPU, no schedule. Which means the scheduler will not put anything there unless the pod tolerates it. For a plain engine X pod, worker three is invisible. Gone. Out of the running. So, let us deploy some pods. Engine X, two replicas, each one asking for eight CPU. That fills most of one node. Kube control applies it. Deployment created. And look at this. Both pods running. One landed on worker one, the other on worker two. The scheduler did its job. Worker three was tainted out. Worker one and worker two were the only options left. Perfect placement. So, before we deploy the next pod, let me show you the spec. Priority class name, high priority payments. That is set to 1 million. Requests eight CPU, same as the engine X pods, but way higher priority. That is what makes this interesting. So, now the fun part. Preemption. Let us apply this critical pod. The priority class and the pod both get created. But remember, worker one and worker two are already full. Worker three is tainted. So, there is no room anywhere. Watch what happens. So, describe the pod and look at these events. Failed scheduling first. Zero of four nodes available. Two insufficient CPU, two had untolerated taints, then scheduled on worker two. But wait, worker two was full a second ago. How? Preemption. The scheduler evicted one of the engine X pods to make room. Post filter did its thing. And the final state. Payments critical running on worker two. That slot belongs to it now. One engine X pod still running on worker one untouched. And the one that got evicted? Pending. No room anywhere in the cluster. That is the trade the scheduler made. Higher priority wins, lower priority yields. Simple rule, huge consequences. That is the scheduler in about two minutes. So, that was the scheduler, every stage, every plugin, all of it in about 12 minutes. If you are enjoying this animated series, hit subscribe on the Cube Simplify channel. It really helps me keep making these. See you in the next one.