Appearance

Scheduler internals

Separate feasible placement from preferences and binding.

What you will understand

  • Distinguish scheduler filtering from scoring.
  • Trace reservation, permit, and binding outcomes.
  • Explain why preemption cannot solve every Pending Pod.

Builds on: Topology and resource governance, Informers, caches, and workqueues

Mental model: rank only nodes that can run the Pod

The scheduler observes Pods without a node assignment and attempts placement. Filtering asks whether a node is feasible under hard constraints: resource requests, required affinity, taints and tolerations, topology, volumes and other configured checks. Scoring then compares feasible nodes using preferences and weighted plugin results. A node with a very attractive score cannot win if it failed a required filter. This distinction is the foundation of useful Pending-Pod diagnosis.

Scheduling uses resource requests and scheduler accounting, not merely a live CPU utilization graph. A node can look quiet while its requested capacity is allocated. Conversely, a node can be busy because applications exceed their requests. The scheduler must also account for Pods assumed during scheduling while their bindings are still being completed. Placement decisions coordinate with concurrent activity; they are not an instantaneous global optimization of every Pod in the cluster.

Feasibility precedes preference

Preferences rank Node B among feasible nodes. A rejected node cannot win through a high score.

Scheduling viewStep 1 of 3
Read hard requirementsThe Pod requires a particular node label and requests CPU and memory.. Pending Pod, Requires accelerator label, active. Filter plugins, Check hard constraints, idle. Node A, No matching label, idle. Node B, Label and free requests, idle. Score plugins, Rank feasible candidates, idlePending Pod: Requires accelerator label
Pending PodRequires accelerator labelIn focus
Filter plugins: Check hard constraints
Filter pluginsCheck hard constraintsContext
Node A: No matching label
Node ANo matching labelContext
Node B: Label and free requests
Node BLabel and free requestsContext
Score plugins: Rank feasible candidates
Score pluginsRank feasible candidatesContext
Pending PodRequires accelerator labelIn focus
Filter pluginsCheck hard constraintsContext
Node ANo matching labelContext
Node BLabel and free requestsContext
Score pluginsRank feasible candidatesContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
No feasible nodes
Read hard requirements

The Pod requires a particular node label and requests CPU and memory.

Read the complete visual sequence
  1. Read hard requirements

    The Pod requires a particular node label and requests CPU and memory.

  2. Filter each candidate

    Node A lacks the required label; Node B satisfies hard constraints and remains feasible.

  3. Score only survivors

    Preferences rank Node B among feasible nodes. A rejected node cannot win through a high score.

No feasible nodes

Every node fails the required label even if it has spare CPU.

  1. All candidates mismatch

    The only available nodes do not carry the required accelerator label.

  2. No useful score stage

    The Pod remains Pending because there is no feasible candidate set to rank.

  3. Preemption cannot add labels

    Removing lower-priority Pods does not repair the required selector mismatch.

About this model

Two nodes and one label rule isolate the ordering; actual scheduler profiles include additional plugins.

Walkthrough: scheduling and binding have separate failure points

The scheduling framework exposes extension points. PreFilter and Filter establish feasibility, then scoring-related plugins rank candidates. Reserve can hold provisional state; Permit can delay or reject progress. PreBind performs required work before binding, and Bind assigns the Pod through the API. If later work fails, Unreserve releases reservations so an abandoned attempt does not permanently consume plugin state. PostBind is notification after a successful bind, not a second scheduling decision.

Scheduling cycles and binding cycles have different concurrency characteristics: the scheduling cycle selects a node, while subsequent binding work can overlap for different Pods. Volume binding is a common reason a placement path includes coordination beyond CPU and memory. A selected node still needs a functioning kubelet, image access, mounted volumes and readiness success. The PodScheduled condition proves a scheduling boundary, not an application health boundary.

A chosen node still needs binding

The API assignment lets the kubelet act, but image pull, mounts and application readiness are later boundaries.

Scheduling viewStep 1 of 3
Hold provisional stateReserve and Permit plugins coordinate work before a final API binding.. Selected node, Feasible, highest total score, idle. Reserve / Permit, Provisional coordination, active. PreBind work, Volume-related preparation, idle. Bind through API, Assign Pod nodeName, idle. Kubelet, Start and monitor containers, idleSelected node: Feasible, highest total score
Selected nodeFeasible, highest total scoreContext
Reserve / Permit: Provisional coordination
Reserve / PermitProvisional coordinationIn focus
PreBind work: Volume-related preparation
PreBind workVolume-related preparationContext
Bind through API: Assign Pod nodeName
Bind through APIAssign Pod nodeNameContext
Kubelet: Start and monitor containers
KubeletStart and monitor containersContext
Selected nodeFeasible, highest total scoreContext
Reserve / PermitProvisional coordinationIn focus
PreBind workVolume-related preparationContext
Bind through APIAssign Pod nodeNameContext
KubeletStart and monitor containersContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
PreBind failure
Hold provisional state

Reserve and Permit plugins coordinate work before a final API binding.

Read the complete visual sequence
  1. Hold provisional state

    Reserve and Permit plugins coordinate work before a final API binding.

  2. Complete prerequisites

    PreBind work such as volume coordination must succeed before the assignment is committed.

  3. Bind is not readiness

    The API assignment lets the kubelet act, but image pull, mounts and application readiness are later boundaries.

PreBind failure

A provisional choice can be abandoned before a Pod is assigned.

  1. Candidate selected

    The node passes filters and scoring, but placement is not yet final.

  2. Preparation fails

    A volume prerequisite fails during PreBind, so the attempt cannot commit a binding.

  3. Release and retry

    Unreserve releases provisional state; the Pod can return for another scheduling attempt.

About this model

Plugin extension points are grouped; scheduling and binding work may overlap for different Pods.

This small Pod is intentionally unschedulable on an ordinary kind cluster unless you add the matching node label. The label represents a hard requirement rather than a preference. Save it as placement.yaml; changing the label key is safer than changing arbitrary node configuration during diagnosis.

apiVersion: v1
kind: Pod
metadata:
  name: m15-placement
  namespace: kv-lab
  labels:
    lab: m15
spec:
  nodeSelector:
    training.kv.example/accelerator: present # Hard feasibility rule.
  containers:
    - name: app
      image: busybox:1.37.0
      command: [sleep, "3600"]
      resources:
        requests:
          cpu: 10m
          memory: 16Mi
kubectl --context kind-kv-lab -n kv-lab apply -f placement.yaml
kubectl --context kind-kv-lab -n kv-lab describe pod m15-placement
kubectl --context kind-kv-lab -n kv-lab get pod m15-placement -o jsonpath='{.status.conditions}{"\n"}'
# Remove only the intentionally impossible selector to retry placement.
kubectl --context kind-kv-lab -n kv-lab delete pod m15-placement

Practice: separate a failed filter from a low score

Use the m15 local exercise to inspect control-loop behavior, and optionally apply the isolated Pod above. Expect Pending with FailedScheduling evidence mentioning the selector mismatch. There is no winning feasible node to score in this case. A lower CPU request would not repair a missing required label. Rewrite the manifest without the selector and reapply if you want to observe scheduling; inspect spec.nodeName and the PodScheduled condition before checking container readiness.

Now predict preemption. If insufficient requested capacity is the obstacle and suitable lower-priority victims exist, preemption may create room. If every node fails an incompatible node selector or required topology constraint, removing other Pods does not make those nodes satisfy the selector. A nominated node is an intermediate scheduling intention and does not guarantee the eventual binding succeeds.

Pitfalls: configuration can encode impossible combinations

Required anti-affinity, a single available zone, a restrictive volume topology and a small node pool can leave no feasible placement. Adding replicas may then increase Pending count without increasing service capacity. Soft preferences are appropriate when violating the preference is better than remaining unavailable; hard requirements are appropriate when placement would be invalid. Document that product decision before converting constraints. Avoid setting spec.nodeName merely to bypass a confusing scheduler event: direct assignment changes which scheduler checks execute and can place the Pod somewhere the intended policy would have rejected.

When reading FailedScheduling events, separate each reported filter failure. The event summarizes rejected candidates; its counts do not establish that every node failed for the same reason. Compare node labels, taints and requested resources before choosing a configuration change.

Check your understanding

A node has the highest preferred-affinity score but lacks a required label. Can it win?

No. Required constraints remove it from the feasible candidate set before scoring. Diagnose the failed hard condition first; adjusting score weights cannot revive a node that does not pass Filter.

Next, examine how scheduler replicas coordinate leadership. Multiple control-plane processes improve availability only when their coordination and failure assumptions are explicit.

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.