Capacity and scale trade-offs
Size headroom around failure, startup and shared dependencies.
What you will understand
- Estimate schedulable capacity from requests and allocatable resources.
- Account for rollouts, failure domains and startup delays.
- Locate control-plane and application scaling bottlenecks separately.
Builds on: Scheduler internals, Autoscaling mechanics and feedback
Mental model: capacity is a constrained budget
Node capacity and node allocatable are different quantities. The operating system, kubelet and other reserved functions consume resources, leaving allocatable capacity for Pods. DaemonSets then occupy part of that budget on each eligible node. The scheduler compares Pod requests with available accounting and placement constraints. Summing physical CPU cores and dividing by a container’s observed average CPU use therefore overstates what you can safely promise a workload.
A simple planning estimate starts with allocatable resources after required system workloads, then checks both CPU and memory requests. The tighter resource sets a rough upper bound, but fragmentation, topology, ports, Pod limits and volumes can lower it further. A Pod requiring one large memory allocation cannot use small free fragments distributed across several nodes. ResourceQuota limits namespace consumption; it does not manufacture feasible capacity or reserve a replacement node during a failure.
Allocatable capacity must satisfy every constraint
The scheduler also checks topology, volumes and other hard constraints; aggregate free CPU is insufficient evidence.
Node reservations reduce physical capacity to the budget available for Pods.
- Node allocatable to Schedulerremaining budget
Read the complete visual sequence
- Start with allocatable
Node reservations reduce physical capacity to the budget available for Pods.
- Subtract required workloads
DaemonSets and existing Pod requests consume that budget before new replicas can fit.
- Check feasibility
The scheduler also checks topology, volumes and other hard constraints; aggregate free CPU is insufficient evidence.
Fragmented free resources
A namespace may be permitted to create a Pod that fits no individual node.
- Quota allows the request
The namespace has unused request budget, so admission can accept another Pod.
- No single node fits
Free memory is fragmented across nodes below this Pod requirement.
- Pod remains Pending
The request needs a feasible individual placement; quota does not merge node memory.
About this model
CPU and memory are highlighted; actual capacity also depends on Pod counts, devices, storage and network limits.
Walkthrough: the spare capacity has a job
Capacity must cover more than the steady-state replica count. A rolling Deployment may temporarily run surge Pods while old replicas remain. Draining a node moves demand onto the remaining pool. Losing a zone can concentrate replicas in fewer failure domains. Startup probes, image pulls, cache warming and storage attachment delay the point when replacements serve traffic. Headroom is the explicit budget for these transitions, rather than an unexplained percentage chosen after an outage.
Horizontal workload scaling and node autoscaling form separate feedback loops. HPA changes desired replicas from metrics, while node autoscaling typically reacts to unschedulable demand and provider-specific signals. New node provisioning may fail because of cloud quota, unavailable instance types, subnet limits or required-zone constraints. A local kind cluster cannot reproduce those provider contracts by creating more Pods. Treat its resource behavior as mechanism practice and validate real capacity decisions in the target environment.
Headroom covers rollout and failure transitions
New Pods supply useful capacity only after startup, readiness and endpoint propagation.
Requests fit during ordinary operation, leaving explicitly planned spare capacity.
- Steady replicas to Remaining node poolbaseline requests
Read the complete visual sequence
- Budget steady state
Requests fit during ordinary operation, leaving explicitly planned spare capacity.
- Spend headroom on transition
A rollout or unavailable node creates temporary placement demand beyond the ordinary count.
- Measure recovery completion
New Pods supply useful capacity only after startup, readiness and endpoint propagation.
Two transitions overlap
A simultaneous rollout and node failure can exhaust individually adequate headroom.
- Surge uses free slots
New revision Pods consume spare requests while old Pods still serve.
- A node fails
Replacement demand arrives before surge capacity has been released.
- Useful capacity lags
Pending Pods do not help traffic; examine rollout policy, failure budget and node provisioning constraints.
About this model
This is a planning model, not a measured guarantee; provider provisioning and application startup must be validated separately.
The following quota is a complete example of a namespace budget. It intentionally does not promise where those requests fit. Save as capacity-quota.yaml only if you want an isolated optional quota experiment; quotas can reject later requests in the namespace until removed.
apiVersion: v1
kind: ResourceQuota
metadata:
name: m17-budget
namespace: kv-lab
labels:
lab: m17
spec:
hard:
requests.cpu: "2"
requests.memory: 2Gi
limits.memory: 4Gi
pods: "20"
kubectl --context kind-kv-lab get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory
kubectl --context kind-kv-lab describe node kv-lab-control-plane
kubectl --context kind-kv-lab -n kv-lab get pods -o wide
kubectl --context kind-kv-lab -n kv-lab get resourcequota
# If the optional example was applied, remove just that budget afterward.
kubectl --context kind-kv-lab -n kv-lab delete resourcequota m17-budget --ignore-not-found
Practice: plan for one failure and one rollout
Use the m17 local exercise to connect a container’s request and limit to observed behavior. Then sketch three identical nodes with one CPU allocatable each after required system workloads. A service has six replicas requesting 250 millicores each. The aggregate request fits, but ask whether it also fits after one node fails and while a rollout creates two surge Pods. Include memory and placement constraints before claiming the plan works.
Compare the sketch with describe node allocated-resource output. That output reflects declared requests and limits, not a complete history of actual resource contention. For production planning, combine declarations with measured distribution, startup peaks and a failure drill. Record the assumptions behind any calculated replica ceiling. A capacity model is useful because it identifies what must be measured and tested; it is not evidence that an untested failure will meet an availability objective.
Pitfalls: more objects also load the control plane
Many short-lived Pods increase API writes, watch events, scheduling work, admission calls and etcd churn. More controller workers can increase throughput until the API, admission dependency or external service becomes the bottleneck. High-cardinality metrics and verbose logs can strain observability systems before CPU becomes the limiting application resource. Use backoff, bounded concurrency, event aggregation and sensible object lifetimes. Avoid treating published scalability envelopes as guarantees for your object mix and add-ons.
A service can have spare Kubernetes capacity while its database, connection pool or upstream quota is exhausted. Inspect latency, errors, queue length and dependency saturation together. Scaling one layer should be an experiment with an expected bottleneck change, not an automatic response to every slowdown.
Check your understanding
The namespace quota allows another Pod, but it stays Pending. Is the quota ineffective?
No. Quota authorizes consumption within a namespace budget; scheduling still requires a node that satisfies requests and placement constraints. Admission success and available physical capacity are different boundaries.
Next, size the coordination system itself. Control-plane availability depends on quorum, disk behavior, membership and failure domains in addition to worker capacity.
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise