Table of Contents
- The Utilisation Gap
- Why Requests Are Always Wrong
- The Bin-Packing Problem Nobody Solves
- Autoscaling That Actually Reduces Cost
- Where the Money Actually Goes
- A Practical Reduction Sequence
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: Kubernetes cost problems are almost never compute pricing problems. They are resource request problems — engineers guessing at CPU and memory numbers, guessing high, and paying for the gap between the guess and reality.
The Utilisation Gap
Pull up your cluster’s actual CPU utilisation against its provisioned capacity. In most organisations the number sits somewhere between 10 and 25 percent.
You are paying for the provisioned capacity. You are using a fifth of it.
This gap is not caused by cloud provider pricing, instance type selection, or reserved capacity decisions — the three things teams usually attack first. It is caused by how Kubernetes schedules workloads, and specifically by the resource requests engineers declare in their manifests.
Understanding the mechanism is the whole game. The scheduler places pods based on their requests, not their actual usage. A pod requesting 2 CPU cores reserves 2 cores worth of schedulable capacity on a node whether it uses 2 cores or 0.05 cores. Twenty pods each requesting 2 cores and each using 0.1 cores will fill a 40-core cluster while consuming 2 cores of real work.
The cluster reports itself as full. Your monitoring shows idle hardware. Both are correct.
Why Requests Are Always Wrong
Resource requests are set by humans, usually once, usually under uncertainty, and almost never revisited.
Consider the incentives facing an engineer writing a deployment manifest. Setting requests too low risks the pod being throttled, evicted, or OOM-killed — a visible production incident with their name on it. Setting requests too high costs money that appears on a bill nobody shows them, attributed to a cluster shared across teams.
The rational individual choice is to overprovision generously. The aggregate organisational outcome is a cluster running at 15 percent utilisation.
Three specific patterns compound this:
Copy-paste inheritance. A manifest gets copied from an existing service. The new service has entirely different resource characteristics, but the requests come along unchanged. Nobody measured either service.
Peak-based sizing. Requests get set to handle the worst traffic spike ever observed, then never lowered. The pod holds peak-sized reservations continuously to survive an event that occurs quarterly.
Requests equal to limits. A common recommendation, and reasonable for latency-critical workloads, but applied indiscriminately it eliminates any possibility of overcommitment. Every pod permanently holds its maximum.
The fix is measurement rather than negotiation. Vertical Pod Autoscaler in recommendation mode observes actual usage over time and reports what requests should be. Running it in recommendation-only mode across a cluster produces a list of overprovisioned workloads within days, with no risk to running services.
# Actual usage vs requests, per pod
kubectl top pods --all-namespaces --no-headers | head -20
# What VPA thinks each workload actually needs
kubectl get vpa -A -o custom-columns=\
'NAME:.metadata.name,CPU:.status.recommendation.containerRecommendations[0].target.cpu,MEM:.status.recommendation.containerRecommendations[0].target.memory'The gap between those two outputs, multiplied across a cluster, is your cost reduction opportunity.
The Bin-Packing Problem Nobody Solves
Even with correct requests, node sizing creates a second layer of waste.
Kubernetes schedules pods onto nodes, and nodes come in fixed sizes. A node with 16 CPU cores hosting pods that request 3 cores each fits five pods and wastes one core permanently. Add system daemons, monitoring agents, and the kubelet’s own reservations, and usable capacity drops further.
This is a bin-packing problem, and it has an unavoidable inefficiency. What is avoidable is making it worse:
Node sizes that fit poorly. If your typical pod requests 3 cores, a 16-core node wastes a core per node. An 18-core node would waste nothing. Matching node size to workload shape is a genuine optimisation, though it constrains flexibility.
Too few, too large nodes. Large nodes amortise system overhead better but produce coarser scaling granularity and larger blast radius. Very large nodes also mean scaling down requires draining substantial workload.
Too many, too small nodes. Each node carries fixed overhead — kubelet, container runtime, monitoring agents, OS reservations — typically 0.5 to 1 core and 1 to 2 GB. Across fifty small nodes that overhead becomes a significant fraction of the bill.
Ignoring pod density limits. Nodes have maximum pod counts independent of resource capacity. A cluster of many small workloads can exhaust pod slots while showing abundant CPU and memory.
The practical approach is a small number of node shapes matched to your actual workload distribution, rather than a single default type chosen at cluster creation and never reconsidered.
Autoscaling That Actually Reduces Cost
Most teams enable Horizontal Pod Autoscaler, observe that it scales up under load, and consider autoscaling solved. Scaling up is the easy half.
Horizontal Pod Autoscaler adjusts replica counts based on metrics. It reduces cost only if replicas actually decrease during quiet periods — which requires a sensible minimum replica count and metrics that genuinely reflect load. An HPA with minReplicas: 10 on a service that needs two replicas overnight saves nothing for two-thirds of every day.
Cluster Autoscaler adds and removes nodes as pods become unschedulable or nodes become empty. Scale-down is where it commonly fails, and the reasons are specific: a single pod without a disruption budget can block draining an otherwise empty node; pods with local storage resist eviction; and system daemons that tolerate all taints prevent nodes ever appearing empty.
Diagnosing blocked scale-down is worth doing explicitly, because it is frequently the difference between an autoscaler that reduces cost and one that only increases it.
Scheduled scaling deserves more attention than it receives. If your traffic follows a predictable daily or weekly pattern — and most business applications do — scaling on a schedule is more effective than reacting to metrics. Reactive scaling always lags; scheduled scaling anticipates.
Spot and preemptible capacity offers the largest single discount available, typically 60 to 90 percent, and is underused because teams assume their workloads cannot tolerate interruption. Many can. Batch jobs, CI runners, stateless request handlers behind a load balancer, and anything already designed to survive a pod restart are all candidates. A mixed node pool running stateful workloads on on-demand capacity and stateless workloads on spot captures most of the discount with modest architectural effort.
Where the Money Actually Goes
A rough breakdown of where waste accumulates in a typical unoptimised cluster:
| Source | Typical share of waste | Effort to fix |
|---|---|---|
| Overprovisioned resource requests | 40–50% | Low |
| Idle nodes autoscaler cannot drain | 15–20% | Low |
| Node size / workload shape mismatch | 10–15% | Moderate |
| No spot capacity for eligible workloads | 10–20% | Moderate |
| Orphaned volumes, load balancers, IPs | 5–10% | Low |
| Cross-zone data transfer | 5–10% | Moderate |
Two items on that list are worth highlighting because they are routinely missed entirely.
Orphaned resources. Persistent volumes whose pods were deleted, load balancers pointing at removed services, unattached IP addresses, and old container images in registries. None appear in cluster utilisation metrics because they are not cluster resources — they are cloud resources the cluster created and abandoned. They bill indefinitely and silently.
Cross-zone traffic. Spreading pods across availability zones is correct for resilience and generates charges for every byte crossing a zone boundary. A chatty service mesh with random pod-to-pod routing across three zones can produce data transfer charges rivalling its compute cost. Topology-aware routing, which prefers same-zone endpoints, addresses this without sacrificing the resilience the multi-zone deployment was for.
A Practical Reduction Sequence
Effort-adjusted ordering, highest return first:
Week one — measure. Deploy cost visibility tooling that attributes spend to namespace and workload. Without attribution, every subsequent conversation is speculation. Simultaneously enable VPA in recommendation mode everywhere.
Week two — delete. Audit orphaned volumes, load balancers, and unused IPs. This is pure waste with no performance trade-off and frequently produces a surprising immediate reduction.
Week three — right-size. Apply VPA recommendations, starting with the largest overprovisioned workloads. Begin conservatively — 80 percent of recommended values rather than exact — and watch for throttling. The top ten workloads usually account for most of the opportunity.
Week four — fix scale-down. Identify why the cluster autoscaler cannot remove nodes. Add pod disruption budgets, adjust daemon tolerations, and confirm nodes actually drain during quiet periods.
Month two — restructure. Introduce spot node pools for eligible workloads. Review node shapes against your workload distribution. Enable topology-aware routing if cross-zone transfer is material.
Ongoing — govern. Namespace resource quotas, admission policies rejecting manifests without requests, and cost reporting visible to the teams that generate it. Attribution changes behaviour more reliably than any policy.
Common Pitfalls
Optimising instance pricing before fixing utilisation. Reserved capacity commitments on an overprovisioned cluster lock in the waste at a discount. Right-size first, commit second.
Setting requests from limits. These serve different purposes. Requests are scheduling reservations; limits are throttling ceilings. Setting them equal removes all overcommitment capability.
Aggressive right-sizing without monitoring. Cutting requests too far causes CPU throttling and OOM kills. Move in steps and watch throttling metrics.
Ignoring memory. Memory is frequently the binding constraint and cannot be throttled — exceeding a memory limit kills the container. Memory right-sizing needs more headroom than CPU.
Treating cost as a platform team problem. The platform team cannot right-size workloads they do not own. Cost data must reach the teams writing the manifests.
Conclusion
Kubernetes cost optimisation is unusually tractable because the waste is concentrated and measurable. Roughly half of it lives in resource requests set by guesswork, and correcting those requires no application changes — only measurement and a manifest edit.
Start with attribution so you know where spend originates. Delete orphaned cloud resources, which is free money. Right-size the largest workloads using observed data rather than estimates. Make sure your autoscaler can actually scale down. Then move eligible workloads to spot capacity.
The clusters running at 60 percent utilisation are not running better hardware or negotiating better contracts. They are running workloads whose resource requests reflect what those workloads actually use.
Frequently Asked Questions
What utilisation should I target? 50 to 70 percent CPU on production clusters is a reasonable goal. Higher risks insufficient headroom for spikes; much lower indicates overprovisioning. Memory targets should be more conservative because memory pressure kills containers rather than slowing them.
Is Vertical Pod Autoscaler safe in automatic mode? It restarts pods to apply changes, which is disruptive for workloads not designed for it. Recommendation mode is safe everywhere and is where most of the value lies. Automatic mode suits batch and stateless workloads.
Can I run production on spot instances? Stateless workloads behind a load balancer with multiple replicas, yes — many organisations do. Stateful services, single-replica deployments, and anything with long graceful shutdown requirements should stay on on-demand capacity. Mixed pools let you have both.
Why does my cluster autoscaler never remove nodes? Usually a pod that cannot be evicted: no disruption budget, local storage, or a daemon tolerating all taints. The autoscaler logs its reasoning, and reading those logs is faster than guessing.
Do requests or limits determine my bill? Requests, indirectly. Requests determine how many nodes you need, and nodes are what you pay for. Limits only throttle usage. This is why request accuracy matters far more than limit tuning for cost.
Is a service mesh worth its cost overhead? It adds a sidecar per pod — real CPU and memory across every workload — plus potential cross-zone traffic charges. Whether that is justified depends on whether you use the traffic management and observability features enough to warrant it. Many installations do not.
