Auto-Scaling Agent Worker Pools in Kubernetes

Auto-scaling agent worker pools in Kubernetes requires a different division of responsibility from CPU-driven replica scaling. KEDA should scale pods on application demand, especially queue depth, while Karpenter or Cluster Autoscaler provisions the nodes those pods need. This two-layer architecture fits bursty, stateful agent work, but its success depends on accounting for cold starts, GPU placement, and disruption.

Why CPU and memory metrics cannot describe an agent workload

A vLLM pod can show low CPU utilization while its GPU, task queue, and KV cache are saturated. For a stateless microservice, CPU often rises as more requests come in, so it can be a useful proxy for demand. For an agent worker, that relationship breaks because the limiting resource may not be the CPU at all. GPU utilization is not exported to the metrics server by default, so HPA cannot see that saturation out of the box.

Demand also arrives in a form that flat utilization averages cannot describe. A single document upload can trigger dozens of analysis tasks, followed by an hour without work and then another spike. Queue depth shows how much work is waiting, while CPU shows how busy the workers already running are. Session count can also matter when demand depends on the agent instances that are active. The diagnostic failure is therefore structural.

What makes agent workloads structurally different from the Kubernetes pod model

Agent workloads combine four properties, which are burstiness, statefulness, long execution times, and heavy GPU demand. Kubernetes Deployments treat replicas as interchangeable, but an agent session may need to return to the same pod because its state lives there. A request that would take milliseconds in a conventional service instead becomes a reasoning loop that runs for minutes or hours and touches many external systems. Killing that pod can lose its reasoning context, plan, and the state it has built up, with damage beyond a request retry.

These differences change the appropriate workload object as well as the scaling metric. A Deployment fits interchangeable stateless replicas, while a Sandbox CRD or StatefulSet better reflects the identity requirements of an agent. Agent Sandbox documentation describes the runtime requirement as “a long-running, stateful, singleton container with a stable identity”. That requirement cannot be supplied by increasing a replica count alone. Scaling must preserve the relationship between a session, its state, and its execution environment.

The cold-start penalty that scale-to-zero creates for agent pools

Removing idle GPU capacity controls cost, but restoring it introduces a delay directly into the agent’s execution path. Loading model weights into VRAM takes 30–120 seconds. A full start from zero also needs the image pull, CUDA graph capture, and KV cache warming, so large models can take many minutes to be ready. Even a well-tuned cluster may need tens of seconds to a couple of minutes to provision a new environment. An agent reasoning loop that needs a fresh sandbox in under two seconds will stall if it has to wait that long.

Scale-to-zero is economically necessary for GPU pools, but it cannot be the only operating posture for latency-sensitive paths. Warm pools and minimum capacity floors bridge that gap by retaining ready resources where allocation latency matters. GPU-backed warm pools remain less mature than CPU-backed pools because fractional GPU sharing and DRA integration are still developing. Consequently, a warm-pool design does not yet eliminate every GPU cold-start risk.

The two-layer architecture: KEDA at the pod layer, Karpenter or Cluster Autoscaler at the node layer

At the pod layer, KEDA sets the replica count based on demand from outside the pods. Its inputs include queue length, Redis streams, HTTP request rate, and custom application metrics. For workers that draw from a task queue, queue depth shows waiting work before CPU utilization is high enough to call for more workers. KEDA can reduce replicas to zero during inactivity and trigger growth when work returns. Kubernetes 1.37 makes HPA scale-to-zero a beta feature enabled by default, but KEDA retains the richer event-source model for heterogeneous agent queues.

At the node layer, Karpenter or Cluster Autoscaler responds when those pods cannot be scheduled. Karpenter provisions nodes in under a minute, while Cluster Autoscaler typically takes several minutes, although provisioning speed does not remove model initialization time. Karpenter selects a fitting instance at scheduling time, whereas Cluster Autoscaler works within predefined node groups. Karpenter’s additional ability to consolidate running workloads onto fewer nodes distinguishes it from Cluster Autoscaler in how it fulfills infrastructure demand.

KEDA responds to application signals within seconds, and the node autoscaler then operates within its provisioning window. Warm pools and minimum replica floors cover the interval between those decisions and usable capacity.

How GPU scheduling with Dynamic Resource Allocation changes what the autoscaler can see

Dynamic Resource Allocation gives placement decisions a richer description than a bare GPU count. DRA reached GA in Kubernetes 1.34 and lets a workload submit a structured resource claim. That claim can express relationships such as an accelerator together with the network interface that feeds it. The scheduler can therefore consider GPU model, node topology, and interconnect rather than treating every free device-plugin slot as equivalent.

DRA-aware autoscalers are beginning to use the same claims when simulating additional capacity. This allows a scale-up decision to consider whether a proposed node has the right resource shape before the pod remains pending on unsuitable infrastructure. DRA addresses workload resource requirements, while utilization export shows how busy an accelerator is. Sajjan Gundapuneedi and Neelendra Bhandari of AWS described DRA-aware autoscaling as emerging on October 19, 2026. Support remains uneven across implementations, creating a fragmentation risk for multi-cloud deployments.

Where the two-layer model breaks down: long-running GPU jobs and the consolidation problem

Karpenter disruption budgets or reserved node groups are necessary options for training jobs that must not be interrupted. Consolidation helps bursty worker pools, but jobs that run for hours or days need a different policy to keep them stable. Moving workloads to reclaim capacity can introduce rescheduling and resource churn during long-running computation. With Cluster Autoscaler’s explicit node groups, a small GPU pool can stay running all the time, including p4d or g5 families on AWS. The practical boundary is to use KEDA with Karpenter for inference, tool execution, and sandbox allocation while protecting sustained training separately.

Scale-down also depends on utilization and eviction rules. Raff’s scaling documentation states that a node is removed only after every pod on it has been evicted. If a PodDisruptionBudget blocks eviction for an extended period, scaling stops, the node is restored, and the pool status identifies the blocking pod. This makes disruption protection a deliberate configuration choice.

Reactive scaling has limits even when disruption is controlled. SPARK, by Zhijun Jiang and Amin Milani Fard of the New York Institute of Technology, reports 32% fewer timeout errors during sudden traffic surges with predictive, traffic-aware scaling than with standard reactive scaling. That result exposes the lag inherent in waiting for thresholds to be crossed. Security creates a separate constraint because autoscalers do not distinguish legitimate demand from malicious traffic. Denial-of-Wallet attacks can make a pool grow out of control, while attackers can use PodDisruptionBudgets to block eviction and keep resources running.

The Agent Sandbox primitive and what it standardizes for teams building at scale

Agent Sandbox turns the stateful-singleton and warm-allocation patterns into reusable Kubernetes objects. It is an open-source Kubernetes SIG Apps subproject at kubernetes-sigs/agent-sandbox. Previewed at KubeCon NA in November 2025, it has been generally available on GKE since 20 May 2026, with GKE serving as its first production platform. The project standardizes stable identity, pre-warmed allocation, and a transactional claim interface that teams otherwise have to assemble themselves.

The Sandbox resource defines a single stateful pod with a stable hostname, persistent storage that survives restarts, and controls for scheduled deletion, pausing, and resuming. SandboxTemplate supplies a reusable security and configuration blueprint that avoids duplicated YAML. SandboxClaim gives callers a ready sandbox from the warm pool without exposing its underlying configuration, providing an interface for frameworks such as ADK or LangChain. Together, these objects separate fleet configuration from the act of requesting an execution environment. Outside the Sandbox CRD, Agent Substrate handles another scheduling need by mapping a large set of mostly idle agents onto a small, fixed pool of physical pods that are ready to use.

Implementing the stack in practice: node pool design, KEDA configuration, and observability

Implementation starts with separating pools by their operating requirements. Using AKS as the reference, a system pool should hold kube-system components and use CriticalAddonsOnly to keep agent workloads out. A general workload pool serves stateless microservices and API gateways within defined autoscaler minimum and maximum bounds. A memory-optimized pool supports in-memory caching and data-intensive processing. GPU pools use NC-series or ND-series nodes on Azure, with nvidia.com/gpu=present:NoSchedule and explicit tolerations, while Spot pools serve interruptible preprocessing, batch jobs, and CI runners with deliberately configured PodDisruptionBudgets.

Spacelift’s Kubernetes worker implementation uses a Worker Pool Controller to manage the pool and a Prometheus Exporter to expose queue length through a PodMonitor. A KEDA ScaledObject queries that metric and drives replica count. Scale-to-zero requires a minimum replica count of zero, with a cooldown period matched to the burst pattern. An aggressive cooldown creates repeated cold starts, while a conservative one retains idle GPU capacity. Agents triggered by HTTP can instead use KEDA’s HTTP add-on or request-rate scalers.

Observability must then show where scaling is held up. DCGM Exporter or an equivalent must expose GPU utilization before scaling decisions can use accelerator saturation. The dashboard should include queue depth, pending pod count, and node utilization by pool type in addition to CPU and memory. Datadog’s Kubernetes autoscaling documentation covers integrating container-level metrics for teams using that observability layer. For GPU paths that cannot tolerate cold starts, minimum node counts should preserve readiness.

Cost discipline: Spot instances, consolidation, and the GPU idle problem

GPU idle time is the largest cost driver in agent worker pools. On-demand NVIDIA H100 pricing varied considerably across AWS, GCP, and Azure as of April 2026, but an idle allocation incurs its provider’s rate continuously. AWS raised H200 Capacity Block pricing in January 2026, reinforcing the pressure to remove unused capacity. The cost objective is therefore not simply fewer replicas, but fewer expensive nodes retained without useful work.

Spot capacity fits batch preprocessing, CI/CD runners, and development workloads where interruption is acceptable. Grover’s migration to Karpenter demonstrates a NodePool combining Spot and On-Demand instances, with automatic Spot pool selection and interruption handling. Consolidation captures further savings between scale events by repacking workloads and terminating unneeded nodes. GPU taints also protect the budget because an unrelated pod on a GPU node can block its removal. Budget alerts and maximum node count bounds remain the minimum safeguards against demand-driven billing abuse.

What to build now versus what to wait for as DRA and predictive scaling mature

The production baseline for CPU-backed agent pools is event-driven pod scaling paired with node provisioning. The available combinations include KEDA with Karpenter on AWS or Azure and KEDA with Cluster Autoscaler on GKE. Agent Sandbox with SandboxWarmPool supports sub-second sandbox allocation, while pool segmentation, Spot mixing, and PDB-aware scale-down supply the surrounding operational controls. GPU-aware decisions additionally require explicit utilization export. These components provide a usable foundation without making newer scheduling capabilities a prerequisite.

GPU-backed SandboxWarmPool, uniformly supported DRA-aware autoscaling, and broadly available predictive controllers remain areas to design for change. Fractional GPU sharing and topology-aware warm-pool placement are still developing, and predictive research has not yet become a standard production option without proprietary infrastructure dependencies. The stable boundary is the division between application demand and infrastructure capacity. Teams should track that boundary from day one, with queue depth, GPU utilization, and pending pod latency in the same observability plane. Those signals provide the basis for deciding when the reactive baseline needs DRA-aware placement or predictive scaling.