Kubernetes Outage Caused by Capacity Provisioning Failure

Incident Summary

A production Kubernetes environment experienced a service outage following a node provisioning configuration change.

The issue prevented new application workloads from being scheduled onto cluster nodes, eventually resulting in customer-facing service degradation.

While existing workloads initially continued operating normally, increasing resource pressure and failed workload scheduling led to partial service disruption.

The outage was resolved by reverting the provisioning configuration, restoring cluster capacity, and validating workload health across affected services.

Environment

Platform:

• Kubernetes
• Amazon EKS
• Karpenter
• AWS EC2
• Application Load Balancer (ALB)

Workloads:

• Customer-facing APIs
• Background processing services
• Internal operational services

Availability Model:

• Multi-AZ deployment
• Auto-scaling worker nodes
• Horizontal pod autoscaling

Detection

The incident was initially detected through a combination of automated alerts and user impact reports.

Observed symptoms included:

• Elevated API latency
• Increasing pod scheduling failures
• Growing workload queues
• Increased error rates
• Customer-facing service degradation

Primary alerts included:

• Pod Pending count exceeded threshold
• Node provisioning failures
• Increased application response times
• Reduced available cluster capacity

Impact

Customer Impact:

• Intermittent API failures
• Increased response times
• Delayed background processing

Business Impact:

• Reduced service availability
• Increased operational response effort
• Elevated support activity

Engineering Impact:

• Incident response activation
• Increased investigation workload
• Temporary suspension of planned deployments

Timeline

09:15

Infrastructure configuration update deployed.

09:22

Cluster scaling activity begins.

09:27

Pods begin entering Pending state.

09:30

First scheduling alerts triggered.

09:35

Customer-facing latency begins increasing.

09:41

Incident declared.

09:48

Investigation identifies node provisioning anomalies.

10:02

Root cause isolated to recent provisioning configuration changes.

10:08

Configuration rollback initiated.

10:17

New nodes successfully provisioned.

10:23

Pending workload count begins decreasing.

10:34

Customer-facing services stabilize.

10:45

Incident resolved.

Investigation

The investigation focused on understanding why workloads were unable to obtain additional cluster capacity.

Initial areas examined included:

• Kubernetes scheduler events
• Node health
• EC2 provisioning activity
• Karpenter controller logs
• Auto-scaling behavior
• Application health

Key indicators included:

```text Pods Pending No suitable nodes available Node provisioning retries Insufficient capacity events ```

Workloads requiring additional compute resources were unable to find schedulable nodes despite cluster demand increasing.

Existing nodes continued operating normally, making the issue initially appear application-related rather than infrastructure-related.

Further analysis showed that new worker nodes were failing to launch successfully.

Root Cause

A provisioning configuration change introduced constraints that prevented Karpenter from successfully creating new worker nodes.

The configuration unintentionally reduced the available EC2 capacity options that satisfied the NodePool and workload scheduling requirements.

When workload demand increased, Karpenter attempted to provision additional nodes but could not identify valid capacity that matched the configured constraints.

As existing nodes became saturated, workloads accumulated in a Pending state.

The outage occurred because the cluster could no longer scale to meet application demand.

Contributing Factors

Several contributing factors increased the severity of the incident.

Restrictive NodePool Configuration

The NodePool configuration contained constraints that significantly reduced the number of valid node provisioning options available to Karpenter.

NodePools define operational requirements such as:

• CPU and memory limits
• capacity type requirements
• taints and tolerations
• disruption policies
• consolidation behavior

Individually, these settings were reasonable.

Collectively, they reduced the number of node configurations capable of satisfying workload scheduling requirements.

As demand increased, Karpenter had fewer viable provisioning options available.

EC2NodeClass Constraints

The NodePool referenced an EC2NodeClass that defined infrastructure-specific requirements including:

• subnet selection
• security groups
• Amazon Machine Images (AMIs)
• IAM roles
• storage configuration
• resource tagging

While each setting was valid, the combination further constrained the available capacity options.

The interaction between NodePool requirements and EC2NodeClass requirements reduced Karpenter's ability to identify launchable nodes during scaling events.

Limited Scale Validation

The configuration passed functional validation and basic testing.

However, testing did not adequately simulate real production scale-out scenarios where multiple constraints interact simultaneously.

As a result, the reduction in provisioning flexibility was not identified before deployment.

Delayed Correlation

Initial investigation focused on application symptoms such as latency, queue growth, and scheduling failures.

The underlying issue existed within the node provisioning workflow.

This increased time-to-diagnosis during the early stages of the incident.

Understanding the Scheduling Chain

Successful node provisioning depends on multiple layers of configuration working together.

Workload Requirements

↓

NodePool Constraints

↓

EC2NodeClass Configuration

↓

AWS Capacity Availability

↓

Node Provisioned

A failure or overly restrictive requirement at any layer can prevent successful node creation.

This is one reason Kubernetes capacity management can be challenging. The root cause often exists several layers below the application symptoms first observed by operators.

Resolution

The response team focused on restoring Karpenter's ability to provision new nodes while minimizing additional customer impact.

The following actions were performed:

1. Reviewed recent NodePool and EC2NodeClass changes. 2. Identified overly restrictive provisioning requirements that reduced the number of valid node launch options. 3. Modified NodePool constraints to increase scheduling flexibility. 4. Validated EC2NodeClass configuration, including subnet, security group, and AMI selection requirements. 5. Confirmed Karpenter could successfully identify launchable capacity. 6. Monitored new node creation and cluster scaling activity. 7. Verified Pending workloads were successfully scheduled. 8. Validated application health and customer-facing services. 9. Confirmed customer impact had ended.

Once Karpenter regained sufficient provisioning flexibility, new nodes were successfully launched and workloads began draining from the Pending queue.

Cluster capacity returned to normal as scheduling pressure decreased and application services recovered.

Lessons Learned

Capacity Is a Critical Dependency

Applications cannot remain healthy when infrastructure provisioning fails.

Capacity management should be treated as a critical production service.

Successful Validation Requires Scale Testing

Configuration validation should include realistic scale-out scenarios.

Passing static validation checks does not guarantee operational success.

Observability Must Span the Entire Workflow

Application monitoring alone is insufficient.

Visibility is required across:

• applications
• Kubernetes scheduling
• node provisioning
• cloud infrastructure

Symptoms Are Not Always Causes

The first visible problem was application degradation.

The actual failure occurred within the infrastructure provisioning workflow.

Understanding the difference reduced future diagnosis time.

Constraint Complexity Requires Careful Review

Node provisioning decisions are influenced by multiple interacting layers of configuration.

NodePools, EC2NodeClasses, workload requirements, taints, tolerations, capacity types, and cloud-provider constraints can all affect whether a node can be launched successfully.

Changes that appear reasonable when reviewed individually may produce unexpected behavior when combined.

Provisioning flexibility should be considered alongside security, cost, and operational requirements.

Preventive Actions

The following improvements were implemented.

Enhanced Alerting

New alerts added for:

• node provisioning failures
• prolonged Pending workloads
• capacity exhaustion risks
• provisioning retry thresholds

Runbook Improvements

Operational runbooks updated with:

• investigation procedures
• common failure indicators
• rollback instructions
• validation steps

Change Management Updates

Infrastructure changes now require:

• scale validation
• rollback plans
• post-deployment verification

Dashboard Enhancements

Additional dashboards created for:

• cluster capacity
• provisioning activity
• scheduling health
• workload placement

Constraint Validation

Additional validation procedures were implemented for NodePool and EC2NodeClass changes.

Reviews now include:

- instance family coverage - capacity type coverage - subnet availability - taint and toleration compatibility - workload placement analysis - scale-out simulation

The goal is to identify restrictive configurations before they reach production.

Observability Improvements

Following the incident, additional monitoring was implemented around:

- Pending workloads - Node provisioning failures - NodePool utilization - NodeClaim lifecycle events - Available cluster capacity - Provisioning latency

The goal was to detect provisioning constraints before customer-facing impact occurred.

Metrics and Signals

Key metrics monitored after the incident include:

Cluster Metrics

• Node count
• Available CPU
• Available memory
• Unschedulable pods

Provisioning Metrics

• Node launch success rate
• Provisioning latency
• Provisioning failures
• Capacity acquisition time

Application Metrics

• Request latency
• Error rate
• Queue depth
• Throughput

Karpenter Metrics

- Node provisioning attempts - Node launch failures - Provisioning duration - Pending workload count - NodePool utilization - Capacity acquisition success rate

How AI Could Assist

AI is unlikely to prevent the incident directly.

However, AI systems can assist operators by improving access to operational knowledge and reducing investigation time.

Potential applications include:

Incident Summarization

Generate concise incident timelines and status updates.

Runbook Retrieval

Retrieve relevant operational procedures during active incidents.

Alert Correlation

Identify relationships between:

• provisioning failures
• scheduling failures
• application symptoms

Knowledge Discovery

Surface similar historical incidents and previous remediation steps.

Configuration Analysis

AI systems could assist by analyzing NodePool, EC2NodeClass, and workload definitions before deployment.

Potential findings might include:

- overly restrictive scheduling constraints - insufficient instance diversity - conflicting taints and tolerations - limited availability zone coverage - elevated provisioning risk

Such analysis would not replace engineering review but could help identify high-risk changes before they impact production.

Post-Incident Analysis

Assist with root cause analysis, lesson extraction, and documentation generation.

AI should support the investigation process rather than replace engineering judgment.

Key Takeaways

• Capacity provisioning failures can create widespread application impact.
• Kubernetes symptoms often originate from infrastructure causes.
• Observability must include provisioning workflows.
• Validation should include realistic scaling scenarios.
• Runbooks and alerting significantly reduce recovery time.
• AI is most effective when supporting operators with context, retrieval, and analysis rather than making autonomous operational decisions.