Contributors to this page
Last update:

Cluster Overprovisioning

This tutorial describes how to overprovisioning of cluster nodes for scaling and failover. This is desired when you have work loads that need to scale up quickly without waiting for the new cluster nodes to be created and join the cluster.

teaser

A similar problem occurs when crashing a node from the Hyperscaler. This must be replaced by Kubernetes as fast as possible. The solution can be overprovisioning of nodes

Overprovisioning: Allocating more computer resources than is strictly necessary

https://en.wikipedia.org/wiki/Overprovisioning

When does the autoscaler change the size of the cluster?

Below is a description of how the cluster behaves when there is a requirement to scale.

Scaling without overprovisioning

  1. load hits the cluster (or a node is crashed)
  2. cannot schedule application-pods due to insufficient resources, scaling fails 💀
  3. cluster-autoscaler notices and begins to provision new instance
  4. wait for instance to be provisioned, boot, join the cluster and become ready
  5. kube-scheduler will notice there is somewhere to put the application-pods and will schedule them

Scaling with Overprovisioning

  1. load hits the cluster (or a node is crashed)
  2. placeholder-pods are evicted,
  3. scaling of application-pod is immediately successful
  4. placeholder-pods cannot be scheduled due to insufficient resources
  5. wait for instance to be provisioned, boot, join the cluster and become ready
  6. kube-scheduler will notice there is somewhere to put the placeholder pods and will schedule them

You can apply the above scenario one-to-one to the case when a node of the Hyperscaler dies.

Real Scenario Test

We executed normal and overprovisioning tests on a gardener cluster on different infrastructure provider (aws, azure, gcp, alicloud). All of them tested the downtime of the application pod running in the cluster, when a node dies.

The test results for the different IaaS provider are shown below.

Results

The results provided should only show how long the downtimes can be approximately.

The downtime results could vary +- 1 min, because the minimum request interval in UpTime is 1 minute

Amazon

Normal

chart

Overprovisioning

chart

Azure

Normal

chart

Overprovisioning

chart

GCP

Normal

chart

Overprovisioning

chart

AliCloud

Normal

chart

Overprovisioning

chart

Summary of results

Normal

ProviderAWSAzureGCPAliCloud
Node deleted08:5609:3209:3909:53
Pod rescheduled09:1709:5009:5310:14
Downtime21 min18 min14 min21 min

Overprovisioning

ProviderAWSAzureGCPAliCloud
Node deleted14:2006:0006:0508:23
Pod rescheduled14:2206:0206:0608:25
Downtime2 min2 min1 min2 min

Test Description

We deployed a nginx web server and a service of type LoadBalancer to expose it. So we are able to call our endpoint with external tools like UpTime to check the availability of our nginx. It takes only a few seconds to deploy a nginx web server on kubernetes, so we could say: when your endpoint works, your node is up and running.

We wanted to test how much time it takes, when your node gets killed and your cluster has to create a new one to run your application on it.

kubectl get nodes

# select the node where your nginx is running on
kubectl delete node <NGINX-HOSTED-NODE>

The downtime is tested with UpTime, which does every minute a request to our endpoint. Further we checked manually, if the node startup time and the timestamps on UpTime are almost similar.

Next, deploy the overprovisioned version of our demo application and kill the node with the NGINX. As you can see - the pod comes up very fast and can serve content again.