Other Components

1 - Dependency Watchdog

Dependency Watchdog

Overview

A watchdog which actively looks out for disruption and recovery of critical services. If there is a disruption then it will prevent cascading failure by conservatively scaling down dependent configured resources and if a critical service has just recovered then it will expedite the recovery of dependent services/pods.

Avoiding cascading failure is handled by Prober component and expediting recovery of dependent services/pods is handled by Weeder component. These are separately deployed as individual pods.

Current Limitation & Future Scope

Although in the current offering the Prober is tailored to handle one such use case of kube-apiserver connectivity, but the usage of prober can be extended to solve similar needs for other scenarios where the components involved might be different.

Start using or developing the Dependency Watchdog

See our documentation in the /docs repository, please find the index here.

Feedback and Support

We always look forward to active community engagement.

Please report bugs or suggestions on how we can enhance dependency-watchdog to address additional recovery scenarios on GitHub issues

1.1 - Concepts

1.1.1 - Prober

Prober

Overview

Prober starts asynchronous and periodic probes for every shoot cluster. The first probe is the api-server probe which checks the reachability of the API Server from the control plane. The second probe is the lease probe which is done after the api server probe is successful and checks if the number of expired node leases is below a certain threshold. If the lease probe fails, it will scale down the dependent kubernetes resources. Once the connectivity to kube-apiserver is reestablished and the number of expired node leases are within the accepted threshold, the prober will then proactively scale up the dependent kubernetes resources it had scaled down earlier. The failure threshold fraction for lease probe and dependent kubernetes resources are defined in configuration that is passed to the prober.

Origin

In a shoot cluster (a.k.a data plane) each node runs a kubelet which periodically renewes its lease. Leases serve as heartbeats informing Kube Controller Manager that the node is alive. The connectivity between the kubelet and the Kube ApiServer can break for different reasons and not recover in time.

As an example, consider a large shoot cluster with several hundred nodes. There is an issue with a NAT gateway on the shoot cluster which prevents the Kubelet from any node in the shoot cluster to reach its control plane Kube ApiServer. As a consequence, Kube Controller Manager transitioned the nodes of this shoot cluster to Unknown state.

Machine Controller Manager which also runs in the shoot control plane reacts to any changes to the Node status and then takes action to recover backing VMs/machine(s). It waits for a grace period and then it will begin to replace the unhealthy machine(s) with new ones.

This replacement of healthy machines due to a broken connectivity between the worker nodes and the control plane Kube ApiServer results in undesired downtimes for customer workloads that were running on these otherwise healthy nodes. It is therefore required that there be an actor which detects the connectivity loss between the the kubelet and shoot cluster’s Kube ApiServer and proactively scales down components in the shoot control namespace which could exacerbate the availability of nodes in the shoot cluster.

Dependency Watchdog Prober in Gardener

Prober is a central component which is deployed in the garden namespace in the seed cluster. Control plane components for a shoot are deployed in a dedicated shoot namespace for the shoot within the seed cluster.

NOTE: If you are not familiar with what gardener components like seed, shoot then please see the appendix for links.

Prober periodically probes Kube ApiServer via two separate probes:

1. API Server Probe: Local cluster DNS name which resolves to the ClusterIP of the Kube Apiserver
2. Lease Probe: Checks for number of expired leases to be within the specified threshold. The threshold defines the limit after which DWD can say that the kubelets are not able to reach the API server.

Behind the scene

For all active shoot clusters (which have not been hibernated or deleted or moved to another seed via control-plane-migration), prober will schedule a probe to run periodically. During each run of a probe it will do the following:

1. Checks if the Kube ApiServer is reachable via local cluster DNS. This should always succeed and will fail only when the Kube ApiServer has gone down. If the Kube ApiServer is down then there can be no further damage to the existing shoot cluster (barring new requests to the Kube Api Server).
2. Only if the probe is able to reach the Kube ApiServer via local cluster DNS, will it attempt to check the number of expired node leases in the shoot. The node lease renewal is done by the Kubelet, and so we can say that the lease probe is checking if the kubelet is able to reach the API server. If the number of expired node leases reaches the threshold, then the probe fails.
3. If and when a lease probe fails, then it will initiate a scale-down operation for dependent resources as defined in the prober configuration.
4. In subsequent runs it will keep performing the lease probe. If it is successful, then it will start the scale-up operation for dependent resources as defined in the configuration.

Prober lifecycle

A reconciler is registered to listen to all events for Cluster resource.

When a Reconciler receives a request for a Cluster change, it will query the extension kube-api server to get the Cluster resource.

In the following cases it will either remove an existing probe for this cluster or skip creating a new probe:

1. Cluster is marked for deletion.
2. Hibernation has been enabled for the cluster.
3. There is an ongoing seed migration for this cluster.
4. If a new cluster is created with no workers.
5. If an update is made to the cluster by removing all workers (in other words making it worker-less).

If none of the above conditions are true and there is no existing probe for this cluster then a new probe will be created, registered and started.

Probe failure identification

DWD probe can either be a success or it could return an error. If the API server probe fails, the lease probe is not done and the probes will be retried. If the error is a TooManyRequests error due to requests to the Kube-API-Server being throttled, then the probes are retried after a backOff of backOffDurationForThrottledRequests.

If the lease probe fails, then the error could be due to failure in listing the leases. In this case, no scaling operations are performed. If the error in listing the leases is a TooManyRequests error due to requests to the Kube-API-Server being throttled, then the probes are retried after a backOff of backOffDurationForThrottledRequests.

If there is no error in listing the leases, then the Lease probe fails if the number of expired leases reaches the threshold fraction specified in the configuration. A lease is considered expired in the following scenario:-

	time.Now() >= lease.Spec.RenewTime + (p.config.KCMNodeMonitorGraceDuration.Duration * expiryBufferFraction)

Here, lease.Spec.RenewTime is the time when current holder of a lease has last updated the lease. config is the probe config generated from the configuration and KCMNodeMonitorGraceDuration is amount of time which KCM allows a running Node to be unresponsive before marking it unhealthy (See ref) . expiryBufferFraction is a hard coded value of 0.75. Using this fraction allows the prober to intervene before KCM marks a node as unknown, but at the same time allowing kubelet sufficient retries to renew the node lease (Kubelet renews the lease every 10s See ref).

Appendix

• Gardener
• Reverse Cluster VPN

1.1.2 - Weeder

Weeder

Overview

Weeder watches for an update to service endpoints and on receiving such an event it will create a time-bound watch for all configured dependent pods that need to be actively recovered in case they have not yet recovered from CrashLoopBackoff state. In a nutshell it accelerates recovery of pods when an upstream service recovers.

An interference in automatic recovery for dependent pods is required because kubernetes pod restarts a container with an exponential backoff when the pod is in CrashLoopBackOff state. This backoff could become quite large if the service stays down for long. Presence of weeder would not let that happen as it’ll restart the pod.

Prerequisites

Before we understand how Weeder works, we need to be familiar with kubernetes services & endpoints.

NOTE: If a kubernetes service is created with selectors then kubernetes will create corresponding endpoint resource which will have the same name as that of the service. In weeder implementation service and endpoint name is used interchangeably.

Config

Weeder can be configured via command line arguments and a weeder configuration. See configure weeder.

Internals

Weeder keeps a watch on the events for the specified endpoints in the config. For every endpoints a list of podSelectors can be specified. It cretes a weeder object per endpoints resource when it receives a satisfactory Create or Update event. Then for every podSelector it creates a goroutine. This goroutine keeps a watch on the pods with labels as per the podSelector and kills any pod which turn into CrashLoopBackOff. Each weeder lives for watchDuration interval which has a default value of 5 mins if not explicitly set.

To understand the actions taken by the weeder lets use the following diagram as a reference. Let us also assume the following configuration for the weeder:

watchDuration: 2m0s
servicesAndDependantSelectors:
  etcd-main-client: # name of the service/endpoint for etcd statefulset that weeder will receive events for.
    podSelectors: # all pods matching the label selector are direct dependencies for etcd service
      - matchExpressions:
          - key: gardener.cloud/role
            operator: In
            values:
              - controlplane
          - key: role
            operator: In
            values:
              - apiserver
  kube-apiserver: # name of the service/endpoint for kube-api-server pods that weeder will receive events for. 
    podSelectors: # all pods matching the label selector are direct dependencies for kube-api-server service
      - matchExpressions:
          - key: gardener.cloud/role
            operator: In
            values:
              - controlplane
          - key: role
            operator: NotIn
            values:
              - main
              - apiserver

Only for the sake of demonstration lets pick the first service -> dependent pods tuple (etcd-main-client as the service endpoint).

1. Assume that there are 3 replicas for etcd statefulset.
2. Time here is just for showing the series of events

• t=0 -> all etcd pods go down
• t=10 -> kube-api-server pods transition to CrashLoopBackOff
• t=100 -> all etcd pods recover together
• t=101 -> Weeder sees Update event for etcd-main-client endpoints resource
• t=102 -> go routine created to keep watch on kube-api-server pods
• t=103 -> Since kube-api-server pods are still in CrashLoopBackOff, weeder deletes the pods to accelerate the recovery.
• t=104 -> new kube-api-server pod created by replica-set controller in kube-controller-manager

Points to Note

• Weeder only respond on Update events where a notReady endpoints resource turn to Ready. Thats why there was no weeder action at time t=10 in the example above.
  • notReady -> no backing pod is Ready
  • Ready -> atleast one backing pod is Ready
• Weeder doesn’t respond on Delete events
• Weeder will always wait for the entire watchDuration. If the dependent pods transition to CrashLoopBackOff after the watch duration or even after repeated deletion of these pods they do not recover then weeder will exit. Quality of service offered via a weeder is only Best-Effort.

1.2 - Deployment

1.2.1 - Configure

Configure Dependency Watchdog Components

Prober

Dependency watchdog prober command takes command-line-flags which are meant to fine-tune the prober. In addition a ConfigMap is also mounted to the container which provides tuning knobs for the all probes that the prober starts.

Command line arguments

Prober can be configured via the following flags:

You can view an example kubernetes prober deployment YAML to see how these command line args are configured.

Prober Configuration

A probe configuration is mounted as ConfigMap to the container. The path to the config file is configured via config-file command line argument as mentioned above. Prober will start one probe per Shoot control plane hosted within the Seed cluster. Each such probe will run asynchronously and will periodically connect to the Kube ApiServer of the Shoot. Configuration below will influence each such probe.

You can view an example YAML configuration provided as data in a ConfigMap here.

DependentResourceInfo

If a lease probe fails, then it scales down the dependent resources defined by this property. Similarly, if the lease probe is now successful, then it scales up the dependent resources defined by this property.

Each dependent resource info has the following properties:

NOTE: Since each dependent resource is a target for scale up/down, therefore it is mandatory that the resource reference points a kubernetes resource which has a scale subresource.

ScaleInfo

How to scale a DependentResourceInfo is captured in ScaleInfo. It has the following properties:

Determining target replicas

Prober cannot assume any target replicas during a scale-up operation for the following reasons:

1. Kubernetes resources could be set to provide highly availability and the number of replicas could wary from one shoot control plane to the other. In gardener the number of replicas of pods in shoot namespace are controlled by the shoot control plane configuration.
2. If Horizontal Pod Autoscaler has been configured for a kubernetes dependent resource then it could potentially change the spec.replicas for a deployment/statefulset.

Given the above constraint lets look at how prober determines the target replicas during scale-down or scale-up operations.

1. Scale-Up: Primary responsibility of a probe while performing a scale-up is to restore the replicas of a kubernetes dependent resource prior to scale-down. In order to do that it updates the following for each dependent resource that requires a scale-up:

   1. spec.replicas: Checks if dependency-watchdog.gardener.cloud/replicas is set. If it is, then it will take the value stored against this key as the target replicas. To be a valid value it should always be greater than 0.
   2. If dependency-watchdog.gardener.cloud/replicas annotation is not present then it falls back to the hard coded default value for scale-up which is set to 1.
   3. Removes the annotation dependency-watchdog.gardener.cloud/replicas if it exists.

2. Scale-Down: To scale down a dependent kubernetes resource it does the following:

   1. Adds an annotation dependency-watchdog.gardener.cloud/replicas and sets its value to the current value of spec.replicas.
   2. Updates spec.replicas to 0.

Level

Each dependent resource that should be scaled up or down is associated to a level. Levels are ordered and processed in ascending order (starting with 0 assigning it the highest priority). Consider the following configuration:

dependentResourceInfos:
  - ref: 
      kind: "Deployment"
      name: "kube-controller-manager"
      apiVersion: "apps/v1"
    scaleUp: 
      level: 1 
    scaleDown: 
      level: 0 
  - ref:
      kind: "Deployment"
      name: "machine-controller-manager"
      apiVersion: "apps/v1"
    scaleUp:
      level: 1
    scaleDown:
      level: 1
  - ref:
      kind: "Deployment"
      name: "cluster-autoscaler"
      apiVersion: "apps/v1"
    scaleUp:
      level: 0
    scaleDown:
      level: 2

Let us order the dependent resources by their respective levels for both scale-up and scale-down. We get the following order:

Scale Up Operation

Order of scale up will be:

1. cluster-autoscaler
2. kube-controller-manager and machine-controller-manager will be scaled up concurrently after cluster-autoscaler has been scaled up.

Scale Down Operation

Order of scale down will be:

1. kube-controller-manager
2. machine-controller-manager after (1) has been scaled down.
3. cluster-autoscaler after (2) has been scaled down.

Disable/Ignore Scaling

A probe can be configured to ignore scaling of configured dependent kubernetes resources. To do that one must set dependency-watchdog.gardener.cloud/ignore-scaling annotation to true on the scalable resource for which scaling should be ignored.

Weeder

Dependency watchdog weeder command also (just like the prober command) takes command-line-flags which are meant to fine-tune the weeder. In addition a ConfigMap is also mounted to the container which helps in defining the dependency of pods on endpoints.

Command Line Arguments

Weeder can be configured with the same flags as that for prober described under command-line-arguments section You can find an example weeder deployment YAML to see how these command line args are configured.

Weeder Configuration

Weeder configuration is mounted as ConfigMap to the container. The path to the config file is configured via config-file command line argument as mentioned above. Weeder will start one go routine per podSelector per endpoint on an endpoint event as described in weeder internal concepts.

You can view the example YAML configuration provided as data in a ConfigMap here.

DependantSelectors

If the service recovers from downtime, then weeder starts to watch for CrashLoopBackOff pods. These pods are identified by info stored in this property.

1.2.2 - Monitor

Monitoring

Work In Progress

We will be introducing metrics for Dependency-Watchdog-Prober and Dependency-Watchdog-Weeder. These metrics will be pushed to prometheus. Once that is completed we will provide details on all the metrics that will be supported here.

1.3 - Contribution

How to contribute?

Contributions are always welcome!

In order to contribute ensure that you have the development environment setup and you familiarize yourself with required steps to build, verify-quality and test.

Setting up development environment

Installing Go

Minimum Golang version required: 1.18. On MacOS run:

brew install go

For other OS, follow the installation instructions.

Installing Git

Git is used as version control for dependency-watchdog. On MacOS run:

brew install git

If you do not have git installed already then please follow the installation instructions.

Installing Docker

In order to test dependency-watchdog containers you will need a local kubernetes setup. Easiest way is to first install Docker. This becomes a pre-requisite to setting up either a vanilla KIND/minikube cluster or a local Gardener cluster.

On MacOS run:

brew install -cash docker

For other OS, follow the installation instructions.

Installing Kubectl

To interact with the local Kubernetes cluster you will need kubectl. On MacOS run:

brew install kubernetes-cli

For other IS, follow the installation instructions.

Get the sources

Clone the repository from Github:

git clone https://github.com/gardener/dependency-watchdog.git

Using Makefile

For every change following make targets are recommended to run.

# build the code changes
> make build
# ensure that all required checks pass
> make verify # this will check formatting, linting and will run unit tests
# if you do not wish to run tests then you can use the following make target.
> make check

All tests should be run and the test coverage should ideally not reduce. Please ensure that you have read testing guidelines.

Before raising a pull request ensure that if you are introducing any new file then you must add licesence header to all new files. To add license header you can run this make target:

> make add-license-headers
# This will add license headers to any file which does not already have it.

NOTE: Also have a look at the Makefile as it has other targets that are not mentioned here.

Raising a Pull Request

To raise a pull request do the following:

1. Create a fork of dependency-watchdog
2. Add dependency-watchdog as upstream remote via

   git remote add upstream https://github.com/gardener/dependency-watchdog

3. It is recommended that you create a git branch and push all your changes for the pull-request.
4. Ensure that while you work on your pull-request, you continue to rebase the changes from upstream to your branch. To do that execute the following command:

   git pull --rebase upstream master

5. We prefer clean commits. If you have multiple commits in the pull-request, then squash the commits to a single commit. You can do this via interactive git rebase command. For example if your PR branch is ahead of remote origin HEAD by 5 commits then you can execute the following command and pick the first commit and squash the remaining commits.

   git rebase -i HEAD~5 #actual number from the head will depend upon how many commits your branch is ahead of remote origin master

1.4 - Dependency Watchdog

Documentation Index

Concepts

• Prober
• Weeder

Development

• Contributions
• Testing
• Setup Dependency Watchdog using local Garden cluster

Deployment

• Configure dependency watchdog

1.5 - Dwd Using Local Garden

Dependency Watchdog with Local Garden Cluster

Setting up Local Garden cluster

A convenient way to test local dependency-watchdog changes is to use a local garden cluster. To setup a local garden cluster you can follow the setup-guide.

Dependency Watchdog resources

As part of the local garden installation, a local seed will be available.

Dependency Watchdog resources created in the seed

Namespaced resources

In the garden namespace of the seed cluster, following resources will be created:

Cluster resources

Dependency Watchdog resources created in Shoot control namespace

Dependency Watchdog resources created in the kube-node-lease namespace of the shoot

These will be created by the GRM and will have a managed resource named shoot-core-dependency-watchdog in the shoot namespace in the seed.

Update Gardener with custom Dependency Watchdog Docker images

Build, Tag and Push docker images

To build dependency watchdog docker images run the following make target:

> make docker-build

Local gardener hosts a docker registry which can be access at localhost:5001. To enable local gardener to be able to access the custom docker images you need to tag and push these images to the embedded docker registry. To do that execute the following commands:

> docker images
# Get the IMAGE ID of the dependency watchdog images that were built using docker-build make target.
> docker tag <IMAGE-ID> localhost:5001/europe-docker.pkg.dev/gardener-project/public/gardener/dependency-watchdog-prober:<TAGNAME>
> docker push localhost:5001/europe-docker.pkg.dev/gardener-project/public/gardener/dependency-watchdog-prober:<TAGNAME>

Update ManagedResource

Garden resource manager will revert back any changes that are done to the kubernetes deployment for dependency watchdog. This is quite useful in live landscapes where only tested and qualified images are used for all gardener managed components. Any change therefore is automatically reverted.

However, during development and testing you will need to use custom docker images. To prevent garden resource manager from reverting the changes done to the kubernetes deployment for dependency watchdog components you must update the respective managed resources first.

# List the managed resources
> kubectl get mr -n garden | grep dependency
# Sample response
dependency-watchdog-weeder            seed    True      True      False         26h
dependency-watchdog-prober            seed    True      True      False         26h
# Lets assume that you are currently testing prober and would like to use a custom docker image
> kubectl edit mr dependency-watchdog-prober -n garden
# This will open the resource YAML for editing. Add the annotation resources.gardener.cloud/ignore=true
# Reference: https://github.com/gardener/gardener/blob/master/docs/concepts/resource-manager.md
# Save the YAML file.

When you are done with your testing then you can again edit the ManagedResource and remove the annotation. Garden resource manager will revert back to the image with which gardener was initially built and started.

Update Kubernetes Deployment

Find and update the kubernetes deployment for dependency watchdog.

> kubectl get deploy -n garden | grep dependency
# Sample response
dependency-watchdog-weeder            1/1     1            1           26h
dependency-watchdog-prober            1/1     1            1           26h

# Lets assume that you are currently testing prober and would like to use a custom docker image
> kubectl edit deploy dependency-watchdog-prober -n garden
# This will open the resource YAML for editing. Change the image or any other changes and save.

1.6 - Testing

Testing Strategy and Developer Guideline

Intent of this document is to introduce you (the developer) to the following:

• Category of tests that exists.
• Libraries that are used to write tests.
• Best practices to write tests that are correct, stable, fast and maintainable.
• How to run each category of tests.

For any new contributions tests are a strict requirement. Boy Scouts Rule is followed: If you touch a code for which either no tests exist or coverage is insufficient then it is expected that you will add relevant tests.

Tools Used for Writing Tests

These are the following tools that were used to write all the tests (unit + envtest + vanilla kind cluster tests), it is preferred not to introduce any additional tools / test frameworks for writing tests:

Gomega

We use gomega as our matcher or assertion library. Refer to Gomega’s official documentation for details regarding its installation and application in tests.

Testing Package from Standard Library

We use the Testing package provided by the standard library in golang for writing all our tests. Refer to its official documentation to learn how to write tests using Testing package. You can also refer to this example.

Writing Tests

Common for All Kinds

• For naming the individual tests (TestXxx and testXxx methods) and helper methods, make sure that the name describes the implementation of the method. For eg: testScalingWhenMandatoryResourceNotFound tests the behaviour of the scaler when a mandatory resource (KCM deployment) is not present.
• Maintain proper logging in tests. Use t.log() method to add appropriate messages wherever necessary to describe the flow of the test. See this for examples.
• Make use of the testdata directory for storing arbitrary sample data needed by tests (YAML manifests, etc.). See this package for examples.
  • From https://pkg.go.dev/cmd/go/internal/test:

    The go tool will ignore a directory named “testdata”, making it available to hold ancillary data needed by the tests.

Table-driven tests

We need a tabular structure in two cases:

• When we have multiple tests which require the same kind of setup:- In this case we have a TestXxxSuite method which will do the setup and run all the tests. We have a slice of test struct which holds all the tests (typically a title and run method). We use a for loop to run all the tests one by one. See this for examples.
• When we have the same code path and multiple possible values to check:- In this case we have the arguments and expectations in a struct. We iterate through the slice of all such structs, passing the arguments to appropriate methods and checking if the expectation is met. See this for examples.

Env Tests

Env tests in Dependency Watchdog use the sigs.k8s.io/controller-runtime/pkg/envtest package. It sets up a temporary control plane (etcd + kube-apiserver) and runs the test against it. The code to set up and teardown the environment can be checked out here.

These are the points to be followed while writing tests that use envtest setup:

• All tests should be divided into two top level partitions:

  1. tests with common environment (testXxxCommonEnvTests)
  2. tests which need a dedicated environment for each one. (testXxxDedicatedEnvTests)

  They should be contained within the TestXxxSuite method. See this for examples. If all tests are of one kind then this is not needed.

• Create a method named setUpXxxTest for performing setup tasks before all/each test. It should either return a method or have a separate method to perform teardown tasks. See this for examples.

• The tests run by the suite can be table-driven as well.

• Use the envtest setup when there is a need of an environment close to an actual setup. Eg: start controllers against a real Kubernetes control plane to catch bugs that can only happen when talking to a real API server.

NOTE: It is currently not possible to bring up more than one envtest environments. See issue#1363. We enforce running serial execution of test suites each of which uses a different envtest environments. See hack/test.sh.

Vanilla Kind Cluster Tests

There are some tests where we need a vanilla kind cluster setup, for eg:- The scaler.go code in the prober package uses the scale subresource to scale the deployments mentioned in the prober config. But the envtest setup does not support the scale subresource as of now. So we need this setup to test if the deployments are scaled as per the config or not. You can check out the code for this setup here. You can add utility methods for different kubernetes and custom resources in there.

These are the points to be followed while writing tests that use Vanilla Kind Cluster setup:

• Use this setup only if there is a need of an actual Kubernetes cluster(api server + control plane + etcd) to write the tests. (Because this is slower than your normal envTest setup)
• Create setUpXxxTest similar to the one in envTest. Follow the same structural pattern used in envTest for writing these tests. See this for examples.

Run Tests

To run unit tests, use the following Makefile target

make test

To run KIND cluster based tests, use the following Makefile target

make kind-tests # these tests will be slower as it brings up a vanilla KIND cluster

To view coverage after running the tests, run :

go tool cover -html=cover.out

Flaky tests

If you see that a test is flaky then you can use make stress target which internally uses stress tool

make stress test-package=<test-package> test-func=<test-func> tool-params="<tool-params>"

An example invocation:

make stress test-package=./internal/util test-func=TestRetryUntilPredicateWithBackgroundContext tool-params="-p 10"

The make target will do the following:

1. It will create a test binary for the package specified via test-package at /tmp/pkg-stress.test directory.
2. It will run stress tool passing the tool-params and targets the function test-func.

2 - Machine Controller Manager

machine-controller-manager

Note One can add support for a new cloud provider by following Adding support for new provider.

Overview

Machine Controller Manager aka MCM is a group of cooperative controllers that manage the lifecycle of the worker machines. It is inspired by the design of Kube Controller Manager in which various sub controllers manage their respective Kubernetes Clients. MCM gives you the following benefits:

• seamlessly manage machines/nodes with a declarative API (of course, across different cloud providers)
• integrate generically with the cluster autoscaler
• plugin with tools such as the node-problem-detector
• transport the immutability design principle to machine/nodes
• implement e.g. rolling upgrades of machines/nodes

MCM supports following providers. These provider code is maintained externally (out-of-tree), and the links for the same are linked below:

• Alicloud
• AWS
• Azure
• Equinix Metal
• GCP
• KubeVirt
• Metal Stack
• Openstack
• V Sphere
• Yandex

It can easily be extended to support other cloud providers as well.

Example of managing machine:

kubectl create/get/delete machine vm1

Key terminologies

Nodes/Machines/VMs are different terminologies used to represent similar things. We use these terms in the following way

1. VM: A virtual machine running on any cloud provider. It could also refer to a physical machine (PM) in case of a bare metal setup.
2. Node: Native kubernetes node objects. The objects you get to see when you do a “kubectl get nodes”. Although nodes can be either physical/virtual machines, for the purposes of our discussions it refers to a VM.
3. Machine: A VM that is provisioned/managed by the Machine Controller Manager.

Design of Machine Controller Manager

The design of the Machine Controller Manager is influenced by the Kube Controller Manager, where-in multiple sub-controllers are used to manage the Kubernetes clients.

Design Principles

It’s designed to run in the master plane of a Kubernetes cluster. It follows the best principles and practices of writing controllers, including, but not limited to:

• Reusing code from kube-controller-manager
• leader election to allow HA deployments of the controller
• workqueues and multiple thread-workers
• SharedInformers that limit to minimum network calls, de-serialization and provide helpful create/update/delete events for resources
• rate-limiting to allow back-off in case of network outages and general instability of other cluster components
• sending events to respected resources for easy debugging and overview
• Prometheus metrics, health and (optional) profiling endpoints

Objects of Machine Controller Manager

Machine Controller Manager reconciles a set of Custom Resources namely MachineDeployment, MachineSet and Machines which are managed & monitored by their controllers MachineDeployment Controller, MachineSet Controller, Machine Controller respectively along with another cooperative controller called the Safety Controller.

Machine Controller Manager makes use of 4 CRD objects and 1 Kubernetes secret object to manage machines. They are as follows:

See here for CRD API Documentation

Components of Machine Controller Manager

Along with the above Custom Controllers and Resources, MCM requires the MachineClass to use K8s Secret that stores cloudconfig (initialization scripts used to create VMs) and cloud specific credentials. All these controllers work in an co-operative manner. They form a parent-child relationship with MachineDeployment Controller being the grandparent, MachineSet Controller being the parent, and Machine Controller being the child.

Development

To start using or developing the Machine Controller Manager, see the documentation in the /docs repository.

FAQ

An FAQ is available here.

cluster-api Implementation

• cluster-api branch of machine-controller-manager implements the machine-api aspect of the cluster-api project.
• Link: https://github.com/gardener/machine-controller-manager/tree/cluster-api
• Once cluster-api project gets stable, we may make master branch of MCM as well cluster-api compliant, with well-defined migration notes.

2.1 - Documents

2.1.1 - Apis

Specification

ProviderSpec Schema

Machine

Machine is the representation of a physical or virtual machine.

MachineClass

MachineClass can be used to templatize and re-use provider configuration across multiple Machines / MachineSets / MachineDeployments.

MachineDeployment

MachineDeployment enables declarative updates for machines and MachineSets.

MachineSet

MachineSet TODO

ClassSpec

(Appears on: MachineSetSpec, MachineSpec)

ClassSpec is the class specification of machine

ConditionStatus (string alias)

(Appears on: MachineDeploymentCondition, MachineSetCondition)

CurrentStatus

(Appears on: MachineStatus)

CurrentStatus contains information about the current status of Machine.

LastOperation

(Appears on: MachineSetStatus, MachineStatus, MachineSummary)

LastOperation suggests the last operation performed on the object

MachineConfiguration

(Appears on: MachineSpec)

MachineConfiguration describes the configurations useful for the machine-controller.

MachineDeploymentCondition

(Appears on: MachineDeploymentStatus)

MachineDeploymentCondition describes the state of a MachineDeployment at a certain point.

MachineDeploymentConditionType (string alias)

(Appears on: MachineDeploymentCondition)

MachineDeploymentSpec

(Appears on: MachineDeployment)

MachineDeploymentSpec is the specification of the desired behavior of the MachineDeployment.

MachineDeploymentStatus

(Appears on: MachineDeployment)

MachineDeploymentStatus is the most recently observed status of the MachineDeployment.

MachineDeploymentStrategy

(Appears on: MachineDeploymentSpec)

MachineDeploymentStrategy describes how to replace existing machines with new ones.

MachineDeploymentStrategyType (string alias)

(Appears on: MachineDeploymentStrategy)

MachineOperationType (string alias)

(Appears on: LastOperation)

MachineOperationType is a label for the operation performed on a machine object.

MachinePhase (string alias)

(Appears on: CurrentStatus)

MachinePhase is a label for the condition of a machine at the current time.

MachineSetCondition

(Appears on: MachineSetStatus)

MachineSetCondition describes the state of a machine set at a certain point.

MachineSetConditionType (string alias)

(Appears on: MachineSetCondition)

MachineSetConditionType is the condition on machineset object

MachineSetSpec

(Appears on: MachineSet)

MachineSetSpec is the specification of a MachineSet.

MachineSetStatus

(Appears on: MachineSet)

MachineSetStatus holds the most recently observed status of MachineSet.

MachineSpec

(Appears on: Machine, MachineTemplateSpec)

MachineSpec is the specification of a Machine.

MachineState (string alias)

(Appears on: LastOperation)

MachineState is a current state of the operation.

MachineStatus

(Appears on: Machine)

MachineStatus holds the most recently observed status of Machine.

MachineSummary

MachineSummary store the summary of machine.

MachineTemplateSpec

(Appears on: MachineDeploymentSpec, MachineSetSpec)

MachineTemplateSpec describes the data a machine should have when created from a template

NodeTemplate

(Appears on: MachineClass)

NodeTemplate contains subfields to track all node resources and other node info required to scale nodegroup from zero

NodeTemplateSpec

(Appears on: MachineSpec)

NodeTemplateSpec describes the data a node should have when created from a template

RollbackConfig

(Appears on: MachineDeploymentSpec)

RollingUpdateMachineDeployment

(Appears on: MachineDeploymentStrategy)

Spec to control the desired behavior of rolling update.

Generated with gen-crd-api-reference-docs

2.2 - Proposals

2.2.1 - Excess Reserve Capacity

Excess Reserve Capacity

• Excess Reserve Capacity
  • Goal
  • Note
  • Possible Approaches
    • Approach 1: Enhance Machine-controller-manager to also entertain the excess machines
    • Approach 2: Enhance Cluster-autoscaler by simulating fake pods in it
    • Approach 3: Enhance cluster-autoscaler to support pluggable scaling-events
    • Approach 4: Make intelligent use of Low-priority pods

Goal

Currently, autoscaler optimizes the number of machines for a given application-workload. Along with effective resource utilization, this feature brings concern where, many times, when new application instances are created - they don’t find space in existing cluster. This leads the cluster-autoscaler to create new machines via MachineDeployment, which can take from 3-4 minutes to ~10 minutes, for the machine to really come-up and join the cluster. In turn, application-instances have to wait till new machines join the cluster.

One of the promising solutions to this issue is Excess Reserve Capacity. Idea is to keep a certain number of machines or percent of resources[cpu/memory] always available, so that new workload, in general, can be scheduled immediately unless huge spike in the workload. Also, the user should be given enough flexibility to choose how many resources or how many machines should be kept alive and non-utilized as this affects the Cost directly.

Note

• We decided to go with Approach-4 which is based on low priority pods. Please find more details here: https://github.com/gardener/gardener/issues/254
• Approach-3 looks more promising in long term, we may decide to adopt that in future based on developments/contributions in autoscaler-community.

Possible Approaches

Following are the possible approaches, we could think of so far.

Approach 1: Enhance Machine-controller-manager to also entertain the excess machines

• Machine-controller-manager currently takes care of the machines in the shoot cluster starting from creation-deletion-health check to efficient rolling-update of the machines. From the architecture point of view, MachineSet makes sure that X number of machines are always running and healthy. MachineDeployment controller smartly uses this facility to perform rolling-updates.

• We can expand the scope of MachineDeployment controller to maintain excess number of machines by introducing new parallel independent controller named MachineTaint controller. This will result in MCM to include Machine, MachineSet, MachineDeployment, MachineSafety, MachineTaint controllers. MachineTaint controller does not need to introduce any new CRD - analogy fits where taint-controller also resides into kube-controller-manager.

• Only Job of MachineTaint controller will be:

  • List all the Machines under each MachineDeployment.
  • Maintain taints of noSchedule and noExecute on X latest MachineObjects.
  • There should be an event-based informer mechanism where MachineTaintController gets to know about any Update/Delete/Create event of MachineObjects - in turn, maintains the noSchedule and noExecute taints on all the latest machines. - Why latest machines? - Whenever autoscaler decides to add new machines - essentially ScaleUp event - taints from the older machines are removed and newer machines get the taints. This way X number of Machines immediately becomes free for new pods to be scheduled. - While ScaleDown event, autoscaler specifically mentions which machines should be deleted, and that should not bring any concerns. Though we will have to put proper label/annotation defined by autoscaler on taintedMachines, so that autoscaler does not consider the taintedMachines for deletion while scale-down. * Annotation on tainted node: "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true"

• Implementation Details:

  • Expect new optional field ExcessReplicas in MachineDeployment.Spec. MachineDeployment controller now adds both Spec.Replicas and Spec.ExcessReplicas[if provided], and considers that as a standard desiredReplicas. - Current working of MCM will not be affected if ExcessReplicas field is kept nil.
  • MachineController currently reads the NodeObject and sets the MachineConditions in MachineObject. Machine-controller will now also read the taints/labels from the MachineObject - and maintains it on the NodeObject.

• We expect cluster-autoscaler to intelligently make use of the provided feature from MCM.

  • CA gets the input of min:max:excess from Gardener. CA continues to set the MachineDeployment.Spec.Replicas as usual based on the application-workload.
  • In addition, CA also sets the MachieDeployment.Spec.ExcessReplicas .
  • Corner-case: * CA should decrement the excessReplicas field accordingly when desiredReplicas+excessReplicas on MachineDeployment goes beyond max.

Approach 2: Enhance Cluster-autoscaler by simulating fake pods in it

• There was already an attempt by community to support this feature.
  • Refer for details to: https://github.com/kubernetes/autoscaler/pull/77/files

Approach 3: Enhance cluster-autoscaler to support pluggable scaling-events

• Forked version of cluster-autoscaler could be improved to plug-in the algorithm for excess-reserve capacity.
• Needs further discussion around upstream support.
• Create golang channel to separate the algorithms to trigger scaling (hard-coded in cluster-autoscaler, currently) from the algorithms about how to to achieve the scaling (already pluggable in cluster-autoscaler). This kind of separation can help us introduce/plug-in new algorithms (such as based node resource utilisation) without affecting existing code-base too much while almost completely re-using the code-base for the actual scaling.
• Also this approach is not specific to our fork of cluster-autoscaler. It can be made upstream eventually as well.

Approach 4: Make intelligent use of Low-priority pods

• Refer to: pod-priority-preemption
• TL; DR:
  • High priority pods can preempt the low-priority pods which are already scheduled.
  • Pre-create bunch[equivivalent of X shoot-control-planes] of low-priority pods with priority of zero, then start creating the workload pods with better priority which will reschedule the low-priority pods or otherwise keep them in pending state if the limit for max-machines has reached.
  • This is still alpha feature.

2.2.2 - GRPC Based Implementation of Cloud Providers

GRPC based implementation of Cloud Providers - WIP

Goal:

Currently the Cloud Providers’ (CP) functionalities ( Create(), Delete(), List() ) are part of the Machine Controller Manager’s (MCM)repository. Because of this, adding support for new CPs into MCM requires merging code into MCM which may not be required for core functionalities of MCM itself. Also, for various reasons it may not be feasible for all CPs to merge their code with MCM which is an Open Source project.

Because of these reasons, it was decided that the CP’s code will be moved out in separate repositories so that they can be maintained separately by the respective teams. Idea is to make MCM act as a GRPC server, and CPs as GRPC clients. The CP can register themselves with the MCM using a GRPC service exposed by the MCM. Details of this approach is discussed below.

How it works:

MCM acts as GRPC server and listens on a pre-defined port 5000. It implements below GRPC services. Details of each of these services are mentioned in next section.

• Register()
• GetMachineClass()
• GetSecret()

GRPC services exposed by MCM:

Register()

rpc Register(stream DriverSide) returns (stream MCMside) {}

The CP GRPC client calls this service to register itself with the MCM. The CP passes the kind and the APIVersion which it implements, and MCM maintains an internal map for all the registered clients. A GRPC stream is returned in response which is kept open througout the life of both the processes. MCM uses this stream to communicate with the client for machine operations: Create(), Delete() or List(). The CP client is responsible for reading the incoming messages continuously, and based on the operationType parameter embedded in the message, it is supposed to take the required action. This part is already handled in the package grpc/infraclient. To add a new CP client, import the package, and implement the ExternalDriverProvider interface:

type ExternalDriverProvider interface {
	Create(machineclass *MachineClassMeta, credentials, machineID, machineName string) (string, string, error)
	Delete(machineclass *MachineClassMeta, credentials, machineID string) error
	List(machineclass *MachineClassMeta, credentials, machineID string) (map[string]string, error)
}

GetMachineClass()

rpc GetMachineClass(MachineClassMeta) returns (MachineClass) {}

As part of the message from MCM for various machine operations, the name of the machine class is sent instead of the full machine class spec. The CP client is expected to use this GRPC service to get the full spec of the machine class. This optionally enables the client to cache the machine class spec, and make the call only if the machine calass spec is not already cached.

GetSecret()

rpc GetSecret(SecretMeta) returns (Secret) {}

As part of the message from MCM for various machine operations, the Cloud Config (CC) and CP credentials are not sent. The CP client is expected to use this GRPC service to get the secret which has CC and CP’s credentials from MCM. This enables the client to cache the CC and credentials, and to make the call only if the data is not already cached.

How to add a new Cloud Provider’s support

Import the package grpc/infraclient and grpc/infrapb from MCM (currently in MCM’s “grpc-driver” branch)

• Implement the interface ExternalDriverProvider
  • Create(): Creates a new machine
  • Delete(): Deletes a machine
  • List(): Lists machines
• Use the interface MachineClassDataProvider
  • GetMachineClass(): Makes the call to MCM to get machine class spec
  • GetSecret(): Makes the call to MCM to get secret containing Cloud Config and CP’s credentials

Example implementation:

Refer GRPC based implementation for AWS client: https://github.com/ggaurav10/aws-driver-grpc

2.2.3 - Hotupdate Instances

Hot-Update VirtualMachine tags without triggering a rolling-update

• Hot-Update VirtualMachine tags without triggering a rolling-update
  • Motivation
  • Boundary Condition
  • What is available today?
  • What are the problems with the current approach?
    • MachineClass Update and its impact
  • Proposal
    • Shoot YAML changes
    • Provider specific WorkerConfig API changes
    • Gardener provider extension changes
    • Driver interface changes
    • Machine Class reconciliation
      • Reconciliation Changes

Motivation

• MCM Issue#750 There is a requirement to provide a way for consumers to add tags which can be hot-updated onto VMs. This requirement can be generalized to also offer a convenient way to specify tags which can be applied to VMs, NICs, Devices etc.

• MCM Issue#635 which in turn points to MCM-Provider-AWS Issue#36 - The issue hints at other fields like enable/disable source/destination checks for NAT instances which needs to be hot-updated on network interfaces.

• In GCP provider - instance.ServiceAccounts can be updated without the need to roll-over the instance. See

Boundary Condition

All tags that are added via means other than MachineClass.ProviderSpec should be preserved as-is. Only updates done to tags in MachineClass.ProviderSpec should be applied to the infra resources (VM/NIC/Disk).

What is available today?

WorkerPool configuration inside shootYaml provides a way to set labels. As per the definition these labels will be applied on Node resources. Currently these labels are also passed to the VMs as tags. There is no distinction made between Node labels and VM tags.

MachineClass has a field which holds provider specific configuration and one such configuration is tags. Gardener provider extensions updates the tags in MachineClass.

• AWS provider extension directly passes the labels to the tag section of machineClass.
• Azure provider extension sanitizes the woker pool labels and adds them as tags in MachineClass.
• GCP provider extension sanitizes them, and then sets them as labels in the MachineClass. In GCP tags only have keys and are currently hard coded.

Let us look at an example of MachineClass.ProviderSpec in AWS:

providerSpec:
  ami: ami-02fe00c0afb75bbd3
  tags:
    #[section-1] pool lables added by gardener extension
    #########################################################
    kubernetes.io/arch: amd64
    networking.gardener.cloud/node-local-dns-enabled: "true"
    node.kubernetes.io/role: node
    worker.garden.sapcloud.io/group: worker-ser234
    worker.gardener.cloud/cri-name: containerd
    worker.gardener.cloud/pool: worker-ser234
    worker.gardener.cloud/system-components: "true"

    #[section-2] Tags defined in the gardener-extension-provider-aws
    ###########################################################
    kubernetes.io/cluster/cluster-full-name: "1"
    kubernetes.io/role/node: "1"

    #[section-3]
    ###########################################################
    user-defined-key1: user-defined-val1
    user-defined-key2: user-defined-val2

Refer src for tags defined in section-1. Refer src for tags defined in section-2. Tags in section-3 are defined by the user.

Out of the above three tag categories, MCM depends section-2 tags (mandatory-tags) for its orphan collection and Driver’s DeleteMachineand GetMachineStatus to work.

ProviderSpec.Tags are transported to the provider specific resources as follows:

What are the problems with the current approach?

There are a few shortcomings in the way tags/labels are handled:

• Tags can only be set at the time a machine is created.
• There is no distinction made amongst tags/labels that are added to VM’s, disks or network interfaces. As stated above for AWS same set of tags are added to all. There is a limit defined on the number of tags/labels that can be associated to the devices (disks, VMs, NICs etc). Example: In AWS a max of 50 user created tags are allowed. Similar restrictions are applied on different resources across providers. Therefore adding all tags to all devices even if the subset of tags are not meant for that resource exhausts the total allowed tags/labels for that resource.
• The only placeholder in shoot yaml as mentioned above is meant to only hold labels that should be applied on primarily on the Node objects. So while you could use the node labels for extended resources, using it also for tags is not clean.
• There is no provision in the shoot YAML today to add tags only to a subset of resources.

MachineClass Update and its impact

When Worker.ProviderConfig is changed then a worker-hash is computed which includes the raw ProviderConfig. This hash value is then used as a suffix when constructing the name for a MachineClass. See aws-extension-provider as an example. A change in the name of the MachineClass will then in-turn trigger a rolling update of machines. Since tags are provider specific and therefore will be part of ProviderConfig, any update to them will result in a rolling-update of machines.

Proposal

Shoot YAML changes

Provider specific configuration is set via providerConfig section for each worker pool.

Example worker provider config (current):

providerConfig:
   apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
   kind: WorkerConfig
   volume:
     iops: 10000
   dataVolumes:
   - name: kubelet-dir
     snapshotID: snap-13234
   iamInstanceProfile: # (specify either ARN or name)
     name: my-profile
     arn: my-instance-profile-arn

It is proposed that an additional field be added for tags under providerConfig. Proposed changed YAML:

providerConfig:
   apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
   kind: WorkerConfig
   volume:
     iops: 10000
   dataVolumes:
   - name: kubelet-dir
     snapshotID: snap-13234
   iamInstanceProfile: # (specify either ARN or name)
     name: my-profile
     arn: my-instance-profile-arn
   tags:
     vm:
       key1: val1
       key2: val2
       ..
     # for GCP network tags are just keys (there is no value associated to them). 
     # What is shown below will work for AWS provider.
     network:
       key3: val3
       key4: val4

Under tags clear distinction is made between tags for VMs, Disks, network interface etc. Each provider has a different allowed-set of characters that it accepts as key names, has different limits on the tags that can be set on a resource (disk, NIC, VM etc.) and also has a different format (GCP network tags are only keys).

TODO:

• Check if worker.labels are getting added as tags on infra resources. We should continue to support it and double check that these should only be added to VMs and not to other resources.

• Should we support users adding VM tags as node labels?

Provider specific WorkerConfig API changes

Taking AWS provider extension as an example to show the changes.

WorkerConfig will now have the following changes:

1. A new field for tags will be introduced.
2. Additional metadata for struct fields will now be added via struct tags.

type WorkerConfig struct {
    metav1.TypeMeta
    Volume *Volume
    // .. all fields are not mentioned here.
    // Tags are a collection of tags to be set on provider resources (e.g. VMs, Disks, Network Interfaces etc.)
    Tags *Tags `hotupdatable:true`
}

// Tags is a placeholder for all tags that can be set/updated on VMs, Disks and Network Interfaces.
type Tags struct {
    // VM tags set on the VM instances.
    VM map[string]string
    // Network tags set on the network interfaces.
    Network map[string]string
    // Disk tags set on the volumes/disks.
    Disk map[string]string
}

There is a need to distinguish fields within ProviderSpec (which is then mapped to the above WorkerConfig) which can be updated without the need to change the hash suffix for MachineClass and thus trigger a rolling update on machines.

To achieve that we propose to use struct tag hotupdatable whose value indicates if the field can be updated without the need to do a rolling update. To ensure backward compatibility, all fields which do not have this tag or have hotupdatable set to false will be considered as immutable and will require a rolling update to take affect.

Gardener provider extension changes

Taking AWS provider extension as an example. Following changes should be made to all gardener provider extensions

AWS Gardener Extension generates machine config using worker pool configuration. As part of that it also computes the workerPoolHash which is then used to create the name of the MachineClass.

Currently WorkerPoolHash function uses the entire providerConfig to compute the hash. Proposal is to do the following:

1. Remove the code from function WorkerPoolHash.
2. Add another function to compute hash using all immutable fields in the provider config struct and then pass that to worker.WorkerPoolHash as additionalData.

The above will ensure that tags and any other field in WorkerConfig which is marked with updatable:true is not considered for hash computation and will therefore not contribute to changing the name of MachineClass object thus preventing a rolling update.

WorkerConfig and therefore the contained tags will be set as ProviderSpec in MachineClass.

If only fields which have updatable:true are changed then it should result in update/patch of MachineClass and not creation.

Driver interface changes

Driver interface which is a facade to provider specific API implementations will have one additional method.

type Driver interface {
    // .. existing methods are not mentioned here for brevity.
    UpdateMachine(context.Context, *UpdateMachineRequest) error
}

// UpdateMachineRequest is the request to update machine tags. 
type UpdateMachineRequest struct {
    ProviderID string
    LastAppliedProviderSpec raw.Extension
    MachineClass *v1alpha1.MachineClass
    Secret *corev1.Secret
}

If any machine-controller-manager-provider-<providername> has not implemented UpdateMachine then updates of tags on Instances/NICs/Disks will not be done. An error message will be logged instead.

Machine Class reconciliation

Current MachineClass reconciliation does not reconcile MachineClass resource updates but it only enqueues associated machines. The reason is that it is assumed that anything that is changed in a MachineClass will result in a creation of a new MachineClass with a different name. This will result in a rolling update of all machines using the MachineClass as a template.

However, it is possible that there is data that all machines in a MachineSet share which do not require a rolling update (e.g. tags), therefore there is a need to reconcile the MachineClass as well.

Reconciliation Changes

In order to ensure that machines get updated eventually with changes to the hot-updatable fields defined in the MachineClass.ProviderConfig as raw.Extension.

We should only fix MCM Issue#751 in the MachineClass reconciliation and let it enqueue the machines as it does today. We additionally propose the following two things:

1. Introduce a new annotation last-applied-providerspec on every machine resource. This will capture the last successfully applied MachineClass.ProviderSpec on this instance.

2. Enhance the machine reconciliation to include code to hot-update machine.

In machine-reconciliation there are currently two flows triggerDeletionFlow and triggerCreationFlow. When a machine gets enqueued due to changes in MachineClass then in this method following changes needs to be introduced:

Check if the machine has last-applied-providerspec annotation.

Case 1.1

If the annotation is not present then there can be just 2 possibilities:

• It is a fresh/new machine and no backing resources (VM/NIC/Disk) exist yet. The current flow checks if the providerID is empty and Status.CurrenStatus.Phase is empty then it enters into the triggerCreationFlow.

• It is an existing machine which does not yet have this annotation. In this case call Driver.UpdateMachine. If the driver returns no error then add last-applied-providerspec annotation with the value of MachineClass.ProviderSpec to this machine.

Case 1.2

If the annotation is present then compare the last applied provider-spec with the current provider-spec. If there are changes (check their hash values) then call Driver.UpdateMachine. If the driver returns no error then add last-applied-providerspec annotation with the value of MachineClass.ProviderSpec to this machine.

NOTE: It is assumed that if there are changes to the fields which are not marked as hotupdatable then it will result in the change of name for MachineClass resulting in a rolling update of machines. If the name has not changed + machine is enqueued + there is a change in machine-class then it will be change to a hotupdatable fields in the spec.

Trigger update flow can be done after reconcileMachineHealth and syncMachineNodeTemplates in machine-reconciliation.

There are 2 edge cases that needs attention and special handling:

Premise: It is identified that there is an update done to one or more hotupdatable fields in the MachineClass.ProviderSpec.

Edge-Case-1

In the machine reconciliation, an update-machine-flow is triggered which in-turn calls Driver.UpdateMachine. Consider the case where the hot update needs to be done to all VM, NIC and Disk resources. The driver returns an error which indicates a partial-failure. As we have mentioned above only when Driver.UpdateMachine returns no error will last-applied-providerspec be updated. In case of partial failure the annotation will not be updated. This event will be re-queued for a re-attempt. However consider a case where before the item is re-queued, another update is done to MachineClass reverting back the changes to the original spec.

At T4 (T4> T3) when a machine is reconciled then it checks that last-applied-providerspec is S1 and current MachineClass.ProviderSpec = S3 and since S3 is same as S1, no update is done. At T2 Driver.UpdateMachine was called to update the machine with S2 but it partially failed. So now you will have resources which are partially updated with S2 and no further updates will be attempted.

Edge-Case-2

The above situation can also happen when Driver.UpdateMachine is in the process of updating resources. It has hot-updated lets say 1 resource. But now MCM crashes. By the time it comes up another update to MachineClass.ProviderSpec is done essentially reverting back the previous change (same case as above). In this case reconciliation loop never got a chance to get any response from the driver.

To handle the above edge cases there are 2 options:

Option #1

Introduce a new annotation inflight-providerspec-hash . The value of this annotation will be the hash value of the MachineClass.ProviderSpec that is in the process of getting applied on this machine. The machine will be updated with this annotation just before calling Driver.UpdateMachine (in the trigger-update-machine-flow). If the driver returns no error then (in a single update):

1. last-applied-providerspec will be updated

2. inflight-providerspec-hash annotation will be removed.

Option #2 - Preferred

Leverage Machine.Status.LastOperation with Type set to MachineOperationUpdate and State set to MachineStateProcessing This status will be updated just before calling Driver.UpdateMachine.

Semantically LastOperation captures the details of the operation post-operation and not pre-operation. So this solution would be a divergence from the norm.

2.2.4 - Initialize Machine

Post-Create Initialization of Machine Instance

Background

Today the driver.Driver facade represents the boundary between the the machine-controller and its various provider specific implementations.

We have abstract operations for creation/deletion and listing of machines (actually compute instances) but we do not correctly handle post-creation initialization logic. Nor do we provide an abstract operation to represent the hot update of an instance after creation.

We have found this to be necessary for several use cases. Today in the MCM AWS Provider, we already misuse driver.GetMachineStatus which is supposed to be a read-only operation obtaining the status of an instance.

1. Each AWS EC2 instance performs source/destination checks by default. For EC2 NAT instances these should be disabled. This is done by issuing a ModifyInstanceAttribute request with the SourceDestCheck set to false. The MCM AWS Provider, decodes the AWSProviderSpec, reads providerSpec.SrcAndDstChecksEnabled and correspondingly issues the call to modify the already launched instance. However, this should be done as an action after creating the instance and should not be part of the VM status retrieval.

2. Similarly, there is a pending PR to add the Ipv6AddessCount and Ipv6PrefixCount to enable the assignment of an ipv6 address and an ipv6 prefix to instances. This requires constructing and issuing an AssignIpv6Addresses request after the EC2 instance is available.

3. We have other uses-cases such as MCM Issue#750 where there is a requirement to provide a way for consumers to add tags which can be hot-updated onto instances. This requirement can be generalized to also offer a convenient way to specify tags which can be applied to VMs, NICs, Devices etc.

4. We have a need for “machine-instance-not-ready” taint as described in MCM#740 which should only get removed once the post creation updates are finished.

Objectives

We will split the fulfilment of this overall need into 2 stages of implementation.

1. Stage-A: Support post-VM creation initialization logic of the instance suing a proposed Driver.InitializeMachine by permitting provider implementors to add initialization logic after VM creation, return with special new error code codes.Initialization for initialization errors and correspondingly support a new machine operation stage InstanceInitialization which will be updated in the machine LastOperation. The triggerCreationFlow - a reconciliation sub-flow of the MCM responsible for orchestrating instance creation and updating machine status will be changed to support this behaviour.

2. Stage-B: Introduction of Driver.UpdateMachine and enhancing the MCM, MCM providers and gardener extension providers to support hot update of instances through Driver.UpdateMachine. The MCM triggerUpdationFlow - a reconciliation sub-flow of the MCM which is supposed to be responsible for orchestrating instance update - but currently not used, will be updated to invoke the provider Driver.UpdateMachine on hot-updates to to the Machine object

Stage-A Proposal

Current MCM triggerCreationFlow

Today, reconcileClusterMachine which is the main routine for the Machine object reconciliation invokes triggerCreationFlow at the end when the machine.Spec.ProviderID is empty or if the machine.Status.CurrentStatus.Phase is empty or in CrashLoopBackOff

%%{ init: {
    'themeVariables':
        { 'fontSize': '12px'}
} }%%
flowchart LR

other["..."]
-->chk{"machine ProviderID empty
OR
Phase empty or CrashLoopBackOff ?
"}--yes-->triggerCreationFlow
chk--noo-->LongRetry["return machineutils.LongRetry"]

Today, the triggerCreationFlow is illustrated below with some minor details omitted/compressed for brevity

NOTES

• The lastop below is an abbreviation for machine.Status.LastOperation. This, along with the machine phase is generally updated on the Machine object just before returning from the method.
• regarding phase=CrashLoopBackOff|Failed. the machine phase may either be CrashLoopBackOff or move to Failed if the difference between current time and the machine.CreationTimestamp has exceeded the configured MachineCreationTimeout.

%%{ init: {
    'themeVariables':
        { 'fontSize': '12px'}
} }%%
flowchart TD


end1(("end"))
begin((" "))
medretry["return MediumRetry, err"]
shortretry["return ShortRetry, err"]
medretry-->end1
shortretry-->end1

begin-->AddBootstrapTokenToUserData
-->gms["statusResp,statusErr=driver.GetMachineStatus(...)"]
-->chkstatuserr{"Check statusErr"}
chkstatuserr--notFound-->chknodelbl{"Chk Node Label"}
chkstatuserr--else-->createFailed["lastop.Type=Create,lastop.state=Failed,phase=CrashLoopBackOff|Failed"]-->medretry
chkstatuserr--nil-->initnodename["nodeName = statusResp.NodeName"]-->setnodename


chknodelbl--notset-->createmachine["createResp, createErr=driver.CreateMachine(...)"]-->chkCreateErr{"Check createErr"}

chkCreateErr--notnil-->createFailed

chkCreateErr--nil-->getnodename["nodeName = createResp.NodeName"]
-->chkstalenode{"nodeName != machine.Name\n//chk stale node"}
chkstalenode--false-->setnodename["if unset machine.Labels['node']= nodeName"]
-->machinepending["if empty/crashloopbackoff lastop.type=Create,lastop.State=Processing,phase=Pending"]
-->shortretry

chkstalenode--true-->delmachine["driver.DeleteMachine(...)"]
-->permafail["lastop.type=Create,lastop.state=Failed,Phase=Failed"]
-->shortretry

subgraph noteA [" "]
    permafail -.- note1(["VM was referring to stale node obj"])
end
style noteA opacity:0


subgraph noteB [" "]
    setnodename-.- note2(["Proposal: Introduce Driver.InitializeMachine after this"])
end

Enhancement of MCM triggerCreationFlow

Relevant Observations on Current Flow

1. Observe that we always perform a call to Driver.GetMachineStatus and only then conditionally perform a call to Driver.CreateMachine if there was was no machine found.
2. Observe that after the call to a successful Driver.CreateMachine, the machine phase is set to Pending, the LastOperation.Type is currently set to Create and the LastOperation.State set to Processing before returning with a ShortRetry. The LastOperation.Description is (unfortunately) set to the fixed message: Creating machine on cloud provider.
3. Observe that after an erroneous call to Driver.CreateMachine, the machine phase is set to CrashLoopBackOff or Failed (in case of creation timeout).

The following changes are proposed with a view towards minimal impact on current code and no introduction of a new Machine Phase.

MCM Changes

1. We propose introducing a new machine operation Driver.InitializeMachine with the following signature

   type Driver interface {
       // .. existing methods are omitted for brevity.
   
       // InitializeMachine call is responsible for post-create initialization of the provider instance.
       InitializeMachine(context.Context, *InitializeMachineRequest) error
   }
   
   // InitializeMachineRequest is the initialization request for machine instance initialization
   type InitializeMachineRequest struct {
       // Machine object whose VM instance should be initialized 
       Machine *v1alpha1.Machine
   
       // MachineClass backing the machine object
       MachineClass *v1alpha1.MachineClass
   
       // Secret backing the machineClass object
       Secret *corev1.Secret
   }
   

2. We propose introducing a new MC error code codes.Initialization indicating that the VM Instance was created but there was an error in initialization after VM creation. The implementor of Driver.InitializeMachine can return this error code, indicating that InitializeMachine needs to be called again. The Machine Controller will change the phase to CrashLoopBackOff as usual when encountering a codes.Initialization error.
3. We will introduce a new machine operation stage InstanceInitialization. In case of an codes.Initialization error
   1. the machine.Status.LastOperation.Description will be set to InstanceInitialization,
   2. machine.Status.LastOperation.ErrorCode will be set to codes.Initialization
   3. the LastOperation.Type will be set to Create
   4. the LastOperation.State set to Failed before returning with a ShortRetry
4. The semantics of Driver.GetMachineStatus will be changed. If the instance associated with machine exists, but the instance was not initialized as expected, the provider implementations of GetMachineStatus should return an error: status.Error(codes.Initialization).
5. If Driver.GetMachineStatus returned an error encapsulating codes.Initialization then Driver.InitializeMachine will be invoked again in the triggerCreationFlow.
6. As according to the usual logic, the main machine controller reconciliation loop will now re-invoke the triggerCreationFlow again if the machine phase is CrashLoopBackOff.

Illustration

AWS Provider Changes

Driver.InitializeMachine

The implementation for the AWS Provider will look something like:

1. After the VM instance is available, check providerSpec.SrcAndDstChecksEnabled, construct ModifyInstanceAttributeInput and call ModifyInstanceAttribute. In case of an error return codes.Initialization instead of the current codes.Internal
2. Check providerSpec.NetworkInterfaces and if Ipv6PrefixCount is not nil, then construct AssignIpv6AddressesInput and call AssignIpv6Addresses. In case of an error return codes.Initialization. Don’t use the generic codes.Internal

The existing Ipv6 PR will need modifications.

Driver.GetMachineStatus

1. If providerSpec.SrcAndDstChecksEnabled is false, check ec2.Instance.SourceDestCheck. If it does not match then return status.Error(codes.Initialization)
2. Check providerSpec.NetworkInterfaces and if Ipv6PrefixCount is not nil, check ec2.Instance.NetworkInterfaces and check if InstanceNetworkInterface.Ipv6Addresses has a non-nil slice. If this is not the case then return status.Error(codes.Initialization)

Instance Not Ready Taint

• Due to the fact that creation flow for machines will now be enhanced to correctly support post-creation startup logic, we should not scheduled workload until this startup logic is complete. Even without this feature we have a need for such a taint as described in MCM#740
• We propose a new taint node.machine.sapcloud.io/instance-not-ready which will be added as a node startup taint in gardener core KubeletConfiguration.RegisterWithTaints
• The will will then removed by MCM in health check reconciliation, once the machine becomes fully ready. (when moving to Running phase)
• We will add this taint as part of --ignore-taint in CA
• We will introduce a disclaimer / prerequisite in the MCM FAQ, to add this taint as part of kubelet config under --register-with-taints, otherwise workload could get scheduled , before machine beomes Running

Stage-B Proposal

Enhancement of Driver Interface for Hot Updation

Kindly refer to the Hot-Update Instances design which provides elaborate detail.

2.3 - ToDo

2.3.1 - Outline

Machine Controller Manager

CORE – ./machine-controller-manager(provider independent) Out of tree : Machine controller (provider specific) MCM is a set controllers:

• Machine Deployment Controller

• Machine Set Controller

• Machine Controller

• Machine Safety Controller

Questions and refactoring Suggestions

Refactoring

Clarification

Bugs/Enhancements

Code references

1.1 code1

       if machineSet.DeletionTimestamp == nil {
        
        		// manageReplicas is the core machineSet method where scale up/down occurs
        
        		// It is not called when deletion timestamp is set
        
        		manageReplicasErr = c.manageReplicas(ctx, filteredMachines, machineSet)
        
        ​
        
        	} else if machineSet.DeletionTimestamp != nil { 
        
            //FIX: change this to simple else without the if

1.2 code2

    defer dc.enqueueMachineDeploymentAfter(deployment, 10*time.Minute)
    
    *  `Clarification`:  Why  is  this  conversion  needed?
    
    err = v1alpha1.Convert_v1alpha1_MachineDeployment_To_machine_MachineDeployment(deployment, internalMachineDeployment, nil)

1.3 code3

// rollback is not re-entrant in case the underlying machine sets are updated with a new

	// revision so we should ensure that we won't proceed to update machine sets until we

	// make sure that the deployment has cleaned up its rollback spec in subsequent enqueues.

	if d.Spec.RollbackTo != nil {

		return dc.rollback(ctx, d, machineSets, machineMap)

	}

2.4 - FAQ

Frequently Asked Questions

The answers in this FAQ apply to the newest (HEAD) version of Machine Controller Manager. If you’re using an older version of MCM please refer to corresponding version of this document. Few of the answers assume that the MCM being used is in conjuction with cluster-autoscaler:

Table of Contents:

• Basics

  • What is Machine Controller Manager?
  • Why is my machine deleted?
  • What are the different sub-controllers in MCM?
  • What is Safety Controller in MCM?

• How to?

  • How to install MCM in a Kubernetes cluster?
  • How to better control the rollout process of the worker nodes?
  • How to scale down MachineDeployment by selective deletion of machines?
  • How to force delete a machine?
  • How to pause the ongoing rolling-update of the machinedeployment?
  • How to delete machine object immedietly if I don’t have access to it?
  • How to avoid garbage collection of your node?
  • How to trigger rolling update of a machinedeployment?

• Internals

  • What is the high level design of MCM?
  • What are the different configuration options in MCM?
  • What are the different timeouts/configurations in a machine’s lifecycle?
  • How is the drain of a machine implemented?
  • How are the stateful applications drained during machine deletion?
  • How does maxEvictRetries configuration work with drainTimeout configuration?
  • What are the different phases of a machine?
  • What health checks are performed on a machine?
  • How does rate limiting replacement of machine work in MCM ? How is it related to meltdown protection?
  • How MCM responds when scale-out/scale-in is done during rolling update of a machinedeployment?
  • How some unhealthy machines are drained quickly?
  • How does MCM prioritize the machines for deletion on scale-down of machinedeployment?

• Troubleshooting

  • My machine is stuck in deletion for 1 hr, why?
  • My machine is not joining the cluster, why?

• Developer

  • How should I test my code before submitting a PR?
  • I need to change the APIs, what are the recommended steps?
  • How can I update the dependencies of MCM?

• In the context of Gardener

  • How can I configure MCM using Shoot resource?
  • How is my worker-pool spread across zones?

Basics

What is Machine Controller Manager?

Machine Controller Manager aka MCM is a bunch of controllers used for the lifecycle management of the worker machines. It reconciles a set of CRDs such as Machine, MachineSet, MachineDeployment which depicts the functionality of Pod, Replicaset, Deployment of the core Kubernetes respectively. Read more about it at README.

• Gardener uses MCM to manage its Kubernetes nodes of the shoot cluster. However, by design, MCM can be used independent of Gardener.

Why is my machine deleted?

A machine is deleted by MCM generally for 2 reasons-

• Machine is unhealthy for at least MachineHealthTimeout period. The default MachineHealthTimeout is 10 minutes.

  • By default, a machine is considered unhealthy if any of the following node conditions - DiskPressure, KernelDeadlock, FileSystem, Readonly is set to true, or KubeletReady is set to false. However, this is something that is configurable using the following flag.

• Machine is scaled down by the MachineDeployment resource.

  • This is very usual when an external controller cluster-autoscaler (aka CA) is used with MCM. CA deletes the under-utilized machines by scaling down the MachineDeployment. Read more about cluster-autoscaler’s scale down behavior here.

What are the different sub-controllers in MCM?

MCM mainly contains the following sub-controllers:

• MachineDeployment Controller: Responsible for reconciling the MachineDeployment objects. It manages the lifecycle of the MachineSet objects.
• MachineSet Controller: Responsible for reconciling the MachineSet objects. It manages the lifecycle of the Machine objects.
• Machine Controller: responsible for reconciling the Machine objects. It manages the lifecycle of the actual VMs/machines created in cloud/on-prem. This controller has been moved out of tree. Please refer an AWS machine controller for more info - link.
• Safety-controller: Responsible for handling the unidentified/unknown behaviors from the cloud providers. Please read more about its functionality below.

What is Safety Controller in MCM?

Safety Controller contains following functions:

• Orphan VM handler:
  • It lists all the VMs in the cloud matching the tag of given cluster name and maps the VMs with the machine objects using the ProviderID field. VMs without any backing machine objects are logged and deleted after confirmation.
  • This handler runs every 30 minutes and is configurable via machine-safety-orphan-vms-period flag.
• Freeze mechanism:
  • Safety Controller freezes the MachineDeployment and MachineSet controller if the number of machine objects goes beyond a certain threshold on top of Spec.Replicas. It can be configured by the flag –safety-up or –safety-down and also machine-safety-overshooting-period.
  • Safety Controller freezes the functionality of the MCM if either of the target-apiserver or the control-apiserver is not reachable.
  • Safety Controller unfreezes the MCM automatically once situation is resolved to normal. A freeze label is applied on MachineDeployment/MachineSet to enforce the freeze condition.

How to?

How to install MCM in a Kubernetes cluster?

MCM can be installed in a cluster with following steps:

• Apply all the CRDs from here

• Apply all the deployment, role-related objects from here.

  • Control cluster is the one where the machine-* objects are stored. Target cluster is where all the node objects are registered.

How to better control the rollout process of the worker nodes?

MCM allows configuring the rollout of the worker machines using maxSurge and maxUnavailable fields. These fields are applicable only during the rollout process and means nothing in general scale up/down scenarios. The overall process is very similar to how the Deployment Controller manages pods during RollingUpdate.

• maxSurge refers to the number of additional machines that can be added on top of the Spec.Replicas of MachineDeployment during rollout process.
• maxUnavailable refers to the number of machines that can be deleted from Spec.Replicas field of the MachineDeployment during rollout process.

How to scale down MachineDeployment by selective deletion of machines?

During scale down, triggered via MachineDeployment/MachineSet, MCM prefers to delete the machine/s which have the least priority set. Each machine object has an annotation machinepriority.machine.sapcloud.io set to 3 by default. Admin can reduce the priority of the given machines by changing the annotation value to 1. The next scale down by MachineDeployment shall delete the machines with the least priority first.

How to force delete a machine?

A machine can be force deleted by adding the label force-deletion: "True" on the machine object before executing the actual delete command. During force deletion, MCM skips the drain function and simply triggers the deletion of the machine. This label should be used with caution as it can violate the PDBs for pods running on the machine.

How to pause the ongoing rolling-update of the machinedeployment?

An ongoing rolling-update of the machine-deployment can be paused by using spec.paused field. See the example below:

apiVersion: machine.sapcloud.io/v1alpha1
kind: MachineDeployment
metadata:
  name: test-machine-deployment
spec:
  paused: true

It can be unpaused again by removing the Paused field from the machine-deployment.

How to delete machine object immedietly if I don’t have access to it?

If the user doesn’t have access to the machine objects (like in case of Gardener clusters) and they would like to replace a node immedietly then they can place the annotation node.machine.sapcloud.io/trigger-deletion-by-mcm: "true" on their node. This will start the replacement of the machine with a new node.

On the other hand if the user deletes the node object immedietly then replacement will start only after MachineHealthTimeout.

This annotation can also be used if the user wants to expedite the replacement of unhealthy nodes

NOTE:

• node.machine.sapcloud.io/trigger-deletion-by-mcm: "false" annotation is NOT acted upon by MCM , neither does it mean that MCM will not replace this machine.
• this annotation would delete the desired machine but another machine would be created to maintain desired replicas specified for the machineDeployment/machineSet. Currently if the user doesn’t have access to machineDeployment/machineSet then they cannot remove a machine without replacement.

How to avoid garbage collection of your node?

MCM provides an in-built safety mechanism to garbage collect VMs which have no corresponding machine object. This is done to save costs and is one of the key features of MCM. However, sometimes users might like to add nodes directly to the cluster without the help of MCM and would prefer MCM to not garbage collect such VMs. To do so they should remove/not-use tags on their VMs containing the following strings:

1. kubernetes.io/cluster/
2. kubernetes.io/role/
3. kubernetes-io-cluster-
4. kubernetes-io-role-

How to trigger rolling update of a machinedeployment?

Rolling update can be triggered for a machineDeployment by updating one of the following:

• .spec.template.annotations
• .spec.template.spec.class.name

Internals

What is the high level design of MCM?

Please refer the following document.

What are the different configuration options in MCM?

MCM allows configuring many knobs to fine-tune its behavior according to the user’s need. Please refer to the link to check the exact configuration options.

What are the different timeouts/configurations in a machine’s lifecycle?

A machine’s lifecycle is governed by mainly following timeouts, which can be configured here.

• MachineDrainTimeout: Amount of time after which drain times out and the machine is force deleted. Default ~2 hours.
• MachineHealthTimeout: Amount of time after which an unhealthy machine is declared Failed and the machine is replaced by MachineSet controller.
• MachineCreationTimeout: Amount of time after which a machine creation is declared Failed and the machine is replaced by the MachineSet controller.
• NodeConditions: List of node conditions which if set to true for MachineHealthTimeout period, the machine is declared Failed and replaced by MachineSet controller.
• MaxEvictRetries: An integer number depicting the number of times a failed eviction should be retried on a pod during drain process. A pod is deleted after max-retries.

How is the drain of a machine implemented?

MCM imports the functionality from the upstream Kubernetes-drain library. Although, few parts have been modified to make it work best in the context of MCM. Drain is executed before machine deletion for graceful migration of the applications. Drain internally uses the EvictionAPI to evict the pods and triggers the Deletion of pods after MachineDrainTimeout. Please note:

• Stateless pods are evicted in parallel.
• Stateful applications (with PVCs) are serially evicted. Please find more info in this answer below.

How are the stateful applications drained during machine deletion?

Drain function serially evicts the stateful-pods. It is observed that serial eviction of stateful pods yields better overall availability of pods as the underlying cloud in most cases detaches and reattaches disks serially anyways. It is implemented in the following manner:

• Drain lists all the pods with attached volumes. It evicts very first stateful-pod and waits for its related entry in Node object’s .status.volumesAttached to be removed by KCM. It does the same for all the stateful-pods.
• It waits for PvDetachTimeout (default 2 minutes) for a given pod’s PVC to be removed, else moves forward.

How does maxEvictRetries configuration work with drainTimeout configuration?

It is recommended to only set MachineDrainTimeout. It satisfies the related requirements. MaxEvictRetries is auto-calculated based on MachineDrainTimeout, if maxEvictRetries is not provided. Following will be the overall behavior of both configurations together:

• If maxEvictRetries isn’t set and only maxDrainTimeout is set:
  • MCM auto calculates the maxEvictRetries based on the drainTimeout.
• If drainTimeout isn’t set and only maxEvictRetries is set:
  • Default drainTimeout and user provided maxEvictRetries for each pod is considered.
• If both maxEvictRetries and drainTimoeut are set:
  • Then both will be respected.
• If none are set:
  • Defaults are respected.

What are the different phases of a machine?

A phase of a machine can be identified with Machine.Status.CurrentStatus.Phase. Following are the possible phases of a machine object:

• Pending: Machine creation call has succeeded. MCM is waiting for machine to join the cluster.

• CrashLoopBackOff: Machine creation call has failed. MCM will retry the operation after a minor delay.

• Running: Machine creation call has succeeded. Machine has joined the cluster successfully and corresponding node doesn’t have node.gardener.cloud/critical-components-not-ready taint.

• Unknown: Machine health checks are failing, eg kubelet has stopped posting the status.

• Failed: Machine health checks have failed for a prolonged time. Hence it is declared failed by Machine controller in a rate limited fashion. Failed machines get replaced immediately.

• Terminating: Machine is being terminated. Terminating state is set immediately when the deletion is triggered for the machine object. It also includes time when it’s being drained.

NOTE: No phase means the machine is being created on the cloud-provider.

Below is a simple phase transition diagram:

What health checks are performed on a machine?

Health check performed on a machine are:

• Existense of corresponding node obj
• Status of certain user-configurable node conditions.
  • These conditions can be specified using the flag --node-conditions for OOT MCM provider or can be specified per machine object.
  • The default user configurable node conditions can be found here
• True status of NodeReady condition . This condition shows kubelet’s status

If any of the above checks fails , the machine turns to Unknown phase.

How does rate limiting replacement of machine work in MCM? How is it related to meltdown protection?

Currently MCM replaces only 1 Unkown machine at a time per machinedeployment. This means until the particular Unknown machine get terminated and its replacement joins, no other Unknown machine would be removed.

The above is achieved by enabling Machine controller to turn machine from Unknown -> Failed only if the above condition is met. MachineSet controller on the other hand marks Failed machine as Terminating immediately.

One reason for this rate limited replacement was to ensure that in case of network failures , where node’s kubelet can’t reach out to kube-apiserver , all nodes are not removed together i.e. meltdown protection. In gardener context however, DWD is deployed to deal with this scenario, but to stay protected from corner cases , this mechanism has been introduced in MCM.

NOTE: Rate limiting replacement is not yet configurable

How MCM responds when scale-out/scale-in is done during rolling update of a machinedeployment?

Machinedeployment controller executes the logic of scaling BEFORE logic of rollout. It identifies scaling by comparing the deployment.kubernetes.io/desired-replicas of each machineset under the machinedeployment with machinedeployment’s .spec.replicas. If the difference is found for any machineSet, a scaling event is detected.

Case scale-out -> ONLY New machineSet is scaled out
Case scale-in -> ALL machineSets(new or old) are scaled in , in proportion to their replica count , any leftover is adjusted in the largest machineSet.

During update for scaling event, a machineSet is updated if any of the below is true for it:

• .spec.Replicas needs update
• deployment.kubernetes.io/desired-replicas needs update

Once scaling is achieved, rollout continues.

How does MCM prioritize the machines for deletion on scale-down of machinedeployment?

There could be many machines under a machinedeployment with different phases, creationTimestamp. When a scale down is triggered, MCM decides to remove the machine using the following logic:

• Machine with least value of machinepriority.machine.sapcloud.io annotation is picked up.
• If all machines have equal priorities, then following precedence is followed:
  • Terminating > Failed > CrashloopBackoff > Unknown > Pending > Available > Running
• If still there is no match, the machine with oldest creation time (.i.e. creationTimestamp) is picked up.

How some unhealthy machines are drained quickly ?

If a node is unhealthy for more than the machine-health-timeout specified for the machine-controller, the controller health-check moves the machine phase to Failed. By default, the machine-health-timeout is 10` minutes.

Failed machines have their deletion timestamp set and the machine then moves to the Terminating phase. The node drain process is initiated. The drain process is invoked either gracefully or forcefully.

The usual drain process is graceful. Pods are evicted from the node and the drain process waits until any existing attached volumes are mounted on new node. However, if the node Ready is False or the ReadonlyFilesystem is True for greater than 5 minutes (non-configurable), then a forceful drain is initiated. In a forceful drain, pods are deleted and VolumeAttachment objects associated with the old node are also marked for deletion. This is followed by the deletion of the cloud provider VM associated with the Machine and then finally ending with the Node object deletion.

During the deletion of the VM we only delete the local data disks and boot disks associated with the VM. The disks associated with persistent volumes are left un-touched as their attach/de-detach, mount/unmount processes are handled by k8s attach-detach controller in conjunction with the CSI driver.

Troubleshooting

My machine is stuck in deletion for 1 hr, why?

In most cases, the Machine.Status.LastOperation provides information around why a machine can’t be deleted. Though following could be the reasons but not limited to:

• Pod/s with misconfigured PDBs block the drain operation. PDBs with maxUnavailable set to 0, doesn’t allow the eviction of the pods. Hence, drain/eviction is retried till MachineDrainTimeout. Default MachineDrainTimeout could be as large as ~2hours. Hence, blocking the machine deletion.
  • Short term: User can manually delete the pod in the question, with caution.
  • Long term: Please set more appropriate PDBs which allow disruption of at least one pod.
• Expired cloud credentials can block the deletion of the machine from infrastructure.
• Cloud provider can’t delete the machine due to internal errors. Such situations are best debugged by using cloud provider specific CLI or cloud console.

My machine is not joining the cluster, why?

In most cases, the Machine.Status.LastOperation provides information around why a machine can’t be created. It could possibly be debugged with following steps:

• Firstly make sure all the relevant controllers like kube-controller-manager , cloud-controller-manager are running.
• Verify if the machine is actually created in the cloud. User can use the Machine.Spec.ProviderId to query the machine in cloud.
• A Kubernetes node is generally bootstrapped with the cloud-config. Please verify, if MachineDeployment is pointing the correct MachineClass, and MachineClass is pointing to the correct Secret. The secret object contains the actual cloud-config in base64 format which will be used to boot the machine.
• User must also check the logs of the MCM pod to understand any broken logical flow of reconciliation.

My rolling update is stuck , why?

The following can be the reason:

• Insufficient capacity for the new instance type the machineClass mentions.
• Old machines are stuck in deletion
• If you are using Gardener for setting up kubernetes cluster, then machine object won’t turn to Running state until node-critical-components are ready. Refer this for more details.

Developer

How should I test my code before submitting a PR?

• Developer can locally setup the MCM using following guide
• Developer must also enhance the unit tests related to the incoming changes.
• Developer can locally run the unit test by executing:

make test-unit

• Developer can locally run integration tests to ensure basic functionality of MCM is not altered.

I need to change the APIs, what are the recommended steps?

Developer should add/update the API fields at both of the following places:

• https://github.com/gardener/machine-controller-manager/blob/master/pkg/apis/machine/types.go
• https://github.com/gardener/machine-controller-manager/tree/master/pkg/apis/machine/v1alpha1

Once API changes are done, auto-generate the code using following command:

make generate

Please ignore the API-violation errors for now.

How can I update the dependencies of MCM?

MCM uses gomod for depedency management. Developer should add/udpate depedency in the go.mod file. Please run following command to automatically tidy the dependencies.

make tidy

In the context of Gardener

How can I configure MCM using Shoot resource?

All of the knobs of MCM can be configured by the workers section of the shoot resource.

• Gardener creates a MachineDeployment per zone for each worker-pool under workers section.
• workers.dataVolumes allows to attach multiple disks to a machine during creation. Refer the link.
• workers.machineControllerManager allows configuration of multiple knobs of the MachineDeployment from the shoot resource.

How is my worker-pool spread across zones?

Shoot resource allows the worker-pool to spread across multiple zones using the field workers.zones. Refer link.

• Gardener creates one MachineDeployment per zone. Each MachineDeployment is initiated with the following replica:

MachineDeployment.Spec.Replicas = (Workers.Minimum)/(Number of availibility zones)

2.5 - Adding Support for a Cloud Provider

Adding support for a new provider

Steps to be followed while implementing a new (hyperscale) provider are mentioned below. This is the easiest way to add new provider support using a blueprint code.

However, you may also develop your machine controller from scratch, which would provide you with more flexibility. First, however, make sure that your custom machine controller adheres to the Machine.Status struct defined in the MachineAPIs. This will make sure the MCM can act with higher-level controllers like MachineSet and MachineDeployment controller. The key is the Machine.Status.CurrentStatus.Phase key that indicates the status of the machine object.

Our strong recommendation would be to follow the steps below. This provides the most flexibility required to support machine management for adding new providers. And if you feel to extend the functionality, feel free to update our machine controller libraries.

Setting up your repository

1. Create a new empty repository named machine-controller-manager-provider-{provider-name} on GitHub username/project. Do not initialize this repository with a README.
2. Copy the remote repository URL (HTTPS/SSH) to this repository displayed once you create this repository.
3. Now, on your local system, create directories as required. {your-github-username} given below could also be {github-project} depending on where you have created the new repository.

   mkdir -p $GOPATH/src/github.com/{your-github-username}
   

4. Navigate to this created directory.

   cd $GOPATH/src/github.com/{your-github-username}
   

5. Clone this repository on your local machine.

   git clone git@github.com:gardener/machine-controller-manager-provider-sampleprovider.git
   

6. Rename the directory from machine-controller-manager-provider-sampleprovider to machine-controller-manager-provider-{provider-name}.

   mv machine-controller-manager-provider-sampleprovider machine-controller-manager-provider-{provider-name}
   

7. Navigate into the newly-created directory.

   cd machine-controller-manager-provider-{provider-name}
   

8. Update the remote origin URL to the newly created repository’s URL you had copied above.

   git remote set-url origin git@github.com:{your-github-username}/machine-controller-manager-provider-{provider-name}.git
   

9. Rename GitHub project from gardener to {github-org/your-github-username} wherever you have cloned the repository above. Also, edit all occurrences of the word sampleprovider to {provider-name} in the code. Then, use the hack script given below to do the same.

   make rename-project PROJECT_NAME={github-org/your-github-username} PROVIDER_NAME={provider-name}
   eg:
       make rename-project PROJECT_NAME=gardener PROVIDER_NAME=AmazonWebServices (or)
       make rename-project PROJECT_NAME=githubusername PROVIDER_NAME=AWS
   

10. Now, commit your changes and push them upstream.

    git add -A
    git commit -m "Renamed SampleProvide to {provider-name}"
    git push origin master
    

Code changes required

The contract between the Machine Controller Manager (MCM) and the Machine Controller (MC) AKA driver has been documented here and the machine error codes can be found here. You may refer to them for any queries.

⚠️

• Keep in mind that there should be a unique way to map between machine objects and VMs. This can be done by mapping machine object names with VM-Name/ tags/ other metadata.
• Optionally, there should also be a unique way to map a VM to its machine class object. This can be done by tagging VM objects with tags/resource groups associated with the machine class.

Steps to integrate

1. Update the pkg/provider/apis/provider_spec.go specification file to reflect the structure of the ProviderSpec blob. It typically contains the machine template details in the MachineClass object. Follow the sample spec provided already in the file. A sample provider specification can be found here.
2. Fill in the methods described at pkg/provider/core.go to manage VMs on your cloud provider. Comments are provided above each method to help you fill them up with desired REQUEST and RESPONSE parameters.
   • A sample provider implementation for these methods can be found here.
   • Fill in the required methods CreateMachine(), and DeleteMachine() methods.
   • Optionally fill in methods like GetMachineStatus(), InitializeMachine, ListMachines(), and GetVolumeIDs(). You may choose to fill these once the working of the required methods seems to be working.
   • GetVolumeIDs() expects VolumeIDs to be decoded from the volumeSpec based on the cloud provider.
   • There is also an OPTIONAL method GenerateMachineClassForMigration() that helps in migration of {ProviderSpecific}MachineClass to MachineClass CR (custom resource). This only makes sense if you have an existing implementation (in-tree) acting on different CRD types. You would like to migrate this. If not, you MUST return an error (machine error UNIMPLEMENTED) to avoid processing this step.
3. Perform validation of APIs that you have described and make it a part of your methods as required at each request.
4. Write unit tests to make it work with your implementation by running make test.

   make test
   

5. Tidy the go dependencies.

   make tidy
   

6. Update the sample YAML files on the kubernetes/ directory to provide sample files through which the working of the machine controller can be tested.
7. Update README.md to reflect any additional changes

Testing your code changes

Make sure $TARGET_KUBECONFIG points to the cluster where you wish to manage machines. Likewise, $CONTROL_NAMESPACE represents the namespaces where MCM is looking for machine CR objects, and $CONTROL_KUBECONFIG points to the cluster that holds these machine CRs.

1. On the first terminal running at $GOPATH/src/github.com/{github-org/your-github-username}/machine-controller-manager-provider-{provider-name},
   • Run the machine controller (driver) using the command below.

     make start
     

2. On the second terminal pointing to $GOPATH/src/github.com/gardener,
   • Clone the latest MCM code

     git clone git@github.com:gardener/machine-controller-manager.git
     

   • Navigate to the newly-created directory.

     cd machine-controller-manager
     

   • Deploy the required CRDs from the machine-controller-manager repo,

     kubectl apply -f kubernetes/crds
     

   • Run the machine-controller-manager in the master branch

     make start
     

3. On the third terminal pointing to $GOPATH/src/github.com/{github-org/your-github-username}/machine-controller-manager-provider-{provider-name}
   • Fill in the object files given below and deploy them as described below.
   • Deploy the machine-class

     kubectl apply -f kubernetes/machine-class.yaml
     

   • Deploy the kubernetes secret if required.

     kubectl apply -f kubernetes/secret.yaml
     

   • Deploy the machine object and make sure it joins the cluster successfully.

     kubectl apply -f kubernetes/machine.yaml
     

   • Once the machine joins, you can test by deploying a machine-deployment.
   • Deploy the machine-deployment object and make sure it joins the cluster successfully.

     kubectl apply -f kubernetes/machine-deployment.yaml
     

   • Make sure to delete both the machine and machine-deployment objects after use.

     kubectl delete -f kubernetes/machine.yaml
     kubectl delete -f kubernetes/machine-deployment.yaml
     

Releasing your docker image

1. Make sure you have logged into gcloud/docker using the CLI.
2. To release your docker image, run the following.

    make release IMAGE_REPOSITORY=<link-to-image-repo>

3. A sample kubernetes deploy file can be found at kubernetes/deployment.yaml. Update the same (with your desired MCM and MC images) to deploy your MCM pod.

2.6 - Deployment

Deploying the Machine Controller Manager into a Kubernetes cluster

• Deploying the Machine Controller Manager into a Kubernetes cluster
  • Prepare the cluster
  • Build the Docker image
  • Configuring optional parameters while deploying
  • Usage

As already mentioned, the Machine Controller Manager is designed to run as controller in a Kubernetes cluster. The existing source code can be compiled and tested on a local machine as described in Setting up a local development environment. You can deploy the Machine Controller Manager using the steps described below.

Prepare the cluster

• Connect to the remote kubernetes cluster where you plan to deploy the Machine Controller Manager using the kubectl. Set the environment variable KUBECONFIG to the path of the yaml file containing the cluster info.
• Now, create the required CRDs on the remote cluster using the following command,

$ kubectl apply -f kubernetes/crds

Build the Docker image

⚠️ Modify the Makefile to refer to your own registry.

• Run the build which generates the binary to bin/machine-controller-manager

$ make build

• Build docker image from latest compiled binary

$ make docker-image

• Push the last created docker image onto the online docker registry.

$ make push

• Now you can deploy this docker image to your cluster. A sample development file is provided. By default, the deployment manages the cluster it is running in. Optionally, the kubeconfig could also be passed as a flag as described in /kubernetes/deployment/out-of-tree/deployment.yaml. This is done when you want your controller running outside the cluster to be managed from.

$ kubectl apply -f kubernetes/deployment/out-of-tree/deployment.yaml

• Also deploy the required clusterRole and clusterRoleBindings

$ kubectl apply -f kubernetes/deployment/out-of-tree/clusterrole.yaml
$ kubectl apply -f kubernetes/deployment/out-of-tree/clusterrolebinding.yaml

Configuring optional parameters while deploying

Machine-controller-manager supports several configurable parameters while deploying. Refer to the following lines, to know how each parameter can be configured, and what it’s purpose is for.

Usage

To start using Machine Controller Manager, follow the links given at usage here.

2.7 - Integration Tests

Integration tests

Usage

General setup & configurations

Integration tests for machine-controller-manager-provider-{provider-name} can be executed manually by following below steps.

1. Clone the repository machine-controller-manager-provider-{provider-name} on the local system.
2. Navigate to machine-controller-manager-provider-{provider-name} directory and create a dev sub-directory in it.
3. If the tags on instances & associated resources on the provider are of String type (for example, GCP tags on its instances are of type String and not key-value pair) then add TAGS_ARE_STRINGS := true in the Makefile and export it. For GCP this has already been hard coded in the Makefile.

Running the tests

1. There is a rule test-integration in the Makefile of the provider repository, which can be used to start the integration test:

   $ make test-integration 
   

2. This will ask for additional inputs. Most of them are self explanatory except:

• The script assumes that both the control and target clusters are already being created.
• In case of non-gardener setup (control cluster is not a gardener seed), the name of the machineclass must be test-mc-v1 and the value of providerSpec.secretRef.name should be test-mc-secret.
• In case of azure, TARGET_CLUSTER_NAME must be same as the name of the Azure ResourceGroup for the cluster.
• If you are deploying the secret manually, a Secret named test-mc-secret (that contains the provider secret and cloud-config) in the default namespace of the Control Cluster should be created.

3. The controllers log files (mcm_process.log and mc_process.log) are stored in .ci/controllers-test/logs repo and can be used later.

Adding Integration Tests for new providers

For a new provider, Running Integration tests works with no changes. But for the orphan resource test cases to work correctly, the provider-specific API calls and the Resource Tracker Interface (RTI) should be implemented. Please check machine-controller-manager-provider-aws for reference.

Extending integration tests

• Update ControllerTests to be extend the testcases for all providers. Common testcases for machine|machineDeployment creation|deletion|scaling are packaged into ControllerTests.
• To extend the provider specfic test cases, the changes should be done in the machine-controller-manager-provider-{provider-name} repository. For example, to extended the testcases for machine-controller-manager-provider-aws, make changes to test/integration/controller/controller_test.go inside the machine-controller-manager-provider-aws repository. commons contains the Cluster and Clientset objects that makes it easy to extend the tests.

2.8 - Local Setup

Preparing the Local Development Setup (Mac OS X)

• Preparing the Local Development Setup (Mac OS X)
  • Installing Golang environment
  • Installing Docker (Optional)
  • Setup Docker Hub account (Optional)
  • Local development
    • Installing the Machine Controller Manager locally
  • Prepare the cluster
  • Getting started
  • Testing Machine Classes
  • Usage

Conceptionally, the Machine Controller Manager is designed to run in a container within a Pod inside a Kubernetes cluster. For development purposes, you can run the Machine Controller Manager as a Go process on your local machine. This process connects to your remote cluster to manage VMs for that cluster. That means that the Machine Controller Manager runs outside a Kubernetes cluster which requires providing a Kubeconfig in your local filesystem and point the Machine Controller Manager to it when running it (see below).

Although the following installation instructions are for Mac OS X, similar alternate commands could be found for any Linux distribution.

Installing Golang environment

Install the latest version of Golang (at least v1.8.3 is required) by using Homebrew:

$ brew install golang

In order to perform linting on the Go source code, install Golint:

$ go get -u golang.org/x/lint/golint

Installing Docker (Optional)

In case you want to build Docker images for the Machine Controller Manager you have to install Docker itself. We recommend using Docker for Mac OS X which can be downloaded from here.

Setup Docker Hub account (Optional)

Create a Docker hub account at Docker Hub if you don’t already have one.

Local development

⚠️ Before you start developing, please ensure to comply with the following requirements:

1. You have understood the principles of Kubernetes, and its components, what their purpose is and how they interact with each other.
2. You have understood the architecture of the Machine Controller Manager

The development of the Machine Controller Manager could happen by targeting any cluster. You basically need a Kubernetes cluster running on a set of machines. You just need the Kubeconfig file with the required access permissions attached to it.

Installing the Machine Controller Manager locally

Clone the repository from GitHub.

$ git clone git@github.com:gardener/machine-controller-manager.git
$ cd machine-controller-manager

Prepare the cluster

• Connect to the remote kubernetes cluster where you plan to deploy the Machine Controller Manager using kubectl. Set the environment variable KUBECONFIG to the path of the yaml file containing your cluster info
• Now, create the required CRDs on the remote cluster using the following command,

$ kubectl apply -f kubernetes/crds.yaml

Getting started

Setup and Restore with Gardener

Setup

In gardener access to static kubeconfig files is no longer supported due to security reasons. One needs to generate short-lived (max TTL = 1 day) admin kube configs for target and control clusters. A convenience script/Makefile target has been provided to do the required initial setup which includes:

• Creating a temporary directory where target and control kubeconfigs will be stored.
• Create a request to generate the short lived admin kubeconfigs. These are downloaded and stored in the temporary folder created above.
• In gardener clusters DWD (Dependency Watchdog) runs as an additional component which can interfere when MCM/CA is scaled down. To prevent that an annotation dependency-watchdog.gardener.cloud/ignore-scaling is added to machine-controller-manager deployment which prevents DWD from scaling up the deployment replicas.
• Scales down machine-controller-manager deployment in the control cluster to 0 replica.
• Creates the required .env file and populates required environment variables which are then used by the Makefile in both machine-controller-manager and in machine-controller-manager-provider-<provider-name> projects.
• Copies the generated and downloaded kubeconfig files for the target and control clusters to machine-controller-manager-provider-<provider-name> project as well.

To do the above you can either invoke make gardener-setup or you can directly invoke the script ./hack/gardener_local_setup.sh. If you invoke the script with -h or --help option then it will give you all CLI options that one can pass.

Restore

Once the testing is over you can invoke a convenience script/Makefile target which does the following:

• Removes all generated admin kubeconfig files from both machine-controller-manager and in machine-controller-manager-provider-<provider-name> projects.
• Removes the .env file that was generated as part of the setup from both machine-controller-manager and in machine-controller-manager-provider-<provider-name> projects.
• Scales up machine-controller-manager deployment in the control cluster back to 1 replica.
• Removes the annotation dependency-watchdog.gardener.cloud/ignore-scaling that was added to prevent DWD to scale up MCM.

To do the above you can either invoke make gardener-restore or you can directly invoke the script ./hack/gardener_local_restore.sh. If you invoke the script with -h or --help option then it will give you all CLI options that one can pass.

Setup and Restore without Gardener

Setup

If you are not running MCM components in a gardener cluster, then it is assumed that there is not going to be any DWD (Dependency Watchdog) component. A convenience script/Makefile target has been provided to the required initial setup which includes:

• Copies the provided control and target kubeconfig files to machine-controller-manager-provider-<provider-name> project.
• Scales down machine-controller-manager deployment in the control cluster to 0 replica.
• Creates the required .env file and populates required environment variables which are then used by the Makefile in both machine-controller-manager and in machine-controller-manager-provider-<provider-name> projects.

To do the above you can either invoke make non-gardener-setup or you can directly invoke the script ./hack/non_gardener_local_setup.sh. If you invoke the script with -h or --help option then it will give you all CLI options that one can pass.

Restore

Once the testing is over you can invoke a convenience script/Makefile target which does the following:

• Removes all provided kubeconfig files from both machine-controller-manager and in machine-controller-manager-provider-<provider-name> projects.
• Removes the .env file that was generated as part of the setup from both machine-controller-manager and in machine-controller-manager-provider-<provider-name> projects.
• Scales up machine-controller-manager deployment in the control cluster back to 1 replica.

To do the above you can either invoke make non-gardener-restore or you can directly invoke the script ./hack/non_gardener_local_restore.sh. If you invoke the script with -h or --help option then it will give you all CLI options that one can pass.

Once the setup is done then you can start the machine-controller-manager as a local process using the following Makefile target:

$ make start
I1227 11:08:19.963638   55523 controllermanager.go:204] Starting shared informers
I1227 11:08:20.766085   55523 controller.go:247] Starting machine-controller-manager

⚠️ The file dev/target-kubeconfig.yaml points to the cluster whose nodes you want to manage. dev/control-kubeconfig.yaml points to the cluster from where you want to manage the nodes from. However, dev/control-kubeconfig.yaml is optional.

The Machine Controller Manager should now be ready to manage the VMs in your kubernetes cluster.

⚠️ This is assuming that your MCM is built to manage machines for any in-tree supported providers. There is a new way to deploy and manage out of tree (external) support for providers whose development can be found here

Testing Machine Classes

To test the creation/deletion of a single instance for one particular machine class you can use the managevm cli. The corresponding INFRASTRUCTURE-machine-class.yaml and the INFRASTRUCTURE-secret.yaml need to be defined upfront. To build and run it

GO111MODULE=on go build -o managevm cmd/machine-controller-manager-cli/main.go
# create machine
./managevm --secret PATH_TO/INFRASTRUCTURE-secret.yaml --machineclass PATH_TO/INFRASTRUCTURE-machine-class.yaml --classkind INFRASTRUCTURE --machinename test
# delete machine
./managevm --secret PATH_TO/INFRASTRUCTURE-secret.yaml --machineclass PATH_TO/INFRASTRUCTURE-machine-class.yaml --classkind INFRASTRUCTURE --machinename test --machineid INFRASTRUCTURE:///REGION/INSTANCE_ID

Usage

To start using Machine Controller Manager, follow the links given at usage here.

2.9 - Machine

Creating/Deleting machines (VM)

• Creating/Deleting machines (VM)
  • Setting up your usage environment
  • Important :
  • Creating machine
  • Inspect status of machine
  • Delete machine

Setting up your usage environment

• Follow the steps described here

Important :

Make sure that the kubernetes/machine_objects/machine.yaml points to the same class name as the kubernetes/machine_classes/aws-machine-class.yaml.

Similarly kubernetes/machine_objects/aws-machine-class.yaml secret name and namespace should be same as that mentioned in kubernetes/secrets/aws-secret.yaml

Creating machine

• Modify kubernetes/machine_objects/machine.yaml as per your requirement and create the VM as shown below:

$ kubectl apply -f kubernetes/machine_objects/machine.yaml

You should notice that the Machine Controller Manager has immediately picked up your manifest and started to create a new machine by talking to the cloud provider.

• Check Machine Controller Manager machines in the cluster

$ kubectl get machine
NAME           STATUS    AGE
test-machine   Running   5m

A new machine is created with the name provided in the kubernetes/machine_objects/machine.yaml file.

• After a few minutes (~3 minutes for AWS), you should notice a new node joining the cluster. You can verify this by running:

$ kubectl get nodes
NAME                                         STATUS     AGE     VERSION
ip-10-250-14-52.eu-east-1.compute.internal.  Ready      1m      v1.8.0

This shows that a new node has successfully joined the cluster.

Inspect status of machine

To inspect the status of any created machine, run the command given below.

$ kubectl get machine test-machine -o yaml

apiVersion: machine.sapcloud.io/v1alpha1
kind: Machine
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"machine.sapcloud.io/v1alpha1","kind":"Machine","metadata":{"annotations":{},"labels":{"test-label":"test-label"},"name":"test-machine","namespace":""},"spec":{"class":{"kind":"AWSMachineClass","name":"test-aws"}}}
  clusterName: ""
  creationTimestamp: 2017-12-27T06:58:21Z
  finalizers:
  - machine.sapcloud.io/operator
  generation: 0
  initializers: null
  labels:
    node: ip-10-250-14-52.eu-east-1.compute.internal
    test-label: test-label
  name: test-machine
  namespace: ""
  resourceVersion: "12616948"
  selfLink: /apis/machine.sapcloud.io/v1alpha1/test-machine
  uid: 535e596c-ead3-11e7-a6c0-828f843e4186
spec:
  class:
    kind: AWSMachineClass
    name: test-aws
  providerID: aws:///eu-east-1/i-00bef3f2618ffef23
status:
  conditions:
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T06:59:16Z
    message: kubelet has sufficient disk space available
    reason: KubeletHasSufficientDisk
    status: "False"
    type: OutOfDisk
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T06:59:16Z
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T06:59:16Z
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T07:00:06Z
    message: kubelet is posting ready status
    reason: KubeletReady
    status: "True"
    type: Ready
  currentStatus:
    lastUpdateTime: 2017-12-27T07:00:06Z
    phase: Running
  lastOperation:
    description: Machine is now ready
    lastUpdateTime: 2017-12-27T07:00:06Z
    state: Successful
    type: Create
  node: ip-10-250-14-52.eu-west-1.compute.internal

Delete machine

To delete the VM using the kubernetes/machine_objects/machine.yaml as shown below

$ kubectl delete -f kubernetes/machine_objects/machine.yaml

Now the Machine Controller Manager picks up the manifest immediately and starts to delete the existing VM by talking to the cloud provider. The node should be detached from the cluster in a few minutes (~1min for AWS).

2.10 - Machine Deployment

Maintaining machine replicas using machines-deployments

• Maintaining machine replicas using machines-deployments
  • Setting up your usage environment
    • Important ⚠️
  • Creating machine-deployment
  • Inspect status of machine-deployment
  • Health monitoring
  • Update your machines
    • Inspect existing cluster configuration
    • Perform a rolling update
    • Re-check cluster configuration
    • More variants of updates
  • Undo an update
  • Pause an update
  • Delete machine-deployment

Setting up your usage environment

Follow the steps described here

Important ⚠️

Make sure that the kubernetes/machine_objects/machine-deployment.yaml points to the same class name as the kubernetes/machine_classes/aws-machine-class.yaml.

Similarly kubernetes/machine_classes/aws-machine-class.yaml secret name and namespace should be same as that mentioned in kubernetes/secrets/aws-secret.yaml

Creating machine-deployment

• Modify kubernetes/machine_objects/machine-deployment.yaml as per your requirement. Modify the number of replicas to the desired number of machines. Then, create an machine-deployment.

$ kubectl apply -f kubernetes/machine_objects/machine-deployment.yaml

Now the Machine Controller Manager picks up the manifest immediately and starts to create a new machines based on the number of replicas you have provided in the manifest.

• Check Machine Controller Manager machine-deployments in the cluster

$ kubectl get machinedeployment
NAME                      READY   DESIRED   UP-TO-DATE   AVAILABLE   AGE
test-machine-deployment   3       3         3            0           10m

You will notice a new machine-deployment with your given name

• Check Machine Controller Manager machine-sets in the cluster

$ kubectl get machineset
NAME                                 DESIRED   CURRENT   READY   AGE
test-machine-deployment-5bc6dd7c8f   3         3         0       10m

You will notice a new machine-set backing your machine-deployment

• Check Machine Controller Manager machines in the cluster

$ kubectl get machine
NAME                                       STATUS    AGE
test-machine-deployment-5bc6dd7c8f-5d24b   Pending   5m
test-machine-deployment-5bc6dd7c8f-6mpn4   Pending   5m
test-machine-deployment-5bc6dd7c8f-dpt2q   Pending   5m

Now you will notice N (number of replicas specified in the manifest) new machines whose name are prefixed with the machine-deployment object name that you created.

• After a few minutes (~3 minutes for AWS), you would see that new nodes have joined the cluster. You can see this using

$  kubectl get nodes
NAME                                          STATUS    AGE       VERSION
ip-10-250-20-19.eu-west-1.compute.internal    Ready     1m        v1.8.0
ip-10-250-27-123.eu-west-1.compute.internal   Ready     1m        v1.8.0
ip-10-250-31-80.eu-west-1.compute.internal    Ready     1m        v1.8.0

This shows how new nodes have joined your cluster

Inspect status of machine-deployment

To inspect the status of any created machine-deployment run the command below,

$ kubectl get machinedeployment test-machine-deployment -o yaml

You should get the following output.

apiVersion: machine.sapcloud.io/v1alpha1
kind: MachineDeployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"machine.sapcloud.io/v1alpha1","kind":"MachineDeployment","metadata":{"annotations":{},"name":"test-machine-deployment","namespace":""},"spec":{"minReadySeconds":200,"replicas":3,"selector":{"matchLabels":{"test-label":"test-label"}},"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"test-label":"test-label"}},"spec":{"class":{"kind":"AWSMachineClass","name":"test-aws"}}}}}
  clusterName: ""
  creationTimestamp: 2017-12-27T08:55:56Z
  generation: 0
  initializers: null
  name: test-machine-deployment
  namespace: ""
  resourceVersion: "12634168"
  selfLink: /apis/machine.sapcloud.io/v1alpha1/test-machine-deployment
  uid: c0b488f7-eae3-11e7-a6c0-828f843e4186
spec:
  minReadySeconds: 200
  replicas: 3
  selector:
    matchLabels:
      test-label: test-label
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        test-label: test-label
    spec:
      class:
        kind: AWSMachineClass
        name: test-aws
status:
  availableReplicas: 3
  conditions:
  - lastTransitionTime: 2017-12-27T08:57:22Z
    lastUpdateTime: 2017-12-27T08:57:22Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  readyReplicas: 3
  replicas: 3
  updatedReplicas: 3

Health monitoring

Health monitor is also applied similar to how it’s described for machine-sets

Update your machines

Let us consider the scenario where you wish to update all nodes of your cluster from t2.xlarge machines to m5.xlarge machines. Assume that your current test-aws has its spec.machineType: t2.xlarge and your deployment test-machine-deployment points to this AWSMachineClass.

Inspect existing cluster configuration

• Check Nodes present in the cluster

$ kubectl get nodes
NAME                                          STATUS    AGE       VERSION
ip-10-250-20-19.eu-west-1.compute.internal    Ready     1m        v1.8.0
ip-10-250-27-123.eu-west-1.compute.internal   Ready     1m        v1.8.0
ip-10-250-31-80.eu-west-1.compute.internal    Ready     1m        v1.8.0

• Check Machine Controller Manager machine-sets in the cluster. You will notice one machine-set backing your machine-deployment

$ kubectl get machineset
NAME                                 DESIRED   CURRENT   READY   AGE
test-machine-deployment-5bc6dd7c8f   3         3         3       10m

• Login to your cloud provider (AWS). In the VM management console, you will find N VMs created of type t2.xlarge.

Perform a rolling update

To update this machine-deployment VMs to m5.xlarge, we would do the following:

• Copy your existing aws-machine-class.yaml

cp kubernetes/machine_classes/aws-machine-class.yaml kubernetes/machine_classes/aws-machine-class-new.yaml

• Modify aws-machine-class-new.yaml, and update its metadata.name: test-aws2 and spec.machineType: m5.xlarge
• Now create this modified MachineClass

kubectl apply -f kubernetes/machine_classes/aws-machine-class-new.yaml

• Edit your existing machine-deployment

kubectl edit machinedeployment test-machine-deployment

• Update from spec.template.spec.class.name: test-aws to spec.template.spec.class.name: test-aws2

Re-check cluster configuration

After a few minutes (~3mins)

• Check nodes present in cluster now. They are different nodes.

$ kubectl get nodes
NAME                                          STATUS    AGE       VERSION
ip-10-250-11-171.eu-west-1.compute.internal   Ready     4m        v1.8.0
ip-10-250-17-213.eu-west-1.compute.internal   Ready     5m        v1.8.0
ip-10-250-31-81.eu-west-1.compute.internal    Ready     5m        v1.8.0

• Check Machine Controller Manager machine-sets in the cluster. You will notice two machine-sets backing your machine-deployment

$ kubectl get machineset
NAME                                 DESIRED   CURRENT   READY   AGE
test-machine-deployment-5bc6dd7c8f   0         0         0       1h
test-machine-deployment-86ff45cc5    3         3         3       20m

• Login to your cloud provider (AWS). In the VM management console, you will find N VMs created of type t2.xlarge in terminated state, and N new VMs of type m5.xlarge in running state.

This shows how a rolling update of a cluster from nodes with t2.xlarge to m5.xlarge went through.

More variants of updates

• The above demonstration was a simple use case. This could be more complex like - updating the system disk image versions/ kubelet versions/ security patches etc.
• You can also play around with the maxSurge and maxUnavailable fields in machine-deployment.yaml
• You can also change the update strategy from rollingupdate to recreate

Undo an update

• Edit the existing machine-deployment

$ kubectl edit machinedeployment test-machine-deployment

• Edit the deployment to have this new field of spec.rollbackTo.revision: 0 as shown as comments in kubernetes/machine_objects/machine-deployment.yaml
• This will undo your update to the previous version.

Pause an update

• You can also pause the update while update is going on by editing the existing machine-deployment

$ kubectl edit machinedeployment test-machine-deployment

• Edit the deployment to have this new field of spec.paused: true as shown as comments in kubernetes/machine_objects/machine-deployment.yaml

• This will pause the rollingUpdate if it’s in process

• To resume the update, edit the deployment as mentioned above and remove the field spec.paused: true updated earlier

Delete machine-deployment

• To delete the VM using the kubernetes/machine_objects/machine-deployment.yaml

$ kubectl delete -f kubernetes/machine_objects/machine-deployment.yaml

The Machine Controller Manager picks up the manifest and starts to delete the existing VMs by talking to the cloud provider. The nodes should be detached from the cluster in a few minutes (~1min for AWS).

2.11 - Machine Error Codes

Machine Error code handling

Notational Conventions

The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” are to be interpreted as described in RFC 2119 (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels”, BCP 14, RFC 2119, March 1997).

The key words “unspecified”, “undefined”, and “implementation-defined” are to be interpreted as described in the rationale for the C99 standard.

An implementation is not compliant if it fails to satisfy one or more of the MUST, REQUIRED, or SHALL requirements for the protocols it implements. An implementation is compliant if it satisfies all the MUST, REQUIRED, and SHALL requirements for the protocols it implements.

Terminology

Pre-requisite

MachineClass Resources

MCM introduces the CRD MachineClass. This is a blueprint for creating machines that join a certain cluster as nodes in a certain role. The provider only works with MachineClass resources that have the structure described here.

ProviderSpec

The MachineClass resource contains a providerSpec field that is passed in the ProviderSpec request field to CMI methods such as CreateMachine. The ProviderSpec can be thought of as a machine template from which the VM specification must be adopted. It can contain key-value pairs of these specs. An example for these key-value pairs are given below.

Most of the ProviderSpec fields are not mandatory. If not specified, the provider passes an empty value in the respective Create VM parameter.

The tags can be used to map a VM to its corresponding machine object’s Name

The ProviderSpec is validated by methods that receive it as a request field for presence of all mandatory parameters and tags, and for validity of all parameters.

Secrets

The MachineClass resource also contains a secretRef field that contains a reference to a secret. The keys of this secret are passed in the Secrets request field to CMI methods.

The secret can contain sensitive data such as

• cloud-credentials secret data used to authenticate at the provider
• cloud-init scripts used to initialize a new VM. The cloud-init script is expected to contain scripts to initialize the Kubelet and make it join the cluster.

Identifying Cluster Machines

To implement certain methods, the provider should be able to identify all machines associated with a particular Kubernetes cluster. This can be achieved using one/more of the below mentioned ways:

• Names of VMs created by the provider are prefixed by the cluster ID specified in the ProviderSpec.
• VMs created by the provider are tagged with the special tags like kubernetes.io/cluster (for the cluster ID) and kubernetes.io/role (for the role), specified in the ProviderSpec.
• Mapping Resource Groups to individual cluster.

Error Scheme

All provider API calls defined in this spec MUST return a machine error status, which is very similar to standard machine status.

Machine Provider Interface

• The provider MUST have a unique way to map a machine object to a VM which triggers the deletion for the corresponding VM backing the machine object.
• The provider SHOULD have a unique way to map the ProviderSpec of a machine-class to a unique Cluster. This avoids deletion of other machines, not backed by the MCM.

CreateMachine

A Provider is REQUIRED to implement this interface method. This interface method will be called by the MCM to provision a new VM on behalf of the requesting machine object.

• This call requests the provider to create a VM backing the machine-object.

• If VM backing the Machine.Name already exists, and is compatible with the specified Machine object in the CreateMachineRequest, the Provider MUST reply 0 OK with the corresponding CreateMachineResponse.

• The provider can OPTIONALLY make use of the MachineClass supplied in the MachineClass in the CreateMachineRequest to communicate with the provider.

• The provider can OPTIONALLY make use of the secrets supplied in the Secret in the CreateMachineRequest to communicate with the provider.

• The provider can OPTIONALLY make use of the Status.LastKnownState in the Machine object to decode the state of the VM operation based on the last known state of the VM. This can be useful to restart/continue an operations which are mean’t to be atomic.

• The provider MUST have a unique way to map a machine object to a VM. This could be implicitly provided by the provider by letting you set VM-names (or) could be explicitly specified by the provider using appropriate tags to map the same.

• This operation SHOULD be idempotent.

• The CreateMachineResponse returned by this method is expected to return

  • ProviderID that uniquely identifys the VM at the provider. This is expected to match with the node.Spec.ProviderID on the node object.
  • NodeName that is the expected name of the machine when it joins the cluster. It must match with the node name.
  • LastKnownState is an OPTIONAL field that can store details of the last known state of the VM. It can be used by future operation calls to determine current infrastucture state. This state is saved on the machine object.

// CreateMachine call is responsible for VM creation on the provider
CreateMachine(context.Context, *CreateMachineRequest) (*CreateMachineResponse, error)

// CreateMachineRequest is the create request for VM creation
type CreateMachineRequest struct {
	// Machine object from whom VM is to be created
	Machine *v1alpha1.Machine

	// MachineClass backing the machine object
	MachineClass *v1alpha1.MachineClass

	//  Secret backing the machineClass object
	Secret *corev1.Secret
}

// CreateMachineResponse is the create response for VM creation
type CreateMachineResponse struct {
	// ProviderID is the unique identification of the VM at the cloud provider.
	// ProviderID typically matches with the node.Spec.ProviderID on the node object.
	// Eg: gce://project-name/region/vm-ID
	ProviderID string

	// NodeName is the name of the node-object registered to kubernetes.
	NodeName string

	// LastKnownState represents the last state of the VM during an creation/deletion error
	LastKnownState string
}

CreateMachine Errors

If the provider is unable to complete the CreateMachine call successfully, it MUST return a non-ok ginterface method code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code. The MCM MUST implement the specified error recovery behavior when it encounters the machine error code.

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

InitializeMachine

Provider can OPTIONALLY implement this driver call. Else should return a UNIMPLEMENTED status in error.
This interface method will be called by the MCM to initialize a new VM just after creation. This can be used to configure network configuration etc.

• This call requests the provider to initialize a newly created VM backing the machine-object.
• The InitializeMachineResponse returned by this method is expected to return
  • ProviderID that uniquely identifys the VM at the provider. This is expected to match with the node.Spec.ProviderID on the node object.
  • NodeName that is the expected name of the machine when it joins the cluster. It must match with the node name.

// InitializeMachine call is responsible for VM initialization on the provider.
InitializeMachine(context.Context, *InitializeMachineRequest) (*InitializeMachineResponse, error)

// InitializeMachineRequest encapsulates params for the VM Initialization operation (Driver.InitializeMachine).
type InitializeMachineRequest struct {
	// Machine object representing VM that must be initialized
	Machine *v1alpha1.Machine

	// MachineClass backing the machine object
	MachineClass *v1alpha1.MachineClass

	// Secret backing the machineClass object
	Secret *corev1.Secret
}

// InitializeMachineResponse is the response for VM instance initialization (Driver.InitializeMachine).
type InitializeMachineResponse struct {
	// ProviderID is the unique identification of the VM at the cloud provider.
	// ProviderID typically matches with the node.Spec.ProviderID on the node object.
	// Eg: gce://project-name/region/vm-ID
	ProviderID string

	// NodeName is the name of the node-object registered to kubernetes.
	NodeName string
}

InitializeMachine Errors

If the provider is unable to complete the InitializeMachine call successfully, it MUST return a non-ok machine code in the machine status.

If the conditions defined below are encountered, the provider MUST return the specified machine error code. The MCM MUST implement the specified error recovery behavior when it encounters the machine error code.

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

DeleteMachine

A Provider is REQUIRED to implement this driver call. This driver call will be called by the MCM to deprovision/delete/terminate a VM backed by the requesting machine object.

• If a VM corresponding to the specified machine-object’s name does not exist or the artifacts associated with the VM do not exist anymore (after deletion), the Provider MUST reply 0 OK.

• The provider SHALL only act on machines belonging to the cluster-id/cluster-name obtained from the ProviderSpec.

• The provider can OPTIONALY make use of the secrets supplied in the Secrets map in the DeleteMachineRequest to communicate with the provider.

• The provider can OPTIONALY make use of the Spec.ProviderID map in the Machine object.

• The provider can OPTIONALLY make use of the Status.LastKnownState in the Machine object to decode the state of the VM operation based on the last known state of the VM. This can be useful to restart/continue an operations which are mean’t to be atomic.

• This operation SHOULD be idempotent.

• The provider must have a unique way to map a machine object to a VM which triggers the deletion for the corresponding VM backing the machine object.

• The DeleteMachineResponse returned by this method is expected to return

  • LastKnownState is an OPTIONAL field that can store details of the last known state of the VM. It can be used by future operation calls to determine current infrastucture state. This state is saved on the machine object.

// DeleteMachine call is responsible for VM deletion/termination on the provider
DeleteMachine(context.Context, *DeleteMachineRequest) (*DeleteMachineResponse, error)

// DeleteMachineRequest is the delete request for VM deletion
type DeleteMachineRequest struct {
	// Machine object from whom VM is to be deleted
	Machine *v1alpha1.Machine

	// MachineClass backing the machine object
	MachineClass *v1alpha1.MachineClass

	// Secret backing the machineClass object
	Secret *corev1.Secret
}

// DeleteMachineResponse is the delete response for VM deletion
type DeleteMachineResponse struct {
	// LastKnownState represents the last state of the VM during an creation/deletion error
	LastKnownState string
}

DeleteMachine Errors

If the provider is unable to complete the DeleteMachine call successfully, it MUST return a non-ok machine code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code.

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

GetMachineStatus

A Provider can OPTIONALLY implement this driver call. Else should return a UNIMPLEMENTED status in error. This call will be invoked by the MC to get the status of a machine. This optional driver call helps in optimizing the working of the provider by avoiding unwanted calls to CreateMachine() and DeleteMachine().

• If a VM corresponding to the specified machine object’s Machine.Name exists on provider the GetMachineStatusResponse fields are to be filled similar to the CreateMachineResponse.
• The provider SHALL only act on machines belonging to the cluster-id/cluster-name obtained from the ProviderSpec.
• The provider can OPTIONALY make use of the secrets supplied in the Secrets map in the GetMachineStatusRequest to communicate with the provider.
• The provider can OPTIONALY make use of the VM unique ID (returned by the provider on machine creation) passed in the ProviderID map in the GetMachineStatusRequest.
• This operation MUST be idempotent.

// GetMachineStatus call get's the status of the VM backing the machine object on the provider
GetMachineStatus(context.Context, *GetMachineStatusRequest) (*GetMachineStatusResponse, error)

// GetMachineStatusRequest is the get request for VM info
type GetMachineStatusRequest struct {
	// Machine object from whom VM status is to be fetched
	Machine *v1alpha1.Machine

	// MachineClass backing the machine object
	MachineClass *v1alpha1.MachineClass

	//  Secret backing the machineClass object
	Secret *corev1.Secret
}

// GetMachineStatusResponse is the get response for VM info
type GetMachineStatusResponse struct {
	// ProviderID is the unique identification of the VM at the cloud provider.
	// ProviderID typically matches with the node.Spec.ProviderID on the node object.
	// Eg: gce://project-name/region/vm-ID
	ProviderID string

	// NodeName is the name of the node-object registered to kubernetes.
	NodeName string
}

GetMachineStatus Errors

If the provider is unable to complete the GetMachineStatus call successfully, it MUST return a non-ok machine code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code.

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

ListMachines

A Provider can OPTIONALLY implement this driver call. Else should return a UNIMPLEMENTED status in error. The Provider SHALL return the information about all the machines associated with the MachineClass. Make sure to use appropriate filters to achieve the same to avoid data transfer overheads. This optional driver call helps in cleaning up orphan VMs present in the cluster. If not implemented, any orphan VM that might have been created incorrectly by the MCM/Provider (due to bugs in code/infra) might require manual clean up.

• If the Provider succeeded in returning a list of Machine.Name with their corresponding ProviderID, then return 0 OK.
• The ListMachineResponse contains a map of MachineList whose
  • Key is expected to contain the ProviderID &
  • Value is expected to contain the Machine.Name corresponding to it’s kubernetes machine CR object
• The provider can OPTIONALY make use of the secrets supplied in the Secrets map in the ListMachinesRequest to communicate with the provider.

// ListMachines lists all the machines that might have been created by the supplied machineClass
ListMachines(context.Context, *ListMachinesRequest) (*ListMachinesResponse, error)

// ListMachinesRequest is the request object to get a list of VMs belonging to a machineClass
type ListMachinesRequest struct {
	// MachineClass object
	MachineClass *v1alpha1.MachineClass

	// Secret backing the machineClass object
	Secret *corev1.Secret
}

// ListMachinesResponse is the response object of the list of VMs belonging to a machineClass
type ListMachinesResponse struct {
	// MachineList is the map of list of machines. Format for the map should be <ProviderID, MachineName>.
	MachineList map[string]string
}

ListMachines Errors

If the provider is unable to complete the ListMachines call successfully, it MUST return a non-ok machine code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code. The MCM MUST implement the specified error recovery behavior when it encounters the machine error code.

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

GetVolumeIDs

A Provider can OPTIONALLY implement this driver call. Else should return a UNIMPLEMENTED status in error. This driver call will be called by the MCM to get the VolumeIDs for the list of PersistentVolumes (PVs) supplied. This OPTIONAL (but recommended) driver call helps in serailzied eviction of pods with PVs while draining of machines. This implies applications backed by PVs would be evicted one by one, leading to shorter application downtimes.

• On succesful returnal of a list of Volume-IDs for all supplied PVSpecs, the Provider MUST reply 0 OK.
• The GetVolumeIDsResponse is expected to return a repeated list of strings consisting of the VolumeIDs for PVSpec that could be extracted.
• If for any PV the Provider wasn’t able to identify the Volume-ID, the provider MAY chose to ignore it and return the Volume-IDs for the rest of the PVs for whom the Volume-ID was found.
• Getting the VolumeID from the PVSpec depends on the Cloud-provider. You can extract this information by parsing the PVSpec based on the ProviderType
  • https://github.com/kubernetes/api/blob/release-1.15/core/v1/types.go#L297-L339
  • https://github.com/kubernetes/api/blob/release-1.15//core/v1/types.go#L175-L257
• This operation MUST be idempotent.

// GetVolumeIDsRequest is the request object to get a list of VolumeIDs for a PVSpec
type GetVolumeIDsRequest struct {
	// PVSpecsList is a list of PV specs for whom volume-IDs are required
	// Plugin should parse this raw data into pre-defined list of PVSpecs
	PVSpecs []*corev1.PersistentVolumeSpec
}

// GetVolumeIDsResponse is the response object of the list of VolumeIDs for a PVSpec
type GetVolumeIDsResponse struct {
	// VolumeIDs is a list of VolumeIDs.
	VolumeIDs []string
}

GetVolumeIDs Errors

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

GenerateMachineClassForMigration

A Provider SHOULD implement this driver call, else it MUST return a UNIMPLEMENTED status in error. This driver call will be called by the Machine Controller to try to perform a machineClass migration for an unknown machineClass Kind. This helps in migration of one kind of machineClass to another kind. For instance an machineClass custom resource of AWSMachineClass to MachineClass.

• On successful generation of machine class the Provider MUST reply 0 OK (or) nil error.
• GenerateMachineClassForMigrationRequest expects the provider-specific machine class (eg. AWSMachineClass) to be supplied as the ProviderSpecificMachineClass. The provider is responsible for unmarshalling the golang struct. It also passes a reference to an existing MachineClass object.
• The provider is expected to fill in thisMachineClass object based on the conversions.
• An optional ClassSpec containing the type ClassSpec struct is also provided to decode the provider info.
• GenerateMachineClassForMigration is only responsible for filling up the passed MachineClass object.
• The task of creating the new CR of the new kind (MachineClass) with the same name as the previous one and also annotating the old machineClass CR with a migrated annotation and migrating existing references is done by the calling library implicitly.
• This operation MUST be idempotent.

// GenerateMachineClassForMigrationRequest is the request for generating the generic machineClass
// for the provider specific machine class
type GenerateMachineClassForMigrationRequest struct {
	// ProviderSpecificMachineClass is provider specfic machine class object.
	// E.g. AWSMachineClass
	ProviderSpecificMachineClass interface{}
	// MachineClass is the machine class object generated that is to be filled up
	MachineClass *v1alpha1.MachineClass
	// ClassSpec contains the class spec object to determine the machineClass kind
	ClassSpec *v1alpha1.ClassSpec
}

// GenerateMachineClassForMigrationResponse is the response for generating the generic machineClass
// for the provider specific machine class
type GenerateMachineClassForMigrationResponse struct{}

MigrateMachineClass Errors

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

Configuration and Operation

Supervised Lifecycle Management

• For Providers packaged in software form:
  • Provider Packages SHOULD use a well-documented container image format (e.g., Docker, OCI).
  • The chosen package image format MAY expose configurable Provider properties as environment variables, unless otherwise indicated in the section below. Variables so exposed SHOULD be assigned default values in the image manifest.
  • A Provider Supervisor MAY programmatically evaluate or otherwise scan a Provider Package’s image manifest in order to discover configurable environment variables.
  • A Provider SHALL NOT assume that an operator or Provider Supervisor will scan an image manifest for environment variables.

Environment Variables

• Variables defined by this specification SHALL be identifiable by their MC_ name prefix.
• Configuration properties not defined by the MC specification SHALL NOT use the same MC_ name prefix; this prefix is reserved for common configuration properties defined by the MC specification.
• The Provider Supervisor SHOULD supply all RECOMMENDED MC environment variables to a Provider.
• The Provider Supervisor SHALL supply all REQUIRED MC environment variables to a Provider.

Logging

• Providers SHOULD generate log messages to ONLY standard output and/or standard error.
  • In this case the Provider Supervisor SHALL assume responsibility for all log lifecycle management.
• Provider implementations that deviate from the above recommendation SHALL clearly and unambiguously document the following:
  • Logging configuration flags and/or variables, including working sample configurations.
  • Default log destination(s) (where do the logs go if no configuration is specified?)
  • Log lifecycle management ownership and related guidance (size limits, rate limits, rolling, archiving, expunging, etc.) applicable to the logging mechanism embedded within the Provider.
• Providers SHOULD NOT write potentially sensitive data to logs (e.g. secrets).

Available Services

• Provider Packages MAY support all or a subset of CMI services; service combinations MAY be configurable at runtime by the Provider Supervisor.
  • This specification does not dictate the mechanism by which mode of operation MUST be discovered, and instead places that burden upon the VM Provider.
• Misconfigured provider software SHOULD fail-fast with an OS-appropriate error code.

Linux Capabilities

• Providers SHOULD clearly document any additionally required capabilities and/or security context.

Cgroup Isolation

• A Provider MAY be constrained by cgroups.

Resource Requirements

• VM Providers SHOULD unambiguously document all of a Provider’s resource requirements.

Deploying

• Recommended: The MCM and Provider are typically expected to run as two containers inside a common Pod.
• However, for the security reasons they could execute on seperate Pods provided they have a secure way to exchange data between them.

2.12 - Machine Set

Maintaining machine replicas using machines-sets

• Maintaining machine replicas using machines-sets
  • Setting up your usage environment
  • Important ⚠️
  • Creating machine-set
  • Inspect status of machine-set
  • Health monitoring
  • Delete machine-set

Setting up your usage environment

• Follow the steps described here

Important ⚠️

Make sure that the kubernetes/machines_objects/machine-set.yaml points to the same class name as the kubernetes/machine_classes/aws-machine-class.yaml.

Similarly kubernetes/machine_classes/aws-machine-class.yaml secret name and namespace should be same as that mentioned in kubernetes/secrets/aws-secret.yaml

Creating machine-set

• Modify kubernetes/machine_objects/machine-set.yaml as per your requirement. You can modify the number of replicas to the desired number of machines. Then, create an machine-set:

$ kubectl apply -f kubernetes/machine_objects/machine-set.yaml

You should notice that the Machine Controller Manager has immediately picked up your manifest and started to create a new machines based on the number of replicas you have provided in the manifest.

• Check Machine Controller Manager machine-sets in the cluster

$ kubectl get machineset
NAME               DESIRED   CURRENT   READY   AGE
test-machine-set   3         3         0       1m

You will see a new machine-set with your given name

• Check Machine Controller Manager machines in the cluster:

$ kubectl get machine
NAME                     STATUS    AGE
test-machine-set-b57zs   Pending   5m
test-machine-set-c4bg8   Pending   5m
test-machine-set-kvskg   Pending   5m

Now you will see N (number of replicas specified in the manifest) new machines whose names are prefixed with the machine-set object name that you created.

• After a few minutes (~3 minutes for AWS), you should notice new nodes joining the cluster. You can verify this by running:

$ kubectl get nodes
NAME                                         STATUS    AGE       VERSION
ip-10-250-0-234.eu-west-1.compute.internal   Ready     3m        v1.8.0
ip-10-250-15-98.eu-west-1.compute.internal   Ready     3m        v1.8.0
ip-10-250-6-21.eu-west-1.compute.internal    Ready     2m        v1.8.0

This shows how new nodes have joined your cluster

Inspect status of machine-set

• To inspect the status of any created machine-set run the following command:

$ kubectl get machineset test-machine-set -o yaml

apiVersion: machine.sapcloud.io/v1alpha1
kind: MachineSet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"machine.sapcloud.io/v1alpha1","kind":"MachineSet","metadata":{"annotations":{},"name":"test-machine-set","namespace":"","test-label":"test-label"},"spec":{"minReadySeconds":200,"replicas":3,"selector":{"matchLabels":{"test-label":"test-label"}},"template":{"metadata":{"labels":{"test-label":"test-label"}},"spec":{"class":{"kind":"AWSMachineClass","name":"test-aws"}}}}}
  clusterName: ""
  creationTimestamp: 2017-12-27T08:37:42Z
  finalizers:
  - machine.sapcloud.io/operator
  generation: 0
  initializers: null
  name: test-machine-set
  namespace: ""
  resourceVersion: "12630893"
  selfLink: /apis/machine.sapcloud.io/v1alpha1/test-machine-set
  uid: 3469faaa-eae1-11e7-a6c0-828f843e4186
spec:
  machineClass: {}
  minReadySeconds: 200
  replicas: 3
  selector:
    matchLabels:
      test-label: test-label
  template:
    metadata:
      creationTimestamp: null
      labels:
        test-label: test-label
    spec:
      class:
        kind: AWSMachineClass
        name: test-aws
status:
  availableReplicas: 3
  fullyLabeledReplicas: 3
  machineSetCondition: null
  lastOperation:
    lastUpdateTime: null
  observedGeneration: 0
  readyReplicas: 3
  replicas: 3

Health monitoring

• If you try to delete/terminate any of the machines backing the machine-set by either talking to the Machine Controller Manager or from the cloud provider, the Machine Controller Manager recreates a matching healthy machine to replace the deleted machine.
• Similarly, if any of your machines are unreachable or in an unhealthy state (kubelet not ready / disk pressure) for longer than the configured timeout (~ 5mins), the Machine Controller Manager recreates the nodes to replace the unhealthy nodes.

Delete machine-set

• To delete the VM using the kubernetes/machine_objects/machine-set.yaml:

$ kubectl delete -f kubernetes/machine-set.yaml

Now the Machine Controller Manager has immediately picked up your manifest and started to delete the existing VMs by talking to the cloud provider. Your nodes should be detached from the cluster in a few minutes (~1min for AWS).

2.13 - Prerequisite

Setting up the usage environment

• Setting up the usage environment
  • Important ⚠️
  • Set KUBECONFIG
  • Replace provider credentials and desired VM configurations
  • Deploy required CRDs and Objects
  • Check current cluster state

Important ⚠️

All paths are relative to the root location of this project repository.

Run the Machine Controller Manager either as described in Setting up a local development environment or Deploying the Machine Controller Manager into a Kubernetes cluster.

Make sure that the following steps are run before managing machines/ machine-sets/ machine-deploys.

Set KUBECONFIG

Using the existing Kubeconfig, open another Terminal panel/window with the KUBECONFIG environment variable pointing to this Kubeconfig file as shown below,

$ export KUBECONFIG=<PATH_TO_REPO>/dev/kubeconfig.yaml

Replace provider credentials and desired VM configurations

Open kubernetes/machine_classes/aws-machine-class.yaml and replace required values there with the desired VM configurations.

Similarily open kubernetes/secrets/aws-secret.yaml and replace - userData, providerAccessKeyId, providerSecretAccessKey with base64 encoded values of cloudconfig file, AWS access key id, and AWS secret access key respectively. Use the following command to get the base64 encoded value of your details

$ echo "sample-cloud-config" | base64
base64-encoded-cloud-config

Do the same for your access key id and secret access key.

Deploy required CRDs and Objects

Create all the required CRDs in the cluster using kubernetes/crds.yaml

$ kubectl apply -f kubernetes/crds.yaml

Create the class template that will be used as an machine template to create VMs using kubernetes/machine_classes/aws-machine-class.yaml

$ kubectl apply -f kubernetes/machine_classes/aws-machine-class.yaml

Create the secret used for the cloud credentials and cloudconfig using kubernetes/secrets/aws-secret.yaml

$ kubectl apply -f kubernetes/secrets/aws-secret.yaml

Check current cluster state

Get to know the current cluster state using the following commands,

• Checking aws-machine-class in the cluster

$ kubectl get awsmachineclass
NAME       MACHINE TYPE   AMI          AGE
test-aws   t2.large       ami-123456   5m

• Checking kubernetes secrets in the cluster

$ kubectl get secret
NAME                  TYPE                                  DATA      AGE
test-secret           Opaque                                3         21h

• Checking kubernetes nodes in the cluster

$ kubectl get nodes

Lists the default set of nodes attached to your cluster

• Checking Machine Controller Manager machines in the cluster

$ kubectl get machine
No resources found.

• Checking Machine Controller Manager machine-sets in the cluster

$ kubectl get machineset
No resources found.

• Checking Machine Controller Manager machine-deploys in the cluster

$ kubectl get machinedeployment
No resources found.

2.14 - Testing And Dependencies

Dependency management

We use golang modules to manage golang dependencies. In order to add a new package dependency to the project, you can perform go get <PACKAGE>@<VERSION> or edit the go.mod file and append the package along with the version you want to use.

Updating dependencies

The Makefile contains a rule called tidy which performs go mod tidy.

go mod tidy makes sure go.mod matches the source code in the module. It adds any missing modules necessary to build the current module’s packages and dependencies, and it removes unused modules that don’t provide any relevant packages.

$ make tidy

The dependencies are installed into the go mod cache folder.

⚠️ Make sure you test the code after you have updated the dependencies!

3 - Etcd Druid

ETCD Druid

Background

Etcd in the control plane of Kubernetes clusters which are managed by Gardener is deployed as a StatefulSet. The statefulset has replica of a pod containing two containers namely, etcd and backup-restore. The etcd container calls components in etcd-backup-restore via REST api to perform data validation before etcd is started. If this validation fails etcd data is restored from the latest snapshot stored in the cloud-provider’s object store. Once etcd has started, the etcd-backup-restore periodically creates full and delta snapshots. It also performs defragmentation of etcd data periodically.

The etcd-backup-restore needs as input the cloud-provider information comprising of security credentials to access the object store, the object store bucket name and prefix for the directory used to store snapshots. Currently, for operations like migration and validation, the bash script has to be updated to initiate the operation.

Goals

• Deploy etcd and etcd-backup-restore using an etcd CRD.
• Support more than one etcd replica.
• Perform scheduled snapshots.
• Support operations such as restores, defragmentation and scaling with zero-downtime.
• Handle cloud-provider specific operation logic.
• Trigger a full backup on request before volume deletion.
• Offline compaction of full and delta snapshots stored in object store.

Proposal

The existing method of deploying etcd and backup-sidecar as a StatefulSet alleviates the pain of ensuring the pods are live and ready after node crashes. However, deploying etcd as a Statefulset introduces a plethora of challenges. The etcd controller should be smart enough to handle etcd statefulsets taking into account limitations imposed by statefulsets. The controller shall update the status regarding how to target the K8s objects it has created. This field in the status can be leveraged by HVPA to scale etcd resources eventually.

CRD specification

The etcd CRD should contain the information required to create the etcd and backup-restore sidecar in a pod/statefulset.

apiVersion: druid.gardener.cloud/v1alpha1
kind: Etcd
metadata:
  finalizers:
  - druid.gardener.cloud/etcd
  name: test
  namespace: demo
spec:
  annotations:
    app: etcd-statefulset
    gardener.cloud/role: controlplane
    networking.gardener.cloud/to-dns: allowed
    networking.gardener.cloud/to-private-networks: allowed
    networking.gardener.cloud/to-public-networks: allowed
    role: test
  backup:
    deltaSnapshotMemoryLimit: 1Gi
    deltaSnapshotPeriod: 300s
    fullSnapshotSchedule: 0 */24 * * *
    garbageCollectionPeriod: 43200s
    garbageCollectionPolicy: Exponential
    imageRepository: europe-docker.pkg.dev/gardener-project/public/gardener/etcdbrctl
    imageVersion: v0.25.0
    port: 8080
    resources:
      limits:
        cpu: 500m
        memory: 2Gi
      requests:
        cpu: 23m
        memory: 128Mi
    snapstoreTempDir: /var/etcd/data/temp
  etcd:
    Quota: 8Gi
    clientPort: 2379
    defragmentationSchedule: 0 */24 * * *
    enableTLS: false
    imageRepository: europe-docker.pkg.dev/gardener-project/public/gardener/etcd-wrapper
    imageVersion: v0.1.0
    initialClusterState: new
    initialClusterToken: new
    metrics: basic
    pullPolicy: IfNotPresent
    resources:
      limits:
        cpu: 2500m
        memory: 4Gi
      requests:
        cpu: 500m
        memory: 1000Mi
    serverPort: 2380
    storageCapacity: 80Gi
    storageClass: gardener.cloud-fast
  sharedConfig:
    autoCompactionMode: periodic
    autoCompactionRetention: 30m
  labels:
    app: etcd-statefulset
    gardener.cloud/role: controlplane
    networking.gardener.cloud/to-dns: allowed
    networking.gardener.cloud/to-private-networks: allowed
    networking.gardener.cloud/to-public-networks: allowed
    role: test
  pvcRetentionPolicy: DeleteAll
  replicas: 1
  storageCapacity: 80Gi
  storageClass: gardener.cloud-fast
  store:
    storageContainer: test
    storageProvider: S3
    storePrefix: etcd-test
    storeSecret: etcd-backup
  tlsClientSecret: etcd-client-tls
  tlsServerSecret: etcd-server-tls
status:
  etcd:
    apiVersion: apps/v1
    kind: StatefulSet
    name: etcd-test