Documentation

1 - Deployment

1.1 - Kubernetes

Deploying the Machine Controller Manager into a Kubernetes cluster

• Deploying the Machine Controller Manager into a Kubernetes cluster
  • Prepare the cluster
  • Build the Docker image
  • Configuring optional parameters while deploying
  • Usage

As already mentioned, the Machine Controller Manager is designed to run as controller in a Kubernetes cluster. The existing source code can be compiled and tested on a local machine as described in Setting up a local development environment. You can deploy the Machine Controller Manager using the steps described below.

Prepare the cluster

• Connect to the remote kubernetes cluster where you plan to deploy the Machine Controller Manager using the kubectl. Set the environment variable KUBECONFIG to the path of the yaml file containing the cluster info.
• Now, create the required CRDs on the remote cluster using the following command,

$ kubectl apply -f kubernetes/crds.yaml

Build the Docker image

⚠️ Modify the Makefile to refer to your own registry.

• Run the build which generates the binary to bin/machine-controller-manager

$ make build

• Build docker image from latest compiled binary

$ make docker-image

• Push the last created docker image onto the online docker registry.

$ make push

• Now you can deploy this docker image to your cluster. A sample development file is given at. By default, the deployment manages the cluster it is running in. Optionally, the kubeconfig could also be passed as a flag as described in /kubernetes/deployment/in-tree/deployment.yaml. This is done when you want your controller running outside the cluster to be managed from.

$ kubectl apply -f kubernetes/deployment/in-tree/deployment.yaml

• Also deploy the required clusterRole and clusterRoleBindings

$ kubectl apply -f kubernetes/deployment/in-tree/clusterrole.yaml
$ kubectl apply -f kubernetes/deployment/in-tree/clusterrolebinding.yaml

Configuring optional parameters while deploying

Machine-controller-manager supports several configurable parameters while deploying. Refer to the following lines, to know how each parameter can be configured, and what it’s purpose is for.

Usage

To start using Machine Controller Manager, follow the links given at usage here.

2 - Development

2.1 - Adding Support for a Cloud Provider

Adding support for a new provider

Steps to be followed while implementing a new (hyperscale) provider are mentioned below. This is the easiest way to add new provider support using a blueprint code.

However, you may also develop your machine controller from scratch, which would provide you with more flexibility. First, however, make sure that your custom machine controller adheres to the Machine.Status struct defined in the MachineAPIs. This will make sure the MCM can act with higher-level controllers like MachineSet and MachineDeployment controller. The key is the Machine.Status.CurrentStatus.Phase key that indicates the status of the machine object.

Our strong recommendation would be to follow the steps below. This provides the most flexibility required to support machine management for adding new providers. And if you feel to extend the functionality, feel free to update our machine controller libraries.

Setting up your repository

1. Create a new empty repository named machine-controller-manager-provider-{provider-name} on GitHub username/project. Do not initialize this repository with a README.
2. Copy the remote repository URL (HTTPS/SSH) to this repository displayed once you create this repository.
3. Now, on your local system, create directories as required. {your-github-username} given below could also be {github-project} depending on where you have created the new repository.

   mkdir -p $GOPATH/src/github.com/{your-github-username}
   

4. Navigate to this created directory.

   cd $GOPATH/src/github.com/{your-github-username}
   

5. Clone this repository on your local machine.

   git clone git@github.com:gardener/machine-controller-manager-provider-sampleprovider.git
   

6. Rename the directory from machine-controller-manager-provider-sampleprovider to machine-controller-manager-provider-{provider-name}.

   mv machine-controller-manager-provider-sampleprovider machine-controller-manager-provider-{provider-name}
   

7. Navigate into the newly-created directory.

   cd machine-controller-manager-provider-{provider-name}
   

8. Update the remote origin URL to the newly created repository’s URL you had copied above.

   git remote set-url origin git@github.com:{your-github-username}/machine-controller-manager-provider-{provider-name}.git
   

9. Rename GitHub project from gardener to {github-org/your-github-username} wherever you have cloned the repository above. Also, edit all occurrences of the word sampleprovider to {provider-name} in the code. Then, use the hack script given below to do the same.

   make rename-project PROJECT_NAME={github-org/your-github-username} PROVIDER_NAME={provider-name}
   eg:
       make rename-project PROJECT_NAME=gardener PROVIDER_NAME=AmazonWebServices (or)
       make rename-project PROJECT_NAME=githubusername PROVIDER_NAME=AWS
   

10. Now, commit your changes and push them upstream.

    git add -A
    git commit -m "Renamed SampleProvide to {provider-name}"
    git push origin master
    

Code changes required

The contract between the Machine Controller Manager (MCM) and the Machine Controller (MC) AKA driver has been documented here and the machine error codes can be found here. You may refer to them for any queries.

⚠️

• Keep in mind that there should be a unique way to map between machine objects and VMs. This can be done by mapping machine object names with VM-Name/ tags/ other metadata.
• Optionally, there should also be a unique way to map a VM to its machine class object. This can be done by tagging VM objects with tags/resource groups associated with the machine class.

Steps to integrate

1. Update the pkg/provider/apis/provider_spec.go specification file to reflect the structure of the ProviderSpec blob. It typically contains the machine template details in the MachineClass object. Follow the sample spec provided already in the file. A sample provider specification can be found here.
2. Fill in the methods described at pkg/provider/core.go to manage VMs on your cloud provider. Comments are provided above each method to help you fill them up with desired REQUEST and RESPONSE parameters.
   • A sample provider implementation for these methods can be found here.
   • Fill in the required methods CreateMachine(), and DeleteMachine() methods.
   • Optionally fill in methods like GetMachineStatus(), ListMachines(), and GetVolumeIDs(). You may choose to fill these once the working of the required methods seems to be working.
   • GetVolumeIDs() expects VolumeIDs to be decoded from the volumeSpec based on the cloud provider.
   • There is also an OPTIONAL method GenerateMachineClassForMigration() that helps in migration of {ProviderSpecific}MachineClass to MachineClass CR (custom resource). This only makes sense if you have an existing implementation (in-tree) acting on different CRD types. You would like to migrate this. If not, you MUST return an error (machine error UNIMPLEMENTED) to avoid processing this step.
3. Perform validation of APIs that you have described and make it a part of your methods as required at each request.
4. Write unit tests to make it work with your implementation by running make test.

   make test
   

5. Re-generate the vendors to update any new vendors imported.

   make revendor
   

6. Update the sample YAML files on the kubernetes/ directory to provide sample files through which the working of the machine controller can be tested.
7. Update README.md to reflect any additional changes

Testing your code changes

Make sure $TARGET_KUBECONFIG points to the cluster where you wish to manage machines. Likewise, $CONTROL_NAMESPACE represents the namespaces where MCM is looking for machine CR objects, and $CONTROL_KUBECONFIG points to the cluster that holds these machine CRs.

1. On the first terminal running at $GOPATH/src/github.com/{github-org/your-github-username}/machine-controller-manager-provider-{provider-name},
   • Run the machine controller (driver) using the command below.

     make start
     

2. On the second terminal pointing to $GOPATH/src/github.com/gardener,
   • Clone the latest MCM code

     git clone git@github.com:gardener/machine-controller-manager.git
     

   • Navigate to the newly-created directory.

     cd machine-controller-manager
     

   • Deploy the required CRDs from the machine-controller-manager repo,

     kubectl apply -f kubernetes/crds
     

   • Run the machine-controller-manager in the master branch

     make start
     

3. On the third terminal pointing to $GOPATH/src/github.com/{github-org/your-github-username}/machine-controller-manager-provider-{provider-name}
   • Fill in the object files given below and deploy them as described below.
   • Deploy the machine-class

     kubectl apply -f kubernetes/machine-class.yaml
     

   • Deploy the kubernetes secret if required.

     kubectl apply -f kubernetes/secret.yaml
     

   • Deploy the machine object and make sure it joins the cluster successfully.

     kubectl apply -f kubernetes/machine.yaml
     

   • Once the machine joins, you can test by deploying a machine-deployment.
   • Deploy the machine-deployment object and make sure it joins the cluster successfully.

     kubectl apply -f kubernetes/machine-deployment.yaml
     

   • Make sure to delete both the machine and machine-deployment objects after use.

     kubectl delete -f kubernetes/machine.yaml
     kubectl delete -f kubernetes/machine-deployment.yaml
     

Releasing your docker image

1. Make sure you have logged into gcloud/docker using the CLI.
2. To release your docker image, run the following.

    make release IMAGE_REPOSITORY=<link-to-image-repo>

3. A sample kubernetes deploy file can be found at kubernetes/deployment.yaml. Update the same (with your desired MCM and MC images) to deploy your MCM pod.

2.2 - Adding Support for a Cloud Provider (Legacy)

Adding support for a new cloud provider

For adding support for a new cloud provider in the Machine Controller Manager, follow the steps described below. Replace provider with your provider-name.

1. Add a ProviderMachineClass CRD similar to existing AWSMachineClass into kubernetes/crds.yaml.
2. Add ProviderMachineClass structs similar to existing AWSMachineClass into the machine APIs into pkg/apis/machine/types.go and pkg/apis/machine/v1alpha1/types.go. This would be the machineClass template used to describe provider specific configurations.
3. Add the Go structures of your machine class (list) to pkg/apis/machine/register.go and pkg/apis/machine/v1alpha1/register.go to allow reporting events on these objects.
4. Regenerate the machine API clients by running ./hack/generate-code
5. Add validation for the new provider machine class at pkg/apis/machine/validation/providermachineclass.go similar to pkg/apis/machine/validation/awsmachineclass.go
6. Update pkg/controller/machine_util.go to allow validation of the new provider.
7. Add a new driver into pkg/driver/driver_provider.go similar to pkg/driver/driver_aws.go to implement the driver interface.
8. Update pkg/driver/driver.go to add a new switch case to support the new provider driver.
9. Add a new method in pkg/controller/machine_safety.go called checkProviderMachineClass similar to the existing method called checkAWSMachineClass present in the same file. Now invoke this method as a go-routine in the method checkVMObjects.
10. Extend the StartControllers() function in cmd/machine-controller-manager/app/controllermanager.go to only start if your new machine class is under the available resources.
11. Update pkg/controller/controller.go to add new providerMachineClassLister, providerMachineClassQueue, awsMachineClassSynced into the controller struct. Also initialize them in NewController() method.
12. Add a new file pkg/controller/providermachineclass.go that allows re-queuing of machines which refer to an modified providerMachineClass.
13. Update pkg/controller/controller.go to extend WaitForCacheSync and .Shutdown() similar to other cloud providers.
14. Update the example ClusterRole in kubernetes/deployment/in-tree/clusterrole.yaml to allow operations on your new machine class.
15. Update pkg/controller/controller.go, pkg/controller/secret.go, pkg/controller/secret_util.go to add event handlers to add/remove finalizers referenced by your machine Class. Refer this commit.

2.3 - Integration Tests

Integration tests

Usage

General setup & configurations

Integration tests for machine-controller-manager-provider-{provider-name} can be executed manually by following below steps.

1. Clone the repository machine-controller-manager-provider-{provider-name} on the local system.
2. Navigate to machine-controller-manager-provider-{provider-name} directory and create a dev sub-directory in it.
3. Copy the kubeconfig of Control Cluster from into dev/control-kubeconfig.yaml.
4. (optional) Copy the kubeconfig of Target Cluster into dev/target-kubeconfig.yaml and update the Makefile variable TARGET_KUBECONFIG to point to dev/target-kubeconfig.yaml.
5. If the tags on instances & associated resources on the provider are of String type (for example, GCP tags on its instances are of type String and not key-value pair) then add TAGS_ARE_STRINGS := true in the Makefile and export it.
6. Atleast, one of the two controllers’ container images must be set in the Makefile variables MCM_IMAGE_TAG and MC_IMAGE_TAG for the controllers to run in the Control Cluster . These images will be used along with kubernetes/deployment.yaml to deploy/update controllers in the Control Cluster . If the intention is to run the controllers locally then unset the variables MCM_IMAGE_TAG and MC_IMAGE_TAG and set variable MACHINE_CONTROLLER_MANAGER_DEPLOYMENT_NAME := machine-controller-manager in the Makefile.
7. In order to apply the CRDs when the Control Cluster is a Gardener Shoot or if none of the controller images are specified, machine-controller-manager repository will be cloned automatically. Incase, this repository already exists in local system, then create a softlink as below which helps to test changes in machine-controller-manager quickly.

   ln -sf <path-for-machine-controller-manager-repo> dev/mcm
   

Scenario based additional configurations

Gardener Shoot as the Control Cluster

If the Control Cluster is a Gardener Shoot cluster then,

1. Deploy a Secret named test-mc-secret (that contains the provider secret and cloud-config) in the default namespace of the Control Cluster. Refer these MachineClass templates for the same.
2. Create a dev/machineclassv1.yaml file in the cloned repository. The name of the MachineClass itself should be test-mc-v1. The value of providerSpec.secretRef.name should be test-mc-secret.
3. (Optional) Create an additional dev/machineclassv2.yaml file similar to above but with a bigger machine type and update the Makefile variable MACHINECLASS_V2 to point to dev/machineclassv2.yaml.

Gardener Seed as the Control Cluster

If the Control Cluster is a Gardener SEED cluster then, the suite ideally employs the already existing MachineClass and Secrets. However,

1. (Optional) User can employ a custom MachineClass for the tests using below steps:
   1. Deploy a Secret named test-mc-secret (that contains the provider secret and cloud-config) in the shoot namespace of the Control Cluster. That is, the value of metadata.namespace should be technicalID of the Shoot and it will be of the pattern shoot--<project>--<shoot-name>. Refer these MachineClass templates for the same.
   2. Create a dev/machineclassv1.yaml file.
      1. providerSpec.secretRef.name should refer the secret created in the previous step.
      2. metadata.namespace and providerSpec.secretRef.namespace should be technicalID (shoot--<project>--<shoot-name>) of the shoot.
      3. The name of the MachineClass itself should be test-mc-v1.

Running the tests

1. There is a rule test-integration in the Makefile, which can be used to start the integration test:

   $ make test-integration 
   Starting integration tests...
   Running Suite: Controller Suite
   ===============================
   

2. The controllers log files (mcm_process.log and mc_process.log) are stored in .ci/controllers-test/logs repo and can be used later.

Adding Integration Tests for new providers

For a new provider, Running Integration tests works with no changes. But for the orphan resource test cases to work correctly, the provider-specific API calls and the Resource Tracker Interface (RTI) should be implemented. Please check machine-controller-manager-provider-aws for reference.

Extending integration tests

• Update ControllerTests to be extend the testcases for all providers. Common testcases for machine|machineDeployment creation|deletion|scaling are packaged into ControllerTests.
• To extend the provider specfic test cases, the changes should be done in the machine-controller-manager-provider-{provider-name} repository. For example, to extended the testcases for machine-controller-manager-provider-aws, make changes to test/integration/controller/controller_test.go inside the machine-controller-manager-provider-aws repository. commons contains the Cluster and Clientset objects that makes it easy to extend the tests.

2.4 - Local Setup

Preparing the Local Development Setup (Mac OS X)

• Preparing the Local Development Setup (Mac OS X)
  • Installing Golang environment
  • Installing Docker (Optional)
  • Setup Docker Hub account (Optional)
  • Local development
    • Installing the Machine Controller Manager locally
  • Prepare the cluster
  • Getting started
  • Testing Machine Classes
  • Usage

Conceptionally, the Machine Controller Manager is designed to run in a container within a Pod inside a Kubernetes cluster. For development purposes, you can run the Machine Controller Manager as a Go process on your local machine. This process connects to your remote cluster to manage VMs for that cluster. That means that the Machine Controller Manager runs outside a Kubernetes cluster which requires providing a Kubeconfig in your local filesystem and point the Machine Controller Manager to it when running it (see below).

Although the following installation instructions are for Mac OS X, similar alternate commands could be found for any Linux distribution.

Installing Golang environment

Install the latest version of Golang (at least v1.8.3 is required) by using Homebrew:

$ brew install golang

In order to perform linting on the Go source code, install Golint:

$ go get -u golang.org/x/lint/golint

Installing Docker (Optional)

In case you want to build Docker images for the Machine Controller Manager you have to install Docker itself. We recommend using Docker for Mac OS X which can be downloaded from here.

Setup Docker Hub account (Optional)

Create a Docker hub account at Docker Hub if you don’t already have one.

Local development

⚠️ Before you start developing, please ensure to comply with the following requirements:

1. You have understood the principles of Kubernetes, and its components, what their purpose is and how they interact with each other.
2. You have understood the architecture of the Machine Controller Manager

The development of the Machine Controller Manager could happen by targetting any cluster. You basically need a Kubernetes cluster running on a set of machines. You just need the Kubeconfig file with the required access permissions attached to it.

Installing the Machine Controller Manager locally

Clone the repository from GitHub.

$ git clone git@github.com:gardener/machine-controller-manager.git
$ cd machine-controller-manager

Prepare the cluster

• Connect to the remote kubernetes cluster where you plan to deploy the Machine Controller Manager using kubectl. Set the environment variable KUBECONFIG to the path of the yaml file containing your cluster info
• Now, create the required CRDs on the remote cluster using the following command,

$ kubectl apply -f kubernetes/crds.yaml

Getting started

• Create a dev directory.
• Copy the kubeconfig of kubernetes cluster where you wish to deploy the machines into dev/target-kubeconfig.yaml.
• (optional) Copy the kubeconfig of kubernetes cluster from where you wish to manage the machines into dev/control-kubeconfig.yaml. If you do this, also update the Makefile variable CONTROL_KUBECONFIG to point to dev/control-kubeconfig.yaml and CONTROL_NAMESPACE to the namespace in which your controller watches over.
• There is a rule dev in the Makefile which will automatically start the Machine Controller Manager with development settings:

$ make start
I1227 11:08:19.963638   55523 controllermanager.go:204] Starting shared informers
I1227 11:08:20.766085   55523 controller.go:247] Starting machine-controller-manager

⚠️ The file dev/target-kubeconfig.yaml points to the cluster whose nodes you want to manage. dev/control-kubeconfig.yaml points to the cluster from where you want to manage the nodes from. However, dev/control-kubeconfig.yaml is optional.

The Machine Controller Manager should now be ready to manage the VMs in your kubernetes cluster.

⚠️ This is assuming that your MCM is built to manage machines for any in-tree supported providers. There is a new way to deploy and manage out of tree (external) support for providers whose development can be found here

Testing Machine Classes

To test the creation/deletion of a single instance for one particular machine class you can use the managevm cli. The corresponding INFRASTRUCTURE-machine-class.yaml and the INFRASTRUCTURE-secret.yaml need to be defined upfront. To build and run it

GO111MODULE=on go build -mod=vendor -o managevm cmd/machine-controller-manager-cli/main.go
# create machine
./managevm --secret PATH_TO/INFRASTRUCTURE-secret.yaml --machineclass PATH_TO/INFRASTRUCTURE-machine-class.yaml --classkind INFRASTRUCTURE --machinename test
# delete machine
./managevm --secret PATH_TO/INFRASTRUCTURE-secret.yaml --machineclass PATH_TO/INFRASTRUCTURE-machine-class.yaml --classkind INFRASTRUCTURE --machinename test --machineid INFRASTRUCTURE:///REGION/INSTANCE_ID

Usage

To start using Machine Controller Manager, follow the links given at usage here.

2.5 - Machine Error Codes

Machine Error code handling

Notational Conventions

The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” are to be interpreted as described in RFC 2119 (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels”, BCP 14, RFC 2119, March 1997).

The key words “unspecified”, “undefined”, and “implementation-defined” are to be interpreted as described in the rationale for the C99 standard.

An implementation is not compliant if it fails to satisfy one or more of the MUST, REQUIRED, or SHALL requirements for the protocols it implements. An implementation is compliant if it satisfies all the MUST, REQUIRED, and SHALL requirements for the protocols it implements.

Terminology

Pre-requisite

MachineClass Resources

MCM introduces the CRD MachineClass. This is a blueprint for creating machines that join a certain cluster as nodes in a certain role. The provider only works with MachineClass resources that have the structure described here.

ProviderSpec

The MachineClass resource contains a providerSpec field that is passed in the ProviderSpec request field to CMI methods such as CreateMachine. The ProviderSpec can be thought of as a machine template from which the VM specification must be adopted. It can contain key-value pairs of these specs. An example for these key-value pairs are given below.

Most of the ProviderSpec fields are not mandatory. If not specified, the provider passes an empty value in the respective Create VM parameter.

The tags can be used to map a VM to its corresponding machine object’s Name

The ProviderSpec is validated by methods that receive it as a request field for presence of all mandatory parameters and tags, and for validity of all parameters.

Secrets

The MachineClass resource also contains a secretRef field that contains a reference to a secret. The keys of this secret are passed in the Secrets request field to CMI methods.

The secret can contain sensitive data such as

• cloud-credentials secret data used to authenticate at the provider
• cloud-init scripts used to initialize a new VM. The cloud-init script is expected to contain scripts to initialize the Kubelet and make it join the cluster.

Identifying Cluster Machines

To implement certain methods, the provider should be able to identify all machines associated with a particular Kubernetes cluster. This can be achieved using one/more of the below mentioned ways:

• Names of VMs created by the provider are prefixed by the cluster ID specified in the ProviderSpec.
• VMs created by the provider are tagged with the special tags like kubernetes.io/cluster (for the cluster ID) and kubernetes.io/role (for the role), specified in the ProviderSpec.
• Mapping Resource Groups to individual cluster.

Error Scheme

All provider API calls defined in this spec MUST return a machine error status, which is very similar to standard machine status.

Machine Provider Interface

• The provider MUST have a unique way to map a machine object to a VM which triggers the deletion for the corresponding VM backing the machine object.
• The provider SHOULD have a unique way to map the ProviderSpec of a machine-class to a unique Cluster. This avoids deletion of other machines, not backed by the MCM.

CreateMachine

A Provider is REQUIRED to implement this interface method. This interface method will be called by the MCM to provision a new VM on behalf of the requesting machine object.

• This call requests the provider to create a VM backing the machine-object.

• If VM backing the Machine.Name already exists, and is compatible with the specified Machine object in the CreateMachineRequest, the Provider MUST reply 0 OK with the corresponding CreateMachineResponse.

• The provider can OPTIONALLY make use of the MachineClass supplied in the MachineClass in the CreateMachineRequest to communicate with the provider.

• The provider can OPTIONALLY make use of the secrets supplied in the Secret in the CreateMachineRequest to communicate with the provider.

• The provider can OPTIONALLY make use of the Status.LastKnownState in the Machine object to decode the state of the VM operation based on the last known state of the VM. This can be useful to restart/continue an operations which are mean’t to be atomic.

• The provider MUST have a unique way to map a machine object to a VM. This could be implicitly provided by the provider by letting you set VM-names (or) could be explicitly specified by the provider using appropriate tags to map the same.

• This operation SHOULD be idempotent.

• The CreateMachineResponse returned by this method is expected to return

  • ProviderID that uniquely identifys the VM at the provider. This is expected to match with the node.Spec.ProviderID on the node object.
  • NodeName that is the expected name of the machine when it joins the cluster. It must match with the node name.
  • LastKnownState is an OPTIONAL field that can store details of the last known state of the VM. It can be used by future operation calls to determine current infrastucture state. This state is saved on the machine object.

// CreateMachine call is responsible for VM creation on the provider
CreateMachine(context.Context, *CreateMachineRequest) (*CreateMachineResponse, error)

// CreateMachineRequest is the create request for VM creation
type CreateMachineRequest struct {
	// Machine object from whom VM is to be created
	Machine *v1alpha1.Machine

	// MachineClass backing the machine object
	MachineClass *v1alpha1.MachineClass

	//  Secret backing the machineClass object
	Secret *corev1.Secret
}

// CreateMachineResponse is the create response for VM creation
type CreateMachineResponse struct {
	// ProviderID is the unique identification of the VM at the cloud provider.
	// ProviderID typically matches with the node.Spec.ProviderID on the node object.
	// Eg: gce://project-name/region/vm-ID
	ProviderID string

	// NodeName is the name of the node-object registered to kubernetes.
	NodeName string

	// LastKnownState represents the last state of the VM during an creation/deletion error
	LastKnownState string
}

CreateMachine Errors

If the provider is unable to complete the CreateMachine call successfully, it MUST return a non-ok ginterface method code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code. The MCM MUST implement the specified error recovery behavior when it encounters the machine error code.

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

DeleteMachine

A Provider is REQUIRED to implement this driver call. This driver call will be called by the MCM to deprovision/delete/terminate a VM backed by the requesting machine object.

• If a VM corresponding to the specified machine-object’s name does not exist or the artifacts associated with the VM do not exist anymore (after deletion), the Provider MUST reply 0 OK.

• The provider SHALL only act on machines belonging to the cluster-id/cluster-name obtained from the ProviderSpec.

• The provider can OPTIONALY make use of the secrets supplied in the Secrets map in the DeleteMachineRequest to communicate with the provider.

• The provider can OPTIONALY make use of the Spec.ProviderID map in the Machine object.

• The provider can OPTIONALLY make use of the Status.LastKnownState in the Machine object to decode the state of the VM operation based on the last known state of the VM. This can be useful to restart/continue an operations which are mean’t to be atomic.

• This operation SHOULD be idempotent.

• The provider must have a unique way to map a machine object to a VM which triggers the deletion for the corresponding VM backing the machine object.

• The DeleteMachineResponse returned by this method is expected to return

  • LastKnownState is an OPTIONAL field that can store details of the last known state of the VM. It can be used by future operation calls to determine current infrastucture state. This state is saved on the machine object.

// DeleteMachine call is responsible for VM deletion/termination on the provider
DeleteMachine(context.Context, *DeleteMachineRequest) (*DeleteMachineResponse, error)

// DeleteMachineRequest is the delete request for VM deletion
type DeleteMachineRequest struct {
	// Machine object from whom VM is to be deleted
	Machine *v1alpha1.Machine

	// MachineClass backing the machine object
	MachineClass *v1alpha1.MachineClass

	// Secret backing the machineClass object
	Secret *corev1.Secret
}

// DeleteMachineResponse is the delete response for VM deletion
type DeleteMachineResponse struct {
	// LastKnownState represents the last state of the VM during an creation/deletion error
	LastKnownState string
}

DeleteMachine Errors

If the provider is unable to complete the DeleteMachine call successfully, it MUST return a non-ok machine code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code.

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

GetMachineStatus

A Provider can OPTIONALLY implement this driver call. Else should return a UNIMPLEMENTED status in error. This call will be invoked by the MC to get the status of a machine. This optional driver call helps in optimizing the working of the provider by avoiding unwanted calls to CreateMachine() and DeleteMachine().

• If a VM corresponding to the specified machine object’s Machine.Name exists on provider the GetMachineStatusResponse fields are to be filled similar to the CreateMachineResponse.
• The provider SHALL only act on machines belonging to the cluster-id/cluster-name obtained from the ProviderSpec.
• The provider can OPTIONALY make use of the secrets supplied in the Secrets map in the GetMachineStatusRequest to communicate with the provider.
• The provider can OPTIONALY make use of the VM unique ID (returned by the provider on machine creation) passed in the ProviderID map in the GetMachineStatusRequest.
• This operation MUST be idempotent.

// GetMachineStatus call get's the status of the VM backing the machine object on the provider
GetMachineStatus(context.Context, *GetMachineStatusRequest) (*GetMachineStatusResponse, error)

// GetMachineStatusRequest is the get request for VM info
type GetMachineStatusRequest struct {
	// Machine object from whom VM status is to be fetched
	Machine *v1alpha1.Machine

	// MachineClass backing the machine object
	MachineClass *v1alpha1.MachineClass

	//  Secret backing the machineClass object
	Secret *corev1.Secret
}

// GetMachineStatusResponse is the get response for VM info
type GetMachineStatusResponse struct {
	// ProviderID is the unique identification of the VM at the cloud provider.
	// ProviderID typically matches with the node.Spec.ProviderID on the node object.
	// Eg: gce://project-name/region/vm-ID
	ProviderID string

	// NodeName is the name of the node-object registered to kubernetes.
	NodeName string
}

GetMachineStatus Errors

If the provider is unable to complete the GetMachineStatus call successfully, it MUST return a non-ok machine code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code.

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

ListMachines

A Provider can OPTIONALLY implement this driver call. Else should return a UNIMPLEMENTED status in error. The Provider SHALL return the information about all the machines associated with the MachineClass. Make sure to use appropriate filters to achieve the same to avoid data transfer overheads. This optional driver call helps in cleaning up orphan VMs present in the cluster. If not implemented, any orphan VM that might have been created incorrectly by the MCM/Provider (due to bugs in code/infra) might require manual clean up.

• If the Provider succeeded in returning a list of Machine.Name with their corresponding ProviderID, then return 0 OK.
• The ListMachineResponse contains a map of MachineList whose
  • Key is expected to contain the ProviderID &
  • Value is expected to contain the Machine.Name corresponding to it’s kubernetes machine CR object
• The provider can OPTIONALY make use of the secrets supplied in the Secrets map in the ListMachinesRequest to communicate with the provider.

// ListMachines lists all the machines that might have been created by the supplied machineClass
ListMachines(context.Context, *ListMachinesRequest) (*ListMachinesResponse, error)

// ListMachinesRequest is the request object to get a list of VMs belonging to a machineClass
type ListMachinesRequest struct {
	// MachineClass object
	MachineClass *v1alpha1.MachineClass

	// Secret backing the machineClass object
	Secret *corev1.Secret
}

// ListMachinesResponse is the response object of the list of VMs belonging to a machineClass
type ListMachinesResponse struct {
	// MachineList is the map of list of machines. Format for the map should be <ProviderID, MachineName>.
	MachineList map[string]string
}

ListMachines Errors

If the provider is unable to complete the ListMachines call successfully, it MUST return a non-ok machine code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code. The MCM MUST implement the specified error recovery behavior when it encounters the machine error code.

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

GetVolumeIDs

A Provider can OPTIONALLY implement this driver call. Else should return a UNIMPLEMENTED status in error. This driver call will be called by the MCM to get the VolumeIDs for the list of PersistentVolumes (PVs) supplied. This OPTIONAL (but recommended) driver call helps in serailzied eviction of pods with PVs while draining of machines. This implies applications backed by PVs would be evicted one by one, leading to shorter application downtimes.

• On succesful returnal of a list of Volume-IDs for all supplied PVSpecs, the Provider MUST reply 0 OK.
• The GetVolumeIDsResponse is expected to return a repeated list of strings consisting of the VolumeIDs for PVSpec that could be extracted.
• If for any PV the Provider wasn’t able to identify the Volume-ID, the provider MAY chose to ignore it and return the Volume-IDs for the rest of the PVs for whom the Volume-ID was found.
• Getting the VolumeID from the PVSpec depends on the Cloud-provider. You can extract this information by parsing the PVSpec based on the ProviderType
  • https://github.com/kubernetes/api/blob/release-1.15/core/v1/types.go#L297-L339
  • https://github.com/kubernetes/api/blob/release-1.15//core/v1/types.go#L175-L257
• This operation MUST be idempotent.

// GetVolumeIDsRequest is the request object to get a list of VolumeIDs for a PVSpec
type GetVolumeIDsRequest struct {
	// PVSpecsList is a list of PV specs for whom volume-IDs are required
	// Plugin should parse this raw data into pre-defined list of PVSpecs
	PVSpecs []*corev1.PersistentVolumeSpec
}

// GetVolumeIDsResponse is the response object of the list of VolumeIDs for a PVSpec
type GetVolumeIDsResponse struct {
	// VolumeIDs is a list of VolumeIDs.
	VolumeIDs []string
}

GetVolumeIDs Errors

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

GenerateMachineClassForMigration

A Provider SHOULD implement this driver call, else it MUST return a UNIMPLEMENTED status in error. This driver call will be called by the Machine Controller to try to perform a machineClass migration for an unknown machineClass Kind. This helps in migration of one kind of machineClass to another kind. For instance an machineClass custom resource of AWSMachineClass to MachineClass.

• On successful generation of machine class the Provider MUST reply 0 OK (or) nil error.
• GenerateMachineClassForMigrationRequest expects the provider-specific machine class (eg. AWSMachineClass) to be supplied as the ProviderSpecificMachineClass. The provider is responsible for unmarshalling the golang struct. It also passes a reference to an existing MachineClass object.
• The provider is expected to fill in thisMachineClass object based on the conversions.
• An optional ClassSpec containing the type ClassSpec struct is also provided to decode the provider info.
• GenerateMachineClassForMigration is only responsible for filling up the passed MachineClass object.
• The task of creating the new CR of the new kind (MachineClass) with the same name as the previous one and also annotating the old machineClass CR with a migrated annotation and migrating existing references is done by the calling library implicitly.
• This operation MUST be idempotent.

// GenerateMachineClassForMigrationRequest is the request for generating the generic machineClass
// for the provider specific machine class
type GenerateMachineClassForMigrationRequest struct {
	// ProviderSpecificMachineClass is provider specfic machine class object.
	// E.g. AWSMachineClass
	ProviderSpecificMachineClass interface{}
	// MachineClass is the machine class object generated that is to be filled up
	MachineClass *v1alpha1.MachineClass
	// ClassSpec contains the class spec object to determine the machineClass kind
	ClassSpec *v1alpha1.ClassSpec
}

// GenerateMachineClassForMigrationResponse is the response for generating the generic machineClass
// for the provider specific machine class
type GenerateMachineClassForMigrationResponse struct{}

MigrateMachineClass Errors

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

Configuration and Operation

Supervised Lifecycle Management

• For Providers packaged in software form:
  • Provider Packages SHOULD use a well-documented container image format (e.g., Docker, OCI).
  • The chosen package image format MAY expose configurable Provider properties as environment variables, unless otherwise indicated in the section below. Variables so exposed SHOULD be assigned default values in the image manifest.
  • A Provider Supervisor MAY programmatically evaluate or otherwise scan a Provider Package’s image manifest in order to discover configurable environment variables.
  • A Provider SHALL NOT assume that an operator or Provider Supervisor will scan an image manifest for environment variables.

Environment Variables

• Variables defined by this specification SHALL be identifiable by their MC_ name prefix.
• Configuration properties not defined by the MC specification SHALL NOT use the same MC_ name prefix; this prefix is reserved for common configuration properties defined by the MC specification.
• The Provider Supervisor SHOULD supply all RECOMMENDED MC environment variables to a Provider.
• The Provider Supervisor SHALL supply all REQUIRED MC environment variables to a Provider.

Logging

• Providers SHOULD generate log messages to ONLY standard output and/or standard error.
  • In this case the Provider Supervisor SHALL assume responsibility for all log lifecycle management.
• Provider implementations that deviate from the above recommendation SHALL clearly and unambiguously document the following:
  • Logging configuration flags and/or variables, including working sample configurations.
  • Default log destination(s) (where do the logs go if no configuration is specified?)
  • Log lifecycle management ownership and related guidance (size limits, rate limits, rolling, archiving, expunging, etc.) applicable to the logging mechanism embedded within the Provider.
• Providers SHOULD NOT write potentially sensitive data to logs (e.g. secrets).

Available Services

• Provider Packages MAY support all or a subset of CMI services; service combinations MAY be configurable at runtime by the Provider Supervisor.
  • This specification does not dictate the mechanism by which mode of operation MUST be discovered, and instead places that burden upon the VM Provider.
• Misconfigured provider software SHOULD fail-fast with an OS-appropriate error code.

Linux Capabilities

• Providers SHOULD clearly document any additionally required capabilities and/or security context.

Cgroup Isolation

• A Provider MAY be constrained by cgroups.

Resource Requirements

• VM Providers SHOULD unambiguously document all of a Provider’s resource requirements.

Deploying

• Recommended: The MCM and Provider are typically expected to run as two containers inside a common Pod.
• However, for the security reasons they could execute on seperate Pods provided they have a secure way to exchange data between them.

2.6 - Testing And Dependencies

Dependency management

We use golang modules to manage golang dependencies. In order to add a new package dependency to the project, you can perform go get <PACKAGE>@<VERSION> or edit the go.mod file and append the package along with the version you want to use.

Updating dependencies

The Makefile contains a rule called revendor which performs go mod vendor and go mod tidy.

go mod vendor resets the main module’s vendor directory to include all packages needed to build and test all the main module’s packages. It does not include test code for vendored packages.

go mod tidy makes sure go.mod matches the source code in the module. It adds any missing modules necessary to build the current module’s packages and dependencies, and it removes unused modules that don’t provide any relevant packages.

$ make revendor

The dependencies are installed into the vendor folder which should be added to the VCS.

⚠️ Make sure you test the code after you have updated the dependencies!

3 - Documents

3.1 - APIs

Specification

ProviderSpec Schema

AWSMachineClass

AWSMachineClass TODO

AlicloudMachineClass

AlicloudMachineClass TODO

AzureMachineClass

AzureMachineClass TODO

GCPMachineClass

GCPMachineClass TODO

Machine

Machine is the representation of a physical or virtual machine.

MachineClass

MachineClass can be used to templatize and re-use provider configuration across multiple Machines / MachineSets / MachineDeployments.

MachineSet

MachineSet TODO

PacketMachineClass

PacketMachineClass TODO

AWSBlockDeviceMappingSpec

(Appears on: AWSMachineClassSpec)

AWSEbsBlockDeviceSpec

(Appears on: AWSBlockDeviceMappingSpec)

Describes a block device for an EBS volume. Please also see https://docs.aws.amazon.com/goto/WebAPI/ec2-2016-11-15/EbsBlockDevice

AWSIAMProfileSpec

(Appears on: AWSMachineClassSpec)

Describes an IAM machine profile.

AWSMachineClassSpec

(Appears on: AWSMachineClass)

AWSMachineClassSpec is the specification of a AWSMachineClass.

AWSNetworkInterfaceSpec

(Appears on: AWSMachineClassSpec)

Describes a network interface. Please also see https://docs.aws.amazon.com/goto/WebAPI/ec2-2016-11-15/MachineAWSNetworkInterfaceSpecification

AlicloudDataDisk

(Appears on: AlicloudMachineClassSpec)

AlicloudMachineClassSpec

(Appears on: AlicloudMachineClass)

AlicloudMachineClassSpec is the specification of a AlicloudMachineClass.

AlicloudSystemDisk

(Appears on: AlicloudMachineClassSpec)

AlicloudSystemDisk describes SystemDisk for Alicloud.

AzureDataDisk

(Appears on: AzureStorageProfile)

AzureHardwareProfile

(Appears on: AzureVirtualMachineProperties)

AzureHardwareProfile is specifies the hardware settings for the virtual machine. Refer github.com/Azure/azure-sdk-for-go/arm/compute/models.go for VMSizes

AzureImageReference

(Appears on: AzureStorageProfile)

AzureImageReference is specifies information about the image to use. You can specify information about platform images, marketplace images, or virtual machine images. This element is required when you want to use a platform image, marketplace image, or virtual machine image, but is not used in other creation operations.

AzureLinuxConfiguration

(Appears on: AzureOSProfile)

AzureLinuxConfiguration is specifies the Linux operating system settings on the virtual machine.

For a list of supported Linux distributions, see Linux on Azure-Endorsed Distributions

For running non-endorsed distributions, see Information for Non-Endorsed Distributions.

AzureMachineClassSpec

(Appears on: AzureMachineClass)

AzureMachineClassSpec is the specification of a AzureMachineClass.

AzureMachineSetConfig

(Appears on: AzureVirtualMachineProperties)

AzureMachineSetConfig contains the information about the machine set

AzureManagedDiskParameters

(Appears on: AzureOSDisk)

AzureManagedDiskParameters is the parameters of a managed disk.

AzureNetworkInterfaceReference

(Appears on: AzureNetworkProfile)

AzureNetworkInterfaceReference is describes a network interface reference.

AzureNetworkInterfaceReferenceProperties

(Appears on: AzureNetworkInterfaceReference)

AzureNetworkInterfaceReferenceProperties is describes a network interface reference properties.

AzureNetworkProfile

(Appears on: AzureVirtualMachineProperties)

AzureNetworkProfile is specifies the network interfaces of the virtual machine.

AzureOSDisk

(Appears on: AzureStorageProfile)

AzureOSDisk is specifies information about the operating system disk used by the virtual machine.

For more information about disks, see About disks and VHDs for Azure virtual machines.

AzureOSProfile

(Appears on: AzureVirtualMachineProperties)

AzureOSProfile is specifies the operating system settings for the virtual machine.

AzureSSHConfiguration

(Appears on: AzureLinuxConfiguration)

AzureSSHConfiguration is SSH configuration for Linux based VMs running on Azure

AzureSSHPublicKey

(Appears on: AzureSSHConfiguration)

AzureSSHPublicKey is contains information about SSH certificate public key and the path on the Linux VM where the public key is placed.

AzureStorageProfile

(Appears on: AzureVirtualMachineProperties)

AzureStorageProfile is specifies the storage settings for the virtual machine disks.

AzureSubResource

(Appears on: AzureVirtualMachineProperties)

AzureSubResource is the Sub Resource definition.

AzureSubnetInfo

(Appears on: AzureMachineClassSpec)

AzureSubnetInfo is the information containing the subnet details

AzureVirtualMachineProperties

(Appears on: AzureMachineClassSpec)

AzureVirtualMachineProperties is describes the properties of a Virtual Machine.

ClassSpec

(Appears on: MachineSetSpec, MachineSpec)

ClassSpec is the class specification of machine

ConditionStatus (string alias)

(Appears on: MachineDeploymentCondition, MachineSetCondition)

CurrentStatus

(Appears on: MachineStatus)

CurrentStatus contains information about the current status of Machine.

GCPDisk

GCPDisk describes disks for GCP.

GCPMachineClassSpec

(Appears on: GCPMachineClass)

GCPMachineClassSpec is the specification of a GCPMachineClass.

GCPMetadata

GCPMetadata describes metadata for GCP.

GCPNetworkInterface

GCPNetworkInterface describes network interfaces for GCP

GCPScheduling

(Appears on: GCPMachineClassSpec)

GCPScheduling describes scheduling configuration for GCP.

GCPServiceAccount

(Appears on: GCPMachineClassSpec)

GCPServiceAccount describes service accounts for GCP.

LastOperation

(Appears on: MachineSetStatus, MachineStatus, MachineSummary)

LastOperation suggests the last operation performed on the object

MachineConfiguration

(Appears on: MachineSpec)

MachineConfiguration describes the configurations useful for the machine-controller.

MachineDeployment

Deployment enables declarative updates for machines and MachineSets.

MachineDeploymentCondition

(Appears on: MachineDeploymentStatus)

MachineDeploymentCondition describes the state of a MachineDeployment at a certain point.

MachineDeploymentConditionType (string alias)

(Appears on: MachineDeploymentCondition)

MachineDeploymentSpec

(Appears on: MachineDeployment)

MachineDeploymentSpec is the specification of the desired behavior of the MachineDeployment.

MachineDeploymentStatus

(Appears on: MachineDeployment)

MachineDeploymentStatus is the most recently observed status of the MachineDeployment.

MachineDeploymentStrategy

(Appears on: MachineDeploymentSpec)

MachineDeploymentStrategy describes how to replace existing machines with new ones.

MachineDeploymentStrategyType (string alias)

(Appears on: MachineDeploymentStrategy)

MachineOperationType (string alias)

(Appears on: LastOperation)

MachineOperationType is a label for the operation performed on a machine object.

MachinePhase (string alias)

(Appears on: CurrentStatus)

MachinePhase is a label for the condition of a machines at the current time.

MachineSetCondition

(Appears on: MachineSetStatus)

MachineSetCondition describes the state of a machine set at a certain point.

MachineSetConditionType (string alias)

(Appears on: MachineSetCondition)

MachineSetConditionType is the condition on machineset object

MachineSetSpec

(Appears on: MachineSet)

MachineSetSpec is the specification of a MachineSet.

MachineSetStatus

(Appears on: MachineSet)

MachineSetStatus holds the most recently observed status of MachineSet.

MachineSpec

(Appears on: Machine, MachineTemplateSpec)

MachineSpec is the specification of a Machine.

MachineState (string alias)

(Appears on: LastOperation)

MachineState is a current state of the machine.

MachineStatus

(Appears on: Machine)

MachineStatus holds the most recently observed status of Machine.

MachineSummary

MachineSummary store the summary of machine.

MachineTemplateSpec

(Appears on: MachineDeploymentSpec, MachineSetSpec)

MachineTemplateSpec describes the data a machine should have when created from a template

NodeTemplate

(Appears on: MachineClass)

NodeTemplate contains subfields to track all node resources and other node info required to scale nodegroup from zero

NodeTemplateSpec

(Appears on: MachineSpec)

NodeTemplateSpec describes the data a node should have when created from a template

OpenStackMachineClass

OpenStackMachineClass TODO

OpenStackMachineClassSpec

(Appears on: OpenStackMachineClass)

OpenStackMachineClassSpec is the specification of a OpenStackMachineClass.

OpenStackNetwork

(Appears on: OpenStackMachineClassSpec)

PacketMachineClassSpec

(Appears on: PacketMachineClass)

PacketMachineClassSpec is the specification of a PacketMachineClass.

RollbackConfig

(Appears on: MachineDeploymentSpec)

RollingUpdateMachineDeployment

(Appears on: MachineDeploymentStrategy)

Spec to control the desired behavior of rolling update.

Generated with gen-crd-api-reference-docs

4 - Proposals

4.1 - Excess Reserve Capacity

Excess Reserve Capacity

• Excess Reserve Capacity
  • Goal
  • Note
  • Possible Approaches
    • Approach 1: Enhance Machine-controller-manager to also entertain the excess machines
    • Approach 2: Enhance Cluster-autoscaler by simulating fake pods in it
    • Approach 3: Enhance cluster-autoscaler to support pluggable scaling-events
    • Approach 4: Make intelligent use of Low-priority pods

Goal

Currently, autoscaler optimizes the number of machines for a given application-workload. Along with effective resource utilization, this feature brings concern where, many times, when new application instances are created - they don’t find space in existing cluster. This leads the cluster-autoscaler to create new machines via MachineDeployment, which can take from 3-4 minutes to ~10 minutes, for the machine to really come-up and join the cluster. In turn, application-instances have to wait till new machines join the cluster.

One of the promising solutions to this issue is Excess Reserve Capacity. Idea is to keep a certain number of machines or percent of resources[cpu/memory] always available, so that new workload, in general, can be scheduled immediately unless huge spike in the workload. Also, the user should be given enough flexibility to choose how many resources or how many machines should be kept alive and non-utilized as this affects the Cost directly.

Note

• We decided to go with Approach-4 which is based on low priority pods. Please find more details here: https://github.com/gardener/gardener/issues/254
• Approach-3 looks more promising in long term, we may decide to adopt that in future based on developments/contributions in autoscaler-community.

Possible Approaches

Following are the possible approaches, we could think of so far.

Approach 1: Enhance Machine-controller-manager to also entertain the excess machines

• Machine-controller-manager currently takes care of the machines in the shoot cluster starting from creation-deletion-health check to efficient rolling-update of the machines. From the architecture point of view, MachineSet makes sure that X number of machines are always running and healthy. MachineDeployment controller smartly uses this facility to perform rolling-updates.

• We can expand the scope of MachineDeployment controller to maintain excess number of machines by introducing new parallel independent controller named MachineTaint controller. This will result in MCM to include Machine, MachineSet, MachineDeployment, MachineSafety, MachineTaint controllers. MachineTaint controller does not need to introduce any new CRD - analogy fits where taint-controller also resides into kube-controller-manager.

• Only Job of MachineTaint controller will be:

  • List all the Machines under each MachineDeployment.
  • Maintain taints of noSchedule and noExecute on X latest MachineObjects.
  • There should be an event-based informer mechanism where MachineTaintController gets to know about any Update/Delete/Create event of MachineObjects - in turn, maintains the noSchedule and noExecute taints on all the latest machines. - Why latest machines? - Whenever autoscaler decides to add new machines - essentially ScaleUp event - taints from the older machines are removed and newer machines get the taints. This way X number of Machines immediately becomes free for new pods to be scheduled. - While ScaleDown event, autoscaler specifically mentions which machines should be deleted, and that should not bring any concerns. Though we will have to put proper label/annotation defined by autoscaler on taintedMachines, so that autoscaler does not consider the taintedMachines for deletion while scale-down. * Annotation on tainted node: "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true"

• Implementation Details:

  • Expect new optional field ExcessReplicas in MachineDeployment.Spec. MachineDeployment controller now adds both Spec.Replicas and Spec.ExcessReplicas[if provided], and considers that as a standard desiredReplicas. - Current working of MCM will not be affected if ExcessReplicas field is kept nil.
  • MachineController currently reads the NodeObject and sets the MachineConditions in MachineObject. Machine-controller will now also read the taints/labels from the MachineObject - and maintains it on the NodeObject.

• We expect cluster-autoscaler to intelligently make use of the provided feature from MCM.

  • CA gets the input of min:max:excess from Gardener. CA continues to set the MachineDeployment.Spec.Replicas as usual based on the application-workload.
  • In addition, CA also sets the MachieDeployment.Spec.ExcessReplicas .
  • Corner-case: * CA should decrement the excessReplicas field accordingly when desiredReplicas+excessReplicas on MachineDeployment goes beyond max.

Approach 2: Enhance Cluster-autoscaler by simulating fake pods in it

• There was already an attempt by community to support this feature.
  • Refer for details to: https://github.com/kubernetes/autoscaler/pull/77/files

Approach 3: Enhance cluster-autoscaler to support pluggable scaling-events

• Forked version of cluster-autoscaler could be improved to plug-in the algorithm for excess-reserve capacity.
• Needs further discussion around upstream support.
• Create golang channel to separate the algorithms to trigger scaling (hard-coded in cluster-autoscaler, currently) from the algorithms about how to to achieve the scaling (already pluggable in cluster-autoscaler). This kind of separation can help us introduce/plug-in new algorithms (such as based node resource utilisation) without affecting existing code-base too much while almost completely re-using the code-base for the actual scaling.
• Also this approach is not specific to our fork of cluster-autoscaler. It can be made upstream eventually as well.

Approach 4: Make intelligent use of Low-priority pods

• Refer to: pod-priority-preemption
• TL; DR:
  • High priority pods can preempt the low-priority pods which are already scheduled.
  • Pre-create bunch[equivivalent of X shoot-control-planes] of low-priority pods with priority of zero, then start creating the workload pods with better priority which will reschedule the low-priority pods or otherwise keep them in pending state if the limit for max-machines has reached.
  • This is still alpha feature.

4.2 - GRPC Based Implementation of Cloud Providers

GRPC based implementation of Cloud Providers - WIP

Goal:

Currently the Cloud Providers’ (CP) functionalities ( Create(), Delete(), List() ) are part of the Machine Controller Manager’s (MCM)repository. Because of this, adding support for new CPs into MCM requires merging code into MCM which may not be required for core functionalities of MCM itself. Also, for various reasons it may not be feasible for all CPs to merge their code with MCM which is an Open Source project.

Because of these reasons, it was decided that the CP’s code will be moved out in separate repositories so that they can be maintained separately by the respective teams. Idea is to make MCM act as a GRPC server, and CPs as GRPC clients. The CP can register themselves with the MCM using a GRPC service exposed by the MCM. Details of this approach is discussed below.

How it works:

MCM acts as GRPC server and listens on a pre-defined port 5000. It implements below GRPC services. Details of each of these services are mentioned in next section.

• Register()
• GetMachineClass()
• GetSecret()

GRPC services exposed by MCM:

Register()

rpc Register(stream DriverSide) returns (stream MCMside) {}

The CP GRPC client calls this service to register itself with the MCM. The CP passes the kind and the APIVersion which it implements, and MCM maintains an internal map for all the registered clients. A GRPC stream is returned in response which is kept open througout the life of both the processes. MCM uses this stream to communicate with the client for machine operations: Create(), Delete() or List(). The CP client is responsible for reading the incoming messages continuously, and based on the operationType parameter embedded in the message, it is supposed to take the required action. This part is already handled in the package grpc/infraclient. To add a new CP client, import the package, and implement the ExternalDriverProvider interface:

type ExternalDriverProvider interface {
	Create(machineclass *MachineClassMeta, credentials, machineID, machineName string) (string, string, error)
	Delete(machineclass *MachineClassMeta, credentials, machineID string) error
	List(machineclass *MachineClassMeta, credentials, machineID string) (map[string]string, error)
}

GetMachineClass()

rpc GetMachineClass(MachineClassMeta) returns (MachineClass) {}

As part of the message from MCM for various machine operations, the name of the machine class is sent instead of the full machine class spec. The CP client is expected to use this GRPC service to get the full spec of the machine class. This optionally enables the client to cache the machine class spec, and make the call only if the machine calass spec is not already cached.

GetSecret()

rpc GetSecret(SecretMeta) returns (Secret) {}

As part of the message from MCM for various machine operations, the Cloud Config (CC) and CP credentials are not sent. The CP client is expected to use this GRPC service to get the secret which has CC and CP’s credentials from MCM. This enables the client to cache the CC and credentials, and to make the call only if the data is not already cached.

How to add a new Cloud Provider’s support

Import the package grpc/infraclient and grpc/infrapb from MCM (currently in MCM’s “grpc-driver” branch)

• Implement the interface ExternalDriverProvider
  • Create(): Creates a new machine
  • Delete(): Deletes a machine
  • List(): Lists machines
• Use the interface MachineClassDataProvider
  • GetMachineClass(): Makes the call to MCM to get machine class spec
  • GetSecret(): Makes the call to MCM to get secret containing Cloud Config and CP’s credentials

Example implementation:

Refer GRPC based implementation for AWS client: https://github.com/ggaurav10/aws-driver-grpc

5 - Usage

5.1 - Machine

Creating/Deleting machines (VM)

• Creating/Deleting machines (VM)
  • Setting up your usage environment
  • Important :
  • Creating machine
  • Inspect status of machine
  • Delete machine

Setting up your usage environment

• Follow the steps described here

Important :

Make sure that the kubernetes/machine_objects/machine.yaml points to the same class name as the kubernetes/machine_classes/aws-machine-class.yaml.

Similarly kubernetes/machine_objects/aws-machine-class.yaml secret name and namespace should be same as that mentioned in kubernetes/secrets/aws-secret.yaml

Creating machine

• Modify kubernetes/machine_objects/machine.yaml as per your requirement and create the VM as shown below:

$ kubectl apply -f kubernetes/machine_objects/machine.yaml

You should notice that the Machine Controller Manager has immediately picked up your manifest and started to create a new machine by talking to the cloud provider.

• Check Machine Controller Manager machines in the cluster

$ kubectl get machine
NAME           STATUS    AGE
test-machine   Running   5m

A new machine is created with the name provided in the kubernetes/machine_objects/machine.yaml file.

• After a few minutes (~3 minutes for AWS), you should notice a new node joining the cluster. You can verify this by running:

$ kubectl get nodes
NAME                                         STATUS     AGE     VERSION
ip-10-250-14-52.eu-east-1.compute.internal.  Ready      1m      v1.8.0

This shows that a new node has successfully joined the cluster.

Inspect status of machine

To inspect the status of any created machine, run the command given below.

$ kubectl get machine test-machine -o yaml

apiVersion: machine.sapcloud.io/v1alpha1
kind: Machine
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"machine.sapcloud.io/v1alpha1","kind":"Machine","metadata":{"annotations":{},"labels":{"test-label":"test-label"},"name":"test-machine","namespace":""},"spec":{"class":{"kind":"AWSMachineClass","name":"test-aws"}}}
  clusterName: ""
  creationTimestamp: 2017-12-27T06:58:21Z
  finalizers:
  - machine.sapcloud.io/operator
  generation: 0
  initializers: null
  labels:
    node: ip-10-250-14-52.eu-east-1.compute.internal
    test-label: test-label
  name: test-machine
  namespace: ""
  resourceVersion: "12616948"
  selfLink: /apis/machine.sapcloud.io/v1alpha1/test-machine
  uid: 535e596c-ead3-11e7-a6c0-828f843e4186
spec:
  class:
    kind: AWSMachineClass
    name: test-aws
  providerID: aws:///eu-east-1/i-00bef3f2618ffef23
status:
  conditions:
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T06:59:16Z
    message: kubelet has sufficient disk space available
    reason: KubeletHasSufficientDisk
    status: "False"
    type: OutOfDisk
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T06:59:16Z
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T06:59:16Z
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T07:00:06Z
    message: kubelet is posting ready status
    reason: KubeletReady
    status: "True"
    type: Ready
  currentStatus:
    lastUpdateTime: 2017-12-27T07:00:06Z
    phase: Running
  lastOperation:
    description: Machine is now ready
    lastUpdateTime: 2017-12-27T07:00:06Z
    state: Successful
    type: Create
  node: ip-10-250-14-52.eu-west-1.compute.internal

Delete machine

To delete the VM using the kubernetes/machine_objects/machine.yaml as shown below

$ kubectl delete -f kubernetes/machine_objects/machine.yaml

Now the Machine Controller Manager picks up the manifest immediately and starts to delete the existing VM by talking to the cloud provider. The node should be detached from the cluster in a few minutes (~1min for AWS).

5.2 - Machine Deployment

Maintaining machine replicas using machines-deployments

• Maintaining machine replicas using machines-deployments
  • Setting up your usage environment
    • Important ⚠️
  • Creating machine-deployment
  • Inspect status of machine-deployment
  • Health monitoring
  • Update your machines
    • Inspect existing cluster configuration
    • Perform a rolling update
    • Re-check cluster configuration
    • More variants of updates
  • Undo an update
  • Pause an update
  • Delete machine-deployment

Setting up your usage environment

Follow the steps described here

Important ⚠️

Make sure that the kubernetes/machine_objects/machine-deployment.yaml points to the same class name as the kubernetes/machine_classes/aws-machine-class.yaml.

Similarly kubernetes/machine_classes/aws-machine-class.yaml secret name and namespace should be same as that mentioned in kubernetes/secrets/aws-secret.yaml

Creating machine-deployment

• Modify kubernetes/machine_objects/machine-deployment.yaml as per your requirement. Modify the number of replicas to the desired number of machines. Then, create an machine-deployment.

$ kubectl apply -f kubernetes/machine_objects/machine-deployment.yaml

Now the Machine Controller Manager picks up the manifest immediately and starts to create a new machines based on the number of replicas you have provided in the manifest.

• Check Machine Controller Manager machine-deployments in the cluster

$ kubectl get machinedeployment
NAME                      READY   DESIRED   UP-TO-DATE   AVAILABLE   AGE
test-machine-deployment   3       3         3            0           10m

You will notice a new machine-deployment with your given name

• Check Machine Controller Manager machine-sets in the cluster

$ kubectl get machineset
NAME                                 DESIRED   CURRENT   READY   AGE
test-machine-deployment-5bc6dd7c8f   3         3         0       10m

You will notice a new machine-set backing your machine-deployment

• Check Machine Controller Manager machines in the cluster

$ kubectl get machine
NAME                                       STATUS    AGE
test-machine-deployment-5bc6dd7c8f-5d24b   Pending   5m
test-machine-deployment-5bc6dd7c8f-6mpn4   Pending   5m
test-machine-deployment-5bc6dd7c8f-dpt2q   Pending   5m

Now you will notice N (number of replicas specified in the manifest) new machines whose name are prefixed with the machine-deployment object name that you created.

• After a few minutes (~3 minutes for AWS), you would see that new nodes have joined the cluster. You can see this using

$  kubectl get nodes
NAME                                          STATUS    AGE       VERSION
ip-10-250-20-19.eu-west-1.compute.internal    Ready     1m        v1.8.0
ip-10-250-27-123.eu-west-1.compute.internal   Ready     1m        v1.8.0
ip-10-250-31-80.eu-west-1.compute.internal    Ready     1m        v1.8.0

This shows how new nodes have joined your cluster

Inspect status of machine-deployment

To inspect the status of any created machine-deployment run the command below,

$ kubectl get machinedeployment test-machine-deployment -o yaml

You should get the following output.

apiVersion: machine.sapcloud.io/v1alpha1
kind: MachineDeployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"machine.sapcloud.io/v1alpha1","kind":"MachineDeployment","metadata":{"annotations":{},"name":"test-machine-deployment","namespace":""},"spec":{"minReadySeconds":200,"replicas":3,"selector":{"matchLabels":{"test-label":"test-label"}},"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"test-label":"test-label"}},"spec":{"class":{"kind":"AWSMachineClass","name":"test-aws"}}}}}
  clusterName: ""
  creationTimestamp: 2017-12-27T08:55:56Z
  generation: 0
  initializers: null
  name: test-machine-deployment
  namespace: ""
  resourceVersion: "12634168"
  selfLink: /apis/machine.sapcloud.io/v1alpha1/test-machine-deployment
  uid: c0b488f7-eae3-11e7-a6c0-828f843e4186
spec:
  minReadySeconds: 200
  replicas: 3
  selector:
    matchLabels:
      test-label: test-label
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        test-label: test-label
    spec:
      class:
        kind: AWSMachineClass
        name: test-aws
status:
  availableReplicas: 3
  conditions:
  - lastTransitionTime: 2017-12-27T08:57:22Z
    lastUpdateTime: 2017-12-27T08:57:22Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  readyReplicas: 3
  replicas: 3
  updatedReplicas: 3

Health monitoring

Health monitor is also applied similar to how it’s described for machine-sets

Update your machines

Let us consider the scenario where you wish to update all nodes of your cluster from t2.xlarge machines to m5.xlarge machines. Assume that your current test-aws has its spec.machineType: t2.xlarge and your deployment test-machine-deployment points to this AWSMachineClass.

Inspect existing cluster configuration

• Check Nodes present in the cluster

$ kubectl get nodes
NAME                                          STATUS    AGE       VERSION
ip-10-250-20-19.eu-west-1.compute.internal    Ready     1m        v1.8.0
ip-10-250-27-123.eu-west-1.compute.internal   Ready     1m        v1.8.0
ip-10-250-31-80.eu-west-1.compute.internal    Ready     1m        v1.8.0

• Check Machine Controller Manager machine-sets in the cluster. You will notice one machine-set backing your machine-deployment

$ kubectl get machineset
NAME                                 DESIRED   CURRENT   READY   AGE
test-machine-deployment-5bc6dd7c8f   3         3         3       10m

• Login to your cloud provider (AWS). In the VM management console, you will find N VMs created of type t2.xlarge.

Perform a rolling update

To update this machine-deployment VMs to m5.xlarge, we would do the following:

• Copy your existing aws-machine-class.yaml

cp kubernetes/machine_classes/aws-machine-class.yaml kubernetes/machine_classes/aws-machine-class-new.yaml

• Modify aws-machine-class-new.yaml, and update its metadata.name: test-aws2 and spec.machineType: m5.xlarge
• Now create this modified MachineClass

kubectl apply -f kubernetes/machine_classes/aws-machine-class-new.yaml

• Edit your existing machine-deployment

kubectl edit machinedeployment test-machine-deployment

• Update from spec.template.spec.class.name: test-aws to spec.template.spec.class.name: test-aws2

Re-check cluster configuration

After a few minutes (~3mins)

• Check nodes present in cluster now. They are different nodes.

$ kubectl get nodes
NAME                                          STATUS    AGE       VERSION
ip-10-250-11-171.eu-west-1.compute.internal   Ready     4m        v1.8.0
ip-10-250-17-213.eu-west-1.compute.internal   Ready     5m        v1.8.0
ip-10-250-31-81.eu-west-1.compute.internal    Ready     5m        v1.8.0

• Check Machine Controller Manager machine-sets in the cluster. You will notice two machine-sets backing your machine-deployment

$ kubectl get machineset
NAME                                 DESIRED   CURRENT   READY   AGE
test-machine-deployment-5bc6dd7c8f   0         0         0       1h
test-machine-deployment-86ff45cc5    3         3         3       20m

• Login to your cloud provider (AWS). In the VM management console, you will find N VMs created of type t2.xlarge in terminated state, and N new VMs of type m5.xlarge in running state.

This shows how a rolling update of a cluster from nodes with t2.xlarge to m5.xlarge went through.

More variants of updates

• The above demonstration was a simple use case. This could be more complex like - updating the system disk image versions/ kubelet versions/ security patches etc.
• You can also play around with the maxSurge and maxUnavailable fields in machine-deployment.yaml
• You can also change the update strategy from rollingupdate to recreate

Undo an update

• Edit the existing machine-deployment

$ kubectl edit machinedeployment test-machine-deployment

• Edit the deployment to have this new field of spec.rollbackTo.revision: 0 as shown as comments in kubernetes/machine_objects/machine-deployment.yaml
• This will undo your update to the previous version.

Pause an update

• You can also pause the update while update is going on by editing the existing machine-deployment

$ kubectl edit machinedeployment test-machine-deployment

• Edit the deployment to have this new field of spec.paused: true as shown as comments in kubernetes/machine_objects/machine-deployment.yaml

• This will pause the rollingUpdate if it’s in process

• To resume the update, edit the deployment as mentioned above and remove the field spec.paused: true updated earlier

Delete machine-deployment

• To delete the VM using the kubernetes/machine_objects/machine-deployment.yaml

$ kubectl delete -f kubernetes/machine_objects/machine-deployment.yaml

The Machine Controller Manager picks up the manifest and starts to delete the existing VMs by talking to the cloud provider. The nodes should be detached from the cluster in a few minutes (~1min for AWS).

5.3 - Machine Set

Maintaining machine replicas using machines-sets

• Maintaining machine replicas using machines-sets
  • Setting up your usage environment
  • Important ⚠️
  • Creating machine-set
  • Inspect status of machine-set
  • Health monitoring
  • Delete machine-set

Setting up your usage environment

• Follow the steps described here

Important ⚠️

Make sure that the kubernetes/machines_objects/machine-set.yaml points to the same class name as the kubernetes/machine_classes/aws-machine-class.yaml.

Similarly kubernetes/machine_classes/aws-machine-class.yaml secret name and namespace should be same as that mentioned in kubernetes/secrets/aws-secret.yaml

Creating machine-set

• Modify kubernetes/machine_objects/machine-set.yaml as per your requirement. You can modify the number of replicas to the desired number of machines. Then, create an machine-set:

$ kubectl apply -f kubernetes/machine_objects/machine-set.yaml

You should notice that the Machine Controller Manager has immediately picked up your manifest and started to create a new machines based on the number of replicas you have provided in the manifest.

• Check Machine Controller Manager machine-sets in the cluster

$ kubectl get machineset
NAME               DESIRED   CURRENT   READY   AGE
test-machine-set   3         3         0       1m

You will see a new machine-set with your given name

• Check Machine Controller Manager machines in the cluster:

$ kubectl get machine
NAME                     STATUS    AGE
test-machine-set-b57zs   Pending   5m
test-machine-set-c4bg8   Pending   5m
test-machine-set-kvskg   Pending   5m

Now you will see N (number of replicas specified in the manifest) new machines whose names are prefixed with the machine-set object name that you created.

• After a few minutes (~3 minutes for AWS), you should notice new nodes joining the cluster. You can verify this by running:

$ kubectl get nodes
NAME                                         STATUS    AGE       VERSION
ip-10-250-0-234.eu-west-1.compute.internal   Ready     3m        v1.8.0
ip-10-250-15-98.eu-west-1.compute.internal   Ready     3m        v1.8.0
ip-10-250-6-21.eu-west-1.compute.internal    Ready     2m        v1.8.0

This shows how new nodes have joined your cluster

Inspect status of machine-set

• To inspect the status of any created machine-set run the following command:

$ kubectl get machineset test-machine-set -o yaml

apiVersion: machine.sapcloud.io/v1alpha1
kind: MachineSet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"machine.sapcloud.io/v1alpha1","kind":"MachineSet","metadata":{"annotations":{},"name":"test-machine-set","namespace":"","test-label":"test-label"},"spec":{"minReadySeconds":200,"replicas":3,"selector":{"matchLabels":{"test-label":"test-label"}},"template":{"metadata":{"labels":{"test-label":"test-label"}},"spec":{"class":{"kind":"AWSMachineClass","name":"test-aws"}}}}}
  clusterName: ""
  creationTimestamp: 2017-12-27T08:37:42Z
  finalizers:
  - machine.sapcloud.io/operator
  generation: 0
  initializers: null
  name: test-machine-set
  namespace: ""
  resourceVersion: "12630893"
  selfLink: /apis/machine.sapcloud.io/v1alpha1/test-machine-set
  uid: 3469faaa-eae1-11e7-a6c0-828f843e4186
spec:
  machineClass: {}
  minReadySeconds: 200
  replicas: 3
  selector:
    matchLabels:
      test-label: test-label
  template:
    metadata:
      creationTimestamp: null
      labels:
        test-label: test-label
    spec:
      class:
        kind: AWSMachineClass
        name: test-aws
status:
  availableReplicas: 3
  fullyLabeledReplicas: 3
  machineSetCondition: null
  lastOperation:
    lastUpdateTime: null
  observedGeneration: 0
  readyReplicas: 3
  replicas: 3

Health monitoring

• If you try to delete/terminate any of the machines backing the machine-set by either talking to the Machine Controller Manager or from the cloud provider, the Machine Controller Manager recreates a matching healthy machine to replace the deleted machine.
• Similarly, if any of your machines are unreachable or in an unhealthy state (kubelet not ready / disk pressure) for longer than the configured timeout (~ 5mins), the Machine Controller Manager recreates the nodes to replace the unhealthy nodes.

Delete machine-set

• To delete the VM using the kubernetes/machine_objects/machine-set.yaml:

$ kubectl delete -f kubernetes/machine-set.yaml

Now the Machine Controller Manager has immediately picked up your manifest and started to delete the existing VMs by talking to the cloud provider. Your nodes should be detached from the cluster in a few minutes (~1min for AWS).

5.4 - Prerequisite

Setting up the usage environment

• Setting up the usage environment
  • Important ⚠️
  • Set KUBECONFIG
  • Replace provider credentials and desired VM configurations
  • Deploy required CRDs and Objects
  • Check current cluster state

Important ⚠️

All paths are relative to the root location of this project repository.

Run the Machine Controller Manager either as described in Setting up a local development environment or Deploying the Machine Controller Manager into a Kubernetes cluster.

Make sure that the following steps are run before managing machines/ machine-sets/ machine-deploys.

Set KUBECONFIG

Using the existing Kubeconfig, open another Terminal panel/window with the KUBECONFIG environment variable pointing to this Kubeconfig file as shown below,

$ export KUBECONFIG=<PATH_TO_REPO>/dev/kubeconfig.yaml

Replace provider credentials and desired VM configurations

Open kubernetes/machine_classes/aws-machine-class.yaml and replace required values there with the desired VM configurations.

Similarily open kubernetes/secrets/aws-secret.yaml and replace - userData, providerAccessKeyId, providerSecretAccessKey with base64 encoded values of cloudconfig file, AWS access key id, and AWS secret access key respectively. Use the following command to get the base64 encoded value of your details

$ echo "sample-cloud-config" | base64
base64-encoded-cloud-config

Do the same for your access key id and secret access key.

Deploy required CRDs and Objects

Create all the required CRDs in the cluster using kubernetes/crds.yaml

$ kubectl apply -f kubernetes/crds.yaml

Create the class template that will be used as an machine template to create VMs using kubernetes/machine_classes/aws-machine-class.yaml

$ kubectl apply -f kubernetes/machine_classes/aws-machine-class.yaml

Create the secret used for the cloud credentials and cloudconfig using kubernetes/secrets/aws-secret.yaml

$ kubectl apply -f kubernetes/secrets/aws-secret.yaml

Check current cluster state

Get to know the current cluster state using the following commands,

• Checking aws-machine-class in the cluster

$ kubectl get awsmachineclass
NAME       MACHINE TYPE   AMI          AGE
test-aws   t2.large       ami-123456   5m

• Checking kubernetes secrets in the cluster

$ kubectl get secret
NAME                  TYPE                                  DATA      AGE
test-secret           Opaque                                3         21h

• Checking kubernetes nodes in the cluster

$ kubectl get nodes

Lists the default set of nodes attached to your cluster

• Checking Machine Controller Manager machines in the cluster

$ kubectl get machine
No resources found.

• Checking Machine Controller Manager machine-sets in the cluster

$ kubectl get machineset
No resources found.

• Checking Machine Controller Manager machine-deploys in the cluster

$ kubectl get machinedeployment
No resources found.

6 - Machine Controller Manager FAQ

Frequently Asked Questions

The answers in this FAQ apply to the newest (HEAD) version of Machine Controller Manager. If you’re using an older version of MCM please refer to corresponding version of this document. Few of the answers assume that the MCM being used is in conjuction with cluster-autoscaler:

Table of Contents:

• Basics

  • What is Machine Controller Manager?
  • Why is my machine deleted?
  • What are the different sub-controllers in MCM?
  • What is Safety Controller in MCM?

• How to?

  • How to install MCM in a Kubernetes cluster?
  • How to better control the rollout process of the worker nodes?
  • How to scale down MachineDeployment by selective deletion of machines?
  • How to force delete a machine?
  • How to pause the ongoing rolling-update of the machinedeployment?
  • How to avoid garbage collection of your node?

• Internals

  • What is the high level design of MCM?
  • What are the different configuration options in MCM?
  • What are the different timeouts/configurations in a machine’s lifecycle?
  • How is the drain of a machine implemented?
  • How are the stateful applications drained during machine deletion?
  • How does maxEvictRetries configuration work with drainTimeout configuration?
  • What are the different phases of a machine?

• Troubleshooting

  • My machine is stuck in deletion for 1 hr, why?
  • My machine is not joining the cluster, why?

• Developer

  • How should I test my code before submitting a PR?
  • I need to change the APIs, what are the recommended steps?
  • How can I update the dependencies of MCM?

• In the context of Gardener

  • How can I configure MCM using Shoot resource?
  • How is my worker-pool spread across zones?

Basics

What is Machine Controller Manager?

Machine Controller Manager aka MCM is a bunch of controllers used for the lifecycle management of the worker machines. It reconciles a set of CRDs such as Machine, MachineSet, MachineDeployment which depicts the functionality of Pod, Replicaset, Deployment of the core Kubernetes respectively. Read more about it at README.

• Gardener uses MCM to manage its Kubernetes nodes of the shoot cluster. However, by design, MCM can be used independent of Gardener.

Why is my machine deleted?

A machine is deleted by MCM generally for 2 reasons-

• Machine is unhealthy for at least MachineHealthTimeout period. The default MachineHealthTimeout is 10 minutes.

  • By default, a machine is considered unhealthy if any of the following node conditions - DiskPressure, KernelDeadlock, FileSystem, Readonly is set to true, or KubeletReady is set to false. However, this is something that is configurable using the following flag.

• Machine is scaled down by the MachineDeployment resource.

  • This is very usual when an external controller cluster-autoscaler (aka CA) is used with MCM. CA deletes the under-utilized machines by scaling down the MachineDeployment. Read more about cluster-autoscaler’s scale down behavior here.

What are the different sub-controllers in MCM?

MCM mainly contains the following sub-controllers:

• MachineDeployment Controller: Responsible for reconciling the MachineDeployment objects. It manages the lifecycle of the MachineSet objects.
• MachineSet Controller: Responsible for reconciling the MachineSet objects. It manages the lifecycle of the Machine objects.
• Machine Controller: responsible for reconciling the Machine objects. It manages the lifecycle of the actual VMs/machines created in cloud/on-prem. This controller has been moved out of tree. Please refer an AWS machine controller for more info - link.
• Safety-controller: Responsible for handling the unidentified/unknown behaviors from the cloud providers. Please read more about its functionality below.

What is Safety Controller in MCM?

Safety Controller contains following functions:

• Orphan VM handler:
  • It lists all the VMs in the cloud matching the tag of given cluster name and maps the VMs with the machine objects using the ProviderID field. VMs without any backing machine objects are logged and deleted after confirmation.
  • This handler runs every 30 minutes and is configurable via machine-safety-orphan-vms-period flag.
• Freeze mechanism:
  • Safety Controller freezes the MachineDeployment and MachineSet controller if the number of machine objects goes beyond a certain threshold on top of Spec.Replicas. It can be configured by the flag –safety-up or –safety-down and also machine-safety-overshooting-period.
  • Safety Controller freezes the functionality of the MCM if either of the target-apiserver or the control-apiserver is not reachable.
  • Safety Controller unfreezes the MCM automatically once situation is resolved to normal. A freeze label is applied on MachineDeployment/MachineSet to enforce the freeze condition.

How to?

How to install MCM in a Kubernetes cluster?

MCM can be installed in a cluster with following steps:

• Apply all the CRDs from here

• Apply all the deployment, role-related objects from here.

  • Control cluster is the one where the machine-* objects are stored. Target cluster is where all the node objects are registered.

How to better control the rollout process of the worker nodes?

MCM allows configuring the rollout of the worker machines using maxSurge and maxUnavailable fields. These fields are applicable only during the rollout process and means nothing in general scale up/down scenarios. The overall process is very similar to how the Deployment Controller manages pods during RollingUpdate.

• maxSurge refers to the number of additional machines that can be added on top of the Spec.Replicas of MachineDeployment during rollout process.
• maxUnavailable refers to the number of machines that can be deleted from Spec.Replicas field of the MachineDeployment during rollout process.

How to scale down MachineDeployment by selective deletion of machines?

During scale down, triggered via MachineDeployment/MachineSet, MCM prefers to delete the machine/s which have the least priority set. Each machine object has an annotation machinepriority.machine.sapcloud.io set to 3 by default. Admin can reduce the priority of the given machines by changing the annotation value to 1. The next scale down by MachineDeployment shall delete the machines with the least priority first.

How to force delete a machine?

A machine can be force deleted by adding the label force-deletion: "True" on the machine object before executing the actual delete command. During force deletion, MCM skips the drain function and simply triggers the deletion of the machine. This label should be used with caution as it can violate the PDBs for pods running on the machine.

How to pause the ongoing rolling-update of the machinedeployment?

An ongoing rolling-update of the machine-deployment can be paused by using spec.paused field. See the example below:

apiVersion: machine.sapcloud.io/v1alpha1
kind: MachineDeployment
metadata:
  name: test-machine-deployment
spec:
  paused: true

It can be unpaused again by removing the Paused field from the machine-deployment.

How to avoid garbage collection of your node?

MCM provides an in-built safety mechanism to garbage collect VMs which have no corresponding machine object. This is done to save costs and is one of the key features of MCM. However, sometimes users might like to add nodes directly to the cluster without the help of MCM and would prefer MCM to not garbage collect such VMs. To do so they should remove/not-use tags on their VMs containing the following strings:

1. kubernetes.io/cluster/
2. kubernetes.io/role/
3. kubernetes-io-cluster-
4. kubernetes-io-role-

Internals

What is the high level design of MCM?

Please refer the following document.

What are the different configuration options in MCM?

MCM allows configuring many knobs to fine-tune its behavior according to the user’s need. Please refer to the link to check the exact configuration options.

What are the different timeouts/configurations in a machine’s lifecycle?

A machine’s lifecycle is governed by mainly following timeouts, which can be configured here.

• MachineDrainTimeout: Amount of time after which drain times out and the machine is force deleted. Default ~2 hours.
• MachineHealthTimeout: Amount of time after which an unhealthy machine is declared Failed and the machine is replaced by MachineSet controller.
• MachineCreationTimeout: Amount of time after which a machine creation is declared Failed and the machine is replaced by the MachineSet controller.
• NodeConditions: List of node conditions which if set to true for MachineHealthTimeout period, the machine is declared Failed and replaced by MachineSet controller.
• MaxEvictRetries: An integer number depicting the number of times a failed eviction should be retried on a pod during drain process. A pod is deleted after max-retries.

How is the drain of a machine implemented?

MCM imports the functionality from the upstream Kubernetes-drain library. Although, few parts have been modified to make it work best in the context of MCM. Drain is executed before machine deletion for graceful migration of the applications. Drain internally uses the EvictionAPI to evict the pods and triggers the Deletion of pods after MachineDrainTimeout. Please note:

• Stateless pods are evicted in parallel.
• Stateful applications (with PVCs) are serially evicted. Please find more info in this answer below.

How are the stateful applications drained during machine deletion?

Drain function serially evicts the stateful-pods. It is observed that serial eviction of stateful pods yields better overall availability of pods as the underlying cloud in most cases detaches and reattaches disks serially anyways. It is implemented in the following manner:

• Drain lists all the pods with attached volumes. It evicts very first stateful-pod and waits for its related entry in Node object’s .status.volumesAttached to be removed by KCM. It does the same for all the stateful-pods.
• It waits for PvDetachTimeout (default 2 minutes) for a given pod’s PVC to be removed, else moves forward.

How does maxEvictRetries configuration work with drainTimeout configuration?

It is recommended to only set MachineDrainTimeout. It satisfies the related requirements. MaxEvictRetries is auto-calculated based on MachineDrainTimeout, if maxEvictRetries is not provided. Following will be the overall behavior of both configurations together:

• If maxEvictRetries isn’t set and only maxDrainTimeout is set:
  • MCM auto calculates the maxEvictRetries based on the drainTimeout.
• If drainTimeout isn’t set and only maxEvictRetries is set:
  • Default drainTimeout and user provided maxEvictRetries for each pod is considered.
• If both maxEvictRetries and drainTimoeut are set:
  • Then both will be respected.
• If none are set:
  • Defaults are respected.

What are the different phases of a machine?

A phase of a machine can be identified with Machine.Status.CurrentStatus.Phase. Following are the possible phases of a machine object:

• Pending: Machine creation call has succeeded. MCM is waiting for machine to join the cluster.
• CrashLoopBackOff: Machine creation call has failed. MCM will retry the operation after a minor delay.
• Running: Machine creation call has succeeded. Machine has joined the cluster successfully.
• Unknown: Machine health checks are failing, eg kubelet has stopped posting the status.
• Failed: Machine health checks have failed for a prolonged time. Hence it is declared failed. MachineSet controller will replace such machines immediately.
• Terminating: Machine is being terminated. Terminating state is set immediately when the deletion is triggered for the machine object. It also includes time when it’s being drained.

Troubleshooting

My machine is stuck in deletion for 1 hr, why?

In most cases, the Machine.Status.LastOperation provides information around why a machine can’t be deleted. Though following could be the reasons but not limited to:

• Pod/s with misconfigured PDBs block the drain operation. PDBs with maxUnavailable set to 0, doesn’t allow the eviction of the pods. Hence, drain/eviction is retried till MachineDrainTimeout. Default MachineDrainTimeout could be as large as ~2hours. Hence, blocking the machine deletion.
  • Short term: User can manually delete the pod in the question, with caution.
  • Long term: Please set more appropriate PDBs which allow disruption of at least one pod.
• Expired cloud credentials can block the deletion of the machine from infrastructure.
• Cloud provider can’t delete the machine due to internal errors. Such situations are best debugged by using cloud provider specific CLI or cloud console.

My machine is not joining the cluster, why?

In most cases, the Machine.Status.LastOperation provides information around why a machine can’t be created. It could possibly be debugged with following steps:

• Verify if the machine is actually created in the cloud. User can use the Machine.Spec.ProviderId to query the machine in cloud.
• A Kubernetes node is generally bootstrapped with the cloud-config. Please verify, if MachineDeployment is pointing the correct MachineClass, and MachineClass is pointing to the correct Secret. The secret object contains the actual cloud-config in base64 format which will be used to boot the machine.
• User must also check the logs of the MCM pod to understand any broken logical flow of reconciliation.

Developer

How should I test my code before submitting a PR?

• Developer can locally setup the MCM using following guide
• Developer must also enhance the unit tests related to the incoming changes.
• Developer can locally run the unit test by executing:

make test-unit

I need to change the APIs, what are the recommended steps?

Developer should add/update the API fields at both of the following places:

• https://github.com/gardener/machine-controller-manager/blob/master/pkg/apis/machine/types.go
• https://github.com/gardener/machine-controller-manager/tree/master/pkg/apis/machine/v1alpha1

Once API changes are done, auto-generate the code using following command:

./hack/generate-code

Please ignore the API-violation errors for now.

How can I update the dependencies of MCM?

MCM uses gomod for depedency management. Developer should add/udpate depedency in the go.mod file. Please run following command to automatically revendor the dependencies.

make revendor

In the context of Gardener

How can I configure MCM using Shoot resource?

All of the knobs of MCM can be configured by the workers section of the shoot resource.

• Gardener creates a MachineDeployment per zone for each worker-pool under workers section.
• workers.dataVolumes allows to attach multiple disks to a machine during creation. Refer the link.
• workers.machineControllerManager allows configuration of multiple knobs of the MachineDeployment from the shoot resource.

How is my worker-pool spread across zones?

Shoot resource allows the worker-pool to spread across multiple zones using the field workers.zones. Refer link.

• Gardener creates one MachineDeployment per zone. Each MachineDeployment is initiated with the following replica:

MachineDeployment.Spec.Replicas = (Workers.Minimum)/(Number of availibility zones)

7 - Outline

Machine Controller Manager

CORE – ./machine-controller-manager(provider independent) Out of tree : Machine controller (provider specific) MCM is a set controllers:

• Machine Deployment Controller

• Machine Set Controller

• Machine Controller

• Machine Safety Controller

Questions and refactoring Suggestions

Refactoring

Clarification

Bugs/Enhancements

Code references

1.1 code1

       if machineSet.DeletionTimestamp == nil {
        
        		// manageReplicas is the core machineSet method where scale up/down occurs
        
        		// It is not called when deletion timestamp is set
        
        		manageReplicasErr = c.manageReplicas(ctx, filteredMachines, machineSet)
        
        ​
        
        	} else if machineSet.DeletionTimestamp != nil { 
        
            //FIX: change this to simple else without the if

1.2 code2

    defer dc.enqueueMachineDeploymentAfter(deployment, 10*time.Minute)
    
    *  `Clarification`:  Why  is  this  conversion  needed?
    
    err = v1alpha1.Convert_v1alpha1_MachineDeployment_To_machine_MachineDeployment(deployment, internalMachineDeployment, nil)

1.3 code3

// rollback is not re-entrant in case the underlying machine sets are updated with a new

	// revision so we should ensure that we won't proceed to update machine sets until we

	// make sure that the deployment has cleaned up its rollback spec in subsequent enqueues.

	if d.Spec.RollbackTo != nil {

		return dc.rollback(ctx, d, machineSets, machineMap)

	}