1 - Deployment

1.1 - Kubernetes

Deploying the Machine Controller Manager into a Kubernetes cluster

As already mentioned, the Machine Controller Manager is designed to run as controller in a Kubernetes cluster. The existing source code can be compiled and tested on a local machine as described in Setting up a local development environment. You can deploy the Machine Controller Manager using the steps described below.

Prepare the cluster

  • Connect to the remote kubernetes cluster where you plan to deploy the Machine Controller Manager using the kubectl. Set the environment variable KUBECONFIG to the path of the yaml file containing the cluster info.
  • Now, create the required CRDs on the remote cluster using the following command,
$ kubectl apply -f kubernetes/crds.yaml

Build the Docker image

⚠️ Modify the Makefile to refer to your own registry.

  • Run the build which generates the binary to bin/machine-controller-manager
$ make build
  • Build docker image from latest compiled binary
$ make docker-image
  • Push the last created docker image onto the online docker registry.
$ make push
  • Now you can deploy this docker image to your cluster. A sample development file is given at. By default, the deployment manages the cluster it is running in. Optionally, the kubeconfig could also be passed as a flag as described in /kubernetes/deployment/in-tree/deployment.yaml. This is done when you want your controller running outside the cluster to be managed from.
$ kubectl apply -f kubernetes/deployment/in-tree/deployment.yaml
  • Also deploy the required clusterRole and clusterRoleBindings
$ kubectl apply -f kubernetes/deployment/in-tree/clusterrole.yaml
$ kubectl apply -f kubernetes/deployment/in-tree/clusterrolebinding.yaml

Configuring optional parameters while deploying

Machine-controller-manager supports several configurable parameters while deploying. Refer to the following lines, to know how each parameter can be configured, and what it’s purpose is for.

Usage

To start using Machine Controller Manager, follow the links given at usage here.

2 - Development

2.1 - Adding Support for a Cloud Provider

Adding support for a new provider

Steps to be followed while implementing a new (hyperscale) provider are mentioned below. This is the easiest way to add new provider support using a blueprint code.

However, you may also develop your machine controller from scratch, which would provide you with more flexibility. First, however, make sure that your custom machine controller adheres to the Machine.Status struct defined in the MachineAPIs. This will make sure the MCM can act with higher-level controllers like MachineSet and MachineDeployment controller. The key is the Machine.Status.CurrentStatus.Phase key that indicates the status of the machine object.

Our strong recommendation would be to follow the steps below. This provides the most flexibility required to support machine management for adding new providers. And if you feel to extend the functionality, feel free to update our machine controller libraries.

Setting up your repository

  1. Create a new empty repository named machine-controller-manager-provider-{provider-name} on GitHub username/project. Do not initialize this repository with a README.
  2. Copy the remote repository URL (HTTPS/SSH) to this repository displayed once you create this repository.
  3. Now, on your local system, create directories as required. {your-github-username} given below could also be {github-project} depending on where you have created the new repository.
    mkdir -p $GOPATH/src/github.com/{your-github-username}
    
  4. Navigate to this created directory.
    cd $GOPATH/src/github.com/{your-github-username}
    
  5. Clone this repository on your local machine.
    git clone git@github.com:gardener/machine-controller-manager-provider-sampleprovider.git
    
  6. Rename the directory from machine-controller-manager-provider-sampleprovider to machine-controller-manager-provider-{provider-name}.
    mv machine-controller-manager-provider-sampleprovider machine-controller-manager-provider-{provider-name}
    
  7. Navigate into the newly-created directory.
    cd machine-controller-manager-provider-{provider-name}
    
  8. Update the remote origin URL to the newly created repository’s URL you had copied above.
    git remote set-url origin git@github.com:{your-github-username}/machine-controller-manager-provider-{provider-name}.git
    
  9. Rename GitHub project from gardener to {github-org/your-github-username} wherever you have cloned the repository above. Also, edit all occurrences of the word sampleprovider to {provider-name} in the code. Then, use the hack script given below to do the same.
    make rename-project PROJECT_NAME={github-org/your-github-username} PROVIDER_NAME={provider-name}
    eg:
        make rename-project PROJECT_NAME=gardener PROVIDER_NAME=AmazonWebServices (or)
        make rename-project PROJECT_NAME=githubusername PROVIDER_NAME=AWS
    
  10. Now, commit your changes and push them upstream.
    git add -A
    git commit -m "Renamed SampleProvide to {provider-name}"
    git push origin master
    

Code changes required

The contract between the Machine Controller Manager (MCM) and the Machine Controller (MC) AKA driver has been documented here and the machine error codes can be found here. You may refer to them for any queries.

⚠️

  • Keep in mind that there should be a unique way to map between machine objects and VMs. This can be done by mapping machine object names with VM-Name/ tags/ other metadata.
  • Optionally, there should also be a unique way to map a VM to its machine class object. This can be done by tagging VM objects with tags/resource groups associated with the machine class.

Steps to integrate

  1. Update the pkg/provider/apis/provider_spec.go specification file to reflect the structure of the ProviderSpec blob. It typically contains the machine template details in the MachineClass object. Follow the sample spec provided already in the file. A sample provider specification can be found here.
  2. Fill in the methods described at pkg/provider/core.go to manage VMs on your cloud provider. Comments are provided above each method to help you fill them up with desired REQUEST and RESPONSE parameters.
    • A sample provider implementation for these methods can be found here.
    • Fill in the required methods CreateMachine(), and DeleteMachine() methods.
    • Optionally fill in methods like GetMachineStatus(), ListMachines(), and GetVolumeIDs(). You may choose to fill these once the working of the required methods seems to be working.
    • GetVolumeIDs() expects VolumeIDs to be decoded from the volumeSpec based on the cloud provider.
    • There is also an OPTIONAL method GenerateMachineClassForMigration() that helps in migration of {ProviderSpecific}MachineClass to MachineClass CR (custom resource). This only makes sense if you have an existing implementation (in-tree) acting on different CRD types. You would like to migrate this. If not, you MUST return an error (machine error UNIMPLEMENTED) to avoid processing this step.
  3. Perform validation of APIs that you have described and make it a part of your methods as required at each request.
  4. Write unit tests to make it work with your implementation by running make test.
    make test
    
  5. Re-generate the vendors to update any new vendors imported.
    make revendor
    
  6. Update the sample YAML files on the kubernetes/ directory to provide sample files through which the working of the machine controller can be tested.
  7. Update README.md to reflect any additional changes

Testing your code changes

Make sure $TARGET_KUBECONFIG points to the cluster where you wish to manage machines. Likewise, $CONTROL_NAMESPACE represents the namespaces where MCM is looking for machine CR objects, and $CONTROL_KUBECONFIG points to the cluster that holds these machine CRs.

  1. On the first terminal running at $GOPATH/src/github.com/{github-org/your-github-username}/machine-controller-manager-provider-{provider-name},
    • Run the machine controller (driver) using the command below.
      make start
      
  2. On the second terminal pointing to $GOPATH/src/github.com/gardener,
    • Clone the latest MCM code
      git clone git@github.com:gardener/machine-controller-manager.git
      
    • Navigate to the newly-created directory.
      cd machine-controller-manager
      
    • Deploy the required CRDs from the machine-controller-manager repo,
      kubectl apply -f kubernetes/crds
      
    • Run the machine-controller-manager in the master branch
      make start
      
  3. On the third terminal pointing to $GOPATH/src/github.com/{github-org/your-github-username}/machine-controller-manager-provider-{provider-name}
    • Fill in the object files given below and deploy them as described below.
    • Deploy the machine-class
      kubectl apply -f kubernetes/machine-class.yaml
      
    • Deploy the kubernetes secret if required.
      kubectl apply -f kubernetes/secret.yaml
      
    • Deploy the machine object and make sure it joins the cluster successfully.
      kubectl apply -f kubernetes/machine.yaml
      
    • Once the machine joins, you can test by deploying a machine-deployment.
    • Deploy the machine-deployment object and make sure it joins the cluster successfully.
      kubectl apply -f kubernetes/machine-deployment.yaml
      
    • Make sure to delete both the machine and machine-deployment objects after use.
      kubectl delete -f kubernetes/machine.yaml
      kubectl delete -f kubernetes/machine-deployment.yaml
      

Releasing your docker image

  1. Make sure you have logged into gcloud/docker using the CLI.
  2. To release your docker image, run the following.
    make release IMAGE_REPOSITORY=<link-to-image-repo>
  1. A sample kubernetes deploy file can be found at kubernetes/deployment.yaml. Update the same (with your desired MCM and MC images) to deploy your MCM pod.

2.2 - Adding Support for a Cloud Provider (Legacy)

Adding support for a new cloud provider

For adding support for a new cloud provider in the Machine Controller Manager, follow the steps described below. Replace provider with your provider-name.

  1. Add a ProviderMachineClass CRD similar to existing AWSMachineClass into kubernetes/crds.yaml.
  2. Add ProviderMachineClass structs similar to existing AWSMachineClass into the machine APIs into pkg/apis/machine/types.go and pkg/apis/machine/v1alpha1/types.go. This would be the machineClass template used to describe provider specific configurations.
  3. Add the Go structures of your machine class (list) to pkg/apis/machine/register.go and pkg/apis/machine/v1alpha1/register.go to allow reporting events on these objects.
  4. Regenerate the machine API clients by running ./hack/generate-code
  5. Add validation for the new provider machine class at pkg/apis/machine/validation/providermachineclass.go similar to pkg/apis/machine/validation/awsmachineclass.go
  6. Update pkg/controller/machine_util.go to allow validation of the new provider.
  7. Add a new driver into pkg/driver/driver_provider.go similar to pkg/driver/driver_aws.go to implement the driver interface.
  8. Update pkg/driver/driver.go to add a new switch case to support the new provider driver.
  9. Add a new method in pkg/controller/machine_safety.go called checkProviderMachineClass similar to the existing method called checkAWSMachineClass present in the same file. Now invoke this method as a go-routine in the method checkVMObjects.
  10. Extend the StartControllers() function in cmd/machine-controller-manager/app/controllermanager.go to only start if your new machine class is under the available resources.
  11. Update pkg/controller/controller.go to add new providerMachineClassLister, providerMachineClassQueue, awsMachineClassSynced into the controller struct. Also initialize them in NewController() method.
  12. Add a new file pkg/controller/providermachineclass.go that allows re-queuing of machines which refer to an modified providerMachineClass.
  13. Update pkg/controller/controller.go to extend WaitForCacheSync and .Shutdown() similar to other cloud providers.
  14. Update the example ClusterRole in kubernetes/deployment/in-tree/clusterrole.yaml to allow operations on your new machine class.
  15. Update pkg/controller/controller.go, pkg/controller/secret.go, pkg/controller/secret_util.go to add event handlers to add/remove finalizers referenced by your machine Class. Refer this commit.

2.3 - Integration Tests

Integration tests

Usage

General setup & configurations

Integration tests for machine-controller-manager-provider-{provider-name} can be executed manually by following below steps.

  1. Clone the repository machine-controller-manager-provider-{provider-name} on the local system.
  2. Navigate to machine-controller-manager-provider-{provider-name} directory and create a dev sub-directory in it.
  3. Copy the kubeconfig of Control Cluster from into dev/control-kubeconfig.yaml.
  4. (optional) Copy the kubeconfig of Target Cluster into dev/target-kubeconfig.yaml and update the Makefile variable TARGET_KUBECONFIG to point to dev/target-kubeconfig.yaml.
  5. If the tags on instances & associated resources on the provider are of String type (for example, GCP tags on its instances are of type String and not key-value pair) then add TAGS_ARE_STRINGS := true in the Makefile and export it.
  6. Atleast, one of the two controllers’ container images must be set in the Makefile variables MCM_IMAGE_TAG and MC_IMAGE_TAG for the controllers to run in the Control Cluster . These images will be used along with kubernetes/deployment.yaml to deploy/update controllers in the Control Cluster . If the intention is to run the controllers locally then unset the variables MCM_IMAGE_TAG and MC_IMAGE_TAG and set variable MACHINE_CONTROLLER_MANAGER_DEPLOYMENT_NAME := machine-controller-manager in the Makefile.
  7. In order to apply the CRDs when the Control Cluster is a Gardener Shoot or if none of the controller images are specified, machine-controller-manager repository will be cloned automatically. Incase, this repository already exists in local system, then create a softlink as below which helps to test changes in machine-controller-manager quickly.
    ln -sf <path-for-machine-controller-manager-repo> dev/mcm
    

Scenario based additional configurations

Gardener Shoot as the Control Cluster

If the Control Cluster is a Gardener Shoot cluster then,

  1. Deploy a Secret named test-mc-secret (that contains the provider secret and cloud-config) in the default namespace of the Control Cluster. Refer these MachineClass templates for the same.
  2. Create a dev/machineclassv1.yaml file in the cloned repository. The name of the MachineClass itself should be test-mc-v1. The value of providerSpec.secretRef.name should be test-mc-secret.
  3. (Optional) Create an additional dev/machineclassv2.yaml file similar to above but with a bigger machine type and update the Makefile variable MACHINECLASS_V2 to point to dev/machineclassv2.yaml.

Gardener Seed as the Control Cluster

If the Control Cluster is a Gardener SEED cluster then, the suite ideally employs the already existing MachineClass and Secrets. However,

  1. (Optional) User can employ a custom MachineClass for the tests using below steps:
    1. Deploy a Secret named test-mc-secret (that contains the provider secret and cloud-config) in the shoot namespace of the Control Cluster. That is, the value of metadata.namespace should be technicalID of the Shoot and it will be of the pattern shoot--<project>--<shoot-name>. Refer these MachineClass templates for the same.
    2. Create a dev/machineclassv1.yaml file.
      1. providerSpec.secretRef.name should refer the secret created in the previous step.
      2. metadata.namespace and providerSpec.secretRef.namespace should be technicalID (shoot--<project>--<shoot-name>) of the shoot.
      3. The name of the MachineClass itself should be test-mc-v1.

Running the tests

  1. There is a rule test-integration in the Makefile, which can be used to start the integration test:
    $ make test-integration 
    Starting integration tests...
    Running Suite: Controller Suite
    ===============================
    
  2. The controllers log files (mcm_process.log and mc_process.log) are stored in .ci/controllers-test/logs repo and can be used later.

Adding Integration Tests for new providers

For a new provider, Running Integration tests works with no changes. But for the orphan resource test cases to work correctly, the provider-specific API calls and the Resource Tracker Interface (RTI) should be implemented. Please check machine-controller-manager-provider-aws for reference.

Extending integration tests

  • Update ControllerTests to be extend the testcases for all providers. Common testcases for machine|machineDeployment creation|deletion|scaling are packaged into ControllerTests.
  • To extend the provider specfic test cases, the changes should be done in the machine-controller-manager-provider-{provider-name} repository. For example, to extended the testcases for machine-controller-manager-provider-aws, make changes to test/integration/controller/controller_test.go inside the machine-controller-manager-provider-aws repository. commons contains the Cluster and Clientset objects that makes it easy to extend the tests.

2.4 - Local Setup

Preparing the Local Development Setup (Mac OS X)

Conceptionally, the Machine Controller Manager is designed to run in a container within a Pod inside a Kubernetes cluster. For development purposes, you can run the Machine Controller Manager as a Go process on your local machine. This process connects to your remote cluster to manage VMs for that cluster. That means that the Machine Controller Manager runs outside a Kubernetes cluster which requires providing a Kubeconfig in your local filesystem and point the Machine Controller Manager to it when running it (see below).

Although the following installation instructions are for Mac OS X, similar alternate commands could be found for any Linux distribution.

Installing Golang environment

Install the latest version of Golang (at least v1.8.3 is required) by using Homebrew:

$ brew install golang

In order to perform linting on the Go source code, install Golint:

$ go get -u golang.org/x/lint/golint

Installing Docker (Optional)

In case you want to build Docker images for the Machine Controller Manager you have to install Docker itself. We recommend using Docker for Mac OS X which can be downloaded from here.

Setup Docker Hub account (Optional)

Create a Docker hub account at Docker Hub if you don’t already have one.

Local development

⚠️ Before you start developing, please ensure to comply with the following requirements:

  1. You have understood the principles of Kubernetes, and its components, what their purpose is and how they interact with each other.
  2. You have understood the architecture of the Machine Controller Manager

The development of the Machine Controller Manager could happen by targetting any cluster. You basically need a Kubernetes cluster running on a set of machines. You just need the Kubeconfig file with the required access permissions attached to it.

Installing the Machine Controller Manager locally

Clone the repository from GitHub.

$ git clone git@github.com:gardener/machine-controller-manager.git
$ cd machine-controller-manager

Prepare the cluster

  • Connect to the remote kubernetes cluster where you plan to deploy the Machine Controller Manager using kubectl. Set the environment variable KUBECONFIG to the path of the yaml file containing your cluster info
  • Now, create the required CRDs on the remote cluster using the following command,
$ kubectl apply -f kubernetes/crds.yaml

Getting started

  • Create a dev directory.
  • Copy the kubeconfig of kubernetes cluster where you wish to deploy the machines into dev/target-kubeconfig.yaml.
  • (optional) Copy the kubeconfig of kubernetes cluster from where you wish to manage the machines into dev/control-kubeconfig.yaml. If you do this, also update the Makefile variable CONTROL_KUBECONFIG to point to dev/control-kubeconfig.yaml and CONTROL_NAMESPACE to the namespace in which your controller watches over.
  • There is a rule dev in the Makefile which will automatically start the Machine Controller Manager with development settings:
$ make start
I1227 11:08:19.963638   55523 controllermanager.go:204] Starting shared informers
I1227 11:08:20.766085   55523 controller.go:247] Starting machine-controller-manager

⚠️ The file dev/target-kubeconfig.yaml points to the cluster whose nodes you want to manage. dev/control-kubeconfig.yaml points to the cluster from where you want to manage the nodes from. However, dev/control-kubeconfig.yaml is optional.

The Machine Controller Manager should now be ready to manage the VMs in your kubernetes cluster.

⚠️ This is assuming that your MCM is built to manage machines for any in-tree supported providers. There is a new way to deploy and manage out of tree (external) support for providers whose development can be found here

Testing Machine Classes

To test the creation/deletion of a single instance for one particular machine class you can use the managevm cli. The corresponding INFRASTRUCTURE-machine-class.yaml and the INFRASTRUCTURE-secret.yaml need to be defined upfront. To build and run it

GO111MODULE=on go build -mod=vendor -o managevm cmd/machine-controller-manager-cli/main.go
# create machine
./managevm --secret PATH_TO/INFRASTRUCTURE-secret.yaml --machineclass PATH_TO/INFRASTRUCTURE-machine-class.yaml --classkind INFRASTRUCTURE --machinename test
# delete machine
./managevm --secret PATH_TO/INFRASTRUCTURE-secret.yaml --machineclass PATH_TO/INFRASTRUCTURE-machine-class.yaml --classkind INFRASTRUCTURE --machinename test --machineid INFRASTRUCTURE:///REGION/INSTANCE_ID

Usage

To start using Machine Controller Manager, follow the links given at usage here.

2.5 - Machine Error Codes

Machine Error code handling

Notational Conventions

The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” are to be interpreted as described in RFC 2119 (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels”, BCP 14, RFC 2119, March 1997).

The key words “unspecified”, “undefined”, and “implementation-defined” are to be interpreted as described in the rationale for the C99 standard.

An implementation is not compliant if it fails to satisfy one or more of the MUST, REQUIRED, or SHALL requirements for the protocols it implements. An implementation is compliant if it satisfies all the MUST, REQUIRED, and SHALL requirements for the protocols it implements.

Terminology

TermDefinition
CRCustom Resource (CR) is defined by a cluster admin using the Kubernetes Custom Resource Definition primitive.
VMA Virtual Machine (VM) provisioned and managed by a provider. It could also refer to a physical machine in case of a bare metal provider.
MachineMachine refers to a VM that is provisioned/managed by MCM. It typically describes the metadata used to store/represent a Virtual Machine
NodeNative kubernetes Node object. The objects you get to see when you do a “kubectl get nodes”. Although nodes can be either physical/virtual machines, for the purposes of our discussions it refers to a VM.
MCMMachine Controller Manager (MCM) is the controller used to manage higher level Machine Custom Resource (CR) such as machine-set and machine-deployment CRs.
Provider/Driver/MCProvider (or) Driver (or) Machine Controller (MC) is the driver responsible for managing machine objects present in the cluster from whom it manages these machines. A simple example could be creation/deletion of VM on the provider.

Pre-requisite

MachineClass Resources

MCM introduces the CRD MachineClass. This is a blueprint for creating machines that join a certain cluster as nodes in a certain role. The provider only works with MachineClass resources that have the structure described here.

ProviderSpec

The MachineClass resource contains a providerSpec field that is passed in the ProviderSpec request field to CMI methods such as CreateMachine. The ProviderSpec can be thought of as a machine template from which the VM specification must be adopted. It can contain key-value pairs of these specs. An example for these key-value pairs are given below.

ParameterMandatoryTypeDescription
vmPoolYesstringVM pool name, e.g. TEST-WOKER-POOL
sizeYesstringVM size, e.g. xsmall, small, etc. Each size maps to a number of CPUs and memory size.
rootFsSizeNointRoot (/) filesystem size in GB
tagsYesmapTags to be put on the created VM

Most of the ProviderSpec fields are not mandatory. If not specified, the provider passes an empty value in the respective Create VM parameter.

The tags can be used to map a VM to its corresponding machine object’s Name

The ProviderSpec is validated by methods that receive it as a request field for presence of all mandatory parameters and tags, and for validity of all parameters.

Secrets

The MachineClass resource also contains a secretRef field that contains a reference to a secret. The keys of this secret are passed in the Secrets request field to CMI methods.

The secret can contain sensitive data such as

  • cloud-credentials secret data used to authenticate at the provider
  • cloud-init scripts used to initialize a new VM. The cloud-init script is expected to contain scripts to initialize the Kubelet and make it join the cluster.

Identifying Cluster Machines

To implement certain methods, the provider should be able to identify all machines associated with a particular Kubernetes cluster. This can be achieved using one/more of the below mentioned ways:

  • Names of VMs created by the provider are prefixed by the cluster ID specified in the ProviderSpec.
  • VMs created by the provider are tagged with the special tags like kubernetes.io/cluster (for the cluster ID) and kubernetes.io/role (for the role), specified in the ProviderSpec.
  • Mapping Resource Groups to individual cluster.

Error Scheme

All provider API calls defined in this spec MUST return a machine error status, which is very similar to standard machine status.

Machine Provider Interface

  • The provider MUST have a unique way to map a machine object to a VM which triggers the deletion for the corresponding VM backing the machine object.
  • The provider SHOULD have a unique way to map the ProviderSpec of a machine-class to a unique Cluster. This avoids deletion of other machines, not backed by the MCM.

CreateMachine

A Provider is REQUIRED to implement this interface method. This interface method will be called by the MCM to provision a new VM on behalf of the requesting machine object.

  • This call requests the provider to create a VM backing the machine-object.

  • If VM backing the Machine.Name already exists, and is compatible with the specified Machine object in the CreateMachineRequest, the Provider MUST reply 0 OK with the corresponding CreateMachineResponse.

  • The provider can OPTIONALLY make use of the MachineClass supplied in the MachineClass in the CreateMachineRequest to communicate with the provider.

  • The provider can OPTIONALLY make use of the secrets supplied in the Secret in the CreateMachineRequest to communicate with the provider.

  • The provider can OPTIONALLY make use of the Status.LastKnownState in the Machine object to decode the state of the VM operation based on the last known state of the VM. This can be useful to restart/continue an operations which are mean’t to be atomic.

  • The provider MUST have a unique way to map a machine object to a VM. This could be implicitly provided by the provider by letting you set VM-names (or) could be explicitly specified by the provider using appropriate tags to map the same.

  • This operation SHOULD be idempotent.

  • The CreateMachineResponse returned by this method is expected to return

    • ProviderID that uniquely identifys the VM at the provider. This is expected to match with the node.Spec.ProviderID on the node object.
    • NodeName that is the expected name of the machine when it joins the cluster. It must match with the node name.
    • LastKnownState is an OPTIONAL field that can store details of the last known state of the VM. It can be used by future operation calls to determine current infrastucture state. This state is saved on the machine object.
// CreateMachine call is responsible for VM creation on the provider
CreateMachine(context.Context, *CreateMachineRequest) (*CreateMachineResponse, error)

// CreateMachineRequest is the create request for VM creation
type CreateMachineRequest struct {
	// Machine object from whom VM is to be created
	Machine *v1alpha1.Machine

	// MachineClass backing the machine object
	MachineClass *v1alpha1.MachineClass

	//  Secret backing the machineClass object
	Secret *corev1.Secret
}

// CreateMachineResponse is the create response for VM creation
type CreateMachineResponse struct {
	// ProviderID is the unique identification of the VM at the cloud provider.
	// ProviderID typically matches with the node.Spec.ProviderID on the node object.
	// Eg: gce://project-name/region/vm-ID
	ProviderID string

	// NodeName is the name of the node-object registered to kubernetes.
	NodeName string

	// LastKnownState represents the last state of the VM during an creation/deletion error
	LastKnownState string
}
CreateMachine Errors

If the provider is unable to complete the CreateMachine call successfully, it MUST return a non-ok ginterface method code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code. The MCM MUST implement the specified error recovery behavior when it encounters the machine error code.

machine CodeConditionDescriptionRecovery BehaviorAuto Retry Required
0 OKSuccessfulThe call was successful in creating/adopting a VM that matches supplied creation request. The CreateMachineResponse is returned with desired valuesN
1 CANCELEDCancelledCall was cancelled. Perform any pending clean-up tasks and return the callN
2 UNKNOWNSomething went wrongNot enough information on what went wrongRetry operation after sometimeY
3 INVALID_ARGUMENTRe-check supplied parametersRe-check the supplied Machine.Name and ProviderSpec. Make sure all parameters are in permitted range of values. Exact issue to be given in .messageUpdate providerSpec to fix issues.N
4 DEADLINE_EXCEEDEDTimeoutThe call processing exceeded supplied deadlineRetry operation after sometimeY
6 ALREADY_EXISTSAlready exists but desired parameters doesn’t matchParameters of the existing VM don’t match the ProviderSpecCreate machine with a different nameN
7 PERMISSION_DENIEDInsufficent permissionsThe requestor doesn’t have enough permissions to create an VM and it’s required dependenciesUpdate requestor permissions to grant the sameN
8 RESOURCE_EXHAUSTEDResource limits have been reachedThe requestor doesn’t have enough resource limits to process this creation requestEnhance resource limits associated with the user/account to process thisN
9 PRECONDITION_FAILEDVM is in inconsistent stateThe VM is in a state that is invalid for this operationManual intervention might be needed to fix the state of the VMN
10 ABORTEDOperation is pendingIndicates that there is already an operation pending for the specified machineWait until previous pending operation is processedY
11 OUT_OF_RANGEResources were out of rangeThe requested number of CPUs, memory size, of FS size in ProviderSpec falls outside of the corresponding valid rangeUpdate request paramaters to request valid resource requestsN
12 UNIMPLEMENTEDNot implementedUnimplemented indicates operation is not implemented or not supported/enabled in this service.Retry with an alternate logic or implement this method at the provider. Most methods by default are in this stateN
13 INTERNALMajor errorMeans some invariants expected by underlying system has been broken. If you see one of these errors, something is very broken.Needs manual intervension to fix thisN
14 UNAVAILABLENot AvailableUnavailable indicates the service is currently unavailable.Retry operation after sometimeY
16 UNAUTHENTICATEDMissing provider credentialsRequest does not have valid authentication credentials for the operationFix the provider credentialsN

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

DeleteMachine

A Provider is REQUIRED to implement this driver call. This driver call will be called by the MCM to deprovision/delete/terminate a VM backed by the requesting machine object.

  • If a VM corresponding to the specified machine-object’s name does not exist or the artifacts associated with the VM do not exist anymore (after deletion), the Provider MUST reply 0 OK.

  • The provider SHALL only act on machines belonging to the cluster-id/cluster-name obtained from the ProviderSpec.

  • The provider can OPTIONALY make use of the secrets supplied in the Secrets map in the DeleteMachineRequest to communicate with the provider.

  • The provider can OPTIONALY make use of the Spec.ProviderID map in the Machine object.

  • The provider can OPTIONALLY make use of the Status.LastKnownState in the Machine object to decode the state of the VM operation based on the last known state of the VM. This can be useful to restart/continue an operations which are mean’t to be atomic.

  • This operation SHOULD be idempotent.

  • The provider must have a unique way to map a machine object to a VM which triggers the deletion for the corresponding VM backing the machine object.

  • The DeleteMachineResponse returned by this method is expected to return

    • LastKnownState is an OPTIONAL field that can store details of the last known state of the VM. It can be used by future operation calls to determine current infrastucture state. This state is saved on the machine object.
// DeleteMachine call is responsible for VM deletion/termination on the provider
DeleteMachine(context.Context, *DeleteMachineRequest) (*DeleteMachineResponse, error)

// DeleteMachineRequest is the delete request for VM deletion
type DeleteMachineRequest struct {
	// Machine object from whom VM is to be deleted
	Machine *v1alpha1.Machine

	// MachineClass backing the machine object
	MachineClass *v1alpha1.MachineClass

	// Secret backing the machineClass object
	Secret *corev1.Secret
}

// DeleteMachineResponse is the delete response for VM deletion
type DeleteMachineResponse struct {
	// LastKnownState represents the last state of the VM during an creation/deletion error
	LastKnownState string
}
DeleteMachine Errors

If the provider is unable to complete the DeleteMachine call successfully, it MUST return a non-ok machine code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code.

machine CodeConditionDescriptionRecovery BehaviorAuto Retry Required
0 OKSuccessfulThe call was successful in deleting a VM that matches supplied deletion request.N
1 CANCELEDCancelledCall was cancelled. Perform any pending clean-up tasks and return the callN
2 UNKNOWNSomething went wrongNot enough information on what went wrongRetry operation after sometimeY
3 INVALID_ARGUMENTRe-check supplied parametersRe-check the supplied Machine.Name and make sure that it is in the desired format and not a blank value. Exact issue to be given in .messageUpdate Machine.Name to fix issues.N
4 DEADLINE_EXCEEDEDTimeoutThe call processing exceeded supplied deadlineRetry operation after sometimeY
7 PERMISSION_DENIEDInsufficent permissionsThe requestor doesn’t have enough permissions to delete an VM and it’s required dependenciesUpdate requestor permissions to grant the sameN
9 PRECONDITION_FAILEDVM is in inconsistent stateThe VM is in a state that is invalid for this operationManual intervention might be needed to fix the state of the VMN
10 ABORTEDOperation is pendingIndicates that there is already an operation pending for the specified machineWait until previous pending operation is processedY
12 UNIMPLEMENTEDNot implementedUnimplemented indicates operation is not implemented or not supported/enabled in this service.Retry with an alternate logic or implement this method at the provider. Most methods by default are in this stateN
13 INTERNALMajor errorMeans some invariants expected by underlying system has been broken. If you see one of these errors, something is very broken.Needs manual intervension to fix thisN
14 UNAVAILABLENot AvailableUnavailable indicates the service is currently unavailable.Retry operation after sometimeY
16 UNAUTHENTICATEDMissing provider credentialsRequest does not have valid authentication credentials for the operationFix the provider credentialsN

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

GetMachineStatus

A Provider can OPTIONALLY implement this driver call. Else should return a UNIMPLEMENTED status in error. This call will be invoked by the MC to get the status of a machine. This optional driver call helps in optimizing the working of the provider by avoiding unwanted calls to CreateMachine() and DeleteMachine().

  • If a VM corresponding to the specified machine object’s Machine.Name exists on provider the GetMachineStatusResponse fields are to be filled similar to the CreateMachineResponse.
  • The provider SHALL only act on machines belonging to the cluster-id/cluster-name obtained from the ProviderSpec.
  • The provider can OPTIONALY make use of the secrets supplied in the Secrets map in the GetMachineStatusRequest to communicate with the provider.
  • The provider can OPTIONALY make use of the VM unique ID (returned by the provider on machine creation) passed in the ProviderID map in the GetMachineStatusRequest.
  • This operation MUST be idempotent.
// GetMachineStatus call get's the status of the VM backing the machine object on the provider
GetMachineStatus(context.Context, *GetMachineStatusRequest) (*GetMachineStatusResponse, error)

// GetMachineStatusRequest is the get request for VM info
type GetMachineStatusRequest struct {
	// Machine object from whom VM status is to be fetched
	Machine *v1alpha1.Machine

	// MachineClass backing the machine object
	MachineClass *v1alpha1.MachineClass

	//  Secret backing the machineClass object
	Secret *corev1.Secret
}

// GetMachineStatusResponse is the get response for VM info
type GetMachineStatusResponse struct {
	// ProviderID is the unique identification of the VM at the cloud provider.
	// ProviderID typically matches with the node.Spec.ProviderID on the node object.
	// Eg: gce://project-name/region/vm-ID
	ProviderID string

	// NodeName is the name of the node-object registered to kubernetes.
	NodeName string
}
GetMachineStatus Errors

If the provider is unable to complete the GetMachineStatus call successfully, it MUST return a non-ok machine code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code.

machine CodeConditionDescriptionRecovery BehaviorAuto Retry Required
0 OKSuccessfulThe call was successful in getting machine details for given machine Machine.NameN
1 CANCELEDCancelledCall was cancelled. Perform any pending clean-up tasks and return the callN
2 UNKNOWNSomething went wrongNot enough information on what went wrongRetry operation after sometimeY
3 INVALID_ARGUMENTRe-check supplied parametersRe-check the supplied Machine.Name and make sure that it is in the desired format and not a blank value. Exact issue to be given in .messageUpdate Machine.Name to fix issues.N
4 DEADLINE_EXCEEDEDTimeoutThe call processing exceeded supplied deadlineRetry operation after sometimeY
5 NOT_FOUNDMachine isn’t found at providerThe machine could not be found at providerNot requiredN
7 PERMISSION_DENIEDInsufficent permissionsThe requestor doesn’t have enough permissions to get details for the VM and it’s required dependenciesUpdate requestor permissions to grant the sameN
9 PRECONDITION_FAILEDVM is in inconsistent stateThe VM is in a state that is invalid for this operationManual intervention might be needed to fix the state of the VMN
11 OUT_OF_RANGEMultiple VMs foundMultiple VMs found with matching machine object namesOrphan VM handler to cleanup orphan VMs / Manual intervention maybe required if orphan VM handler isn’t enabled.Y
12 UNIMPLEMENTEDNot implementedUnimplemented indicates operation is not implemented or not supported/enabled in this service.Retry with an alternate logic or implement this method at the provider. Most methods by default are in this stateN
13 INTERNALMajor errorMeans some invariants expected by underlying system has been broken. If you see one of these errors, something is very broken.Needs manual intervension to fix thisN
14 UNAVAILABLENot AvailableUnavailable indicates the service is currently unavailable.Retry operation after sometimeY
16 UNAUTHENTICATEDMissing provider credentialsRequest does not have valid authentication credentials for the operationFix the provider credentialsN

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

ListMachines

A Provider can OPTIONALLY implement this driver call. Else should return a UNIMPLEMENTED status in error. The Provider SHALL return the information about all the machines associated with the MachineClass. Make sure to use appropriate filters to achieve the same to avoid data transfer overheads. This optional driver call helps in cleaning up orphan VMs present in the cluster. If not implemented, any orphan VM that might have been created incorrectly by the MCM/Provider (due to bugs in code/infra) might require manual clean up.

  • If the Provider succeeded in returning a list of Machine.Name with their corresponding ProviderID, then return 0 OK.
  • The ListMachineResponse contains a map of MachineList whose
    • Key is expected to contain the ProviderID &
    • Value is expected to contain the Machine.Name corresponding to it’s kubernetes machine CR object
  • The provider can OPTIONALY make use of the secrets supplied in the Secrets map in the ListMachinesRequest to communicate with the provider.
// ListMachines lists all the machines that might have been created by the supplied machineClass
ListMachines(context.Context, *ListMachinesRequest) (*ListMachinesResponse, error)

// ListMachinesRequest is the request object to get a list of VMs belonging to a machineClass
type ListMachinesRequest struct {
	// MachineClass object
	MachineClass *v1alpha1.MachineClass

	// Secret backing the machineClass object
	Secret *corev1.Secret
}

// ListMachinesResponse is the response object of the list of VMs belonging to a machineClass
type ListMachinesResponse struct {
	// MachineList is the map of list of machines. Format for the map should be <ProviderID, MachineName>.
	MachineList map[string]string
}
ListMachines Errors

If the provider is unable to complete the ListMachines call successfully, it MUST return a non-ok machine code in the machine status. If the conditions defined below are encountered, the provider MUST return the specified machine error code. The MCM MUST implement the specified error recovery behavior when it encounters the machine error code.

machine CodeConditionDescriptionRecovery BehaviorAuto Retry Required
0 OKSuccessfulThe call for listing all VMs associated with ProviderSpec was successful.N
1 CANCELEDCancelledCall was cancelled. Perform any pending clean-up tasks and return the callN
2 UNKNOWNSomething went wrongNot enough information on what went wrongRetry operation after sometimeY
3 INVALID_ARGUMENTRe-check supplied parametersRe-check the supplied ProviderSpec and make sure that all required fields are present in their desired value format. Exact issue to be given in .messageUpdate ProviderSpec to fix issues.N
4 DEADLINE_EXCEEDEDTimeoutThe call processing exceeded supplied deadlineRetry operation after sometimeY
7 PERMISSION_DENIEDInsufficent permissionsThe requestor doesn’t have enough permissions to list VMs and it’s required dependenciesUpdate requestor permissions to grant the sameN
12 UNIMPLEMENTEDNot implementedUnimplemented indicates operation is not implemented or not supported/enabled in this service.Retry with an alternate logic or implement this method at the provider. Most methods by default are in this stateN
13 INTERNALMajor errorMeans some invariants expected by underlying system has been broken. If you see one of these errors, something is very broken.Needs manual intervension to fix thisN
14 UNAVAILABLENot AvailableUnavailable indicates the service is currently unavailable.Retry operation after sometimeY
16 UNAUTHENTICATEDMissing provider credentialsRequest does not have valid authentication credentials for the operationFix the provider credentialsN

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

GetVolumeIDs

A Provider can OPTIONALLY implement this driver call. Else should return a UNIMPLEMENTED status in error. This driver call will be called by the MCM to get the VolumeIDs for the list of PersistentVolumes (PVs) supplied. This OPTIONAL (but recommended) driver call helps in serailzied eviction of pods with PVs while draining of machines. This implies applications backed by PVs would be evicted one by one, leading to shorter application downtimes.

// GetVolumeIDsRequest is the request object to get a list of VolumeIDs for a PVSpec
type GetVolumeIDsRequest struct {
	// PVSpecsList is a list of PV specs for whom volume-IDs are required
	// Plugin should parse this raw data into pre-defined list of PVSpecs
	PVSpecs []*corev1.PersistentVolumeSpec
}

// GetVolumeIDsResponse is the response object of the list of VolumeIDs for a PVSpec
type GetVolumeIDsResponse struct {
	// VolumeIDs is a list of VolumeIDs.
	VolumeIDs []string
}
GetVolumeIDs Errors
machine CodeConditionDescriptionRecovery BehaviorAuto Retry Required
0 OKSuccessfulThe call getting list of VolumeIDs for the list of PersistentVolumes was successful.N
1 CANCELEDCancelledCall was cancelled. Perform any pending clean-up tasks and return the callN
2 UNKNOWNSomething went wrongNot enough information on what went wrongRetry operation after sometimeY
3 INVALID_ARGUMENTRe-check supplied parametersRe-check the supplied PVSpecList and make sure that it is in the desired format. Exact issue to be given in .messageUpdate PVSpecList to fix issues.N
4 DEADLINE_EXCEEDEDTimeoutThe call processing exceeded supplied deadlineRetry operation after sometimeY
12 UNIMPLEMENTEDNot implementedUnimplemented indicates operation is not implemented or not supported/enabled in this service.Retry with an alternate logic or implement this method at the provider. Most methods by default are in this stateN
13 INTERNALMajor errorMeans some invariants expected by underlying system has been broken. If you see one of these errors, something is very broken.Needs manual intervension to fix thisN
14 UNAVAILABLENot AvailableUnavailable indicates the service is currently unavailable.Retry operation after sometimeY

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

GenerateMachineClassForMigration

A Provider SHOULD implement this driver call, else it MUST return a UNIMPLEMENTED status in error. This driver call will be called by the Machine Controller to try to perform a machineClass migration for an unknown machineClass Kind. This helps in migration of one kind of machineClass to another kind. For instance an machineClass custom resource of AWSMachineClass to MachineClass.

  • On successful generation of machine class the Provider MUST reply 0 OK (or) nil error.
  • GenerateMachineClassForMigrationRequest expects the provider-specific machine class (eg. AWSMachineClass) to be supplied as the ProviderSpecificMachineClass. The provider is responsible for unmarshalling the golang struct. It also passes a reference to an existing MachineClass object.
  • The provider is expected to fill in thisMachineClass object based on the conversions.
  • An optional ClassSpec containing the type ClassSpec struct is also provided to decode the provider info.
  • GenerateMachineClassForMigration is only responsible for filling up the passed MachineClass object.
  • The task of creating the new CR of the new kind (MachineClass) with the same name as the previous one and also annotating the old machineClass CR with a migrated annotation and migrating existing references is done by the calling library implicitly.
  • This operation MUST be idempotent.
// GenerateMachineClassForMigrationRequest is the request for generating the generic machineClass
// for the provider specific machine class
type GenerateMachineClassForMigrationRequest struct {
	// ProviderSpecificMachineClass is provider specfic machine class object.
	// E.g. AWSMachineClass
	ProviderSpecificMachineClass interface{}
	// MachineClass is the machine class object generated that is to be filled up
	MachineClass *v1alpha1.MachineClass
	// ClassSpec contains the class spec object to determine the machineClass kind
	ClassSpec *v1alpha1.ClassSpec
}

// GenerateMachineClassForMigrationResponse is the response for generating the generic machineClass
// for the provider specific machine class
type GenerateMachineClassForMigrationResponse struct{}
MigrateMachineClass Errors
machine CodeConditionDescriptionRecovery BehaviorAuto Retry Required
0 OKSuccessfulMigration of provider specific machine class was successfulMachine reconcilation is retried once the new class has been createdY
12 UNIMPLEMENTEDNot implementedUnimplemented indicates operation is not implemented or not supported/enabled in this provider.NoneN
13 INTERNALMajor errorMeans some invariants expected by underlying system has been broken. If you see one of these errors, something is very broken.Might need manual intervension to fix thisY

The status message MUST contain a human readable description of error, if the status code is not OK. This string MAY be surfaced by MCM to end users.

Configuration and Operation

Supervised Lifecycle Management

  • For Providers packaged in software form:
    • Provider Packages SHOULD use a well-documented container image format (e.g., Docker, OCI).
    • The chosen package image format MAY expose configurable Provider properties as environment variables, unless otherwise indicated in the section below. Variables so exposed SHOULD be assigned default values in the image manifest.
    • A Provider Supervisor MAY programmatically evaluate or otherwise scan a Provider Package’s image manifest in order to discover configurable environment variables.
    • A Provider SHALL NOT assume that an operator or Provider Supervisor will scan an image manifest for environment variables.

Environment Variables

  • Variables defined by this specification SHALL be identifiable by their MC_ name prefix.
  • Configuration properties not defined by the MC specification SHALL NOT use the same MC_ name prefix; this prefix is reserved for common configuration properties defined by the MC specification.
  • The Provider Supervisor SHOULD supply all RECOMMENDED MC environment variables to a Provider.
  • The Provider Supervisor SHALL supply all REQUIRED MC environment variables to a Provider.
Logging
  • Providers SHOULD generate log messages to ONLY standard output and/or standard error.
    • In this case the Provider Supervisor SHALL assume responsibility for all log lifecycle management.
  • Provider implementations that deviate from the above recommendation SHALL clearly and unambiguously document the following:
    • Logging configuration flags and/or variables, including working sample configurations.
    • Default log destination(s) (where do the logs go if no configuration is specified?)
    • Log lifecycle management ownership and related guidance (size limits, rate limits, rolling, archiving, expunging, etc.) applicable to the logging mechanism embedded within the Provider.
  • Providers SHOULD NOT write potentially sensitive data to logs (e.g. secrets).
Available Services
  • Provider Packages MAY support all or a subset of CMI services; service combinations MAY be configurable at runtime by the Provider Supervisor.
    • This specification does not dictate the mechanism by which mode of operation MUST be discovered, and instead places that burden upon the VM Provider.
  • Misconfigured provider software SHOULD fail-fast with an OS-appropriate error code.
Linux Capabilities
  • Providers SHOULD clearly document any additionally required capabilities and/or security context.
Cgroup Isolation
  • A Provider MAY be constrained by cgroups.
Resource Requirements
  • VM Providers SHOULD unambiguously document all of a Provider’s resource requirements.

Deploying

  • Recommended: The MCM and Provider are typically expected to run as two containers inside a common Pod.
  • However, for the security reasons they could execute on seperate Pods provided they have a secure way to exchange data between them.

2.6 - Testing And Dependencies

Dependency management

We use golang modules to manage golang dependencies. In order to add a new package dependency to the project, you can perform go get <PACKAGE>@<VERSION> or edit the go.mod file and append the package along with the version you want to use.

Updating dependencies

The Makefile contains a rule called revendor which performs go mod vendor and go mod tidy.

go mod vendor resets the main module’s vendor directory to include all packages needed to build and test all the main module’s packages. It does not include test code for vendored packages.

go mod tidy makes sure go.mod matches the source code in the module. It adds any missing modules necessary to build the current module’s packages and dependencies, and it removes unused modules that don’t provide any relevant packages.

$ make revendor

The dependencies are installed into the vendor folder which should be added to the VCS.

⚠️ Make sure you test the code after you have updated the dependencies!

3 - Documents

3.1 - APIs

Specification

ProviderSpec Schema


AWSMachineClass

AWSMachineClass TODO

FieldTypeDescription
apiVersionstringmachine.sapcloud.io.v1alpha1
kindstringAWSMachineClass
metadataKubernetes meta/v1.ObjectMeta(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
specAWSMachineClassSpec(Optional)

amistring
regionstring
blockDevices[]AWSBlockDeviceMappingSpec
ebsOptimizedbool
iamAWSIAMProfileSpec
machineTypestring
keyNamestring
monitoringbool
networkInterfaces[]AWSNetworkInterfaceSpec
tagsmap[string]string
spotPrice*string
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference

AlicloudMachineClass

AlicloudMachineClass TODO

FieldTypeDescription
apiVersionstringmachine.sapcloud.io.v1alpha1
kindstringAlicloudMachineClass
metadataKubernetes meta/v1.ObjectMeta(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
specAlicloudMachineClassSpec(Optional)

imageIDstring
instanceTypestring
regionstring
zoneIDstring
securityGroupIDstring
vSwitchIDstring
privateIPAddressstring
systemDiskAlicloudSystemDisk
dataDisks[]AlicloudDataDisk
instanceChargeTypestring
internetChargeTypestring
internetMaxBandwidthIn*int
internetMaxBandwidthOut*int
spotStrategystring
IoOptimizedstring
tagsmap[string]string
keyPairNamestring
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference

AzureMachineClass

AzureMachineClass TODO

FieldTypeDescription
apiVersionstringmachine.sapcloud.io.v1alpha1
kindstringAzureMachineClass
metadataKubernetes meta/v1.ObjectMeta(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
specAzureMachineClassSpec(Optional)

locationstring
tagsmap[string]string
propertiesAzureVirtualMachineProperties
resourceGroupstring
subnetInfoAzureSubnetInfo
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference

GCPMachineClass

GCPMachineClass TODO

FieldTypeDescription
apiVersionstringmachine.sapcloud.io.v1alpha1
kindstringGCPMachineClass
metadataKubernetes meta/v1.ObjectMeta(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
specGCPMachineClassSpec(Optional)

canIpForwardbool
deletionProtectionbool
description*string
disks[]*github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1.GCPDisk
labelsmap[string]string
machineTypestring
metadata[]*github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1.GCPMetadata
networkInterfaces[]*github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1.GCPNetworkInterface
schedulingGCPScheduling
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference
serviceAccounts[]GCPServiceAccount
tags[]string
regionstring
zonestring

Machine

Machine is the representation of a physical or virtual machine.

FieldTypeDescription
apiVersionstringmachine.sapcloud.io.v1alpha1
kindstringMachine
metadataKubernetes meta/v1.ObjectMeta

ObjectMeta for machine object

Refer to the Kubernetes API documentation for the fields of the metadata field.
specMachineSpec

Spec contains the specification of the machine



classClassSpec(Optional)

Class contains the machineclass attributes of a machine

providerIDstring(Optional)

ProviderID represents the provider’s unique ID given to a machine

nodeTemplateNodeTemplateSpec(Optional)

NodeTemplateSpec describes the data a node should have when created from a template

MachineConfigurationMachineConfiguration

(Members of MachineConfiguration are embedded into this type.)

(Optional)

Configuration for the machine-controller.

statusMachineStatus

Status contains fields depicting the status


MachineClass

MachineClass can be used to templatize and re-use provider configuration across multiple Machines / MachineSets / MachineDeployments.

FieldTypeDescription
apiVersionstringmachine.sapcloud.io.v1alpha1
kindstringMachineClass
metadataKubernetes meta/v1.ObjectMeta(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
nodeTemplateNodeTemplate(Optional)

NodeTemplate contains subfields to track all node resources and other node info required to scale nodegroup from zero

credentialsSecretRefKubernetes core/v1.SecretReference

CredentialsSecretRef can optionally store the credentials (in this case the SecretRef does not need to store them). This might be useful if multiple machine classes with the same credentials but different user-datas are used.

providerSpeck8s.io/apimachinery/pkg/runtime.RawExtension

Provider-specific configuration to use during node creation.

providerstring

Provider is the combination of name and location of cloud-specific drivers.

secretRefKubernetes core/v1.SecretReference

SecretRef stores the necessary secrets such as credentials or userdata.


MachineSet

MachineSet TODO

FieldTypeDescription
apiVersionstringmachine.sapcloud.io.v1alpha1
kindstringMachineSet
metadataKubernetes meta/v1.ObjectMeta(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
specMachineSetSpec(Optional)

replicasint32(Optional)
selectorKubernetes meta/v1.LabelSelector(Optional)
machineClassClassSpec(Optional)
templateMachineTemplateSpec(Optional)
minReadySecondsint32(Optional)
statusMachineSetStatus(Optional)

PacketMachineClass

PacketMachineClass TODO

FieldTypeDescription
apiVersionstringmachine.sapcloud.io.v1alpha1
kindstringPacketMachineClass
metadataKubernetes meta/v1.ObjectMeta(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
specPacketMachineClassSpec(Optional)

facility[]string
machineTypestring
billingCyclestring
OSstring
projectIDstring
tags[]string
sshKeys[]string
userdatastring
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference

AWSBlockDeviceMappingSpec

(Appears on: AWSMachineClassSpec)

FieldTypeDescription
deviceNamestring

The device name exposed to the machine (for example, /dev/sdh or xvdh).

ebsAWSEbsBlockDeviceSpec

Parameters used to automatically set up EBS volumes when the machine is launched.

noDevicestring

Suppresses the specified device included in the block device mapping of the AMI.

virtualNamestring

The virtual device name (ephemeralN). Machine store volumes are numbered starting from 0. An machine type with 2 available machine store volumes can specify mappings for ephemeral0 and ephemeral1.The number of available machine store volumes depends on the machine type. After you connect to the machine, you must mount the volume.

Constraints: For M3 machines, you must specify machine store volumes in the block device mapping for the machine. When you launch an M3 machine, we ignore any machine store volumes specified in the block device mapping for the AMI.


AWSEbsBlockDeviceSpec

(Appears on: AWSBlockDeviceMappingSpec)

Describes a block device for an EBS volume. Please also see https://docs.aws.amazon.com/goto/WebAPI/ec2-2016-11-15/EbsBlockDevice

FieldTypeDescription
deleteOnTermination*bool

Indicates whether the EBS volume is deleted on machine termination.

encryptedbool

Indicates whether the EBS volume is encrypted. Encrypted Amazon EBS volumes may only be attached to machines that support Amazon EBS encryption.

iopsint64

The number of I/O operations per second (IOPS) that the volume supports. For io1, this represents the number of IOPS that are provisioned for the volume. For gp2, this represents the baseline performance of the volume and the rate at which the volume accumulates I/O credits for bursting. For more information about General Purpose SSD baseline performance, I/O credits, and bursting, see Amazon EBS Volume Types (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html) in the Amazon Elastic Compute Cloud User Guide.

Constraint: Range is 100-20000 IOPS for io1 volumes and 100-10000 IOPS for gp2 volumes.

Condition: This parameter is required for requests to create io1 volumes; it is not used in requests to create gp2, st1, sc1, or standard volumes.

kmsKeyID*string

Identifier (key ID, key alias, ID ARN, or alias ARN) for a customer managed CMK under which the EBS volume is encrypted.

This parameter is only supported on BlockDeviceMapping objects called by RunInstances (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html), RequestSpotFleet (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RequestSpotFleet.html), and RequestSpotInstances (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RequestSpotInstances.html).

snapshotID*string

The ID of the snapshot.

volumeSizeint64

The size of the volume, in GiB.

Constraints: 1-16384 for General Purpose SSD (gp2), 4-16384 for Provisioned IOPS SSD (io1), 500-16384 for Throughput Optimized HDD (st1), 500-16384 for Cold HDD (sc1), and 1-1024 for Magnetic (standard) volumes. If you specify a snapshot, the volume size must be equal to or larger than the snapshot size.

Default: If you’re creating the volume from a snapshot and don’t specify a volume size, the default is the snapshot size.

volumeTypestring

The volume type: gp2, io1, st1, sc1, or standard.

Default: standard


AWSIAMProfileSpec

(Appears on: AWSMachineClassSpec)

Describes an IAM machine profile.

FieldTypeDescription
arnstring

The Amazon Resource Name (ARN) of the machine profile.

namestring

The name of the machine profile.


AWSMachineClassSpec

(Appears on: AWSMachineClass)

AWSMachineClassSpec is the specification of a AWSMachineClass.

FieldTypeDescription
amistring
regionstring
blockDevices[]AWSBlockDeviceMappingSpec
ebsOptimizedbool
iamAWSIAMProfileSpec
machineTypestring
keyNamestring
monitoringbool
networkInterfaces[]AWSNetworkInterfaceSpec
tagsmap[string]string
spotPrice*string
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference

AWSNetworkInterfaceSpec

(Appears on: AWSMachineClassSpec)

Describes a network interface. Please also see https://docs.aws.amazon.com/goto/WebAPI/ec2-2016-11-15/MachineAWSNetworkInterfaceSpecification

FieldTypeDescription
associatePublicIPAddress*bool

Indicates whether to assign a public IPv4 address to an machine you launch in a VPC. The public IP address can only be assigned to a network interface for eth0, and can only be assigned to a new network interface, not an existing one. You cannot specify more than one network interface in the request. If launching into a default subnet, the default value is true.

deleteOnTermination*bool

If set to true, the interface is deleted when the machine is terminated. You can specify true only if creating a new network interface when launching an machine.

description*string

The description of the network interface. Applies only if creating a network interface when launching an machine.

securityGroupIDs[]string

The IDs of the security groups for the network interface. Applies only if creating a network interface when launching an machine.

subnetIDstring

The ID of the subnet associated with the network string. Applies only if creating a network interface when launching an machine.


AlicloudDataDisk

(Appears on: AlicloudMachineClassSpec)

FieldTypeDescription
namestring
categorystring
descriptionstring(Optional)
encryptedbool
deleteWithInstance*bool
sizeint

AlicloudMachineClassSpec

(Appears on: AlicloudMachineClass)

AlicloudMachineClassSpec is the specification of a AlicloudMachineClass.

FieldTypeDescription
imageIDstring
instanceTypestring
regionstring
zoneIDstring
securityGroupIDstring
vSwitchIDstring
privateIPAddressstring
systemDiskAlicloudSystemDisk
dataDisks[]AlicloudDataDisk
instanceChargeTypestring
internetChargeTypestring
internetMaxBandwidthIn*int
internetMaxBandwidthOut*int
spotStrategystring
IoOptimizedstring
tagsmap[string]string
keyPairNamestring
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference

AlicloudSystemDisk

(Appears on: AlicloudMachineClassSpec)

AlicloudSystemDisk describes SystemDisk for Alicloud.

FieldTypeDescription
categorystring
sizeint

AzureDataDisk

(Appears on: AzureStorageProfile)

FieldTypeDescription
namestring
lun*int32
cachingstring
storageAccountTypestring
diskSizeGBint32

AzureHardwareProfile

(Appears on: AzureVirtualMachineProperties)

AzureHardwareProfile is specifies the hardware settings for the virtual machine. Refer github.com/Azure/azure-sdk-for-go/arm/compute/models.go for VMSizes

FieldTypeDescription
vmSizestring

AzureImageReference

(Appears on: AzureStorageProfile)

AzureImageReference is specifies information about the image to use. You can specify information about platform images, marketplace images, or virtual machine images. This element is required when you want to use a platform image, marketplace image, or virtual machine image, but is not used in other creation operations.

FieldTypeDescription
idstring
urn*string

Uniform Resource Name of the OS image to be used , it has the format ‘publisher:offer:sku:version’


AzureLinuxConfiguration

(Appears on: AzureOSProfile)

AzureLinuxConfiguration is specifies the Linux operating system settings on the virtual machine.

For a list of supported Linux distributions, see Linux on Azure-Endorsed Distributions

For running non-endorsed distributions, see Information for Non-Endorsed Distributions.

FieldTypeDescription
disablePasswordAuthenticationbool
sshAzureSSHConfiguration

AzureMachineClassSpec

(Appears on: AzureMachineClass)

AzureMachineClassSpec is the specification of a AzureMachineClass.

FieldTypeDescription
locationstring
tagsmap[string]string
propertiesAzureVirtualMachineProperties
resourceGroupstring
subnetInfoAzureSubnetInfo
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference

AzureMachineSetConfig

(Appears on: AzureVirtualMachineProperties)

AzureMachineSetConfig contains the information about the machine set

FieldTypeDescription
idstring
kindstring

AzureManagedDiskParameters

(Appears on: AzureOSDisk)

AzureManagedDiskParameters is the parameters of a managed disk.

FieldTypeDescription
idstring
storageAccountTypestring

AzureNetworkInterfaceReference

(Appears on: AzureNetworkProfile)

AzureNetworkInterfaceReference is describes a network interface reference.

FieldTypeDescription
idstring
propertiesAzureNetworkInterfaceReferenceProperties

AzureNetworkInterfaceReferenceProperties

(Appears on: AzureNetworkInterfaceReference)

AzureNetworkInterfaceReferenceProperties is describes a network interface reference properties.

FieldTypeDescription
primarybool

AzureNetworkProfile

(Appears on: AzureVirtualMachineProperties)

AzureNetworkProfile is specifies the network interfaces of the virtual machine.

FieldTypeDescription
networkInterfacesAzureNetworkInterfaceReference
acceleratedNetworking*bool

AzureOSDisk

(Appears on: AzureStorageProfile)

AzureOSDisk is specifies information about the operating system disk used by the virtual machine.

For more information about disks, see About disks and VHDs for Azure virtual machines.

FieldTypeDescription
namestring
cachingstring
managedDiskAzureManagedDiskParameters
diskSizeGBint32
createOptionstring

AzureOSProfile

(Appears on: AzureVirtualMachineProperties)

AzureOSProfile is specifies the operating system settings for the virtual machine.

FieldTypeDescription
computerNamestring
adminUsernamestring
adminPasswordstring
customDatastring
linuxConfigurationAzureLinuxConfiguration

AzureSSHConfiguration

(Appears on: AzureLinuxConfiguration)

AzureSSHConfiguration is SSH configuration for Linux based VMs running on Azure

FieldTypeDescription
publicKeysAzureSSHPublicKey

AzureSSHPublicKey

(Appears on: AzureSSHConfiguration)

AzureSSHPublicKey is contains information about SSH certificate public key and the path on the Linux VM where the public key is placed.

FieldTypeDescription
pathstring
keyDatastring

AzureStorageProfile

(Appears on: AzureVirtualMachineProperties)

AzureStorageProfile is specifies the storage settings for the virtual machine disks.

FieldTypeDescription
imageReferenceAzureImageReference
osDiskAzureOSDisk
dataDisks[]AzureDataDisk

AzureSubResource

(Appears on: AzureVirtualMachineProperties)

AzureSubResource is the Sub Resource definition.

FieldTypeDescription
idstring

AzureSubnetInfo

(Appears on: AzureMachineClassSpec)

AzureSubnetInfo is the information containing the subnet details

FieldTypeDescription
vnetNamestring
vnetResourceGroup*string
subnetNamestring

AzureVirtualMachineProperties

(Appears on: AzureMachineClassSpec)

AzureVirtualMachineProperties is describes the properties of a Virtual Machine.

FieldTypeDescription
hardwareProfileAzureHardwareProfile
storageProfileAzureStorageProfile
osProfileAzureOSProfile
networkProfileAzureNetworkProfile
availabilitySetAzureSubResource
identityID*string
zone*int
machineSetAzureMachineSetConfig

ClassSpec

(Appears on: MachineSetSpec, MachineSpec)

ClassSpec is the class specification of machine

FieldTypeDescription
apiGroupstring

API group to which it belongs

kindstring

Kind for machine class

namestring

Name of machine class


ConditionStatus (string alias)

(Appears on: MachineDeploymentCondition, MachineSetCondition)


CurrentStatus

(Appears on: MachineStatus)

CurrentStatus contains information about the current status of Machine.

FieldTypeDescription
phaseMachinePhase
timeoutActivebool
lastUpdateTimeKubernetes meta/v1.Time

Last update time of current status


GCPDisk

GCPDisk describes disks for GCP.

FieldTypeDescription
autoDelete*bool
bootbool
sizeGbint64
typestring
interfacestring
imagestring
labelsmap[string]string

GCPMachineClassSpec

(Appears on: GCPMachineClass)

GCPMachineClassSpec is the specification of a GCPMachineClass.

FieldTypeDescription
canIpForwardbool
deletionProtectionbool
description*string
disks[]*github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1.GCPDisk
labelsmap[string]string
machineTypestring
metadata[]*github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1.GCPMetadata
networkInterfaces[]*github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1.GCPNetworkInterface
schedulingGCPScheduling
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference
serviceAccounts[]GCPServiceAccount
tags[]string
regionstring
zonestring

GCPMetadata

GCPMetadata describes metadata for GCP.

FieldTypeDescription
keystring
value*string

GCPNetworkInterface

GCPNetworkInterface describes network interfaces for GCP

FieldTypeDescription
disableExternalIPbool
networkstring
subnetworkstring

GCPScheduling

(Appears on: GCPMachineClassSpec)

GCPScheduling describes scheduling configuration for GCP.

FieldTypeDescription
automaticRestartbool
onHostMaintenancestring
preemptiblebool

GCPServiceAccount

(Appears on: GCPMachineClassSpec)

GCPServiceAccount describes service accounts for GCP.

FieldTypeDescription
emailstring
scopes[]string

LastOperation

(Appears on: MachineSetStatus, MachineStatus, MachineSummary)

LastOperation suggests the last operation performed on the object

FieldTypeDescription
descriptionstring

Description of the current operation

lastUpdateTimeKubernetes meta/v1.Time

Last update time of current operation

stateMachineState

State of operation

typeMachineOperationType

Type of operation


MachineConfiguration

(Appears on: MachineSpec)

MachineConfiguration describes the configurations useful for the machine-controller.

FieldTypeDescription
drainTimeoutKubernetes meta/v1.Duration(Optional)

MachineDraintimeout is the timeout after which machine is forcefully deleted.

healthTimeoutKubernetes meta/v1.Duration(Optional)

MachineHealthTimeout is the timeout after which machine is declared unhealhty/failed.

creationTimeoutKubernetes meta/v1.Duration(Optional)

MachineCreationTimeout is the timeout after which machinie creation is declared failed.

maxEvictRetries*int32(Optional)

MaxEvictRetries is the number of retries that will be attempted while draining the node.

nodeConditions*string(Optional)

NodeConditions are the set of conditions if set to true for MachineHealthTimeOut, machine will be declared failed.


MachineDeployment

Deployment enables declarative updates for machines and MachineSets.

FieldTypeDescription
metadataKubernetes meta/v1.ObjectMeta(Optional)

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
specMachineDeploymentSpec(Optional)

Specification of the desired behavior of the MachineDeployment.



replicasint32(Optional)

Number of desired machines. This is a pointer to distinguish between explicit zero and not specified. Defaults to 0.

selectorKubernetes meta/v1.LabelSelector(Optional)

Label selector for machines. Existing MachineSets whose machines are selected by this will be the ones affected by this MachineDeployment.

templateMachineTemplateSpec

Template describes the machines that will be created.

strategyMachineDeploymentStrategy(Optional)

The MachineDeployment strategy to use to replace existing machines with new ones.

minReadySecondsint32(Optional)

Minimum number of seconds for which a newly created machine should be ready without any of its container crashing, for it to be considered available. Defaults to 0 (machine will be considered available as soon as it is ready)

revisionHistoryLimit*int32(Optional)

The number of old MachineSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified.

pausedbool(Optional)

Indicates that the MachineDeployment is paused and will not be processed by the MachineDeployment controller.

rollbackToRollbackConfig(Optional)

DEPRECATED. The config this MachineDeployment is rolling back to. Will be cleared after rollback is done.

progressDeadlineSeconds*int32(Optional)

The maximum time in seconds for a MachineDeployment to make progress before it is considered to be failed. The MachineDeployment controller will continue to process failed MachineDeployments and a condition with a ProgressDeadlineExceeded reason will be surfaced in the MachineDeployment status. Note that progress will not be estimated during the time a MachineDeployment is paused. This is not set by default.

statusMachineDeploymentStatus(Optional)

Most recently observed status of the MachineDeployment.


MachineDeploymentCondition

(Appears on: MachineDeploymentStatus)

MachineDeploymentCondition describes the state of a MachineDeployment at a certain point.

FieldTypeDescription
typeMachineDeploymentConditionType

Type of MachineDeployment condition.

statusConditionStatus

Status of the condition, one of True, False, Unknown.

lastUpdateTimeKubernetes meta/v1.Time

The last time this condition was updated.

lastTransitionTimeKubernetes meta/v1.Time

Last time the condition transitioned from one status to another.

reasonstring

The reason for the condition’s last transition.

messagestring

A human readable message indicating details about the transition.


MachineDeploymentConditionType (string alias)

(Appears on: MachineDeploymentCondition)


MachineDeploymentSpec

(Appears on: MachineDeployment)

MachineDeploymentSpec is the specification of the desired behavior of the MachineDeployment.

FieldTypeDescription
replicasint32(Optional)

Number of desired machines. This is a pointer to distinguish between explicit zero and not specified. Defaults to 0.

selectorKubernetes meta/v1.LabelSelector(Optional)

Label selector for machines. Existing MachineSets whose machines are selected by this will be the ones affected by this MachineDeployment.

templateMachineTemplateSpec

Template describes the machines that will be created.

strategyMachineDeploymentStrategy(Optional)

The MachineDeployment strategy to use to replace existing machines with new ones.

minReadySecondsint32(Optional)

Minimum number of seconds for which a newly created machine should be ready without any of its container crashing, for it to be considered available. Defaults to 0 (machine will be considered available as soon as it is ready)

revisionHistoryLimit*int32(Optional)

The number of old MachineSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified.

pausedbool(Optional)

Indicates that the MachineDeployment is paused and will not be processed by the MachineDeployment controller.

rollbackToRollbackConfig(Optional)

DEPRECATED. The config this MachineDeployment is rolling back to. Will be cleared after rollback is done.

progressDeadlineSeconds*int32(Optional)

The maximum time in seconds for a MachineDeployment to make progress before it is considered to be failed. The MachineDeployment controller will continue to process failed MachineDeployments and a condition with a ProgressDeadlineExceeded reason will be surfaced in the MachineDeployment status. Note that progress will not be estimated during the time a MachineDeployment is paused. This is not set by default.


MachineDeploymentStatus

(Appears on: MachineDeployment)

MachineDeploymentStatus is the most recently observed status of the MachineDeployment.

FieldTypeDescription
observedGenerationint64(Optional)

The generation observed by the MachineDeployment controller.

replicasint32(Optional)

Total number of non-terminated machines targeted by this MachineDeployment (their labels match the selector).

updatedReplicasint32(Optional)

Total number of non-terminated machines targeted by this MachineDeployment that have the desired template spec.

readyReplicasint32(Optional)

Total number of ready machines targeted by this MachineDeployment.

availableReplicasint32(Optional)

Total number of available machines (ready for at least minReadySeconds) targeted by this MachineDeployment.

unavailableReplicasint32(Optional)

Total number of unavailable machines targeted by this MachineDeployment. This is the total number of machines that are still required for the MachineDeployment to have 100% available capacity. They may either be machines that are running but not yet available or machines that still have not been created.

conditions[]MachineDeploymentCondition

Represents the latest available observations of a MachineDeployment’s current state.

collisionCount*int32(Optional)

Count of hash collisions for the MachineDeployment. The MachineDeployment controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest MachineSet.

failedMachines[]*github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1.MachineSummary(Optional)

FailedMachines has summary of machines on which lastOperation Failed


MachineDeploymentStrategy

(Appears on: MachineDeploymentSpec)

MachineDeploymentStrategy describes how to replace existing machines with new ones.

FieldTypeDescription
typeMachineDeploymentStrategyType(Optional)

Type of MachineDeployment. Can be “Recreate” or “RollingUpdate”. Default is RollingUpdate.

rollingUpdateRollingUpdateMachineDeployment(Optional)

Rolling update config params. Present only if MachineDeploymentStrategyType =

RollingUpdate.

TODO: Update this to follow our convention for oneOf, whatever we decide it to be.


MachineDeploymentStrategyType (string alias)

(Appears on: MachineDeploymentStrategy)


MachineOperationType (string alias)

(Appears on: LastOperation)

MachineOperationType is a label for the operation performed on a machine object.


MachinePhase (string alias)

(Appears on: CurrentStatus)

MachinePhase is a label for the condition of a machines at the current time.


MachineSetCondition

(Appears on: MachineSetStatus)

MachineSetCondition describes the state of a machine set at a certain point.

FieldTypeDescription
typeMachineSetConditionType

Type of machine set condition.

statusConditionStatus

Status of the condition, one of True, False, Unknown.

lastTransitionTimeKubernetes meta/v1.Time(Optional)

The last time the condition transitioned from one status to another.

reasonstring(Optional)

The reason for the condition’s last transition.

messagestring(Optional)

A human readable message indicating details about the transition.


MachineSetConditionType (string alias)

(Appears on: MachineSetCondition)

MachineSetConditionType is the condition on machineset object


MachineSetSpec

(Appears on: MachineSet)

MachineSetSpec is the specification of a MachineSet.

FieldTypeDescription
replicasint32(Optional)
selectorKubernetes meta/v1.LabelSelector(Optional)
machineClassClassSpec(Optional)
templateMachineTemplateSpec(Optional)
minReadySecondsint32(Optional)

MachineSetStatus

(Appears on: MachineSet)

MachineSetStatus holds the most recently observed status of MachineSet.

FieldTypeDescription
replicasint32

Replicas is the number of actual replicas.

fullyLabeledReplicasint32(Optional)

The number of pods that have labels matching the labels of the pod template of the replicaset.

readyReplicasint32(Optional)

The number of ready replicas for this replica set.

availableReplicasint32(Optional)

The number of available replicas (ready for at least minReadySeconds) for this replica set.

observedGenerationint64(Optional)

ObservedGeneration is the most recent generation observed by the controller.

machineSetCondition[]MachineSetCondition(Optional)

Represents the latest available observations of a replica set’s current state.

lastOperationLastOperation

LastOperation performed

failedMachines[]github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1.MachineSummary(Optional)

FailedMachines has summary of machines on which lastOperation Failed


MachineSpec

(Appears on: Machine, MachineTemplateSpec)

MachineSpec is the specification of a Machine.

FieldTypeDescription
classClassSpec(Optional)

Class contains the machineclass attributes of a machine

providerIDstring(Optional)

ProviderID represents the provider’s unique ID given to a machine

nodeTemplateNodeTemplateSpec(Optional)

NodeTemplateSpec describes the data a node should have when created from a template

MachineConfigurationMachineConfiguration

(Members of MachineConfiguration are embedded into this type.)

(Optional)

Configuration for the machine-controller.


MachineState (string alias)

(Appears on: LastOperation)

MachineState is a current state of the machine.


MachineStatus

(Appears on: Machine)

MachineStatus holds the most recently observed status of Machine.

FieldTypeDescription
nodestring

Node string

conditions[]Kubernetes core/v1.NodeCondition

Conditions of this machine, same as node

lastOperationLastOperation

Last operation refers to the status of the last operation performed

currentStatusCurrentStatus

Current status of the machine object

lastKnownStatestring(Optional)

LastKnownState can store details of the last known state of the VM by the plugins. It can be used by future operation calls to determine current infrastucture state


MachineSummary

MachineSummary store the summary of machine.

FieldTypeDescription
namestring

Name of the machine object

providerIDstring

ProviderID represents the provider’s unique ID given to a machine

lastOperationLastOperation

Last operation refers to the status of the last operation performed

ownerRefstring

OwnerRef


MachineTemplateSpec

(Appears on: MachineDeploymentSpec, MachineSetSpec)

MachineTemplateSpec describes the data a machine should have when created from a template

FieldTypeDescription
metadataKubernetes meta/v1.ObjectMeta(Optional)

Standard object’s metadata. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata

Refer to the Kubernetes API documentation for the fields of the metadata field.
specMachineSpec(Optional)

Specification of the desired behavior of the machine. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status



classClassSpec(Optional)

Class contains the machineclass attributes of a machine

providerIDstring(Optional)

ProviderID represents the provider’s unique ID given to a machine

nodeTemplateNodeTemplateSpec(Optional)

NodeTemplateSpec describes the data a node should have when created from a template

MachineConfigurationMachineConfiguration

(Members of MachineConfiguration are embedded into this type.)

(Optional)

Configuration for the machine-controller.


NodeTemplate

(Appears on: MachineClass)

NodeTemplate contains subfields to track all node resources and other node info required to scale nodegroup from zero

FieldTypeDescription
capacityKubernetes core/v1.ResourceList

Capacity contains subfields to track all node resources required to scale nodegroup from zero

instanceTypestring

Instance type of the node belonging to nodeGroup

regionstring

Region of the expected node belonging to nodeGroup

zonestring

Zone of the expected node belonging to nodeGroup


NodeTemplateSpec

(Appears on: MachineSpec)

NodeTemplateSpec describes the data a node should have when created from a template

FieldTypeDescription
metadataKubernetes meta/v1.ObjectMeta(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
specKubernetes core/v1.NodeSpec(Optional)

NodeSpec describes the attributes that a node is created with.



podCIDRstring(Optional)

PodCIDR represents the pod IP range assigned to the node.

podCIDRs[]string(Optional)

podCIDRs represents the IP ranges assigned to the node for usage by Pods on that node. If this field is specified, the 0th entry must match the podCIDR field. It may contain at most 1 value for each of IPv4 and IPv6.

providerIDstring(Optional)

ID of the node assigned by the cloud provider in the format: ://

unschedulablebool(Optional)

Unschedulable controls node schedulability of new pods. By default, node is schedulable. More info: https://kubernetes.io/docs/concepts/nodes/node/#manual-node-administration

taints[]Kubernetes core/v1.Taint(Optional)

If specified, the node’s taints.

configSourceKubernetes core/v1.NodeConfigSource(Optional)

Deprecated. If specified, the source of the node’s configuration. The DynamicKubeletConfig feature gate must be enabled for the Kubelet to use this field. This field is deprecated as of 1.22: https://git.k8s.io/enhancements/keps/sig-node/281-dynamic-kubelet-configuration

externalIDstring(Optional)

Deprecated. Not all kubelets will set this field. Remove field after 1.13. see: https://issues.k8s.io/61966


OpenStackMachineClass

OpenStackMachineClass TODO

FieldTypeDescription
metadataKubernetes meta/v1.ObjectMeta(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
specOpenStackMachineClassSpec(Optional)

imageIDstring
imageNamestring
regionstring
availabilityZonestring
flavorNamestring
keyNamestring
securityGroups[]string
tagsmap[string]string
networkIDstring
networks[]OpenStackNetwork
subnetID*string
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference
podNetworkCidrstring
rootDiskSizeint
useConfigDrive*bool

in GB

serverGroupID*string

OpenStackMachineClassSpec

(Appears on: OpenStackMachineClass)

OpenStackMachineClassSpec is the specification of a OpenStackMachineClass.

FieldTypeDescription
imageIDstring
imageNamestring
regionstring
availabilityZonestring
flavorNamestring
keyNamestring
securityGroups[]string
tagsmap[string]string
networkIDstring
networks[]OpenStackNetwork
subnetID*string
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference
podNetworkCidrstring
rootDiskSizeint
useConfigDrive*bool

in GB

serverGroupID*string

OpenStackNetwork

(Appears on: OpenStackMachineClassSpec)

FieldTypeDescription
idstring
namestring

takes priority before name

podNetworkbool

PacketMachineClassSpec

(Appears on: PacketMachineClass)

PacketMachineClassSpec is the specification of a PacketMachineClass.

FieldTypeDescription
facility[]string
machineTypestring
billingCyclestring
OSstring
projectIDstring
tags[]string
sshKeys[]string
userdatastring
secretRefKubernetes core/v1.SecretReference
credentialsSecretRefKubernetes core/v1.SecretReference

RollbackConfig

(Appears on: MachineDeploymentSpec)

FieldTypeDescription
revisionint64(Optional)

The revision to rollback to. If set to 0, rollback to the last revision.


RollingUpdateMachineDeployment

(Appears on: MachineDeploymentStrategy)

Spec to control the desired behavior of rolling update.

FieldTypeDescription
maxUnavailablek8s.io/apimachinery/pkg/util/intstr.IntOrString(Optional)

The maximum number of machines that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired machines (ex: 10%). Absolute number is calculated from percentage by rounding down. This can not be 0 if MaxSurge is 0. By default, a fixed value of 1 is used. Example: when this is set to 30%, the old MC can be scaled down to 70% of desired machines immediately when the rolling update starts. Once new machines are ready, old MC can be scaled down further, followed by scaling up the new MC, ensuring that the total number of machines available at all times during the update is at least 70% of desired machines.

maxSurgek8s.io/apimachinery/pkg/util/intstr.IntOrString(Optional)

The maximum number of machines that can be scheduled above the desired number of machines. Value can be an absolute number (ex: 5) or a percentage of desired machines (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is calculated from percentage by rounding up. By default, a value of 1 is used. Example: when this is set to 30%, the new MC can be scaled up immediately when the rolling update starts, such that the total number of old and new machines do not exceed 130% of desired machines. Once old machines have been killed, new MC can be scaled up further, ensuring that total number of machines running at any time during the update is atmost 130% of desired machines.


Generated with gen-crd-api-reference-docs

4 - Proposals

4.1 - Excess Reserve Capacity

Excess Reserve Capacity

Goal

Currently, autoscaler optimizes the number of machines for a given application-workload. Along with effective resource utilization, this feature brings concern where, many times, when new application instances are created - they don’t find space in existing cluster. This leads the cluster-autoscaler to create new machines via MachineDeployment, which can take from 3-4 minutes to ~10 minutes, for the machine to really come-up and join the cluster. In turn, application-instances have to wait till new machines join the cluster.

One of the promising solutions to this issue is Excess Reserve Capacity. Idea is to keep a certain number of machines or percent of resources[cpu/memory] always available, so that new workload, in general, can be scheduled immediately unless huge spike in the workload. Also, the user should be given enough flexibility to choose how many resources or how many machines should be kept alive and non-utilized as this affects the Cost directly.

Note

  • We decided to go with Approach-4 which is based on low priority pods. Please find more details here: https://github.com/gardener/gardener/issues/254
  • Approach-3 looks more promising in long term, we may decide to adopt that in future based on developments/contributions in autoscaler-community.

Possible Approaches

Following are the possible approaches, we could think of so far.

Approach 1: Enhance Machine-controller-manager to also entertain the excess machines

  • Machine-controller-manager currently takes care of the machines in the shoot cluster starting from creation-deletion-health check to efficient rolling-update of the machines. From the architecture point of view, MachineSet makes sure that X number of machines are always running and healthy. MachineDeployment controller smartly uses this facility to perform rolling-updates.

  • We can expand the scope of MachineDeployment controller to maintain excess number of machines by introducing new parallel independent controller named MachineTaint controller. This will result in MCM to include Machine, MachineSet, MachineDeployment, MachineSafety, MachineTaint controllers. MachineTaint controller does not need to introduce any new CRD - analogy fits where taint-controller also resides into kube-controller-manager.

  • Only Job of MachineTaint controller will be:

    • List all the Machines under each MachineDeployment.
    • Maintain taints of noSchedule and noExecute on X latest MachineObjects.
    • There should be an event-based informer mechanism where MachineTaintController gets to know about any Update/Delete/Create event of MachineObjects - in turn, maintains the noSchedule and noExecute taints on all the latest machines. - Why latest machines? - Whenever autoscaler decides to add new machines - essentially ScaleUp event - taints from the older machines are removed and newer machines get the taints. This way X number of Machines immediately becomes free for new pods to be scheduled. - While ScaleDown event, autoscaler specifically mentions which machines should be deleted, and that should not bring any concerns. Though we will have to put proper label/annotation defined by autoscaler on taintedMachines, so that autoscaler does not consider the taintedMachines for deletion while scale-down. * Annotation on tainted node: "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true"
  • Implementation Details:

    • Expect new optional field ExcessReplicas in MachineDeployment.Spec. MachineDeployment controller now adds both Spec.Replicas and Spec.ExcessReplicas[if provided], and considers that as a standard desiredReplicas. - Current working of MCM will not be affected if ExcessReplicas field is kept nil.
    • MachineController currently reads the NodeObject and sets the MachineConditions in MachineObject. Machine-controller will now also read the taints/labels from the MachineObject - and maintains it on the NodeObject.
  • We expect cluster-autoscaler to intelligently make use of the provided feature from MCM.

    • CA gets the input of min:max:excess from Gardener. CA continues to set the MachineDeployment.Spec.Replicas as usual based on the application-workload.
    • In addition, CA also sets the MachieDeployment.Spec.ExcessReplicas .
    • Corner-case: * CA should decrement the excessReplicas field accordingly when desiredReplicas+excessReplicas on MachineDeployment goes beyond max.

Approach 2: Enhance Cluster-autoscaler by simulating fake pods in it

Approach 3: Enhance cluster-autoscaler to support pluggable scaling-events

  • Forked version of cluster-autoscaler could be improved to plug-in the algorithm for excess-reserve capacity.
  • Needs further discussion around upstream support.
  • Create golang channel to separate the algorithms to trigger scaling (hard-coded in cluster-autoscaler, currently) from the algorithms about how to to achieve the scaling (already pluggable in cluster-autoscaler). This kind of separation can help us introduce/plug-in new algorithms (such as based node resource utilisation) without affecting existing code-base too much while almost completely re-using the code-base for the actual scaling.
  • Also this approach is not specific to our fork of cluster-autoscaler. It can be made upstream eventually as well.

Approach 4: Make intelligent use of Low-priority pods

  • Refer to: pod-priority-preemption
  • TL; DR:
    • High priority pods can preempt the low-priority pods which are already scheduled.
    • Pre-create bunch[equivivalent of X shoot-control-planes] of low-priority pods with priority of zero, then start creating the workload pods with better priority which will reschedule the low-priority pods or otherwise keep them in pending state if the limit for max-machines has reached.
    • This is still alpha feature.

4.2 - GRPC Based Implementation of Cloud Providers

GRPC based implementation of Cloud Providers - WIP

Goal:

Currently the Cloud Providers’ (CP) functionalities ( Create(), Delete(), List() ) are part of the Machine Controller Manager’s (MCM)repository. Because of this, adding support for new CPs into MCM requires merging code into MCM which may not be required for core functionalities of MCM itself. Also, for various reasons it may not be feasible for all CPs to merge their code with MCM which is an Open Source project.

Because of these reasons, it was decided that the CP’s code will be moved out in separate repositories so that they can be maintained separately by the respective teams. Idea is to make MCM act as a GRPC server, and CPs as GRPC clients. The CP can register themselves with the MCM using a GRPC service exposed by the MCM. Details of this approach is discussed below.

How it works:

MCM acts as GRPC server and listens on a pre-defined port 5000. It implements below GRPC services. Details of each of these services are mentioned in next section.

  • Register()
  • GetMachineClass()
  • GetSecret()

GRPC services exposed by MCM:

Register()

rpc Register(stream DriverSide) returns (stream MCMside) {}

The CP GRPC client calls this service to register itself with the MCM. The CP passes the kind and the APIVersion which it implements, and MCM maintains an internal map for all the registered clients. A GRPC stream is returned in response which is kept open througout the life of both the processes. MCM uses this stream to communicate with the client for machine operations: Create(), Delete() or List(). The CP client is responsible for reading the incoming messages continuously, and based on the operationType parameter embedded in the message, it is supposed to take the required action. This part is already handled in the package grpc/infraclient. To add a new CP client, import the package, and implement the ExternalDriverProvider interface:

type ExternalDriverProvider interface {
	Create(machineclass *MachineClassMeta, credentials, machineID, machineName string) (string, string, error)
	Delete(machineclass *MachineClassMeta, credentials, machineID string) error
	List(machineclass *MachineClassMeta, credentials, machineID string) (map[string]string, error)
}

GetMachineClass()

rpc GetMachineClass(MachineClassMeta) returns (MachineClass) {}

As part of the message from MCM for various machine operations, the name of the machine class is sent instead of the full machine class spec. The CP client is expected to use this GRPC service to get the full spec of the machine class. This optionally enables the client to cache the machine class spec, and make the call only if the machine calass spec is not already cached.

GetSecret()

rpc GetSecret(SecretMeta) returns (Secret) {}

As part of the message from MCM for various machine operations, the Cloud Config (CC) and CP credentials are not sent. The CP client is expected to use this GRPC service to get the secret which has CC and CP’s credentials from MCM. This enables the client to cache the CC and credentials, and to make the call only if the data is not already cached.

How to add a new Cloud Provider’s support

Import the package grpc/infraclient and grpc/infrapb from MCM (currently in MCM’s “grpc-driver” branch)

  • Implement the interface ExternalDriverProvider
    • Create(): Creates a new machine
    • Delete(): Deletes a machine
    • List(): Lists machines
  • Use the interface MachineClassDataProvider
    • GetMachineClass(): Makes the call to MCM to get machine class spec
    • GetSecret(): Makes the call to MCM to get secret containing Cloud Config and CP’s credentials

Example implementation:

Refer GRPC based implementation for AWS client: https://github.com/ggaurav10/aws-driver-grpc

5 - Usage

5.1 - Machine

Creating/Deleting machines (VM)

Setting up your usage environment

Important :

Make sure that the kubernetes/machine_objects/machine.yaml points to the same class name as the kubernetes/machine_classes/aws-machine-class.yaml.

Similarly kubernetes/machine_objects/aws-machine-class.yaml secret name and namespace should be same as that mentioned in kubernetes/secrets/aws-secret.yaml

Creating machine

  • Modify kubernetes/machine_objects/machine.yaml as per your requirement and create the VM as shown below:
$ kubectl apply -f kubernetes/machine_objects/machine.yaml

You should notice that the Machine Controller Manager has immediately picked up your manifest and started to create a new machine by talking to the cloud provider.

  • Check Machine Controller Manager machines in the cluster
$ kubectl get machine
NAME           STATUS    AGE
test-machine   Running   5m

A new machine is created with the name provided in the kubernetes/machine_objects/machine.yaml file.

  • After a few minutes (~3 minutes for AWS), you should notice a new node joining the cluster. You can verify this by running:
$ kubectl get nodes
NAME                                         STATUS     AGE     VERSION
ip-10-250-14-52.eu-east-1.compute.internal.  Ready      1m      v1.8.0

This shows that a new node has successfully joined the cluster.

Inspect status of machine

To inspect the status of any created machine, run the command given below.

$ kubectl get machine test-machine -o yaml
apiVersion: machine.sapcloud.io/v1alpha1
kind: Machine
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"machine.sapcloud.io/v1alpha1","kind":"Machine","metadata":{"annotations":{},"labels":{"test-label":"test-label"},"name":"test-machine","namespace":""},"spec":{"class":{"kind":"AWSMachineClass","name":"test-aws"}}}
  clusterName: ""
  creationTimestamp: 2017-12-27T06:58:21Z
  finalizers:
  - machine.sapcloud.io/operator
  generation: 0
  initializers: null
  labels:
    node: ip-10-250-14-52.eu-east-1.compute.internal
    test-label: test-label
  name: test-machine
  namespace: ""
  resourceVersion: "12616948"
  selfLink: /apis/machine.sapcloud.io/v1alpha1/test-machine
  uid: 535e596c-ead3-11e7-a6c0-828f843e4186
spec:
  class:
    kind: AWSMachineClass
    name: test-aws
  providerID: aws:///eu-east-1/i-00bef3f2618ffef23
status:
  conditions:
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T06:59:16Z
    message: kubelet has sufficient disk space available
    reason: KubeletHasSufficientDisk
    status: "False"
    type: OutOfDisk
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T06:59:16Z
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T06:59:16Z
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: 2017-12-27T07:00:46Z
    lastTransitionTime: 2017-12-27T07:00:06Z
    message: kubelet is posting ready status
    reason: KubeletReady
    status: "True"
    type: Ready
  currentStatus:
    lastUpdateTime: 2017-12-27T07:00:06Z
    phase: Running
  lastOperation:
    description: Machine is now ready
    lastUpdateTime: 2017-12-27T07:00:06Z
    state: Successful
    type: Create
  node: ip-10-250-14-52.eu-west-1.compute.internal

Delete machine

To delete the VM using the kubernetes/machine_objects/machine.yaml as shown below

$ kubectl delete -f kubernetes/machine_objects/machine.yaml

Now the Machine Controller Manager picks up the manifest immediately and starts to delete the existing VM by talking to the cloud provider. The node should be detached from the cluster in a few minutes (~1min for AWS).

5.2 - Machine Deployment

Maintaining machine replicas using machines-deployments

Setting up your usage environment

Follow the steps described here

Important ⚠️

Make sure that the kubernetes/machine_objects/machine-deployment.yaml points to the same class name as the kubernetes/machine_classes/aws-machine-class.yaml.

Similarly kubernetes/machine_classes/aws-machine-class.yaml secret name and namespace should be same as that mentioned in kubernetes/secrets/aws-secret.yaml

Creating machine-deployment

  • Modify kubernetes/machine_objects/machine-deployment.yaml as per your requirement. Modify the number of replicas to the desired number of machines. Then, create an machine-deployment.
$ kubectl apply -f kubernetes/machine_objects/machine-deployment.yaml

Now the Machine Controller Manager picks up the manifest immediately and starts to create a new machines based on the number of replicas you have provided in the manifest.

  • Check Machine Controller Manager machine-deployments in the cluster
$ kubectl get machinedeployment
NAME                      READY   DESIRED   UP-TO-DATE   AVAILABLE   AGE
test-machine-deployment   3       3         3            0           10m

You will notice a new machine-deployment with your given name

  • Check Machine Controller Manager machine-sets in the cluster
$ kubectl get machineset
NAME                                 DESIRED   CURRENT   READY   AGE
test-machine-deployment-5bc6dd7c8f   3         3         0       10m

You will notice a new machine-set backing your machine-deployment

  • Check Machine Controller Manager machines in the cluster
$ kubectl get machine
NAME                                       STATUS    AGE
test-machine-deployment-5bc6dd7c8f-5d24b   Pending   5m
test-machine-deployment-5bc6dd7c8f-6mpn4   Pending   5m
test-machine-deployment-5bc6dd7c8f-dpt2q   Pending   5m

Now you will notice N (number of replicas specified in the manifest) new machines whose name are prefixed with the machine-deployment object name that you created.

  • After a few minutes (~3 minutes for AWS), you would see that new nodes have joined the cluster. You can see this using
$  kubectl get nodes
NAME                                          STATUS    AGE       VERSION
ip-10-250-20-19.eu-west-1.compute.internal    Ready     1m        v1.8.0
ip-10-250-27-123.eu-west-1.compute.internal   Ready     1m        v1.8.0
ip-10-250-31-80.eu-west-1.compute.internal    Ready     1m        v1.8.0

This shows how new nodes have joined your cluster

Inspect status of machine-deployment

To inspect the status of any created machine-deployment run the command below,

$ kubectl get machinedeployment test-machine-deployment -o yaml

You should get the following output.

apiVersion: machine.sapcloud.io/v1alpha1
kind: MachineDeployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"machine.sapcloud.io/v1alpha1","kind":"MachineDeployment","metadata":{"annotations":{},"name":"test-machine-deployment","namespace":""},"spec":{"minReadySeconds":200,"replicas":3,"selector":{"matchLabels":{"test-label":"test-label"}},"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"test-label":"test-label"}},"spec":{"class":{"kind":"AWSMachineClass","name":"test-aws"}}}}}
  clusterName: ""
  creationTimestamp: 2017-12-27T08:55:56Z
  generation: 0
  initializers: null
  name: test-machine-deployment
  namespace: ""
  resourceVersion: "12634168"
  selfLink: /apis/machine.sapcloud.io/v1alpha1/test-machine-deployment
  uid: c0b488f7-eae3-11e7-a6c0-828f843e4186
spec:
  minReadySeconds: 200
  replicas: 3
  selector:
    matchLabels:
      test-label: test-label
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        test-label: test-label
    spec:
      class:
        kind: AWSMachineClass
        name: test-aws
status:
  availableReplicas: 3
  conditions:
  - lastTransitionTime: 2017-12-27T08:57:22Z
    lastUpdateTime: 2017-12-27T08:57:22Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  readyReplicas: 3
  replicas: 3
  updatedReplicas: 3

Health monitoring

Health monitor is also applied similar to how it’s described for machine-sets

Update your machines

Let us consider the scenario where you wish to update all nodes of your cluster from t2.xlarge machines to m5.xlarge machines. Assume that your current test-aws has its spec.machineType: t2.xlarge and your deployment test-machine-deployment points to this AWSMachineClass.

Inspect existing cluster configuration

  • Check Nodes present in the cluster
$ kubectl get nodes
NAME                                          STATUS    AGE       VERSION
ip-10-250-20-19.eu-west-1.compute.internal    Ready     1m        v1.8.0
ip-10-250-27-123.eu-west-1.compute.internal   Ready     1m        v1.8.0
ip-10-250-31-80.eu-west-1.compute.internal    Ready     1m        v1.8.0
  • Check Machine Controller Manager machine-sets in the cluster. You will notice one machine-set backing your machine-deployment
$ kubectl get machineset
NAME                                 DESIRED   CURRENT   READY   AGE
test-machine-deployment-5bc6dd7c8f   3         3         3       10m
  • Login to your cloud provider (AWS). In the VM management console, you will find N VMs created of type t2.xlarge.

Perform a rolling update

To update this machine-deployment VMs to m5.xlarge, we would do the following:

  • Copy your existing aws-machine-class.yaml
cp kubernetes/machine_classes/aws-machine-class.yaml kubernetes/machine_classes/aws-machine-class-new.yaml
  • Modify aws-machine-class-new.yaml, and update its metadata.name: test-aws2 and spec.machineType: m5.xlarge
  • Now create this modified MachineClass
kubectl apply -f kubernetes/machine_classes/aws-machine-class-new.yaml
  • Edit your existing machine-deployment
kubectl edit machinedeployment test-machine-deployment
  • Update from spec.template.spec.class.name: test-aws to spec.template.spec.class.name: test-aws2

Re-check cluster configuration

After a few minutes (~3mins)

  • Check nodes present in cluster now. They are different nodes.
$ kubectl get nodes
NAME                                          STATUS    AGE       VERSION
ip-10-250-11-171.eu-west-1.compute.internal   Ready     4m        v1.8.0
ip-10-250-17-213.eu-west-1.compute.internal   Ready     5m        v1.8.0
ip-10-250-31-81.eu-west-1.compute.internal    Ready     5m        v1.8.0
  • Check Machine Controller Manager machine-sets in the cluster. You will notice two machine-sets backing your machine-deployment
$ kubectl get machineset
NAME                                 DESIRED   CURRENT   READY   AGE
test-machine-deployment-5bc6dd7c8f   0         0         0       1h
test-machine-deployment-86ff45cc5    3         3         3       20m
  • Login to your cloud provider (AWS). In the VM management console, you will find N VMs created of type t2.xlarge in terminated state, and N new VMs of type m5.xlarge in running state.

This shows how a rolling update of a cluster from nodes with t2.xlarge to m5.xlarge went through.

More variants of updates

  • The above demonstration was a simple use case. This could be more complex like - updating the system disk image versions/ kubelet versions/ security patches etc.
  • You can also play around with the maxSurge and maxUnavailable fields in machine-deployment.yaml
  • You can also change the update strategy from rollingupdate to recreate

Undo an update

  • Edit the existing machine-deployment
$ kubectl edit machinedeployment test-machine-deployment
  • Edit the deployment to have this new field of spec.rollbackTo.revision: 0 as shown as comments in kubernetes/machine_objects/machine-deployment.yaml
  • This will undo your update to the previous version.

Pause an update

  • You can also pause the update while update is going on by editing the existing machine-deployment
$ kubectl edit machinedeployment test-machine-deployment
  • Edit the deployment to have this new field of spec.paused: true as shown as comments in kubernetes/machine_objects/machine-deployment.yaml

  • This will pause the rollingUpdate if it’s in process

  • To resume the update, edit the deployment as mentioned above and remove the field spec.paused: true updated earlier

Delete machine-deployment

  • To delete the VM using the kubernetes/machine_objects/machine-deployment.yaml
$ kubectl delete -f kubernetes/machine_objects/machine-deployment.yaml

The Machine Controller Manager picks up the manifest and starts to delete the existing VMs by talking to the cloud provider. The nodes should be detached from the cluster in a few minutes (~1min for AWS).

5.3 - Machine Set

Maintaining machine replicas using machines-sets

Setting up your usage environment

Important ⚠️

Make sure that the kubernetes/machines_objects/machine-set.yaml points to the same class name as the kubernetes/machine_classes/aws-machine-class.yaml.

Similarly kubernetes/machine_classes/aws-machine-class.yaml secret name and namespace should be same as that mentioned in kubernetes/secrets/aws-secret.yaml

Creating machine-set

  • Modify kubernetes/machine_objects/machine-set.yaml as per your requirement. You can modify the number of replicas to the desired number of machines. Then, create an machine-set:
$ kubectl apply -f kubernetes/machine_objects/machine-set.yaml

You should notice that the Machine Controller Manager has immediately picked up your manifest and started to create a new machines based on the number of replicas you have provided in the manifest.

  • Check Machine Controller Manager machine-sets in the cluster
$ kubectl get machineset
NAME               DESIRED   CURRENT   READY   AGE
test-machine-set   3         3         0       1m

You will see a new machine-set with your given name

  • Check Machine Controller Manager machines in the cluster:
$ kubectl get machine
NAME                     STATUS    AGE
test-machine-set-b57zs   Pending   5m
test-machine-set-c4bg8   Pending   5m
test-machine-set-kvskg   Pending   5m

Now you will see N (number of replicas specified in the manifest) new machines whose names are prefixed with the machine-set object name that you created.

  • After a few minutes (~3 minutes for AWS), you should notice new nodes joining the cluster. You can verify this by running:
$ kubectl get nodes
NAME                                         STATUS    AGE       VERSION
ip-10-250-0-234.eu-west-1.compute.internal   Ready     3m        v1.8.0
ip-10-250-15-98.eu-west-1.compute.internal   Ready     3m        v1.8.0
ip-10-250-6-21.eu-west-1.compute.internal    Ready     2m        v1.8.0

This shows how new nodes have joined your cluster

Inspect status of machine-set

  • To inspect the status of any created machine-set run the following command:
$ kubectl get machineset test-machine-set -o yaml
apiVersion: machine.sapcloud.io/v1alpha1
kind: MachineSet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"machine.sapcloud.io/v1alpha1","kind":"MachineSet","metadata":{"annotations":{},"name":"test-machine-set","namespace":"","test-label":"test-label"},"spec":{"minReadySeconds":200,"replicas":3,"selector":{"matchLabels":{"test-label":"test-label"}},"template":{"metadata":{"labels":{"test-label":"test-label"}},"spec":{"class":{"kind":"AWSMachineClass","name":"test-aws"}}}}}
  clusterName: ""
  creationTimestamp: 2017-12-27T08:37:42Z
  finalizers:
  - machine.sapcloud.io/operator
  generation: 0
  initializers: null
  name: test-machine-set
  namespace: ""
  resourceVersion: "12630893"
  selfLink: /apis/machine.sapcloud.io/v1alpha1/test-machine-set
  uid: 3469faaa-eae1-11e7-a6c0-828f843e4186
spec:
  machineClass: {}
  minReadySeconds: 200
  replicas: 3
  selector:
    matchLabels:
      test-label: test-label
  template:
    metadata:
      creationTimestamp: null
      labels:
        test-label: test-label
    spec:
      class:
        kind: AWSMachineClass
        name: test-aws
status:
  availableReplicas: 3
  fullyLabeledReplicas: 3
  machineSetCondition: null
  lastOperation:
    lastUpdateTime: null
  observedGeneration: 0
  readyReplicas: 3
  replicas: 3

Health monitoring

  • If you try to delete/terminate any of the machines backing the machine-set by either talking to the Machine Controller Manager or from the cloud provider, the Machine Controller Manager recreates a matching healthy machine to replace the deleted machine.
  • Similarly, if any of your machines are unreachable or in an unhealthy state (kubelet not ready / disk pressure) for longer than the configured timeout (~ 5mins), the Machine Controller Manager recreates the nodes to replace the unhealthy nodes.

Delete machine-set

  • To delete the VM using the kubernetes/machine_objects/machine-set.yaml:
$ kubectl delete -f kubernetes/machine-set.yaml

Now the Machine Controller Manager has immediately picked up your manifest and started to delete the existing VMs by talking to the cloud provider. Your nodes should be detached from the cluster in a few minutes (~1min for AWS).

5.4 - Prerequisite

Setting up the usage environment

Important ⚠️

All paths are relative to the root location of this project repository.

Run the Machine Controller Manager either as described in Setting up a local development environment or Deploying the Machine Controller Manager into a Kubernetes cluster.

Make sure that the following steps are run before managing machines/ machine-sets/ machine-deploys.

Set KUBECONFIG

Using the existing Kubeconfig, open another Terminal panel/window with the KUBECONFIG environment variable pointing to this Kubeconfig file as shown below,

$ export KUBECONFIG=<PATH_TO_REPO>/dev/kubeconfig.yaml

Replace provider credentials and desired VM configurations

Open kubernetes/machine_classes/aws-machine-class.yaml and replace required values there with the desired VM configurations.

Similarily open kubernetes/secrets/aws-secret.yaml and replace - userData, providerAccessKeyId, providerSecretAccessKey with base64 encoded values of cloudconfig file, AWS access key id, and AWS secret access key respectively. Use the following command to get the base64 encoded value of your details

$ echo "sample-cloud-config" | base64
base64-encoded-cloud-config

Do the same for your access key id and secret access key.

Deploy required CRDs and Objects

Create all the required CRDs in the cluster using kubernetes/crds.yaml

$ kubectl apply -f kubernetes/crds.yaml

Create the class template that will be used as an machine template to create VMs using kubernetes/machine_classes/aws-machine-class.yaml

$ kubectl apply -f kubernetes/machine_classes/aws-machine-class.yaml

Create the secret used for the cloud credentials and cloudconfig using kubernetes/secrets/aws-secret.yaml

$ kubectl apply -f kubernetes/secrets/aws-secret.yaml

Check current cluster state

Get to know the current cluster state using the following commands,

  • Checking aws-machine-class in the cluster
$ kubectl get awsmachineclass
NAME       MACHINE TYPE   AMI          AGE
test-aws   t2.large       ami-123456   5m
  • Checking kubernetes secrets in the cluster
$ kubectl get secret
NAME                  TYPE                                  DATA      AGE
test-secret           Opaque                                3         21h
  • Checking kubernetes nodes in the cluster
$ kubectl get nodes

Lists the default set of nodes attached to your cluster

  • Checking Machine Controller Manager machines in the cluster
$ kubectl get machine
No resources found.
  • Checking Machine Controller Manager machine-sets in the cluster
$ kubectl get machineset
No resources found.
  • Checking Machine Controller Manager machine-deploys in the cluster
$ kubectl get machinedeployment
No resources found.

6 - Machine Controller Manager FAQ

Commonly asked questions about MCM

Frequently Asked Questions

The answers in this FAQ apply to the newest (HEAD) version of Machine Controller Manager. If you’re using an older version of MCM please refer to corresponding version of this document. Few of the answers assume that the MCM being used is in conjuction with cluster-autoscaler:

Table of Contents:

Basics

What is Machine Controller Manager?

Machine Controller Manager aka MCM is a bunch of controllers used for the lifecycle management of the worker machines. It reconciles a set of CRDs such as Machine, MachineSet, MachineDeployment which depicts the functionality of Pod, Replicaset, Deployment of the core Kubernetes respectively. Read more about it at README.

  • Gardener uses MCM to manage its Kubernetes nodes of the shoot cluster. However, by design, MCM can be used independent of Gardener.

Why is my machine deleted?

A machine is deleted by MCM generally for 2 reasons-

  • Machine is unhealthy for at least MachineHealthTimeout period. The default MachineHealthTimeout is 10 minutes.

    • By default, a machine is considered unhealthy if any of the following node conditions - DiskPressure, KernelDeadlock, FileSystem, Readonly is set to true, or KubeletReady is set to false. However, this is something that is configurable using the following flag.
  • Machine is scaled down by the MachineDeployment resource.

    • This is very usual when an external controller cluster-autoscaler (aka CA) is used with MCM. CA deletes the under-utilized machines by scaling down the MachineDeployment. Read more about cluster-autoscaler’s scale down behavior here.

What are the different sub-controllers in MCM?

MCM mainly contains the following sub-controllers:

  • MachineDeployment Controller: Responsible for reconciling the MachineDeployment objects. It manages the lifecycle of the MachineSet objects.
  • MachineSet Controller: Responsible for reconciling the MachineSet objects. It manages the lifecycle of the Machine objects.
  • Machine Controller: responsible for reconciling the Machine objects. It manages the lifecycle of the actual VMs/machines created in cloud/on-prem. This controller has been moved out of tree. Please refer an AWS machine controller for more info - link.
  • Safety-controller: Responsible for handling the unidentified/unknown behaviors from the cloud providers. Please read more about its functionality below.

What is Safety Controller in MCM?

Safety Controller contains following functions:

  • Orphan VM handler:
    • It lists all the VMs in the cloud matching the tag of given cluster name and maps the VMs with the machine objects using the ProviderID field. VMs without any backing machine objects are logged and deleted after confirmation.
    • This handler runs every 30 minutes and is configurable via machine-safety-orphan-vms-period flag.
  • Freeze mechanism:
    • Safety Controller freezes the MachineDeployment and MachineSet controller if the number of machine objects goes beyond a certain threshold on top of Spec.Replicas. It can be configured by the flag –safety-up or –safety-down and also machine-safety-overshooting-period.
    • Safety Controller freezes the functionality of the MCM if either of the target-apiserver or the control-apiserver is not reachable.
    • Safety Controller unfreezes the MCM automatically once situation is resolved to normal. A freeze label is applied on MachineDeployment/MachineSet to enforce the freeze condition.

How to?

How to install MCM in a Kubernetes cluster?

MCM can be installed in a cluster with following steps:

  • Apply all the CRDs from here

  • Apply all the deployment, role-related objects from here.

    • Control cluster is the one where the machine-* objects are stored. Target cluster is where all the node objects are registered.

How to better control the rollout process of the worker nodes?

MCM allows configuring the rollout of the worker machines using maxSurge and maxUnavailable fields. These fields are applicable only during the rollout process and means nothing in general scale up/down scenarios. The overall process is very similar to how the Deployment Controller manages pods during RollingUpdate.

  • maxSurge refers to the number of additional machines that can be added on top of the Spec.Replicas of MachineDeployment during rollout process.
  • maxUnavailable refers to the number of machines that can be deleted from Spec.Replicas field of the MachineDeployment during rollout process.

How to scale down MachineDeployment by selective deletion of machines?

During scale down, triggered via MachineDeployment/MachineSet, MCM prefers to delete the machine/s which have the least priority set. Each machine object has an annotation machinepriority.machine.sapcloud.io set to 3 by default. Admin can reduce the priority of the given machines by changing the annotation value to 1. The next scale down by MachineDeployment shall delete the machines with the least priority first.

How to force delete a machine?

A machine can be force deleted by adding the label force-deletion: "True" on the machine object before executing the actual delete command. During force deletion, MCM skips the drain function and simply triggers the deletion of the machine. This label should be used with caution as it can violate the PDBs for pods running on the machine.

How to pause the ongoing rolling-update of the machinedeployment?

An ongoing rolling-update of the machine-deployment can be paused by using spec.paused field. See the example below:

apiVersion: machine.sapcloud.io/v1alpha1
kind: MachineDeployment
metadata:
  name: test-machine-deployment
spec:
  paused: true

It can be unpaused again by removing the Paused field from the machine-deployment.

How to avoid garbage collection of your node?

MCM provides an in-built safety mechanism to garbage collect VMs which have no corresponding machine object. This is done to save costs and is one of the key features of MCM. However, sometimes users might like to add nodes directly to the cluster without the help of MCM and would prefer MCM to not garbage collect such VMs. To do so they should remove/not-use tags on their VMs containing the following strings:

  1. kubernetes.io/cluster/
  2. kubernetes.io/role/
  3. kubernetes-io-cluster-
  4. kubernetes-io-role-

Internals

What is the high level design of MCM?

Please refer the following document.

What are the different configuration options in MCM?

MCM allows configuring many knobs to fine-tune its behavior according to the user’s need. Please refer to the link to check the exact configuration options.

What are the different timeouts/configurations in a machine’s lifecycle?

A machine’s lifecycle is governed by mainly following timeouts, which can be configured here.

  • MachineDrainTimeout: Amount of time after which drain times out and the machine is force deleted. Default ~2 hours.
  • MachineHealthTimeout: Amount of time after which an unhealthy machine is declared Failed and the machine is replaced by MachineSet controller.
  • MachineCreationTimeout: Amount of time after which a machine creation is declared Failed and the machine is replaced by the MachineSet controller.
  • NodeConditions: List of node conditions which if set to true for MachineHealthTimeout period, the machine is declared Failed and replaced by MachineSet controller.
  • MaxEvictRetries: An integer number depicting the number of times a failed eviction should be retried on a pod during drain process. A pod is deleted after max-retries.

How is the drain of a machine implemented?

MCM imports the functionality from the upstream Kubernetes-drain library. Although, few parts have been modified to make it work best in the context of MCM. Drain is executed before machine deletion for graceful migration of the applications. Drain internally uses the EvictionAPI to evict the pods and triggers the Deletion of pods after MachineDrainTimeout. Please note:

  • Stateless pods are evicted in parallel.
  • Stateful applications (with PVCs) are serially evicted. Please find more info in this answer below.

How are the stateful applications drained during machine deletion?

Drain function serially evicts the stateful-pods. It is observed that serial eviction of stateful pods yields better overall availability of pods as the underlying cloud in most cases detaches and reattaches disks serially anyways. It is implemented in the following manner:

  • Drain lists all the pods with attached volumes. It evicts very first stateful-pod and waits for its related entry in Node object’s .status.volumesAttached to be removed by KCM. It does the same for all the stateful-pods.
  • It waits for PvDetachTimeout (default 2 minutes) for a given pod’s PVC to be removed, else moves forward.

How does maxEvictRetries configuration work with drainTimeout configuration?

It is recommended to only set MachineDrainTimeout. It satisfies the related requirements. MaxEvictRetries is auto-calculated based on MachineDrainTimeout, if maxEvictRetries is not provided. Following will be the overall behavior of both configurations together:

  • If maxEvictRetries isn’t set and only maxDrainTimeout is set:
    • MCM auto calculates the maxEvictRetries based on the drainTimeout.
  • If drainTimeout isn’t set and only maxEvictRetries is set:
    • Default drainTimeout and user provided maxEvictRetries for each pod is considered.
  • If both maxEvictRetries and drainTimoeut are set:
    • Then both will be respected.
  • If none are set:
    • Defaults are respected.

What are the different phases of a machine?

A phase of a machine can be identified with Machine.Status.CurrentStatus.Phase. Following are the possible phases of a machine object:

  • Pending: Machine creation call has succeeded. MCM is waiting for machine to join the cluster.
  • CrashLoopBackOff: Machine creation call has failed. MCM will retry the operation after a minor delay.
  • Running: Machine creation call has succeeded. Machine has joined the cluster successfully.
  • Unknown: Machine health checks are failing, eg kubelet has stopped posting the status.
  • Failed: Machine health checks have failed for a prolonged time. Hence it is declared failed. MachineSet controller will replace such machines immediately.
  • Terminating: Machine is being terminated. Terminating state is set immediately when the deletion is triggered for the machine object. It also includes time when it’s being drained.

Troubleshooting

My machine is stuck in deletion for 1 hr, why?

In most cases, the Machine.Status.LastOperation provides information around why a machine can’t be deleted. Though following could be the reasons but not limited to:

  • Pod/s with misconfigured PDBs block the drain operation. PDBs with maxUnavailable set to 0, doesn’t allow the eviction of the pods. Hence, drain/eviction is retried till MachineDrainTimeout. Default MachineDrainTimeout could be as large as ~2hours. Hence, blocking the machine deletion.
    • Short term: User can manually delete the pod in the question, with caution.
    • Long term: Please set more appropriate PDBs which allow disruption of at least one pod.
  • Expired cloud credentials can block the deletion of the machine from infrastructure.
  • Cloud provider can’t delete the machine due to internal errors. Such situations are best debugged by using cloud provider specific CLI or cloud console.

My machine is not joining the cluster, why?

In most cases, the Machine.Status.LastOperation provides information around why a machine can’t be created. It could possibly be debugged with following steps:

  • Verify if the machine is actually created in the cloud. User can use the Machine.Spec.ProviderId to query the machine in cloud.
  • A Kubernetes node is generally bootstrapped with the cloud-config. Please verify, if MachineDeployment is pointing the correct MachineClass, and MachineClass is pointing to the correct Secret. The secret object contains the actual cloud-config in base64 format which will be used to boot the machine.
  • User must also check the logs of the MCM pod to understand any broken logical flow of reconciliation.

Developer

How should I test my code before submitting a PR?

  • Developer can locally setup the MCM using following guide
  • Developer must also enhance the unit tests related to the incoming changes.
  • Developer can locally run the unit test by executing:
make test-unit

Developer should add/update the API fields at both of the following places:

Once API changes are done, auto-generate the code using following command:

./hack/generate-code

Please ignore the API-violation errors for now.

How can I update the dependencies of MCM?

MCM uses gomod for depedency management. Developer should add/udpate depedency in the go.mod file. Please run following command to automatically revendor the dependencies.

make revendor

In the context of Gardener

How can I configure MCM using Shoot resource?

All of the knobs of MCM can be configured by the workers section of the shoot resource.

  • Gardener creates a MachineDeployment per zone for each worker-pool under workers section.
  • workers.dataVolumes allows to attach multiple disks to a machine during creation. Refer the link.
  • workers.machineControllerManager allows configuration of multiple knobs of the MachineDeployment from the shoot resource.

How is my worker-pool spread across zones?

Shoot resource allows the worker-pool to spread across multiple zones using the field workers.zones. Refer link.

  • Gardener creates one MachineDeployment per zone. Each MachineDeployment is initiated with the following replica:
MachineDeployment.Spec.Replicas = (Workers.Minimum)/(Number of availibility zones)

7 - Outline

Machine Controller Manager

CORE – ./machine-controller-manager(provider independent) Out of tree : Machine controller (provider specific) MCM is a set controllers:

  • Machine Deployment Controller

  • Machine Set Controller

  • Machine Controller

  • Machine Safety Controller

Questions and refactoring Suggestions

Refactoring

StatementFilePathStatus
ConcurrentNodeSyncs” bad name - nothing to do with node syncs actually.
If its value is ’10’ then it will start 10 goroutines (workers) per resource type (machine, machinist, machinedeployment, provider-specific-class, node - study the different resource types.
cmd/machine-controller-manager/app/options/options.gopending
LeaderElectionConfiguration is very similar to the one present in “client-go/tools/leaderelection/leaderelection.go” - can we simply used the one in client-go instead of defining again?pkg/options/types.go - MachineControllerManagerConfigurationpending
Have all userAgents as constant. Right now there is just one.cmd/app/controllermanager.gopending
Shouldn’t run function be defined on MCMServer struct itself?cmd/app/controllermanager.gopending
clientcmd.BuildConfigFromFlags fallsback to inClusterConfig which will surely not work as that is not the target. Should it not check and exit early?cmd/app/controllermanager.go - run Functionpending
A more direct way to create an in cluster config is using k8s.io/client-go/rest -> rest.InClusterConfig instead of using clientcmd.BuildConfigFromFlags passing empty arguments and depending upon the implementation to fallback to creating a inClusterConfig. If they change the implementation that you get affected.cmd/app/controllermanager.go - run Functionpending
Introduce a method on MCMServer which gets a target KubeConfig and controlKubeConfig or alternatively which creates respective clients.cmd/app/controllermanager.go - run Functionpending
Why can’t we use Kubernetes.NewConfigOrDie also for kubeClientControl?cmd/app/controllermanager.go - run Functionpending
I do not see any benefit of client builders actually. All you need to do is pass in a config and then directly use client-go functions to create a client.cmd/app/controllermanager.go - run Functionpending
Function: getAvailableResources - rename this to getApiServerResourcescmd/app/controllermanager.gopending
Move the method which waits for API server to up and ready to a separate method which returns a discoveryClient when the API server is ready.cmd/app/controllermanager.go - getAvailableResources functionpending
Many methods in client-go used are now deprecated. Switch to the ones that are now recommended to be used instead.cmd/app/controllermanager.go - startControllerspending
This method needs a general overhaulcmd/app/controllermanager.go - startControllerspending
If the design is influenced/copied from KCM then its very different. There are different controller structs defined for deployment, replicaset etc which makes the code much more clearer. You can see “kubernetes/cmd/kube-controller-manager/apps.go” and then follow the trail from there. - agreed needs to be changed in future (if time permits)pkg/controller/controller.gopending
I am not sure why “MachineSetControlInterface”, “RevisionControlInterface”, “MachineControlInterface”, “FakeMachineControl” are defined in this file?pkg/controller/controller_util.gopending
IsMachineActive - combine the first 2 conditions into one with OR.pkg/controller/controller_util.gopending
Minor change - correct the comment, first word should always be the method name. Currently none of the comments have correct names.pkg/controller/controller_util.gopending
There are too many deep copies made. What is the need to make another deep copy in this method? You are not really changing anything here.pkg/controller/deployment.go - updateMachineDeploymentFinalizerspending
Why can’t these validations be done as part of a validating webhook?pkg/controller/machineset.go - reconcileClusterMachineSetpending
Small change to the following if condition. else if is not required a simple else is sufficient. Code1
pkg/controller/machineset.go - reconcileClusterMachineSetpending
Why call these inactiveMachines, these are live and running and therefore active.pkg/controller/machineset.go - terminateMachinespending

Clarification

StatementFilePathStatus
Why are there 2 versions - internal and external versions?Generalpending
Safety controller freezes MCM controllers in the following cases:
* Num replicas go beyond a threshold (above the defined replicas)
* Target API service is not reachable
There seems to be an overlap between DWD and MCM Safety controller. In the meltdown scenario why is MCM being added to DWD, you could have used Safety controller for that.
Generalpending
All machine resources are v1alpha1 - should we not promote it to beta. V1alpha1 has a different semantic and does not give any confidence to the consumers.cmd/app/controllermanager.gopending
Shouldn’t controller manager use context.Context instead of creating a stop channel? - Check if signals (os.Interrupt and SIGTERM are handled properly. Do not see code where this is handled currently.)cmd/app/controllermanager.gopending
What is the rationale behind a timeout of 10s? If the API server is not up, should this not just block as it can anyways not do anything. Also, if there is an error returned then you exit the MCM which does not make much sense actually as it will be started again and you will again do the poll for the API server to come back up. Forcing an exit of MCM will not have any impact on the reachability of the API server in anyway so why exit?cmd/app/controllermanager.go - getAvailableResourcespending
There is a very weird check - availableResources[machineGVR] || availableResources[machineSetGVR] || availableResources[machineDeploymentGVR]
* Shouldn’t this be conjunction instead of disjunction?
* What happens if you do not find one or all of these resources?
Currently an error log is printed and nothing else is done. MCM can be used outside gardener context where consumers can directly create MachineClass and Machine and not create MachineSet / Maching Deployment. There is no distinction made between context (gardener or outside-gardener).
cmd/app/controllermanager.go - StartControllerspending
Instead of having an empty select {} to block forever, isn’t it better to wait on the stop channel?cmd/app/controllermanager.go - StartControllerspending
Do we need provider specific queues and syncs and listerspkg/controller/controller.gopending
Why are resource types prefixed with “Cluster”? - not sure , check PRpkg/controller/controller.gopending
When will forgetAfterSuccess be false and why? - as per the current code this is never the case. - Himanshu will checkcmd/app/controllermanager.go - createWorkerpending
What is the use of “ExpectationsInterface” and “UIDTrackingContExpectations”?
* All expectations related code should be in its own file “expectations.go” and not in this file.
pkg/controller/controller_util.gopending
Why do we not use lister but directly use the controlMachingClient to get the deployment? Is it because you want to avoid any potential delays caused by update of the local cache held by the informer and accessed by the lister? What is the load on API server due to this?pkg/controller/deployment.go - reconcileClusterMachineDeploymentpending
Why is this conversion needed? code2pkg/controller/deployment.go - reconcileClusterMachineDeploymentpending
A deep copy of machineDeployment is already passed and within the function another deepCopy is made. Any reason for it?pkg/controller/deployment.go - addMachineDeploymentFinalizerspending
What is an Status.ObservedGeneration?
*Read more about generations and observedGeneration at:
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata
https://alenkacz.medium.com/kubernetes-operator-best-practices-implementing-observedgeneration-250728868792
Ideally the update to the ObservedGeneration should only be made after successful reconciliation and not before. I see that this is just copied from deployment_controller.go as is
pkg/controller/deployment.go - reconcileClusterMachineDeploymentpending
Why and when will a MachineDeployment be marked as frozen and when will it be un-frozen?pkg/controller/deployment.go - reconcileClusterMachineDeploymentpending
Shoudn’t the validation of the machine deployment be done during the creation via a validating webhook instead of allowing it to be stored in etcd and then failing the validation during sync? I saw the checks and these can be done via validation webhook.pkg/controller/deployment.go - reconcileClusterMachineDeploymentpending
RollbackTo has been marked as deprecated. What is the replacement? code3pkg/controller/deployment.go - reconcileClusterMachineDeploymentpending
What is the max machineSet deletions that you could process in a single run? The reason for asking this question is that for every machineSetDeletion a new goroutine spawned.
* Is the Delete call a synchrounous call? Which means it blocks till the machineset deletion is triggered which then also deletes the machines (due to cascade-delete and blockOwnerDeletion= true)?
pkg/controller/deployment.go - terminateMachineSetspending
If there are validation errors or error when creating label selector then a nil is returned. In the worker reconcile loop if the return value is nil then it will remove it from the queue (forget + done). What is the way to see any errors? Typically when we describe a resource the errors are displayed. Will these be displayed when we discribe a MachineDeployment?pkg/controller/deployment.go - reconcileClusterMachineSetpending
If an error is returned by updateMachineSetStatus and it is IsNotFound error then returning an error will again queue the MachineSet. Is this desired as IsNotFound indicates the MachineSet has been deleted and is no longer there?pkg/controller/deployment.go - reconcileClusterMachineSetpending
is machineControl.DeleteMachine a synchronous operation which will wait till the machine has been deleted? Also where is the DeletionTimestamp set on the Machine? Will it be automatically done by the API server?pkg/controller/deployment.go - prepareMachineForDeletionpending

Bugs/Enhancements

Statement + TODOFilePathStatus
This defines QPS and Burst for its requests to the KAPI. Check if it would make sense to explicitly define a FlowSchema and PriorityLevelConfiguration to ensure that the requests from this controller are given a well-defined preference. What is the rational behind deciding these values?pkg/options/types.go - MachineControllerManagerConfigurationpending
In function “validateMachineSpec” fldPath func parameter is never used.pkg/apis/machine/validation/machine.gopending
If there is an update failure then this method recursively calls itself without any sort of delays which could lead to a LOT of load on the API server. (opened: https://github.com/gardener/machine-controller-manager/issues/686)pkg/controller/deployment.go - updateMachineDeploymentFinalizerspending
We are updating filteredMachines by invoking syncMachinesNodeTemplates, syncMachinesConfig and syncMachinesClassKind but we do not create any deepCopy here. Everywhere else the general principle is when you mutate always make a deepCopy and then mutate the copy instead of the original as a lister is used and that changes the cached copy.
Fix: SatisfiedExpectations check has been commented and there is a TODO there to fix it. Is there a PR for this?
pkg/controller/machineset.go - reconcileClusterMachineSetpending

Code references

1.1 code1

       if machineSet.DeletionTimestamp == nil {
        
        		// manageReplicas is the core machineSet method where scale up/down occurs
        
        		// It is not called when deletion timestamp is set
        
        		manageReplicasErr = c.manageReplicas(ctx, filteredMachines, machineSet)
        
        
        
        	} else if machineSet.DeletionTimestamp != nil { 
        
            //FIX: change this to simple else without the if

1.2 code2

    defer dc.enqueueMachineDeploymentAfter(deployment, 10*time.Minute)
    
    *  `Clarification`:  Why  is  this  conversion  needed?
    
    err = v1alpha1.Convert_v1alpha1_MachineDeployment_To_machine_MachineDeployment(deployment, internalMachineDeployment, nil)

1.3 code3


// rollback is not re-entrant in case the underlying machine sets are updated with a new

	// revision so we should ensure that we won't proceed to update machine sets until we

	// make sure that the deployment has cleaned up its rollback spec in subsequent enqueues.

	if d.Spec.RollbackTo != nil {

		return dc.rollback(ctx, d, machineSets, machineMap)

	}