Machine Controller Manager
Kubernetes is a cloud-native enabler built around the principles for a resilient, manageable, observable, highly automated, loosely coupled system. We know that Kubernetes is infrastructure agnostic with the help of provider specific Cloud Controller Manager. But Kubernetes has explicitly externalized the mangement of the nodes. Once they appear - correctly configured - in the cluster, Kubernetes can use them. If nodes fail, Kubernetes can’t do anything about it, external tooling is required. But every tool, every provider is different. So, why not elevate node management to a first class Kubernetes citizen? Why not create a Kubernetes native resource that manages machines just like pods? Such an approach is brought to you by the Machine Controller Manager (aka MCM), which, of course, is an open sourced project. MCM gives you the following benefits:
- seamlessly manage machines/nodes with a declarative API (of course, across different cloud providers),
- integrate generically with the cluster autoscaler,
- plugin with tools such as the node-problem-detector,
- transport the immutability design principle to machine/nodes as well, and last but not least,
- implement e.g. rolling upgrades of machines/nodes.
Machine Controller Manager aka MCM
Machine Controller Manager is a group of cooperative controllers that manage the lifecycle of the worker machines. It is inspired by the design of Kube Controller Manager in which various sub controllers manage their respective Kubernetes Clients.
Machine Controller Manager reconciles a set of Custom Resources namely
Machines which are managed & monitored by their controllers
Machine Controller respectively along with another cooperative controller called the
Understanding the sub-controllers and Custom Resources of MCM
The Custom Resources
Machines are very much analogous to the native K8s resources of
Pods respectively. So, in the context of MCM:
MachineDeploymentprovides a declarative update for
MachineDeployment Controllerreconciles the
MachineDeploymentobjects and manages the lifecycle of
MachineDeploymentconsumes provider specific
spec.template.specwhich is the template of the VM spec that would be spawned on the cloud by MCM.
MachineSetensures that the specified number of
Machinereplicas are running at a given point of time.
MachineSet Controllerreconciles the
MachineSetobjects and manages the lifecycle of
Machinesare the actual VMs running on the cloud platform provided by one of the supported cloud providers.
Machine Controlleris the controller that actually communicates with the cloud provider to create/update/delete machines on the cloud.
- There is a
Safety Controllerresponsible for handling the unidentified or unknown behaviours from the cloud providers.
- Along with the above Custom Controllers and Resources, MCM requires the
MachineClassto use K8s
Secretthat stores cloudconfig (initialization scripts used to create VMs) and cloud specific credentials.
Working of MCM
In MCM, there are two K8s clusters in the scope — a Control Cluster and a Target Cluster. Control Cluster is the K8s cluster where the MCM is installed to manage the machine lifecycle of the Target Cluster. In other words, Control Cluster is the one where the machine-* objects are stored. Target Cluster is where all the node objects are registered. These clusters can be two distinct clusters or the same cluster, whichever fits.
MachineDeployment object is created,
MachineDeployment Controller creates the corresponding
MachineSet object. The
MachineSet Controller in-turn creates the
Machine objects. The
Machine Controller then talks to the cloud provider API and actually creates the VMs on the cloud.
The cloud initialization script that is introduced into the VMs via the K8s
Secret consumed by the
MachineClasses talks to the KCM (K8s Controller Manager) and creates the node objects. Nodes after registering themselves to the Target Cluster, start sending health signals to the machine objects. That is when MCM updates the status of the machine object from
More on Safety Controller
Safety Controller contains following functions:
Orphan VM handling:
- It lists all the VMs in the cloud; matching the tag of given cluster name and maps the VMs with the
Machineobjects using the
ProviderIDfield. VMs without any backing
Machineobjects are logged and deleted after confirmation.
- This handler runs every 30 minutes and is configurable via
Safety Controllerfreezes the
MachineSet controllerif the number of
Machineobjects goes beyond a certain threshold on top of
Spec.Replicas. It can be configured by the flag
Safety Controllerfreezes the functionality of the MCM if either of the
control-apiserveris not reachable.
Safety Controllerunfreezes the MCM automatically once situation is resolved to normal. A
freezelabel is applied on
MachineSetto enforce the freeze condition.
Evolution of MCM from In-Tree to Out-of-Tree (OOT)
MCM supports declarative management of machines in a K8s Cluster on various cloud providers like AWS, Azure, GCP, AliCloud, OpenStack, Metal-stack, Packet, KubeVirt, VMWare, Yandex. It can, of course, be easily extended to support other cloud providers.
Going ahead having the implementation of the Machine Controller Manager supporting too many cloud providers would be too much upkeep from both a development and a maintenance point of view. Which is why, the
Machine Controller component of MCM has been moved to Out-of-Tree design where
Machine Controller for respective cloud provider runs as an independent executable; even though typically packaged under the same deployment.
This OOT Machine Controller will implement a common interface to manage the VMs on the respective cloud provider. Now, while
Machine Controller deals with the
Machine objects, Machine Controller Manager (MCM) deals with higher level objects such as
A lot of contributions are already being made towards OOT Machine Controller Manager for various cloud providers. Below are the links to the repositories:
- Out of Tree Machine Controller Manager for AliCloud
- Out of Tree Machine Controller Manager for AWS
- Out of Tree Machine Controller Manager for Azure
- Out of Tree Machine Controller Manager for GCP
- Out of Tree Machine Controller Manager for KubeVirt
- Out of Tree Machine Controller Manager for Metal
- Out of Tree Machine Controller Manager for vSphere
- Out of Tree Machine Controller Manager for Yandex
Who uses MCM?
MCM is originally developed and employed by a K8s Control Plane as a Service called Gardener. However, the MCM’s design is elegant enough to be employed when managing the machines of any independent K8s clusters, without having to necessarily associate it with Gardener.
Metal-stack is a set of microservices that implements Metal as a Service (MaaS). It enables you to turn your hardware into elastic cloud infrastructure. Metal-stack employs the Machine Controller Manager adopted to their Metal API. Check out an introduction to here.
Also, other interesting use cases with MCM are implemented by Kubernetes enthusiasts, who for example adjusted the Machine Controller Manager to provision machines in the cloud to extend a local Raspberry-Pi K3s cluster. Read more about it here or Check out this video on our YouTube Gardener Project channel.
Machine Controller Manager is the leading automation tool for machine management for, and in, Kubernetes. And the best part is that it is open sourced. It is freely (and easily) usable and extensible, and the community more than welcomes contributions.
Whether you want to know more about Machine Controller Manager or find out about a similar scope for your solutions, then visit the GitHub page machine-controller-manager. We are so excited to see what you achieve with Machine Controller Manager.