Preservation of Machines

• Preservation of Machines
  • Objective
  • Proposal
  • State Diagrams
  • Use Cases

Objective

Currently, the Machine Controller Manager(MCM) moves Machines with errors to the Unknown phase, and after the configured machineHealthTimeout, to the Failed phase. Failed machines are swiftly moved to the Terminating phase during which the node is drained and the Machine object is deleted. This rapid cleanup prevents SRE/operators/support from conducting an analysis on the VM and makes finding root cause of failure more difficult.

Moreover, in cases where a node seems healthy but all the workload on it are facing issues, there is a need for operators to be able to cordon/drain the node and conduct their analysis without the cluster-autoscaler (CA) scaling down the node.

This document proposes enhancing MCM, such that:

• VMs of machines are retained temporarily for analysis.
• There is a configurable limit to the number of machines that can be preserved automatically on failure (auto-preservation).
• There is a configurable limit to the duration for which machines are preserved.
• Users can specify which healthy machines they would like to preserve in case of failure, or for diagnoses in current state (prevent scale down by CA).
• Users can request MCM to release a preserved machine, even before the timeout expires, so that MCM can transition the machine to either Running or Terminating phase, as the case may be.

Related Issue: https://github.com/gardener/machine-controller-manager/issues/1008

Proposal

In order to achieve the objectives mentioned, the following are proposed:

1. Enhance worker configuration in the ShootSpec, to specify the maximum number of failed machines that will be auto-preserved and the time duration for which machines will be preserved.

   workers:
   - name: example-worker 
     autoPreserveFailedMachineMax: 2
     machineControllerManager:
          machinePreserveTimeout: 72h

• This configuration will be set per worker pool.
• Since gardener worker pool can correspond to 1..N MachineDeployments depending on number of zones, autoPreserveFailedMachineMax will be distributed across N machine deployments.
• autoPreserveFailedMachineMax must be chosen such that it can be appropriately distributed across the MachineDeployments.
• Example: if autoPreserveFailedMachineMax is set to 2, and the worker pool has 2 zones, then the maximum number of machines that will be preserved per zone is 1.

2. MCM will be modified to include a new sub-phase Preserved to indicate that the machine has been preserved by MCM.
3. Allow user/operator to request for preservation of a specific machine/node with the use of annotations : node.machine.sapcloud.io/preserve=now and node.machine.sapcloud.io/preserve=when-failed.
4. When annotation node.machine.sapcloud.io/preserve=now is added to a Running machine, the following will take place:
   • cluster-autoscaler.kubernetes.io/scale-down-disabled: "true" is added to the node to prevent CA from scaling it down.
   • machine.CurrentStatus.PreserveExpiryTime is updated by MCM as $machine.CurrentStatus.PreserveExpiryTime = currentTime+machinePreserveTimeout$.
   • The machine's phase is changed to Running:Preserved.
   • After timeout, the node.machine.sapcloud.io/preserve=now and cluster-autoscaler.kubernetes.io/scale-down-disabled: "true" are deleted. The machine.CurrentStatus.PreserveExpiryTime is set to nil. The machine phase is changed to Running and the CA may delete the node.
     • If a machine in Running:Preserved fails, it is moved to Failed:Preserved.
5. When annotation node.machine.sapcloud.io/preserve=when-failed is added to a Running machine and the machine goes to Failed, the following will take place:
   • Pods (other than DaemonSet pods) are drained.
   • The machine phase is changed to Failed:Preserved.
   • cluster-autoscaler.kubernetes.io/scale-down-disabled: "true" is added to the node to prevent CA from scaling it down.
   • machine.CurrentStatus.PreserveExpiryTime is updated by MCM as $machine.CurrentStatus.PreserveExpiryTime = currentTime+machinePreserveTimeout$.
   • After timeout, the annotations node.machine.sapcloud.io/preserve=when-failed and cluster-autoscaler.kubernetes.io/scale-down-disabled: "true" are deleted. machine.CurrentStatus.PreserveExpiryTime is set to nil. The phase is changed to Terminating.
6. When an un-annotated machine goes to Failed phase and autoPreserveFailedMachineMax is not breached:
   • Pods (other than DaemonSet pods) are drained.
   • The machine's phase is changed to Failed:Preserved.
   • cluster-autoscaler.kubernetes.io/scale-down-disabled: "true" is added to the node to prevent CA from scaling it down.
   • machine.CurrentStatus.PreserveExpiryTime is updated by MCM as $machine.CurrentStatus.PreserveExpiryTime = currentTime+machinePreserveTimeout$.
   • After timeout, the annotation cluster-autoscaler.kubernetes.io/scale-down-disabled: "true" is deleted. machine.CurrentStatus.PreserveExpiryTime is set to nil. The phase is changed to Terminating.
   • Number of machines in Failed:Preserved phase count towards enforcing autoPreserveFailedMachineMax. .
7. A user/operator can request MCM to stop preserving a machine/node in Running:Preserved or Failed:Preserved phase by deleting the annotation: node.machine.sapcloud.io/preserve.
   • MCM will move a machine thus annotated either to Running phase or Terminating depending on the phase of the machine before it was preserved.
8. Machines of a MachineDeployment in Preserved sub-phase will also be counted towards the replica count and in the enforcement of maximum machines allowed for the MachineDeployment.
9. MCM will be modified to perform drain in Failed phase for preserved machines.

State Diagrams:

1. State Diagram for when a machine or its node is explicitly annotated for preservation:

    stateDiagram-v2
        state "Running" as R
        state "Running + Requested" as RR
        state "Running:Preserved" as RP
        state "Failed
        (node drained)" as F
        state "Failed:Preserved" as FP
        state "Terminating" as T
        [*]-->R
        R --> RR: annotated with preserve=when-failed
        RP-->R: on timeout or preserve=false
        RR --> F: on failure
        F --> FP
        FP --> T: on timeout or preserve=false
        FP --> R: if node Healthy before timeout
        T --> [*]
        R-->RP: annotated with preserve=now
        RP-->F: if node/VM not healthy

2. State Diagram for when an un-annotated Running machine fails (Auto-preservation):

   stateDiagram-v2
       state "Running" as R
       state "Failed
       (node drained)" as F
       state "Failed:Preserved" as FP
       state "Terminating" as T
       [*] --> R
       R-->F: on failure
       F --> FP: if autoPreserveFailedMachineMax not breached
       F --> T: if autoPreserveFailedMachineMax breached
       FP --> T: on timeout or value=false
       FP --> R : if node Healthy before timeout
       T --> [*]

Use Cases:

Use Case 1: Preservation Request for Analysing Running Machine

Scenario: Workload on machine failing. Operator wishes to diagnose.

Steps:

1. Operator annotates node with node.machine.sapcloud.io/preserve=now.
2. MCM preserves the machine, and prevents CA from scaling it down.
3. Operator analyzes the VM.

Use Case 2: Proactive Preservation Request

Scenario: Operator suspects a machine might fail and wants to ensure preservation for analysis.

Steps:

1. Operator annotates node with node.machine.sapcloud.io/preserve=when-failed.
2. Machine fails later.
3. MCM preserves the machine.
4. Operator analyzes the VM.

Use Case 3: Auto-Preservation of Failed Machine aiding in Failure Analysis and Recovery

Scenario: Machine fails unexpectedly, no prior annotation.

Steps:

1. Machine transitions to Failed phase.
2. Machine is drained.
3. If autoPreserveFailedMachineMax is not breached, machine moved to Failed:Preserved phase by MCM.
4. After machinePreserveTimeout, machine is terminated by MCM.
5. If machine is brought back to Running phase before timeout, pods can be scheduled on it again.

Use Case 4: Early Release

Scenario: Operator has performed his analysis and no longer requires machine to be preserved.

Steps:

1. Machine is in Running:Preserved or Failed:Preserved phase.
2. Operator removes node.machine.sapcloud.io/preserve from node.
3. MCM transitions machine to Running or Terminating for Running:Preserved or Failed:Preserved respectively, even though machinePreserveTimeout has not expired.
4. If machine was auto-preserved, capacity becomes available for auto-preservation.

Points to Note

1. During rolling updates MCM will NOT honor preserving Machines. The Machine will be replaced with a healthy one if it moves to Failed phase.
2. Hibernation policy will override machine preservation.
3. Consumers (with access to shoot cluster) can annotate Nodes they would like to preserve.
4. Operators (with access to control plane) can additionally annotate Machines that they would like to preserve. This feature can be used when a Machine does not have a backing Node and the operator wishes to preserve the backing VM.
5. If the backing Node object exists but does not have the preservation annotation, preservation annotations added on the Machine will be honoured.
6. However, if a backing Node exists for a Machine and has the preservation annotation, the Node's annotation value will override the Machine annotation value.
7. If autoPreserveFailedMachineMax is reduced in the Shoot Spec, older machines are moved to Terminating phase before newer ones.
8. In case of a scale down of an MCD's replica count, Preserved machines will be the last to be scaled down. Replica count will always be honoured.
9. On increase/decrease of machinePreserveTimeout, the new value will only apply to machines that go into Preserved phase after the change. Operators can always edit machine.CurrentStatus.PreserveExpiryTime to prolong the expiry time of existing Preserved machines.
10. Modify CA FAQ once feature is developed to use node.machine.sapcloud.io/preserve=now instead of the cluster-autoscaler.kubernetes.io/scale-down-disabled=true currently suggested. This would:
    • harmonise machine flow
    • shield from CA's internals
    • make it generic and no longer CA specific
    • allow a timeout to be specified