Etcd Druid

1 - API Reference

Packages:

• druid.gardener.cloud/v1alpha1

druid.gardener.cloud/v1alpha1

Package v1alpha1 is the v1alpha1 version of the etcd-druid API.

BackupSpec

(Appears on: EtcdSpec)

BackupSpec defines parameters associated with the full and delta snapshots of etcd.

ClientService

(Appears on: EtcdConfig)

ClientService defines the parameters of the client service that a user can specify

CompactionMode (string alias)

(Appears on: SharedConfig)

CompactionMode defines the auto-compaction-mode: ‘periodic’ or ‘revision’. ‘periodic’ for duration based retention and ‘revision’ for revision number based retention.

CompressionPolicy (string alias)

(Appears on: CompressionSpec)

CompressionPolicy defines the type of policy for compression of snapshots.

CompressionSpec

(Appears on: BackupSpec)

CompressionSpec defines parameters related to compression of Snapshots(full as well as delta).

Condition

(Appears on: EtcdCopyBackupsTaskStatus, EtcdStatus)

Condition holds the information about the state of a resource.

ConditionStatus (string alias)

(Appears on: Condition)

ConditionStatus is the status of a condition.

ConditionType (string alias)

(Appears on: Condition)

ConditionType is the type of condition.

CrossVersionObjectReference

(Appears on: EtcdStatus)

CrossVersionObjectReference contains enough information to let you identify the referred resource.

Etcd

Etcd is the Schema for the etcds API

EtcdConfig

(Appears on: EtcdSpec)

EtcdConfig defines parameters associated etcd deployed

EtcdCopyBackupsTask

EtcdCopyBackupsTask is a task for copying etcd backups from a source to a target store.

EtcdCopyBackupsTaskSpec

(Appears on: EtcdCopyBackupsTask)

EtcdCopyBackupsTaskSpec defines the parameters for the copy backups task.

EtcdCopyBackupsTaskStatus

(Appears on: EtcdCopyBackupsTask)

EtcdCopyBackupsTaskStatus defines the observed state of the copy backups task.

EtcdMemberConditionStatus (string alias)

(Appears on: EtcdMemberStatus)

EtcdMemberConditionStatus is the status of an etcd cluster member.

EtcdMemberStatus

(Appears on: EtcdStatus)

EtcdMemberStatus holds information about a etcd cluster membership.

EtcdRole (string alias)

(Appears on: EtcdMemberStatus)

EtcdRole is the role of an etcd cluster member.

EtcdSpec

(Appears on: Etcd)

EtcdSpec defines the desired state of Etcd

EtcdStatus

(Appears on: Etcd)

EtcdStatus defines the observed state of Etcd.

GarbageCollectionPolicy (string alias)

(Appears on: BackupSpec)

GarbageCollectionPolicy defines the type of policy for snapshot garbage collection.

LeaderElectionSpec

(Appears on: BackupSpec)

LeaderElectionSpec defines parameters related to the LeaderElection configuration.

MetricsLevel (string alias)

(Appears on: EtcdConfig)

MetricsLevel defines the level ‘basic’ or ‘extensive’.

SchedulingConstraints

(Appears on: EtcdSpec)

SchedulingConstraints defines the different scheduling constraints that must be applied to the pod spec in the etcd statefulset. Currently supported constraints are Affinity and TopologySpreadConstraints.

SecretReference

(Appears on: TLSConfig)

SecretReference defines a reference to a secret.

SharedConfig

(Appears on: EtcdSpec)

SharedConfig defines parameters shared and used by Etcd as well as backup-restore sidecar.

StorageProvider (string alias)

(Appears on: StoreSpec)

StorageProvider defines the type of object store provider for storing backups.

StoreSpec

(Appears on: BackupSpec, EtcdCopyBackupsTaskSpec)

StoreSpec defines parameters related to ObjectStore persisting backups

TLSConfig

(Appears on: BackupSpec, EtcdConfig)

TLSConfig hold the TLS configuration details.

WaitForFinalSnapshotSpec

(Appears on: EtcdCopyBackupsTaskSpec)

WaitForFinalSnapshotSpec defines the parameters for waiting for a final full snapshot before copying backups.

Generated with gen-crd-api-reference-docs

2 - 01 Multi Node Etcd Clusters

DEP-01: Multi-node etcd cluster instances via etcd-druid

This document proposes an approach (along with some alternatives) to support provisioning and management of multi-node etcd cluster instances via etcd-druid and etcd-backup-restore.

Goal

• Enhance etcd-druid and etcd-backup-restore to support provisioning and management of multi-node etcd cluster instances within a single Kubernetes cluster.
• The etcd CRD interface should be simple to use. It should preferably work with just setting the spec.replicas field to the desired value and should not require any more configuration in the CRD than currently required for the single-node etcd instances. The spec.replicas field is part of the scale sub-resource implementation in Etcd CRD.
• The single-node and multi-node scenarios must be automatically identified and managed by etcd-druid and etcd-backup-restore.
• The etcd clusters (single-node or multi-node) managed by etcd-druid and etcd-backup-restore must automatically recover from failures (even quorum loss) and disaster (e.g. etcd member persistence/data loss) as much as possible.
• It must be possible to dynamically scale an etcd cluster horizontally (even between single-node and multi-node scenarios) by simply scaling the Etcd scale sub-resource.
• It must be possible to (optionally) schedule the individual members of an etcd clusters on different nodes or even infrastructure availability zones (within the hosting Kubernetes cluster).

Though this proposal tries to cover most aspects related to single-node and multi-node etcd clusters, there are some more points that are not goals for this document but are still in the scope of either etcd-druid/etcd-backup-restore and/or gardener. In such cases, a high-level description of how they can be addressed in the future are mentioned at the end of the document.

Background and Motivation

Single-node etcd cluster

At present, etcd-druid supports only single-node etcd cluster instances. The advantages of this approach are given below.

• The problem domain is smaller. There are no leader election and quorum related issues to be handled. It is simpler to setup and manage a single-node etcd cluster.
• Single-node etcd clusters instances have less request latency than multi-node etcd clusters because there is no requirement to replicate the changes to the other members before committing the changes.
• etcd-druid provisions etcd cluster instances as pods (actually as statefulsets) in a Kubernetes cluster and Kubernetes is quick (<20s) to restart container/pods if they go down.
• Also, etcd-druid is currently only used by gardener to provision etcd clusters to act as back-ends for Kubernetes control-planes and Kubernetes control-plane components (kube-apiserver, kubelet, kube-controller-manager, kube-scheduler etc.) can tolerate etcd going down and recover when it comes back up.
• Single-node etcd clusters incur less cost (CPU, memory and storage)
• It is easy to cut-off client requests if backups fail by using readinessProbe on the etcd-backup-restore healthz endpoint to minimize the gap between the latest revision and the backup revision.

The disadvantages of using single-node etcd clusters are given below.

• The database verification step by etcd-backup-restore can introduce additional delays whenever etcd container/pod restarts (in total ~20-25s). This can be much longer if a database restoration is required. Especially, if there are incremental snapshots that need to be replayed (this can be mitigated by compacting the incremental snapshots in the background).
• Kubernetes control-plane components can go into CrashloopBackoff if etcd is down for some time. This is mitigated by the dependency-watchdog. But Kubernetes control-plane components require a lot of resources and create a lot of load on the etcd cluster and the apiserver when they come out of CrashloopBackoff. Especially, in medium or large sized clusters (> 20 nodes).
• Maintenance operations such as updates to etcd (and updates to etcd-druid of etcd-backup-restore), rolling updates to the nodes of the underlying Kubernetes cluster and vertical scaling of etcd pods are disruptive because they cause etcd pods to be restarted. The vertical scaling of etcd pods is somewhat mitigated during scale down by doing it only during the target clusters’ maintenance window. But scale up is still disruptive.
• We currently use some form of elastic storage (via persistentvolumeclaims) for storing which have some upper-bounds on the I/O latency and throughput. This can be potentially be a problem for large clusters (> 220 nodes). Also, some cloud providers (e.g. Azure) take a long time to attach/detach volumes to and from machines which increases the down time to the Kubernetes components that depend on etcd. It is difficult to use ephemeral/local storage (to achieve better latency/throughput as well as to circumvent volume attachment/detachment) for single-node etcd cluster instances.

Multi-node etcd-cluster

The advantages of introducing support for multi-node etcd clusters via etcd-druid are below.

• Multi-node etcd cluster is highly-available. It can tolerate disruption to individual etcd pods as long as the quorum is not lost (i.e. more than half the etcd member pods are healthy and ready).
• Maintenance operations such as updates to etcd (and updates to etcd-druid of etcd-backup-restore), rolling updates to the nodes of the underlying Kubernetes cluster and vertical scaling of etcd pods can be done non-disruptively by respecting poddisruptionbudgets for the various multi-node etcd cluster instances hosted on that cluster.
• Kubernetes control-plane components do not see any etcd cluster downtime unless quorum is lost (which is expected to be lot less frequent than current frequency of etcd container/pod restarts).
• We can consider using ephemeral/local storage for multi-node etcd cluster instances because individual member restarts can afford to take time to restore from backup before (re)joining the etcd cluster because the remaining members serve the requests in the meantime.
• High-availability across availability zones is also possible by specifying (anti)affinity for the etcd pods (possibly via kupid).

Some disadvantages of using multi-node etcd clusters due to which it might still be desirable, in some cases, to continue to use single-node etcd cluster instances in the gardener context are given below.

• Multi-node etcd cluster instances are more complex to manage. The problem domain is larger including the following.
  • Leader election
  • Quorum loss
  • Managing rolling changes
  • Backups to be taken from only the leading member.
  • More complex to cut-off client requests if backups fail to minimize the gap between the latest revision and the backup revision is under control.
• Multi-node etcd cluster instances incur more cost (CPU, memory and storage).

Dynamic multi-node etcd cluster

Though it is not part of this proposal, it is conceivable to convert a single-node etcd cluster into a multi-node etcd cluster temporarily to perform some disruptive operation (etcd, etcd-backup-restore or etcd-druid updates, etcd cluster vertical scaling and perhaps even node rollout) and convert it back to a single-node etcd cluster once the disruptive operation has been completed. This will necessarily still involve a down-time because scaling from a single-node etcd cluster to a three-node etcd cluster will involve etcd pod restarts, it is still probable that it can be managed with a shorter down time than we see at present for single-node etcd clusters (on the other hand, converting a three-node etcd cluster to five node etcd cluster can be non-disruptive).

This is definitely not to argue in favour of such a dynamic approach in all cases (eventually, if/when dynamic multi-node etcd clusters are supported). On the contrary, it makes sense to make use of static (fixed in size) multi-node etcd clusters for production scenarios because of the high-availability.

Prior Art

ETCD Operator from CoreOS

etcd operator

Project status: archived

This project is no longer actively developed or maintained. The project exists here for historical reference. If you are interested in the future of the project and taking over stewardship, please contact etcd-dev@googlegroups.com.

etcdadm from kubernetes-sigs

etcdadm is a command-line tool for operating an etcd cluster. It makes it easy to create a new cluster, add a member to, or remove a member from an existing cluster. Its user experience is inspired by kubeadm.

It is a tool more tailored for manual command-line based management of etcd clusters with no API’s. It also makes no assumptions about the underlying platform on which the etcd clusters are provisioned and hence, doesn’t leverage any capabilities of Kubernetes.

Etcd Cluster Operator from Improbable-Engineering

Etcd Cluster Operator

Etcd Cluster Operator is an Operator for automating the creation and management of etcd inside of Kubernetes. It provides a custom resource definition (CRD) based API to define etcd clusters with Kubernetes resources, and enable management with native Kubernetes tooling._

Out of all the alternatives listed here, this one seems to be the only possible viable alternative. Parts of its design/implementations are similar to some of the approaches mentioned in this proposal. However, we still don’t propose to use it as -

1. The project is still in early phase and is not mature enough to be consumed as is in productive scenarios of ours.
2. The resotration part is completely different which makes it difficult to adopt as-is and requries lot of re-work with the current restoration semantics with etcd-backup-restore making the usage counter-productive.

General Approach to ETCD Cluster Management

Bootstrapping

There are three ways to bootstrap an etcd cluster which are static, etcd discovery and DNS discovery. Out of these, the static way is the simplest (and probably faster to bootstrap the cluster) and has the least external dependencies. Hence, it is preferred in this proposal. But it requires that the initial (during bootstrapping) etcd cluster size (number of members) is already known before bootstrapping and that all of the members are already addressable (DNS,IP,TLS etc.). Such information needs to be passed to the individual members during startup using the following static configuration.

• ETCD_INITIAL_CLUSTER
  • The list of peer URLs including all the members. This must be the same as the advertised peer URLs configuration. This can also be passed as initial-cluster flag to etcd.
• ETCD_INITIAL_CLUSTER_STATE
  • This should be set to new while bootstrapping an etcd cluster.
• ETCD_INITIAL_CLUSTER_TOKEN
  • This is a token to distinguish the etcd cluster from any other etcd cluster in the same network.

Assumptions

• ETCD_INITIAL_CLUSTER can use DNS instead of IP addresses. We need to verify this by deleting a pod (as against scaling down the statefulset) to ensure that the pod IP changes and see if the recreated pod (by the statefulset controller) re-joins the cluster automatically.
• DNS for the individual members is known or computable. This is true in the case of etcd-druid setting up an etcd cluster using a single statefulset. But it may not necessarily be true in other cases (multiple statefulset per etcd cluster or deployments instead of statefulsets or in the case of etcd cluster with members distributed across more than one Kubernetes cluster.

Adding a new member to an etcd cluster

A new member can be added to an existing etcd cluster instance using the following steps.

1. If the latest backup snapshot exists, restore the member’s etcd data to the latest backup snapshot. This can reduce the load on the leader to bring the new member up to date when it joins the cluster.
   1. If the latest backup snapshot doesn’t exist or if the latest backup snapshot is not accessible (please see backup failure) and if the cluster itself is quorate, then the new member can be started with an empty data. But this will will be suboptimal because the new member will fetch all the data from the leading member to get up-to-date.
2. The cluster is informed that a new member is being added using the MemberAdd API including information like the member name and its advertised peer URLs.
3. The new etcd member is then started with ETCD_INITIAL_CLUSTER_STATE=existing apart from other required configuration.

This proposal recommends this approach.

Note

• If there are incremental snapshots (taken by etcd-backup-restore), they cannot be applied because that requires the member to be started in isolation without joining the cluster which is not possible. This is acceptable if the amount of incremental snapshots are managed to be relatively small. This adds one more reason to increase the priority of the issue of incremental snapshot compaction.
• There is a time window, between the MemberAdd call and the new member joining the cluster and getting up to date, where the cluster is vulnerable to leader elections which could be disruptive.

Alternative

With v3.4, the new raft learner approach can be used to mitigate some of the possible disruptions mentioned above. Then the steps will be as follows.

1. If the latest backup snapshot exists, restore the member’s etcd data to the latest backup snapshot. This can reduce the load on the leader to bring the new member up to date when it joins the cluster.
2. The cluster is informed that a new member is being added using the MemberAddAsLearner API including information like the member name and its advertised peer URLs.
3. The new etcd member is then started with ETCD_INITIAL_CLUSTER_STATE=existing apart from other required configuration.
4. Once the new member (learner) is up to date, it can be promoted to a full voting member by using the MemberPromote API

This approach is new and involves more steps and is not recommended in this proposal. It can be considered in future enhancements.

Managing Failures

A multi-node etcd cluster may face failures of diffent kinds during its life-cycle. The actions that need to be taken to manage these failures depend on the failure mode.

Removing an existing member from an etcd cluster

If a member of an etcd cluster becomes unhealthy, it must be explicitly removed from the etcd cluster, as soon as possible. This can be done by using the MemberRemove API. This ensures that only healthy members participate as voting members.

A member of an etcd cluster may be removed not just for managing failures but also for other reasons such as -

• The etcd cluster is being scaled down. I.e. the cluster size is being reduced
• An existing member is being replaced by a new one for some reason (e.g. upgrades)

If the majority of the members of the etcd cluster are healthy and the member that is unhealthy/being removed happens to be the leader at that moment then the etcd cluster will automatically elect a new leader. But if only a minority of etcd clusters are healthy after removing the member then the the cluster will no longer be quorate and will stop accepting write requests. Such an etcd cluster needs to be recovered via some kind of disaster-recovery.

Restarting an existing member of an etcd cluster

If the existing member of an etcd cluster restarts and retains an uncorrupted data directory after the restart, then it can simply re-join the cluster as an existing member without any API calls or configuration changes. This is because the relevant metadata (including member ID and cluster ID) are maintained in the write ahead logs. However, if it doesn’t retain an uncorrupted data directory after the restart, then it must first be removed and added as a new member.

Recovering an etcd cluster from failure of majority of members

If a majority of members of an etcd cluster fail but if they retain their uncorrupted data directory then they can be simply restarted and they will re-form the existing etcd cluster when they come up. However, if they do not retain their uncorrupted data directory, then the etcd cluster must be recovered from latest snapshot in the backup. This is very similar to bootstrapping with the additional initial step of restoring the latest snapshot in each of the members. However, the same limitation about incremental snapshots, as in the case of adding a new member, applies here. But unlike in the case of adding a new member, not applying incremental snapshots is not acceptable in the case of etcd cluster recovery. Hence, if incremental snapshots are required to be applied, the etcd cluster must be recovered in the following steps.

1. Restore a new single-member cluster using the latest snapshot.
2. Apply incremental snapshots on the single-member cluster.
3. Take a full snapshot which can now be used while adding the remaining members.
4. Add new members using the latest snapshot created in the step above.

Kubernetes Context

• Users will provision an etcd cluster in a Kubernetes cluster by creating an etcd CRD resource instance.
• A multi-node etcd cluster is indicated if the spec.replicas field is set to any value greater than 1. The etcd-druid will add validation to ensure that the spec.replicas value is an odd number according to the requirements of etcd.
• The etcd-druid controller will provision a statefulset with the etcd main container and the etcd-backup-restore sidecar container. It will pass on the spec.replicas field from the etcd resource to the statefulset. It will also supply the right pre-computed configuration to both the containers.
• The statefulset controller will create the pods based on the pod template in the statefulset spec and these individual pods will be the members that form the etcd cluster.

This approach makes it possible to satisfy the assumption that the DNS for the individual members of the etcd cluster must be known/computable. This can be achieved by using a headless service (along with the statefulset) for each etcd cluster instance. Then we can address individual pods/etcd members via the predictable DNS name of <statefulset_name>-{0|1|2|3|…|n}.<headless_service_name> from within the Kubernetes namespace (or from outside the Kubernetes namespace by appending .<namespace>.svc.<cluster_domain> suffix). The etcd-druid controller can compute the above configurations automatically based on the spec.replicas in the etcd resource.

This proposal recommends this approach.

Alternative

One statefulset is used for each member (instead of one statefulset for all members). While this approach gives a flexibility to have different pod specifications for the individual members, it makes managing the individual members (e.g. rolling updates) more complicated. Hence, this approach is not recommended.

ETCD Configuration

As mentioned in the general approach section, there are differences in the configuration that needs to be passed to individual members of an etcd cluster in different scenarios such as bootstrapping, adding a new member, removing a member, restarting an existing member etc. Managing such differences in configuration for individual pods of a statefulset is tricky in the recommended approach of using a single statefulset to manage all the member pods of an etcd cluster. This is because statefulset uses the same pod template for all its pods.

The recommendation is for etcd-druid to provision the base configuration template in a ConfigMap which is passed to all the pods via the pod template in the StatefulSet. The initialization flow of etcd-backup-restore (which is invoked every time the etcd container is (re)started) is then enhanced to generate the customized etcd configuration for the corresponding member pod (in a shared volume between etcd and the backup-restore containers) based on the supplied template configuration. This will require that etcd-backup-restore will have to have a mechanism to detect which scenario listed above applies during any given member container/pod restart.

Alternative

As mentioned above, one statefulset is used for each member of the etcd cluster. Then different configuration (generated directly by etcd-druid) can be passed in the pod templates of the different statefulsets. Though this approach is advantageous in the context of managing the different configuration, it is not recommended in this proposal because it makes the rest of the management (e.g. rolling updates) more complicated.

Data Persistence

The type of persistence used to store etcd data (including the member ID and cluster ID) has an impact on the steps that are needed to be taken when the member pods or containers (minority of them or majority) need to be recovered.

Persistent

Like the single-node case, persistentvolumes can be used to persist ETCD data for all the member pods. The individual member pods then get their own persistentvolumes. The advantage is that individual members retain their member ID across pod restarts and even pod deletion/recreation across Kubernetes nodes. This means that member pods that crash (or are unhealthy) can be restarted automatically (by configuring livenessProbe) and they will re-join the etcd cluster using their existing member ID without any need for explicit etcd cluster management).

The disadvantages of this approach are as follows.

• The number of persistentvolumes increases linearly with the cluster size which is a cost-related concern.
• Network-mounted persistentvolumes might eventually become a performance bottleneck under heavy load for a latency-sensitive component like ETCD.
• Volume attach/detach issues when associated with etcd cluster instances cause downtimes to the target shoot clusters that are backed by those etcd cluster instances.

Ephemeral

The ephemeral volumes use-case is considered as an optimization and may be planned as a follow-up action.

Disk

Ephemeral persistence can be achieved in Kubernetes by using either emptyDir volumes or local persistentvolumes to persist ETCD data. The advantages of this approach are as follows.

• Potentially faster disk I/O.
• The number of persistent volumes does not increase linearly with the cluster size (at least not technically).
• Issues related volume attachment/detachment can be avoided.

The main disadvantage of using ephemeral persistence is that the individual members may retain their identity and data across container restarts but not across pod deletion/recreation across Kubernetes nodes. If the data is lost then on restart of the member pod, the older member (represented by the container) has to be removed and a new member has to be added.

Using emptyDir ephemeral persistence has the disadvantage that the volume doesn’t have its own identity. So, if the member pod is recreated but scheduled on the same node as before then it will not retain the identity as the persistence is lost. But it has the advantage that scheduling of pods is unencumbered especially during pod recreation as they are free to be scheduled anywhere.

Using local persistentvolumes has the advantage that the volume has its own indentity and hence, a recreated member pod will retain its identity if scheduled on the same node. But it has the disadvantage of tying down the member pod to a node which is a problem if the node becomes unhealthy requiring etcd druid to take additional actions (such as deleting the local persistent volume).

Based on these constraints, if ephemeral persistence is opted for, it is recommended to use emptyDir ephemeral persistence.

In-memory

In-memory ephemeral persistence can be achieved in Kubernetes by using emptyDir with medium: Memory. In this case, a tmpfs (RAM-backed file-system) volume will be used. In addition to the advantages of ephemeral persistence, this approach can achieve the fastest possible disk I/O. Similarly, in addition to the disadvantages of ephemeral persistence, in-memory persistence has the following additional disadvantages.

• More memory required for the individual member pods.
• Individual members may not at all retain their data and identity across container restarts let alone across pod restarts/deletion/recreation across Kubernetes nodes. I.e. every time an etcd container restarts, the old member (represented by the container) will have to be removed and a new member has to be added.

How to detect if valid metadata exists in an etcd member

Since the likelyhood of a member not having valid metadata in the WAL files is much more likely in the ephemeral persistence scenario, one option is to pass the information that ephemeral persistence is being used to the etcd-backup-restore sidecar (say, via command-line flags or environment variables).

But in principle, it might be better to determine this from the WAL files directly so that the possibility of corrupted WAL files also gets handled correctly. To do this, the wal package has some functions that might be useful.

Recommendation

It might be possible that using the wal package for verifying if valid metadata exists might be performance intensive. So, the performance impact needs to be measured. If the performance impact is acceptable (both in terms of resource usage and time), it is recommended to use this way to verify if the member contains valid metadata. Otherwise, alternatives such as a simple check that WAL folder exists coupled with the static information about use of persistent or ephemeral storage might be considered.

How to detect if valid data exists in an etcd member

The initialization sequence in etcd-backup-restore already includes database verification. This would suffice to determine if the member has valid data.

Recommendation

Though ephemeral persistence has performance and logistics advantages, it is recommended to start with persistent data for the member pods. In addition to the reasons and concerns listed above, there is also the additional concern that in case of backup failure, the risk of additional data loss is a bit higher if ephemeral persistence is used (simultaneous quoram loss is sufficient) when compared to persistent storage (simultaenous quorum loss with majority persistence loss is needed). The risk might still be acceptable but the idea is to gain experience about how frequently member containers/pods get restarted/recreated, how frequently leader election happens among members of an etcd cluster and how frequently etcd clusters lose quorum. Based on this experience, we can move towards using ephemeral (perhaps even in-memory) persistence for the member pods.

Separating peer and client traffic

The current single-node ETCD cluster implementation in etcd-druid and etcd-backup-restore uses a single service object to act as the entry point for the client traffic. There is no separation or distinction between the client and peer traffic because there is not much benefit to be had by making that distinction.

In the multi-node ETCD cluster scenario, it makes sense to distinguish between and separate the peer and client traffic. This can be done by using two services.

• peer
  • To be used for peer communication. This could be a headless service.
• client
  • To be used for client communication. This could be a normal ClusterIP service like it is in the single-node case.

The main advantage of this approach is that it makes it possible (if needed) to allow only peer to peer communication while blocking client communication. Such a thing might be required during some phases of some maintenance tasks (manual or automated).

Cutting off client requests

At present, in the single-node ETCD instances, etcd-druid configures the readinessProbe of the etcd main container to probe the healthz endpoint of the etcd-backup-restore sidecar which considers the status of the latest backup upload in addition to the regular checks about etcd and the side car being up and healthy. This has the effect of setting the etcd main container (and hence the etcd pod) as not ready if the latest backup upload failed. This results in the endpoints controller removing the pod IP address from the endpoints list for the service which eventually cuts off ingress traffic coming into the etcd pod via the etcd client service. The rationale for this is to fail early when the backup upload fails rather than continuing to serve requests while the gap between the last backup and the current data increases which might lead to unacceptably large amount of data loss if disaster strikes.

This approach will not work in the multi-node scenario because we need the individual member pods to be able to talk to each other to maintain the cluster quorum when backup upload fails but need to cut off only client ingress traffic.

It is recommended to separate the backup health condition tracking taking appropriate remedial actions. With that, the backup health condition tracking is now separated to the BackupReady condition in the Etcd resource status and the cutting off of client traffic (which could now be done for more reasons than failed backups) can be achieved in a different way described below.

Manipulating Client Service podSelector

The client traffic can be cut off by updating (manually or automatically by some component) the podSelector of the client service to add an additional label (say, unhealthy or disabled) such that the podSelector no longer matches the member pods created by the statefulset. This will result in the client ingress traffic being cut off. The peer service is left unmodified so that peer communication is always possible.

Health Check

The etcd main container and the etcd-backup-restore sidecar containers will be configured with livenessProbe and readinessProbe which will indicate the health of the containers and effectively the corresponding ETCD cluster member pod.

Backup Failure

As described above using readinessProbe failures based on latest backup failure is not viable in the multi-node ETCD scenario.

Though cutting off traffic by manipulating client service podSelector is workable, it may not be desirable.

It is recommended that on backup failure, the leading etcd-backup-restore sidecar (the one that is responsible for taking backups at that point in time, as explained in the backup section below, updates the BackupReady condition in the Etcd status and raises a high priority alert to the landscape operators but does not cut off the client traffic.

The reasoning behind this decision to not cut off the client traffic on backup failure is to allow the Kubernetes cluster’s control plane (which relies on the ETCD cluster) to keep functioning as long as possible and to avoid bringing down the control-plane due to a missed backup.

The risk of this approach is that with a cascaded sequence of failures (on top of the backup failure), there is a chance of more data loss than the frequency of backup would otherwise indicate.

To be precise, the risk of such an additional data loss manifests only when backup failure as well as a special case of quorum loss (majority of the members are not ready) happen in such a way that the ETCD cluster needs to be re-bootstrapped from the backup. As described here, re-bootstrapping the ETCD cluster requires restoration from the latest backup only when a majority of members no longer have uncorrupted data persistence.

If persistent storage is used, this will happen only when backup failure as well as a majority of the disks/volumes backing the ETCD cluster members fail simultaneously. This would indeed be rare and might be an acceptable risk.

If ephemeral storage is used (especially, in-memory), the data loss will happen if a majority of the ETCD cluster members become NotReady (requiring a pod restart) at the same time as the backup failure. This may not be as rare as majority members’ disk/volume failure. The risk can be somewhat mitigated at least for planned maintenance operations by postponing potentially disruptive maintenance operations when BackupReady condition is false (vertical scaling, rolling updates, evictions due to node roll-outs).

But in practice (when ephemeral storage is used), the current proposal suggests restoring from the latest full backup even when a minority of ETCD members (even a single pod) restart both to speed up the process of the new member catching up to the latest revision but also to avoid load on the leading member which needs to supply the data to bring the new member up-to-date. But as described here, in case of a minority member failure while using ephemeral storage, it is possible to restart the new member with empty data and let it fetch all the data from the leading member (only if backup is not accessible). Though this is suboptimal, it is workable given the constraints and conditions. With this, the risk of additional data loss in the case of ephemeral storage is only if backup failure as well as quorum loss happens. While this is still less rare than the risk of additional data loss in case of persistent storage, the risk might be tolerable. Provided the risk of quorum loss is not too high. This needs to be monitored/evaluated before opting for ephemeral storage.

Given these constraints, it is better to dynamically avoid/postpone some potentially disruptive operations when BackupReady condition is false. This has the effect of allowing n/2 members to be evicted when the backups are healthy and completely disabling evictions when backups are not healthy.

1. Skip/postpone potentially disruptive maintenance operations (listed below) when the BackupReady condition is false.
2. Vertical scaling.
3. Rolling updates, Basically, any updates to the StatefulSet spec which includes vertical scaling.
4. Dynamically toggle the minAvailable field of the PodDisruptionBudget between n/2 + 1 and n (where n is the ETCD desired cluster size) whenever the BackupReady condition toggles between true and false.

This will mean that etcd-backup-restore becomes Kubernetes-aware. But there might be reasons for making etcd-backup-restore Kubernetes-aware anyway (e.g. to update the etcd resource status with latest full snapshot details). This enhancement should keep etcd-backup-restore backward compatible. I.e. it should be possible to use etcd-backup-restore Kubernetes-unaware as before this proposal. This is possible either by auto-detecting the existence of kubeconfig or by an explicit command-line flag (such as --enable-client-service-updates which can be defaulted to false for backward compatibility).

Alternative

The alternative is for etcd-druid to implement the above functionality.

But etcd-druid is centrally deployed in the host Kubernetes cluster and cannot scale well horizontally. So, it can potentially be a bottleneck if it is involved in regular health check mechanism for all the etcd clusters it manages. Also, the recommended approach above is more robust because it can work even if etcd-druid is down when the backup upload of a particular etcd cluster fails.

Status

It is desirable (for the etcd-druid and landscape administrators/operators) to maintain/expose status of the etcd cluster instances in the status sub-resource of the Etcd CRD. The proposed structure for maintaining the status is as shown in the example below.

apiVersion: druid.gardener.cloud/v1alpha1
kind: Etcd
metadata:
  name: etcd-main
spec:
  replicas: 3
  ...
...
status:
  ...
  conditions:
  - type: Ready                 # Condition type for the readiness of the ETCD cluster
    status: "True"              # Indicates of the ETCD Cluster is ready or not
    lastHeartbeatTime:          "2020-11-10T12:48:01Z"
    lastTransitionTime:         "2020-11-10T12:48:01Z"
    reason: Quorate             # Quorate|QuorumLost
  - type: AllMembersReady       # Condition type for the readiness of all the member of the ETCD cluster
    status: "True"              # Indicates if all the members of the ETCD Cluster are ready
    lastHeartbeatTime:          "2020-11-10T12:48:01Z"
    lastTransitionTime:         "2020-11-10T12:48:01Z"
    reason: AllMembersReady     # AllMembersReady|NotAllMembersReady
  - type: BackupReady           # Condition type for the readiness of the backup of the ETCD cluster
    status: "True"              # Indicates if the backup of the ETCD cluster is ready
    lastHeartbeatTime:          "2020-11-10T12:48:01Z"
    lastTransitionTime:         "2020-11-10T12:48:01Z"
    reason: FullBackupSucceeded # FullBackupSucceeded|IncrementalBackupSucceeded|FullBackupFailed|IncrementalBackupFailed
  ...
  clusterSize: 3
  ...
  replicas: 3
  ...
  members:
  - name: etcd-main-0          # member pod name
    id: 272e204152             # member Id
    role: Leader               # Member|Leader
    status: Ready              # Ready|NotReady|Unknown
    lastTransitionTime:        "2020-11-10T12:48:01Z"
    reason: LeaseSucceeded     # LeaseSucceeded|LeaseExpired|UnknownGracePeriodExceeded|PodNotRead
  - name: etcd-main-1          # member pod name
    id: 272e204153             # member Id
    role: Member               # Member|Leader
    status: Ready              # Ready|NotReady|Unknown
    lastTransitionTime:        "2020-11-10T12:48:01Z"
    reason: LeaseSucceeded     # LeaseSucceeded|LeaseExpired|UnknownGracePeriodExceeded|PodNotRead

This proposal recommends that etcd-druid (preferrably, the custodian controller in etcd-druid) maintains most of the information in the status of the Etcd resources described above.

One exception to this is the BackupReady condition which is recommended to be maintained by the leading etcd-backup-restore sidecar container. This will mean that etcd-backup-restore becomes Kubernetes-aware. But there are other reasons for making etcd-backup-restore Kubernetes-aware anyway (e.g. to maintain health conditions). This enhancement should keep etcd-backup-restore backward compatible. But it should be possible to use etcd-backup-restore Kubernetes-unaware as before this proposal. This is possible either by auto-detecting the existence of kubeconfig or by an explicit command-line flag (such as --enable-etcd-status-updates which can be defaulted to false for backward compatibility).

Members

The members section of the status is intended to be maintained by etcd-druid (preferraby, the custodian controller of etcd-druid) based on the leases of the individual members.

Note

An earlier design in this proposal was for the individual etcd-backup-restore sidecars to update the corresponding status.members entries themselves. But this was redesigned to use member leases to avoid conflicts rising from frequent updates and the limitations in the support for Server-Side Apply in some versions of Kubernetes.

The spec.holderIdentity field in the leases is used to communicate the ETCD member id and role between the etcd-backup-restore sidecars and etcd-druid.

Member name as the key

In an ETCD cluster, the member id is the unique identifier for a member. However, this proposal recommends using a single StatefulSet whose pods form the members of the ETCD cluster and Pods of a StatefulSet have uniquely indexed names as well as uniquely addressible DNS.

This proposal recommends that the name of the member (which is the same as the name of the member Pod) be used as the unique key to identify a member in the members array. This can minimise the need to cleanup superfluous entries in the members array after the member pods are gone to some extent because the replacement pods for any member will share the same name and will overwrite the entry with a possibly new member id.

There is still the possibility of not only superfluous entries in the members array but also superfluous members in the ETCD cluster for which there is no corresponding pod in the StatefulSet anymore.

For example, if an ETCD cluster is scaled up from 3 to 5 and the new members were failing constantly due to insufficient resources and then if the ETCD client is scaled back down to 3 and failing member pods may not have the chance to clean up their member entries (from the members array as well as from the ETCD cluster) leading to superfluous members in the cluster that may have adverse effect on quorum of the cluster.

Hence, the superfluous entries in both members array as well as the ETCD cluster need to be cleaned up as appropriate.

Member Leases

One Kubernetes lease object per desired ETCD member is maintained by etcd-druid (preferrably, the custodian controller in etcd-druid). The lease objects will be created in the same namespace as their owning Etcd object and will have the same name as the member to which they correspond (which, in turn would be the same as the pod name in which the member ETCD process runs).

The lease objects are created and deleted only by etcd-druid but are continually renewed within the leaseDurationSeconds by the individual etcd-backup-restore sidecars (corresponding to their members) if the the corresponding ETCD member is ready and is part of the ETCD cluster.

This will mean that etcd-backup-restore becomes Kubernetes-aware. But there are other reasons for making etcd-backup-restore Kubernetes-aware anyway (e.g. to maintain health conditions). This enhancement should keep etcd-backup-restore backward compatible. But it should be possible to use etcd-backup-restore Kubernetes-unaware as before this proposal. This is possible either by auto-detecting the existence of kubeconfig or by an explicit command-line flag (such as --enable-etcd-lease-renewal which can be defaulted to false for backward compatibility).

A member entry in the Etcd resource status would be marked as Ready (with reason: LeaseSucceeded) if the corresponding pod is ready and the corresponding lease has not yet expired. The member entry would be marked as NotReady if the corresponding pod is not ready (with reason PodNotReady) or as Unknown if the corresponding lease has expired (with reason: LeaseExpired).

While renewing the lease, the etcd-backup-restore sidecars also maintain the ETCD member id and their role (Leader or Member) separated by : in the spec.holderIdentity field of the corresponding lease object since this information is only available to the ETCD member processes and the etcd-backup-restore sidecars (e.g. 272e204152:Leader or 272e204153:Member). When the lease objects are created by etcd-druid, the spec.holderIdentity field would be empty.

The value in spec.holderIdentity in the leases is parsed and copied onto the id and role fields of the corresponding status.members by etcd-druid.

Conditions

The conditions section in the status describe the overall condition of the ETCD cluster. The condition type Ready indicates if the ETCD cluster as a whole is ready to serve requests (i.e. the cluster is quorate) even though some minority of the members are not ready. The condition type AllMembersReady indicates of all the members of the ETCD cluster are ready. The distinction between these conditions could be significant for both external consumers of the status as well as etcd-druid itself. Some maintenance operations might be safe to do (e.g. rolling updates) only when all members of the cluster are ready. The condition type BackupReady indicates of the most recent backup upload (full or incremental) succeeded. This information also might be significant because some maintenance operations might be safe to do (e.g. anything that involves re-bootstrapping the ETCD cluster) only when backup is ready.

The Ready and AllMembersReady conditions can be maintained by etcd-druid based on the status in the members section. The BackupReady condition will be maintained by the leading etcd-backup-restore sidecar that is in charge of taking backups.

More condition types could be introduced in the future if specific purposes arise.

ClusterSize

The clusterSize field contains the current size of the ETCD cluster. It will be actively kept up-to-date by etcd-druid in all scenarios.

• Before bootstrapping the ETCD cluster (during cluster creation or later bootstrapping because of quorum failure), etcd-druid will clear the status.members array and set status.clusterSize to be equal to spec.replicas.
• While the ETCD cluster is quorate, etcd-druid will actively set status.clusterSize to be equal to length of the status.members whenever the length of the array changes (say, due to scaling of the ETCD cluster).

Given that clusterSize reliably represents the size of the ETCD cluster, it can be used to calculate the Ready condition.

Alternative

The alternative is for etcd-druid to maintain the status in the Etcd status sub-resource. But etcd-druid is centrally deployed in the host Kubernetes cluster and cannot scale well horizontally. So, it can potentially be a bottleneck if it is involved in regular health check mechanism for all the etcd clusters it manages. Also, the recommended approach above is more robust because it can work even if etcd-druid is down when the backup upload of a particular etcd cluster fails.

Decision table for etcd-druid based on the status

The following decision table describes the various criteria etcd-druid takes into consideration to determine the different etcd cluster management scenarios and the corresponding reconciliation actions it must take. The general principle is to detect the scenario and take the minimum action to move the cluster along the path to good health. The path from any one scenario to a state of good health will typically involve going through multiple reconciliation actions which probably take the cluster through many other cluster management scenarios. Especially, it is proposed that individual members auto-heal where possible, even in the case of the failure of a majority of members of the etcd cluster and that etcd-druid takes action only if the auto-healing doesn’t happen for a configured period of time.

1. Pink of health

Observed state

• Cluster Size
  • Desired: n
  • Current: n
• StatefulSet replicas
  • Desired: n
  • Ready: n
• Etcd status
  • members
    • Total: n
    • Ready: n
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: 0
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: 0
    • Members with expired lease: 0
  • conditions:
    • Ready: true
    • AllMembersReady: true
    • BackupReady: true

Recommended Action

Nothing to do

2. Member status is out of sync with their leases

Observed state

• Cluster Size
  • Desired: n
  • Current: n
• StatefulSet replicas
  • Desired: n
  • Ready: n
• Etcd status
  • members
    • Total: n
    • Ready: r
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: 0
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: 0
    • Members with expired lease: l
  • conditions:
    • Ready: true
    • AllMembersReady: true
    • BackupReady: true

Recommended Action

Mark the l members corresponding to the expired leases as Unknown with reason LeaseExpired and with id populated from spec.holderIdentity of the lease if they are not already updated so.

Mark the n - l members corresponding to the active leases as Ready with reason LeaseSucceeded and with id populated from spec.holderIdentity of the lease if they are not already updated so.

Please refer here for more details.

3. All members are Ready but AllMembersReady condition is stale

Observed state

• Cluster Size
  • Desired: N/A
  • Current: N/A
• StatefulSet replicas
  • Desired: n
  • Ready: N/A
• Etcd status
  • members
    • Total: n
    • Ready: n
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: 0
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: 0
    • Members with expired lease: 0
  • conditions:
    • Ready: N/A
    • AllMembersReady: false
    • BackupReady: N/A

Recommended Action

Mark the status condition type AllMembersReady to true.

4. Not all members are Ready but AllMembersReady condition is stale

Observed state

• Cluster Size

  • Desired: N/A
  • Current: N/A

• StatefulSet replicas

  • Desired: n
  • Ready: N/A

• Etcd status

  • members
    • Total: N/A
    • Ready: r where 0 <= r < n
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: nr where 0 < nr < n
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: u where 0 < u < n
    • Members with expired lease: h where 0 < h < n
  • conditions:
    • Ready: N/A
    • AllMembersReady: true
    • BackupReady: N/A

  where (nr + u + h) > 0 or r < n

Recommended Action

Mark the status condition type AllMembersReady to false.

5. Majority members are Ready but Ready condition is stale

Observed state

• Cluster Size

  • Desired: N/A
  • Current: N/A

• StatefulSet replicas

  • Desired: n
  • Ready: N/A

• Etcd status

  • members
    • Total: n
    • Ready: r where r > n/2
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: nr where 0 < nr < n/2
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: u where 0 < u < n/2
    • Members with expired lease: N/A
  • conditions:
    • Ready: false
    • AllMembersReady: N/A
    • BackupReady: N/A

  where 0 < (nr + u + h) < n/2

Recommended Action

Mark the status condition type Ready to true.

6. Majority members are NotReady but Ready condition is stale

Observed state

• Cluster Size

  • Desired: N/A
  • Current: N/A

• StatefulSet replicas

  • Desired: n
  • Ready: N/A

• Etcd status

  • members
    • Total: n
    • Ready: r where 0 < r < n
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: nr where 0 < nr < n
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: u where 0 < u < n
    • Members with expired lease: N/A
  • conditions:
    • Ready: true
    • AllMembersReady: N/A
    • BackupReady: N/A

  where (nr + u + h) > n/2 or r < n/2

Recommended Action

Mark the status condition type Ready to false.

7. Some members have been in Unknown status for a while

Observed state

• Cluster Size
  • Desired: N/A
  • Current: n
• StatefulSet replicas
  • Desired: N/A
  • Ready: N/A
• Etcd status
  • members
    • Total: N/A
    • Ready: N/A
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: N/A
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: u where u <= n
    • Members with expired lease: N/A
  • conditions:
    • Ready: N/A
    • AllMembersReady: N/A
    • BackupReady: N/A

Recommended Action

Mark the u members as NotReady in Etcd status with reason: UnknownGracePeriodExceeded.

8. Some member pods are not Ready but have not had the chance to update their status

Observed state

• Cluster Size
  • Desired: N/A
  • Current: n
• StatefulSet replicas
  • Desired: n
  • Ready: s where s < n
• Etcd status
  • members
    • Total: N/A
    • Ready: N/A
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: N/A
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: N/A
    • Members with expired lease: N/A
  • conditions:
    • Ready: N/A
    • AllMembersReady: N/A
    • BackupReady: N/A

Recommended Action

Mark the n - s members (corresponding to the pods that are not Ready) as NotReady in Etcd status with reason: PodNotReady

9. Quorate cluster with a minority of members NotReady

Observed state

• Cluster Size
  • Desired: N/A
  • Current: n
• StatefulSet replicas
  • Desired: N/A
  • Ready: N/A
• Etcd status
  • members
    • Total: n
    • Ready: n - f
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: f where f < n/2
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: 0
    • Members with expired lease: N/A
  • conditions:
    • Ready: true
    • AllMembersReady: false
    • BackupReady: true

Recommended Action

Delete the f NotReady member pods to force restart of the pods if they do not automatically restart via failed livenessProbe. The expectation is that they will either re-join the cluster as an existing member or remove themselves and join as new members on restart of the container or pod and renew their leases.

10. Quorum lost with a majority of members NotReady

Observed state

• Cluster Size
  • Desired: N/A
  • Current: n
• StatefulSet replicas
  • Desired: N/A
  • Ready: N/A
• Etcd status
  • members
    • Total: n
    • Ready: n - f
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: f where f >= n/2
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: N/A
    • Members with expired lease: N/A
  • conditions:
    • Ready: false
    • AllMembersReady: false
    • BackupReady: true

Recommended Action

Scale down the StatefulSet to replicas: 0. Ensure that all member pods are deleted. Ensure that all the members are removed from Etcd status. Delete and recreate all the member leases. Recover the cluster from loss of quorum as discussed here.

11. Scale up of a healthy cluster

Observed state

• Cluster Size
  • Desired: d
  • Current: n where d > n
• StatefulSet replicas
  • Desired: N/A
  • Ready: n
• Etcd status
  • members
    • Total: n
    • Ready: n
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: 0
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: 0
    • Members with expired lease: 0
  • conditions:
    • Ready: true
    • AllMembersReady: true
    • BackupReady: true

Recommended Action

Add d - n new members by scaling the StatefulSet to replicas: d. The rest of the StatefulSet spec need not be updated until the next cluster bootstrapping (alternatively, the rest of the StatefulSet spec can be updated pro-actively once the new members join the cluster. This will trigger a rolling update).

Also, create the additional member leases for the d - n new members.

12. Scale down of a healthy cluster

Observed state

• Cluster Size
  • Desired: d
  • Current: n where d < n
• StatefulSet replicas
  • Desired: n
  • Ready: n
• Etcd status
  • members
    • Total: n
    • Ready: n
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: 0
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: 0
    • Members with expired lease: 0
  • conditions:
    • Ready: true
    • AllMembersReady: true
    • BackupReady: true

Recommended Action

Remove d - n existing members (numbered d, d + 1 … n) by scaling the StatefulSet to replicas: d. The StatefulSet spec need not be updated until the next cluster bootstrapping (alternatively, the StatefulSet spec can be updated pro-actively once the superfluous members exit the cluster. This will trigger a rolling update).

Also, delete the member leases for the d - n members being removed.

The superfluous entries in the members array will be cleaned up as explained here. The superfluous members in the ETCD cluster will be cleaned up by the leading etcd-backup-restore sidecar.

13. Superfluous member entries in Etcd status

Observed state

• Cluster Size
  • Desired: N/A
  • Current: n
• StatefulSet replicas
  • Desired: n
  • Ready: n
• Etcd status
  • members
    • Total: m where m > n
    • Ready: N/A
    • Members NotReady for long enough to be evicted, i.e. lastTransitionTime > notReadyGracePeriod: N/A
    • Members with readiness status Unknown long enough to be considered NotReady, i.e. lastTransitionTime > unknownGracePeriod: N/A
    • Members with expired lease: N/A
  • conditions:
    • Ready: N/A
    • AllMembersReady: N/A
    • BackupReady: N/A

Recommended Action

Remove the superfluous m - n member entries from Etcd status (numbered n, n+1 … m). Remove the superfluous m - n member leases if they exist. The superfluous members in the ETCD cluster will be cleaned up by the leading etcd-backup-restore sidecar.

Decision table for etcd-backup-restore during initialization

As discussed above, the initialization sequence of etcd-backup-restore in a member pod needs to generate suitable etcd configuration for its etcd container. It also might have to handle the etcd database verification and restoration functionality differently in different scenarios.

The initialization sequence itself is proposed to be as follows. It is an enhancement of the existing initialization sequence.

The details of the decisions to be taken during the initialization are given below.

1. First member during bootstrap of a fresh etcd cluster

Observed state

• Cluster Size: n
• Etcd status members:
  • Total: 0
  • Ready: 0
  • Status contains own member: false
• Data persistence
  • WAL directory has cluster/ member metadata: false
  • Data directory is valid and up-to-date: false
• Backup
  • Backup exists: false
  • Backup has incremental snapshots: false

Recommended Action

Generate etcd configuration with n initial cluster peer URLs and initial cluster state new and return success.

2. Addition of a new following member during bootstrap of a fresh etcd cluster

Observed state

• Cluster Size: n
• Etcd status members:
  • Total: m where 0 < m < n
  • Ready: m
  • Status contains own member: false
• Data persistence
  • WAL directory has cluster/ member metadata: false
  • Data directory is valid and up-to-date: false
• Backup
  • Backup exists: false
  • Backup has incremental snapshots: false

Recommended Action

Generate etcd configuration with n initial cluster peer URLs and initial cluster state new and return success.

3. Restart of an existing member of a quorate cluster with valid metadata and data

Observed state

• Cluster Size: n
• Etcd status members:
  • Total: m where m > n/2
  • Ready: r where r > n/2
  • Status contains own member: true
• Data persistence
  • WAL directory has cluster/ member metadata: true
  • Data directory is valid and up-to-date: true
• Backup
  • Backup exists: N/A
  • Backup has incremental snapshots: N/A

Recommended Action

Re-use previously generated etcd configuration and return success.

4. Restart of an existing member of a quorate cluster with valid metadata but without valid data

Observed state

• Cluster Size: n
• Etcd status members:
  • Total: m where m > n/2
  • Ready: r where r > n/2
  • Status contains own member: true
• Data persistence
  • WAL directory has cluster/ member metadata: true
  • Data directory is valid and up-to-date: false
• Backup
  • Backup exists: N/A
  • Backup has incremental snapshots: N/A

Recommended Action

Remove self as a member (old member ID) from the etcd cluster as well as Etcd status. Add self as a new member of the etcd cluster as well as in the Etcd status. If backups do not exist, create an empty data and WAL directory. If backups exist, restore only the latest full snapshot (please see here for the reason for not restoring incremental snapshots). Generate etcd configuration with n initial cluster peer URLs and initial cluster state existing and return success.

5. Restart of an existing member of a quorate cluster without valid metadata

Observed state

• Cluster Size: n
• Etcd status members:
  • Total: m where m > n/2
  • Ready: r where r > n/2
  • Status contains own member: true
• Data persistence
  • WAL directory has cluster/ member metadata: false
  • Data directory is valid and up-to-date: N/A
• Backup
  • Backup exists: N/A
  • Backup has incremental snapshots: N/A

Recommended Action

Remove self as a member (old member ID) from the etcd cluster as well as Etcd status. Add self as a new member of the etcd cluster as well as in the Etcd status. If backups do not exist, create an empty data and WAL directory. If backups exist, restore only the latest full snapshot (please see here for the reason for not restoring incremental snapshots). Generate etcd configuration with n initial cluster peer URLs and initial cluster state existing and return success.

6. Restart of an existing member of a non-quorate cluster with valid metadata and data

Observed state

• Cluster Size: n
• Etcd status members:
  • Total: m where m < n/2
  • Ready: r where r < n/2
  • Status contains own member: true
• Data persistence
  • WAL directory has cluster/ member metadata: true
  • Data directory is valid and up-to-date: true
• Backup
  • Backup exists: N/A
  • Backup has incremental snapshots: N/A

Recommended Action

Re-use previously generated etcd configuration and return success.

7. Restart of the first member of a non-quorate cluster without valid data

Observed state

• Cluster Size: n
• Etcd status members:
  • Total: 0
  • Ready: 0
  • Status contains own member: false
• Data persistence
  • WAL directory has cluster/ member metadata: N/A
  • Data directory is valid and up-to-date: false
• Backup
  • Backup exists: N/A
  • Backup has incremental snapshots: N/A

Recommended Action

If backups do not exist, create an empty data and WAL directory. If backups exist, restore the latest full snapshot. Start a single-node embedded etcd with initial cluster peer URLs containing only own peer URL and initial cluster state new. If incremental snapshots exist, apply them serially (honouring source transactions). Take and upload a full snapshot after incremental snapshots are applied successfully (please see here for more reasons why). Generate etcd configuration with n initial cluster peer URLs and initial cluster state new and return success.

8. Restart of a following member of a non-quorate cluster without valid data

Observed state

• Cluster Size: n
• Etcd status members:
  • Total: m where 1 < m < n
  • Ready: r where 1 < r < n
  • Status contains own member: false
• Data persistence
  • WAL directory has cluster/ member metadata: N/A
  • Data directory is valid and up-to-date: false
• Backup
  • Backup exists: N/A
  • Backup has incremental snapshots: N/A

Recommended Action

If backups do not exist, create an empty data and WAL directory. If backups exist, restore only the latest full snapshot (please see here for the reason for not restoring incremental snapshots). Generate etcd configuration with n initial cluster peer URLs and initial cluster state existing and return success.

Backup

Only one of the etcd-backup-restore sidecars among the members are required to take the backup for a given ETCD cluster. This can be called a backup leader. There are two possibilities to ensure this.

Leading ETCD main container’s sidecar is the backup leader

The backup-restore sidecar could poll the etcd cluster and/or its own etcd main container to see if it is the leading member in the etcd cluster. This information can be used by the backup-restore sidecars to decide that sidecar of the leading etcd main container is the backup leader (i.e. responsible to for taking/uploading backups regularly).

The advantages of this approach are as follows.

• The approach is operationally and conceptually simple. The leading etcd container and backup-restore sidecar are always located in the same pod.
• Network traffic between the backup container and the etcd cluster will always be local.

The disadvantage is that this approach may not age well in the future if we think about moving the backup-restore container as a separate pod rather than a sidecar container.

Independent leader election between backup-restore sidecars

We could use the etcd lease mechanism to perform leader election among the backup-restore sidecars. For example, using something like go.etcd.io/etcd/clientv3/concurrency.

The advantage and disadvantages are pretty much the opposite of the approach above. The advantage being that this approach may age well in the future if we think about moving the backup-restore container as a separate pod rather than a sidecar container.

The disadvantages are as follows.

• The approach is operationally and conceptually a bit complex. The leading etcd container and backup-restore sidecar might potentially belong to different pods.
• Network traffic between the backup container and the etcd cluster might potentially be across nodes.

History Compaction

This proposal recommends to configure automatic history compaction on the individual members.

Defragmentation

Defragmentation is already triggered periodically by etcd-backup-restore. This proposal recommends to enhance this functionality to be performed only by the leading backup-restore container. The defragmentation must be performed only when etcd cluster is in full health and must be done in a rolling manner for each members to avoid disruption. The leading member should be defragmented last after all the rest of the members have been defragmented to minimise potential leadership changes caused by defragmentation. If the etcd cluster is unhealthy when it is time to trigger scheduled defragmentation, the defragmentation must be postponed until the cluster becomes healthy. This check must be done before triggering defragmentation for each member.

Work-flows in etcd-backup-restore

There are different work-flows in etcd-backup-restore. Some existing flows like initialization, scheduled backups and defragmentation have been enhanced or modified. Some new work-flows like status updates have been introduced. Some of these work-flows are sensitive to which etcd-backup-restore container is leading and some are not.

The life-cycle of these work-flows is shown below.

Work-flows independent of leader election in all members

• Serve the HTTP API that all members are expected to support currently but some HTTP API call which are used to take out-of-sync delta or full snapshot should delegate the incoming HTTP requests to the leading-sidecar and one of the possible approach to achieve this is via an HTTP reverse proxy.
• Check the health of the respective etcd member and renew the corresponding member lease.

Work-flows only on the leading member

• Take backups (full and incremental) at configured regular intervals
• Defragment all the members sequentially at configured regular intervals
• Cleanup superflous members from the ETCD cluster for which there is no corresponding pod (the ordinal in the pod name is greater than the cluster size) at regular intervals (or whenever the Etcd resource status changes by watching it)
  • The cleanup of superfluous entries in status.members array is already covered here

High Availability

Considering that high-availability is the primary reason for using a multi-node etcd cluster, it makes sense to distribute the individual member pods of the etcd cluster across different physical nodes. If the underlying Kubernetes cluster has nodes from multiple availability zones, it makes sense to also distribute the member pods across nodes from different availability zones.

One possibility to do this is via SelectorSpreadPriority of kube-scheduler but this is only best-effort and may not always be enforced strictly.

It is better to use pod anti-affinity to enforce such distribution of member pods.

Zonal Cluster - Single Availability Zone

A zonal cluster is configured to consist of nodes belonging to only a single availability zone in a region of the cloud provider. In such a case, we can at best distribute the member pods of a multi-node etcd cluster instance only across different nodes in the configured availability zone.

This can be done by specifying pod anti-affinity in the specification of the member pods using kubernetes.io/hostname as the topology key.

apiVersion: apps/v1
kind: StatefulSet
...
spec:
  ...
  template:
    ...
    spec:
      ...
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector: {} # podSelector that matches the member pods of the given etcd cluster instance
            topologyKey: "kubernetes.io/hostname"
      ...
    ...
  ...

The recommendation is to keep etcd-druid agnostic of such topics related scheduling and cluster-topology and to use kupid to orthogonally inject the desired pod anti-affinity.

Alternative

Another option is to build the functionality into etcd-druid to include the required pod anti-affinity when it provisions the StatefulSet that manages the member pods. While this has the advantage of avoiding a dependency on an external component like kupid, the disadvantage is that we might need to address development or testing use-cases where it might be desirable to avoid distributing member pods and schedule them on as less number of nodes as possible. Also, as mentioned below, kupid can be used to distribute member pods of an etcd cluster instance across nodes in a single availability zone as well as across nodes in multiple availability zones with very minor variation. This keeps the solution uniform regardless of the topology of the underlying Kubernetes cluster.

Regional Cluster - Multiple Availability Zones

A regional cluster is configured to consist of nodes belonging to multiple availability zones (typically, three) in a region of the cloud provider. In such a case, we can distribute the member pods of a multi-node etcd cluster instance across nodes belonging to different availability zones.

This can be done by specifying pod anti-affinity in the specification of the member pods using topology.kubernetes.io/zone as the topology key. In Kubernetes clusters using Kubernetes release older than 1.17, the older (and now deprecated) failure-domain.beta.kubernetes.io/zone might have to be used as the topology key.

apiVersion: apps/v1
kind: StatefulSet
...
spec:
  ...
  template:
    ...
    spec:
      ...
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector: {} # podSelector that matches the member pods of the given etcd cluster instance
            topologyKey: "topology.kubernetes.io/zone
      ...
    ...
  ...

The recommendation is to keep etcd-druid agnostic of such topics related scheduling and cluster-topology and to use kupid to orthogonally inject the desired pod anti-affinity.

Alternative

Another option is to build the functionality into etcd-druid to include the required pod anti-affinity when it provisions the StatefulSet that manages the member pods. While this has the advantage of avoiding a dependency on an external component like kupid, the disadvantage is that such built-in support necessarily limits what kind of topologies of the underlying cluster will be supported. Hence, it is better to keep etcd-druid altogether agnostic of issues related to scheduling and cluster-topology.

PodDisruptionBudget

This proposal recommends that etcd-druid should deploy PodDisruptionBudget (minAvailable set to floor(<cluster size>/2) + 1) for multi-node etcd clusters (if AllMembersReady condition is true) to ensure that any planned disruptive operation can try and honour the disruption budget to ensure high availability of the etcd cluster while making potentially disrupting maintenance operations.

Also, it is recommended to toggle the minAvailable field between floor(<cluster size>/2) and <number of members with status Ready true> whenever the AllMembersReady condition toggles between true and false. This is to disable eviction of any member pods when not all members are Ready.

In case of a conflict, the recommendation is to use the highest of the applicable values for minAvailable.

Rolling updates to etcd members

Any changes to the Etcd resource spec that might result in a change to StatefulSet spec or otherwise result in a rolling update of member pods should be applied/propagated by etcd-druid only when the etcd cluster is fully healthy to reduce the risk of quorum loss during the updates. This would include vertical autoscaling changes (via, HVPA). If the cluster status unhealthy (i.e. if either AllMembersReady or BackupReady conditions are false), etcd-druid must restore it to full health before proceeding with such operations that lead to rolling updates. This can be further optimized in the future to handle the cases where rolling updates can still be performed on an etcd cluster that is not fully healthy.

Follow Up

Ephemeral Volumes

See section Ephemeral Volumes.

Shoot Control-Plane Migration

This proposal adds support for multi-node etcd clusters but it should not have significant impact on shoot control-plane migration any more than what already present in the single-node etcd cluster scenario. But to be sure, this needs to be discussed further.

Performance impact of multi-node etcd clusters

Multi-node etcd clusters incur a cost on write performance as compared to single-node etcd clusters. This performance impact needs to be measured and documented. Here, we should compare different persistence option for the multi-nodeetcd clusters so that we have all the information necessary to take the decision balancing the high-availability, performance and costs.

Metrics, Dashboards and Alerts

There are already metrics exported by etcd and etcd-backup-restore which are visualized in monitoring dashboards and also used in triggering alerts. These might have hidden assumptions about single-node etcd clusters. These might need to be enhanced and potentially new metrics, dashboards and alerts configured to cover the multi-node etcd cluster scenario.

Especially, a high priority alert must be raised if BackupReady condition becomes false.

Costs

Multi-node etcd clusters will clearly involve higher cost (when compared with single-node etcd clusters) just going by the CPU and memory usage for the additional members. Also, the different options for persistence for etcd data for the members will have different cost implications. Such cost impact needs to be assessed and documented to help navigate the trade offs between high availability, performance and costs.

Future Work

Gardener Ring

Gardener Ring, requires provisioning and management of an etcd cluster with the members distributed across more than one Kubernetes cluster. This cannot be achieved by etcd-druid alone which has only the view of a single Kubernetes cluster. An additional component that has the view of all the Kubernetes clusters involved in setting up the gardener ring will be required to achieve this. However, etcd-druid can be used by such a higher-level component/controller (for example, by supplying the initial cluster configuration) such that individual etcd-druid instances in the individual Kubernetes clusters can manage the corresponding etcd cluster members.

Autonomous Shoot Clusters

Autonomous Shoot Clusters also will require a highly availble etcd cluster to back its control-plane and the multi-node support proposed here can be leveraged in that context. However, the current proposal will not meet all the needs of a autonomous shoot cluster. Some additional components will be required that have the overall view of the autonomous shoot cluster and they can use etcd-druid to manage the multi-node etcd cluster. But this scenario may be different from that of Gardener Ring in that the individual etcd members of the cluster may not be hosted on different Kubernetes clusters.

Optimization of recovery from non-quorate cluster with some member containing valid data

It might be possible to optimize the actions during the recovery of a non-quorate cluster where some of the members contain valid data and some other don’t. The optimization involves verifying the data of the valid members to determine the data of which member is the most recent (even considering the latest backup) so that the full snapshot can be taken from it before recovering the etcd cluster. Such an optimization can be attempted in the future.

Optimization of rolling updates to unhealthy etcd clusters

As mentioned above, optimizations to proceed with rolling updates to unhealthy etcd clusters (without first restoring the cluster to full health) can be pursued in future work.

3 - 02 Snapshot Compaction

DEP-02: Snapshot Compaction for Etcd

Current Problem

To ensure recoverability of Etcd, backups of the database are taken at regular interval. Backups are of two types: Full Snapshots and Incremental Snapshots.

Full Snapshots

Full snapshot is a snapshot of the complete database at given point in time.The size of the database keeps changing with time and typically the size is relatively large (measured in 100s of megabytes or even in gigabytes. For this reason, full snapshots are taken after some large intervals.

Incremental Snapshots

Incremental Snapshots are collection of events on Etcd database, obtained through running WATCH API Call on Etcd. After some short intervals, all the events that are accumulated through WATCH API Call are saved in a file and named as Incremental Snapshots at relatively short time intervals.

Recovery from the Snapshots

Recovery from Full Snapshots

As the full snapshots are snapshots of the complete database, the whole database can be recovered from a full snapshot in one go. Etcd provides API Call to restore the database from a full snapshot file.

Recovery from Incremental Snapshots

Delta snapshots are collection of retrospective Etcd events. So, to restore from Incremental snapshot file, the events from the file are needed to be applied sequentially on Etcd database through Etcd Put/Delete API calls. As it is heavily dependent on Etcd calls sequentially, restoring from Incremental Snapshot files can take long if there are numerous commands captured in Incremental Snapshot files.

Delta snapshots are applied on top of running Etcd database. So, if there is inconsistency between the state of database at the point of applying and the state of the database when the delta snapshot commands were captured, restoration will fail.

Currently, in Gardener setup, Etcd is restored from the last full snapshot and then the delta snapshots, which were captured after the last full snapshot.

The main problem with this is that the complete restoration time can be unacceptably large if the rate of change coming into the etcd database is quite high because there are large number of events in the delta snapshots to be applied sequentially. A secondary problem is that, though auto-compaction is enabled for etcd, it is not quick enough to compact all the changes from the incremental snapshots being re-applied during the relatively short period of time of restoration (as compared to the actual period of time when the incremental snapshots were accumulated). This may lead to the etcd pod (the backup-restore sidecar container, to be precise) to run out of memory and/or storage space even if it is sufficient for normal operations.

Solution

Compaction command

To help with the problem mentioned earlier, our proposal is to introduce compact subcommand with etcdbrctl. On execution of compact command, A separate embedded Etcd process will be started where the Etcd data will be restored from the snapstore (exactly as in the restoration scenario today). Then the new Etcd database will be compacted and defragmented using Etcd API calls. The compaction will strip off the Etcd database of old revisions as per the Etcd auto-compaction configuration. The defragmentation will free up the unused fragment memory space released after compaction. Then a full snapshot of the compacted database will be saved in snapstore which then can be used as the base snapshot during any subsequent restoration (or backup compaction).

How the solution works

The newly introduced compact command does not disturb the running Etcd while compacting the backup snapshots. The command is designed to run potentially separately (from the main Etcd process/container/pod). Etcd Druid can be configured to run the newly introduced compact command as a separate job (scheduled periodically) based on total number of Etcd events accumulated after the most recent full snapshot.

Etcd-druid flags:

Etcd-druid introduces the following flags to configure the compaction job:

• --enable-backup-compaction (default false): Set this flag to true to enable the automatic compaction of etcd backups when the threshold value denoted by CLI flag --etcd-events-threshold is exceeded.
• --compaction-workers (default 3): Number of worker threads of the CompactionJob controller. The controller creates a backup compaction job if a certain etcd event threshold is reached. If compaction is enabled, the value for this flag must be greater than zero.
• --etcd-events-threshold (default 1000000): Total number of etcd events that can be allowed before a backup compaction job is triggered.
• --active-deadline-duration (default 3h): Duration after which a running backup compaction job will be terminated.
• --metrics-scrape-wait-duration (default 0s): Duration to wait for after compaction job is completed, to allow Prometheus metrics to be scraped.

Points to take care while saving the compacted snapshot:

As compacted snapshot and the existing periodic full snapshots are taken by different processes running in different pods but accessing same store to save the snapshots, some problems may arise:

1. When uploading the compacted snapshot to the snapstore, there is the problem of how does the restorer know when to start using the newly compacted snapshot. This communication needs to be atomic.
2. With a regular schedule for compaction that happens potentially separately from the main etcd pod, is there a need for regular scheduled full snapshots anymore?
3. We are planning to introduce new directory structure, under v2 prefix, for saving the snapshots (compacted and full), as mentioned in details below. But for backward compatibility, we also need to consider the older directory, which is currently under v1 prefix, during accessing snapshots.

How to swap full snapshot with compacted snapshot atomically

Currently, full snapshots and the subsequent delta snapshots are grouped under same prefix path in the snapstore. When a full snapshot is created, it is placed under a prefix/directory with the name comprising of timestamp. Then subsequent delta snapshots are also pushed into the same directory. Thus each prefix/directory contains a single full snapshot and the subsequent delta snapshots. So far, it is the job of ETCDBR to start main Etcd process and snapshotter process which takes full snapshot and delta snapshot periodically. But as per our proposal, compaction will be running as parallel process to main Etcd process and snapshotter process. So we can’t reliably co-ordinate between the processes to achieve switching to the compacted snapshot as the base snapshot atomically.

Current Directory Structure

- Backup-192345
    - Full-Snapshot-0-1-192345
    - Incremental-Snapshot-1-100-192355
    - Incremental-Snapshot-100-200-192365
    - Incremental-Snapshot-200-300-192375
- Backup-192789
    - Full-Snapshot-0-300-192789
    - Incremental-Snapshot-300-400-192799
    - Incremental-Snapshot-400-500-192809
    - Incremental-Snapshot-500-600-192819

To solve the problem, proposal is:

1. ETCDBR will take the first full snapshot after it starts main Etcd Process and snapshotter process. After taking the first full snapshot, snapshotter will continue taking full snapshots. On the other hand, ETCDBR compactor command will be run as periodic job in a separate pod and use the existing full or compacted snapshots to produce further compacted snapshots. Full snapshots and compacted snapshots will be named after same fashion. So, there is no need of any mechanism to choose which snapshots(among full and compacted snapshot) to consider as base snapshots.
2. Flatten the directory structure of backup folder. Save all the full snapshots, delta snapshots and compacted snapshots under same directory/prefix. Restorer will restore from full/compacted snapshots and delta snapshots sorted based on the revision numbers in name (or timestamp if the revision numbers are equal).

Proposed Directory Structure

Backup :
    - Full-Snapshot-0-1-192355 (Taken by snapshotter)
    - Incremental-Snapshot-revision-1-100-192365
    - Incremental-Snapshot-revision-100-200-192375
    - Full-Snapshot-revision-0-200-192379 (Taken by snapshotter)
    - Incremental-Snapshot-revision-200-300-192385
    - Full-Snapshot-revision-0-300-192386 (Taken by compaction job)
    - Incremental-Snapshot-revision-300-400-192396
    - Incremental-Snapshot-revision-400-500-192406
    - Incremental-Snapshot-revision-500-600-192416
    - Full-Snapshot-revision-0-600-192419 (Taken by snapshotter)
    - Full-Snapshot-revision-0-600-192420 (Taken by compaction job)

What happens to the delta snapshots that were compacted?

The proposed compaction sub-command in etcdbrctl (and hence, the CronJob provisioned by etcd-druid that will schedule it at a regular interval) would only upload the compacted full snapshot. It will not delete the snapshots (delta or full snapshots) that were compacted. These snapshots which were superseded by a freshly uploaded compacted snapshot would follow the same life-cycle as other older snapshots. I.e. they will be garbage collected according to the configured backup snapshot retention policy. For example, if an exponential retention policy is configured and if compaction is done every 30m then there might be at most 48 additional (compacted) full snapshots (24h * 2) in the backup for the latest day. As time rolls forward to the next day, these additional compacted snapshots (along with the delta snapshots that were compacted into them) will get garbage collected retaining only one full snapshot for the day before according to the retention policy.

Future work

In the future, we have plan to stop the snapshotter just after taking the first full snapshot. Then, the compaction job will be solely responsible for taking subsequent full snapshots. The directory structure would be looking like following:

Backup :
    - Full-Snapshot-0-1-192355 (Taken by snapshotter)
    - Incremental-Snapshot-revision-1-100-192365
    - Incremental-Snapshot-revision-100-200-192375
    - Incremental-Snapshot-revision-200-300-192385
    - Full-Snapshot-revision-0-300-192386 (Taken by compaction job)
    - Incremental-Snapshot-revision-300-400-192396
    - Incremental-Snapshot-revision-400-500-192406
    - Incremental-Snapshot-revision-500-600-192416
    - Full-Snapshot-revision-0-600-192420 (Taken by compaction job)

Backward Compatibility

1. Restoration : The changes to handle the newly proposed backup directory structure must be backward compatible with older structures at least for restoration because we need have to restore from backups in the older structure. This includes the support for restoring from a backup without a metadata file if that is used in the actual implementation.
2. Backup : For new snapshots (even on a backup containing the older structure), the new structure may be used. The new structure must be setup automatically including creating the base full snapshot.
3. Garbage collection : The existing functionality of garbage collection of snapshots (full and incremental) according to the backup retention policy must be compatible with both old and new backup folder structure. I.e. the snapshots in the older backup structure must be retained in their own structure and the snapshots in the proposed backup structure should be retained in the proposed structure. Once all the snapshots in the older backup structure go out of the retention policy and are garbage collected, we can think of removing the support for older backup folder structure.

Note: Compactor will run parallel to current snapshotter process and work only if there is any full snapshot already present in the store. By current design, a full snapshot will be taken if there is already no full snapshot or the existing full snapshot is older than 24 hours. It is not limitation but a design choice. As per proposed design, the backup storage will contain both periodic full snapshots as well as periodic compacted snapshot. Restorer will pickup the base snapshot whichever is latest one.

4 - 03 Scaling Up An Etcd Cluster

DEP-03: Scaling-up a single-node to multi-node etcd cluster deployed by etcd-druid

To mark a cluster for scale-up from single node to multi-node etcd, just patch the etcd custom resource’s .spec.replicas from 1 to 3 (for example).

Challenges for scale-up

1. Etcd cluster with single replica don’t have any peers, so no peer communication is required hence peer URL may or may not be TLS enabled. However, while scaling up from single node etcd to multi-node etcd, there will be a requirement to have peer communication between members of the etcd cluster. Peer communication is required for various reasons, for instance for members to sync up cluster state, data, and to perform leader election or any cluster wide operation like removal or addition of a member etc. Hence in a multi-node etcd cluster we need to have TLS enable peer URL for peer communication.
2. Providing the correct configuration to start new etcd members as it is different from boostrapping a cluster since these new etcd members will join an existing cluster.

Approach

We first went through the etcd doc of update-advertise-peer-urls to find out information regarding peer URL updation. Interestingly, etcd doc has mentioned the following:

To update the advertise peer URLs of a member, first update it explicitly via member command and then restart the member.

But we can’t assume peer URL is not TLS enabled for single-node cluster as it depends on end-user. A user may or may not enable the TLS for peer URL for a single node etcd cluster. So, How do we detect whether peer URL was enabled or not when cluster is marked for scale-up?

Detecting if peerURL TLS is enabled or not

For this we use an annotation in member lease object member.etcd.gardener.cloud/tls-enabled set by backup-restore sidecar of etcd. As etcd configuration is provided by backup-restore, so it can find out whether TLS is enabled or not and accordingly set this annotation member.etcd.gardener.cloud/tls-enabled to either true or false in member lease object. And with the help of this annotation and config-map values etcd-druid is able to detect whether there is a change in a peer URL or not.

Etcd-Druid helps in scaling up etcd cluster

Now, it is detected whether peer URL was TLS enabled or not for single node etcd cluster. Etcd-druid can now use this information to take action:

• If peer URL was already TLS enabled then no action is required from etcd-druid side. Etcd-druid can proceed with scaling up the cluster.
• If peer URL was not TLS enabled then etcd-druid has to intervene and make sure peer URL should be TLS enabled first for the single node before marking the cluster for scale-up.

Action taken by etcd-druid to enable the peerURL TLS

1. Etcd-druid will update the etcd-bootstrap config-map with new config like initial-cluster,initial-advertise-peer-urls etc. Backup-restore will detect this change and update the member lease annotation to member.etcd.gardener.cloud/tls-enabled: "true".
2. In case the peer URL TLS has been changed to enabled: Etcd-druid will add tasks to the deployment flow:
   • Check if peer TLS has been enabled for existing StatefulSet pods, by checking the member leases for the annotation member.etcd.gardener.cloud/tls-enabled.
   • If peer TLS enablement is pending for any of the members, then check and patch the StatefulSet with the peer TLS volume mounts, if not already patched. This will cause a rolling update of the existing StatefulSet pods, which allows etcd-backup-restore to update the member peer URL in the etcd cluster.
   • Requeue this reconciliation flow until peer TLS has been enabled for all the existing etcd members.

After PeerURL is TLS enabled

After peer URL TLS enablement for single node etcd cluster, now etcd-druid adds a scale-up annotation: gardener.cloud/scaled-to-multi-node to the etcd statefulset and etcd-druid will patch the statefulsets .spec.replicas to 3(for example). The statefulset controller will then bring up new pods(etcd with backup-restore as a sidecar). Now etcd’s sidecar i.e backup-restore will check whether this member is already a part of a cluster or not and incase it is unable to check (may be due to some network issues) then backup-restore checks presence of this annotation: gardener.cloud/scaled-to-multi-node in etcd statefulset to detect scale-up. If it finds out it is the scale-up case then backup-restore adds new etcd member as a learner first and then starts the etcd learner by providing the correct configuration. Once learner gets in sync with the etcd cluster leader, it will get promoted to a voting member.

Providing the correct etcd config

As backup-restore detects that it’s a scale-up scenario, backup-restore sets initial-cluster-state to existing as this member will join an existing cluster and it calculates the rest of the config from the updated config-map provided by etcd-druid.

Future improvements:

The need of restarting etcd pods twice will change in the future. please refer: https://github.com/gardener/etcd-backup-restore/issues/538

5 - Add New Etcd Cluster Component

Add A New Etcd Cluster Component

etcd-druid defines an Operator which is responsible for creation, deletion and update of a resource that is created for an Etcd cluster. If you want to introduce a new resource for an Etcd cluster then you must do the following:

• Add a dedicated package for the resource under component.

• Implement Operator interface.

• Define a new Kind for this resource in the operator Registry.

• Every resource a.k.a Component needs to have the following set of default labels:

  • app.kubernetes.io/name - value of this label is the name of this component. Helper functions are defined here to create the name of each component using the parent Etcd resource. Please define a new helper function to generate the name of your resource using the parent Etcd resource.
  • app.kubernetes.io/component - value of this label is the type of the component. All component type label values are defined here where you can add an entry for your component.
  • In addition to the above component specific labels, each resource/component should have default labels defined on the Etcd resource. You can use GetDefaultLabels function.

  These labels are also part of recommended labels by kubernetes. NOTE: Constants for the label keys are already defined here.

• Ensure that there is no wait introduced in any Operator method implementation in your component. In case there are multiple steps to be executed in a sequence then re-queue the event with a special error code in case there is an error or if the pre-conditions check to execute the next step are not yet satisfied.

• All errors should be wrapped with a custom DruidError.

6 - Changing Api

Change the API

This guide provides detailed information on what needs to be done when the API needs to be changed.

etcd-druid API follows the same API conventions and guidelines which Kubernetes defines and adopts. The Kubernetes API Conventions as well as Changing the API topics already provide a good overview and general explanation of the basic concepts behind it. We adhere to the principles laid down by Kubernetes.

Etcd Druid API

The etcd-druid API is defined here.

!!! info The current version of the API is v1alpha1. We are currently working on migration to v1beta1 API.

Changing the API

If there is a need to make changes to the API, then one should do the following:

• If new fields are added then ensure that these are added as optional fields. They should have the +optional comment and an omitempty JSON tag should be added against the field.
• Ensure that all new fields or changing the existing fields are well documented with doc-strings.
• Care should be taken that incompatible API changes should not be made in the same version of the API. If there is a real necessity to introduce a backward incompatible change then a newer version of the API should be created and an API conversion webhook should be put in place to support more than one version of the API.
• After the changes to the API are finalized, run make generate to ensure that the changes are also reflected in the CRD.
• If necessary, implement or adapt the validation for the API.
• If necessary, adapt the samples YAML manifests.
• When opening a pull-request, always add a release note informing the end-users of the changes that are coming in.

Removing a Field

If field(s) needs to be removed permanently from the API, then one should ensure the following:

• Field should not be directly removed, instead first a deprecation notice should be put which should follow a well-defined deprecation period. Ensure that the release note in the pull-request is properly categorized so that this is easily visible to the end-users and clearly mentiones which field(s) have been deprecated. Clearly suggest a way in which clients need to adapt.
• To allow sufficient time to the end-users to adapt to the API changes, deprecated field(s) should only be removed once the deprecation period is over. It is generally recommended that this be done in 2 stages:
  • First stage: Remove the code that refers to the deprecated fields. This ensures that the code no longer has dependency on the deprecated field(s).
  • Second Stage: Remove the field from the API.

7 - Configure Etcd Druid

etcd-druid CLI Flags

etcd-druid process can be started with the following command line flags.

Command line flags

Leader election

If you wish to setup etcd-druid in high-availability mode then leader election needs to be enabled to ensure that at a time only one replica services the incoming events and does the reconciliation.

Metrics

etcd-druid exposes a /metrics endpoint which can be scrapped by tools like Prometheus. If the default metrics endpoint configuration is not suitable then consumers can change it via the following options.

Metrics bind-address is computed by joining the host and port. By default its value is computed as :8080.

!!! tip Ensure that the metrics-port is also reflected in the etcd-druid deployment specification.

Webhook Server

etcd-druid provides the following CLI flags to configure webhook server. These CLI flags are used to construct a new webhook.Server by configuring Options.

Etcd-Components Webhook

etcd-druid provisions and manages several Kubernetes resources which we call Etcdcluster components. To ensure that there is no accidental changes done to these managed resources, a webhook is put in place to check manual changes done to any managed etcd-cluster Kubernetes resource. It rejects most of these changes except a few. The details on how to enable the etcd-components webhook, which resources are protected and in which scenarios is the change allowed is documented here.

Following CLI flags are provided to configure the etcd-components webhook:

Reconcilers

Following set of flags configures the reconcilers running within etcd-druid. To know more about different reconcilers read this document.

Etcd Reconciler

Compaction Reconciler

Etcd Copy-Backup Task & Secret Reconcilers

Miscellaneous

8 - Contribution

Contributors Guide

etcd-druid is an actively maintained project which has organically evolved to be a mature and stable etcd operator. We welcome active participation from the community and to this end this guide serves as a good starting point.

Code of Conduct

All maintainers and contributors must abide by Contributor Covenant. Real progress can only happen in a collaborative environment which fosters mutual respect, openeness and disruptive innovation.

Developer Certificate of Origin

Due to legal reasons, contributors will be asked to accept a Developer Certificate of Origin (DCO) before they submit the first pull request to the IronCore project, this happens in an automated fashion during the submission process. We use the standard DCO text of the Linux Foundation.

License

Your contributions to etcd-druid must be licensed properly:

• Code contributions must be licensed under the Apache 2.0 License.
• Documentation contributions must be licensed under the Creative Commons Attribution 4.0 International License.

Contributing

etcd-druid use Github to manage reviews of pull requests.

• If you are looking to make your first contribution, follow Steps to Contribute.
• If you have a trivial fix or improvement, go ahead and create an issue first followed by a pull request.
• If you plan to do something more involved, first discuss your ideas by creating an issue. This will avoid unnecessary work and surely give you and us a good deal of inspiration.

Steps to Contribute

• If you wish to contribute and have not done that in the past, then first try and filter the list of issues with label exp/beginner. Once you find the issue that interests you, add a comment stating that you would like to work on it. This is to prevent duplicated efforts from contributors on the same issue.
• If you have questions about one of the issues please comment on them and one of the maintainers will clarify it.

We kindly ask you to follow the Pull Request Checklist to ensure reviews can happen accordingly.

Issues and Planning

We use GitHub issues to track bugs and enhancement requests. Please provide as much context as possible when you open an issue. The information you provide must be comprehensive enough to understand, reproduce the behavior and find related reports of that issue for the assignee. Therefore, contributors may use but aren’t restricted to the issue template provided by the etcd-druid maintainers.

9 - Controllers

Controllers

etcd-druid is an operator to manage etcd clusters, and follows the Operator pattern for Kubernetes. It makes use of the Kubebuilder framework which makes it quite easy to define Custom Resources (CRs) such as Etcds and EtcdCopyBackupTasks through Custom Resource Definitions (CRDs), and define controllers for these CRDs. etcd-druid uses Kubebuilder to define the Etcd CR and its corresponding controllers.

All controllers that are a part of etcd-druid reside in package internal/controller, as sub-packages.

Etcd-druid currently consists of the following controllers, each having its own responsibility:

• etcd : responsible for the reconciliation of the Etcd CR spec, which allows users to run etcd clusters within the specified Kubernetes cluster, and also responsible for periodically updating the Etcd CR status with the up-to-date state of the managed etcd cluster.
• compaction : responsible for snapshot compaction.
• etcdcopybackupstask : responsible for the reconciliation of the EtcdCopyBackupsTask CR, which helps perform the job of copying snapshot backups from one object store to another.
• secret : responsible in making sure Secrets being referenced by Etcd resources are not deleted while in use.

Package Structure

The typical package structure for the controllers that are part of etcd-druid is shown with the compaction controller:

internal/controller/compaction
├── config.go
├── reconciler.go
└── register.go

• config.go: contains all the logic for the configuration of the controller, including feature gate activations, CLI flag parsing and validations.
• register.go: contains the logic for registering the controller with the etcd-druid controller manager.
• reconciler.go: contains the controller reconciliation logic.

Each controller package also contains auxiliary files which are relevant to that specific controller.

Controller Manager

A manager is first created for all controllers that are a part of etcd-druid. The controller manager is responsible for all the controllers that are associated with CRDs. Once the manager is Start()ed, all the controllers that are registered with it are started.

Each controller is built using a controller builder, configured with details such as the type of object being reconciled, owned objects whose owner object is reconciled, event filters (predicates), etc. Predicates are filters which allow controllers to filter which type of events the controller should respond to and which ones to ignore.

The logic relevant to the controller manager like the creation of the controller manager and registering each of the controllers with the manager, is contained in internal/manager/manager.go.

Etcd Controller

The etcd controller is responsible for the reconciliation of the Etcd resource spec and status. It handles the provisioning and management of the etcd cluster. Different components that are required for the functioning of the cluster like Leases, ConfigMaps, and the Statefulset for the etcd cluster are all deployed and managed by the etcd controller.

Additionally, etcd controller also periodically updates the Etcd resource status with the latest available information from the etcd cluster, as well as results and errors from the recent-most reconciliation of the Etcd resource spec.

The etcd controller is essential to the functioning of the etcd cluster and etcd-druid, thus the minimum number of worker threads is 1 (default being 3), controlled by the CLI flag --etcd-workers.

Etcd Spec Reconciliation

While building the controller, an event filter is set such that the behavior of the controller, specifically for Etcd update operations, depends on the gardener.cloud/operation: reconcile annotation. This is controlled by the --enable-etcd-spec-auto-reconcile CLI flag, which, if set to false, tells the controller to perform reconciliation only when this annotation is present. If the flag is set to true, the controller will reconcile the etcd cluster anytime the Etcd spec, and thus generation, changes, and the next queued event for it is triggered.

!!! note Creation and deletion of Etcd resources are not affected by the above flag or annotation.

The reason this filter is present is that any disruption in the Etcd resource due to reconciliation (due to changes in the Etcd spec, for example) while workloads are being run would cause unwanted downtimes to the etcd cluster. Hence, any user who wishes to avoid such disruptions, can choose to set the --enable-etcd-spec-auto-reconcile CLI flag to false. An example of this is Gardener’s gardenlet, which reconciles the Etcd resource only during a shoot cluster’s maintenance window.

The controller adds a finalizer to the Etcd resource in order to ensure that it does not get deleted until all dependent resources managed by etcd-druid, aka managed components, are properly cleaned up. Only the etcd controller can delete a resource once it adds finalizers to it. This ensures that the proper deletion flow steps are followed while deleting the resource. During deletion flow, managed components are deleted in parallel.

Etcd Status Updates

The Etcd resource status is updated periodically by etcd controller, the interval for which is determined by the CLI flag --etcd-status-sync-period.

Status fields of the Etcd resource such as LastOperation, LastErrors and ObservedGeneration, are updated to reflect the result of the recent reconciliation of the Etcd resource spec.

• LastOperation holds information about the last operation performed on the etcd cluster, indicated by fields Type, State, Description and LastUpdateTime. Additionally, a field RunID indicates the unique ID assigned to the specific reconciliation run, to allow for better debugging of issues.
• LastErrors is a slice of errors encountered by the last reconciliation run. Each error consists of fields Code to indicate the custom etcd-druid error code for the error, a human-readable Description, and the ObservedAt time when the error was seen.
• ObservedGeneration indicates the latest generation of the Etcd resource that etcd-druid has “observed” and consequently reconciled. It helps identify whether a change in the Etcd resource spec was acted upon by druid or not.

Status fields of the Etcd resource which correspond to the StatefulSet like CurrentReplicas, ReadyReplicas and Replicas are updated to reflect those of the StatefulSet by the controller.

Status fields related to the etcd cluster itself, such as Members, PeerUrlTLSEnabled and Ready are updated as follows:

• Cluster Membership: The controller updates the information about etcd cluster membership like Role, Status, Reason, LastTransitionTime and identifying information like the Name and ID. For the Status field, the member is checked for the Ready condition, where the member can be in Ready, NotReady and Unknown statuses.

Etcd resource conditions are indicated by status field Conditions. The condition checks that are currently performed are:

• AllMembersReady: indicates readiness of all members of the etcd cluster.
• Ready: indicates overall readiness of the etcd cluster in serving traffic.
• BackupReady: indicates health of the etcd backups, i.e., whether etcd backups are being taken regularly as per schedule. This condition is applicable only when backups are enabled for the etcd cluster.
• DataVolumesReady: indicates health of the persistent volumes containing the etcd data.

Compaction Controller

The compaction controller deploys the snapshot compaction job whenever required. To understand the rationale behind this controller, please read snapshot-compaction.md. The controller watches the number of events accumulated as part of delta snapshots in the etcd cluster’s backups, and triggers a snapshot compaction when the number of delta events crosses the set threshold, which is configurable through the --etcd-events-threshold CLI flag (1M events by default).

The controller watches for changes in snapshot Leases associated with Etcd resources. It checks the full and delta snapshot Leases and calculates the difference in events between the latest delta snapshot and the previous full snapshot, and initiates the compaction job if the event threshold is crossed.

The number of worker threads for the compaction controller needs to be greater than or equal to 0 (default 3), controlled by the CLI flag --compaction-workers. This is unlike other controllers which need at least one worker thread for the proper functioning of etcd-druid as snapshot compaction is not a core functionality for the etcd clusters to be deployed. The compaction controller should be explicitly enabled by the user, through the --enable-backup-compaction CLI flag.

EtcdCopyBackupsTask Controller

The etcdcopybackupstask controller is responsible for deploying the etcdbrctl copy command as a job. This controller reacts to create/update events arising from EtcdCopyBackupsTask resources, and deploys the EtcdCopyBackupsTask job with source and target backup storage providers as arguments, which are derived from source and target bucket secrets referenced by the EtcdCopyBackupsTask resource.

The number of worker threads for the etcdcopybackupstask controller needs to be greater than or equal to 0 (default being 3), controlled by the CLI flag --etcd-copy-backups-task-workers. This is unlike other controllers who need at least one worker thread for the proper functioning of etcd-druid as EtcdCopyBackupsTask is not a core functionality for the etcd clusters to be deployed.

Secret Controller

The secret controller’s primary responsibility is to add a finalizer on Secrets referenced by the Etcd resource. The secret controller is registered for Secrets, and the controller keeps a watch on the Etcd CR. This finalizer is added to ensure that Secrets which are referenced by the Etcd CR aren’t deleted while still being used by the Etcd resource.

Events arising from the Etcd resource are mapped to a list of Secrets such as backup and TLS secrets that are referenced by the Etcd resource, and are enqueued into the request queue, which the reconciler then acts on.

The number of worker threads for the secret controller must be at least 1 (default being 10) for this core controller, controlled by the CLI flag --secret-workers, since the referenced TLS and infrastructure access secrets are essential to the proper functioning of the etcd cluster.

10 - DEP Title

DEP-NN: Your short, descriptive title

Summary

Motivation

Goals

Non-Goals

Proposal

Alternatives

11 - Dependency Management

Dependency Management

We use Go Modules for dependency management. In order to add a new package dependency to the project, you can perform go get <package@version> or edit the go.mod file and append the package along with the version you want to use.

Organize Dependencies

Unfortunately go does not differentiate between dev and test dependencies. It becomes cleaner to organize dev and test dependencies in their respective require clause which gives a clear view on existing set of dependencies. The goal is to keep the dependencies to a minimum and only add a dependency when absolutely required.

Updating Dependencies

The Makefile contains a rule called tidy which performs go mod tidy which ensures that the go.mod file matches the source code in the module. It adds any missing module requirements necessary to build the current module’s packages and dependencies, and it removes requirements on modules that don’t provide any relevant packages. It also adds any missing entries to go.sum and removes unnecessary entries.

make tidy

!!! warning Make sure that you test the code after you have updated the dependencies!

12 - Etcd Cluster Components

Etcd Cluster Components

For every Etcd cluster that is provisioned by etcd-druid it deploys a set of resources. Following sections provides information and code reference to each such resource.

StatefulSet

StatefulSet is the primary kubernetes resource that gets provisioned for an etcd cluster.

• Replicas for the StatefulSet are derived from Etcd.Spec.Replicas in the custom resource.

• Each pod comprises of two containers:

  • etcd-wrapper : This is the main container which runs an etcd process.

  • etcd-backup-restore : This is a side-container which does the following:

    • Orchestrates the initialization of etcd. This includes validation of any existing etcd data directory, restoration in case of corrupt etcd data directory files for a single-member etcd cluster.
    • Periodically renewes member lease.
    • Optionally takes schedule and thresold based delta and full snapshots and pushes them to a configured object store.
    • Orchestrates scheduled etcd-db defragmentation.

    NOTE: This is not a complete list of functionalities offered out of etcd-backup-restore.

Code reference: StatefulSet-Component

For detailed information on each container you can visit etcd-wrapper and etcd-backup-restore respositories.

ConfigMap

Every etcd member requires configuration with which it must be started. etcd-druid creates a ConfigMap which gets mounted onto the etcd-backup-restore container. etcd-backup-restore container will modify the etcd configuration and serve it to the etcd-wrapper container upon request.

Code reference: ConfigMap-Component

PodDisruptionBudget

An etcd cluster requires quorum for all write operations. Clients can additionally configure quorum based reads as well to ensure linearizable reads (kube-apiserver’s etcd client is configured for linearizable reads and writes). In a cluster of size 3, only 1 member failure is tolerated. Failure tolerance for an etcd cluster with replicas n is computed as (n-1)/2.

To ensure that etcd pods are not evicted more than its failure tolerance, etcd-druid creates a PodDisruptionBudget.

!!! note For a single node etcd cluster a PodDisruptionBudget will be created, however pdb.spec.minavailable is set to 0 effectively disabling it.

Code reference: PodDisruptionBudget-Component

ServiceAccount

etch-backup-restore container running as a side-car in every etcd-member, requires permissions to access resources like Lease, StatefulSet etc. A dedicated ServiceAccount is created per Etcd cluster for this purpose.

Code reference: ServiceAccount-Component

Role & RoleBinding

etch-backup-restore container running as a side-car in every etcd-member, requires permissions to access resources like Lease, StatefulSet etc. A dedicated Role and RoleBinding is created and linked to the ServiceAccount created per Etcd cluster.

Code reference: Role-Component & RoleBinding-Component

Client & Peer Service

To enable clients to connect to an etcd cluster a ClusterIP Client Service is created. To enable etcd members to talk to each other(for discovery, leader-election, raft consensus etc.) etcd-druid also creates a Headless Service.

Code reference: Client-Service-Component & Peer-Service-Component

Member Lease

Every member in an Etcd cluster has a dedicated Lease that gets created which signifies that the member is alive. It is the responsibility of the etcd-backup-store side-car container to periodically renew the lease.

!!! note Today the lease object is also used to indicate the member-ID and the role of the member in an etcd cluster. Possible roles are Leader, Member(which denotes that this is a member but not a leader). This will change in the future with EtcdMember resource.

Code reference: Member-Lease-Component

Delta & Full Snapshot Leases

One of the responsibilities of etcd-backup-restore container is to take periodic or threshold based snapshots (delta and full) of the etcd DB. Today etcd-backup-restore communicates the end-revision of the latest full/delta snapshots to etcd-druid operator via leases.

etcd-druid creates two Lease resources one for delta and another for full snapshot. This information is used by the operator to trigger snapshot-compaction jobs. Snapshot leases are also used to derive the health of backups which gets updated in the Status subresource of every Etcd resource.

In future these leases will be replaced by EtcdMember resource.

Code reference: Snapshot-Lease-Component

13 - Etcd Cluster Resource Protection

Etcd Cluster Resource Protection

etcd-druid provisions and manages kubernetes resources (a.k.a components) for each Etcd cluster. To ensure that each component’s specification is in line with the configured attributes defined in Etcd custom resource and to protect unintended changes done to any of these managed components a Validating Webhook is employed.

Etcd Components Webhook is the validating webhook which prevents unintended UPDATE and DELETE operations on all managed resources. Following sections describe what is prohibited and in which specific conditions the changes are permitted.

Configure Etcd Components Webhook

Prerequisite to enable the validation webhook is to configure the Webhook Server. Additionally you need to enable the Etcd Components validating webhook and optionally configure other options. You can look at all the options here.

What is allowed?

Modifications to managed resources under the following circumstances will be allowed:

• Create and Connect operations are allowed and no validation is done.
• Changes to a kubernetes resource (e.g. StatefulSet, ConfigMap etc) not managed by etcd-druid are allowed.
• Changes to a resource whose Group-Kind is amongst the resources managed by etcd-druid but does not have a parent Etcd resource are allowed.
• It is possible that an operator wishes to explicitly disable etcd-component protection. This can be done by setting druid.gardener.cloud/disable-etcd-component-protection annotation on an Etcd resource. If this annotation is present then changes to managed components will be allowed.
• If Etcd resource has a deletion timestamp set indicating that it is marked for deletion and is awaiting etcd-druid to delete all managed resources then deletion requests for all managed resources for this etcd cluster will be allowed if:
  • The deletion request has come from a ServiceAccount associated to etcd-druid. If not explicitly specified via --reconciler-service-account then a default-reconciler-service-account will be assumed.
  • The deletion request has come from a ServiceAccount configured via --etcd-components-webhook-exempt-service-accounts.
• Lease objects are periodically updated by each etcd member pod. A single ServiceAccount is created for all members. Update operation on Lease objects from this ServiceAccount is allowed.
• If an active reconciliation is in-progress then only allow operations that are initiated by etcd-druid.
• If no active reconciliation is currently in-progress, then allow updates to managed resource from ServiceAccounts configured via --etcd-components-webhook-exempt-service-accounts.

14 - Etcd Druid Api

API Reference

Packages

• druid.gardener.cloud/v1alpha1

druid.gardener.cloud/v1alpha1

Package v1alpha1 contains API Schema definitions for the druid v1alpha1 API group

Resource Types

• Etcd
• EtcdCopyBackupsTask

BackupSpec

BackupSpec defines parameters associated with the full and delta snapshots of etcd.

Appears in:

• EtcdSpec

ClientService

ClientService defines the parameters of the client service that a user can specify

Appears in:

• EtcdConfig

CompactionMode

Underlying type: string

CompactionMode defines the auto-compaction-mode: ‘periodic’ or ‘revision’. ‘periodic’ for duration based retention and ‘revision’ for revision number based retention.

Validation:

• Enum: [periodic revision]

Appears in:

• SharedConfig

CompressionPolicy

Underlying type: string

CompressionPolicy defines the type of policy for compression of snapshots.

Validation:

• Enum: [gzip lzw zlib]

Appears in:

• CompressionSpec

CompressionSpec

CompressionSpec defines parameters related to compression of Snapshots(full as well as delta).

Appears in:

• BackupSpec

Condition

Condition holds the information about the state of a resource.

Appears in:

• EtcdCopyBackupsTaskStatus
• EtcdStatus

ConditionStatus

Underlying type: string

ConditionStatus is the status of a condition.

Appears in:

• Condition

ConditionType

Underlying type: string

ConditionType is the type of condition.

Appears in:

• Condition

CrossVersionObjectReference

CrossVersionObjectReference contains enough information to let you identify the referred resource.

Appears in:

• EtcdStatus

ErrorCode

Underlying type: string

ErrorCode is a string alias representing an error code that identifies an error.

Appears in:

• LastError

Etcd

Etcd is the Schema for the etcds API

EtcdConfig

EtcdConfig defines the configuration for the etcd cluster to be deployed.

Appears in:

• EtcdSpec

EtcdCopyBackupsTask

EtcdCopyBackupsTask is a task for copying etcd backups from a source to a target store.

EtcdCopyBackupsTaskSpec

EtcdCopyBackupsTaskSpec defines the parameters for the copy backups task.

Appears in:

• EtcdCopyBackupsTask

EtcdCopyBackupsTaskStatus

EtcdCopyBackupsTaskStatus defines the observed state of the copy backups task.

Appears in:

• EtcdCopyBackupsTask

EtcdMemberConditionStatus

Underlying type: string

EtcdMemberConditionStatus is the status of an etcd cluster member.

Appears in:

• EtcdMemberStatus

EtcdMemberStatus

EtcdMemberStatus holds information about etcd cluster membership.

Appears in:

• EtcdStatus

EtcdRole

Underlying type: string

EtcdRole is the role of an etcd cluster member.

Appears in:

• EtcdMemberStatus

EtcdSpec

EtcdSpec defines the desired state of Etcd

Appears in:

• Etcd

EtcdStatus

EtcdStatus defines the observed state of Etcd.

Appears in:

• Etcd

GarbageCollectionPolicy

Underlying type: string

GarbageCollectionPolicy defines the type of policy for snapshot garbage collection.

Validation:

• Enum: [Exponential LimitBased]

Appears in:

• BackupSpec

LastError

LastError stores details of the most recent error encountered for a resource.

Appears in:

• EtcdStatus

LastOperation

LastOperation holds the information on the last operation done on the Etcd resource.

Appears in:

• EtcdStatus

LastOperationState

Underlying type: string

LastOperationState is a string alias representing the state of the last operation.

Appears in:

• LastOperation

LastOperationType

Underlying type: string

LastOperationType is a string alias representing type of the last operation.

Appears in:

• LastOperation

LeaderElectionSpec

LeaderElectionSpec defines parameters related to the LeaderElection configuration.

Appears in:

• BackupSpec

MetricsLevel

Underlying type: string

MetricsLevel defines the level ‘basic’ or ’extensive’.

Validation:

• Enum: [basic extensive]

Appears in:

• EtcdConfig

SchedulingConstraints

SchedulingConstraints defines the different scheduling constraints that must be applied to the pod spec in the etcd statefulset. Currently supported constraints are Affinity and TopologySpreadConstraints.

Appears in:

• EtcdSpec

SecretReference

SecretReference defines a reference to a secret.

Appears in:

• TLSConfig

SharedConfig

SharedConfig defines parameters shared and used by Etcd as well as backup-restore sidecar.

Appears in:

• EtcdSpec

StorageProvider

Underlying type: string

StorageProvider defines the type of object store provider for storing backups.

Appears in:

• StoreSpec

StoreSpec

StoreSpec defines parameters related to ObjectStore persisting backups

Appears in:

• BackupSpec
• EtcdCopyBackupsTaskSpec

TLSConfig

TLSConfig hold the TLS configuration details.

Appears in:

• BackupSpec
• EtcdConfig

WaitForFinalSnapshotSpec

WaitForFinalSnapshotSpec defines the parameters for waiting for a final full snapshot before copying backups.

Appears in:

• EtcdCopyBackupsTaskSpec

15 - etcd Network Latency

Network Latency analysis: sn-etcd-sz vs mn-etcd-sz vs mn-etcd-mz

This page captures the etcd cluster latency analysis for below scenarios using the benchmark tool (build from etcd benchmark tool).

sn-etcd-sz -> single-node etcd single zone (Only single replica of etcd will be running)

mn-etcd-sz -> multi-node etcd single zone (Multiple replicas of etcd pods will be running across nodes in a single zone)

mn-etcd-mz -> multi-node etcd multi zone (Multiple replicas of etcd pods will be running across nodes in multiple zones)

PUT Analysis

Summary

• sn-etcd-sz latency is ~20% less than mn-etcd-sz when benchmark tool with single client.
• mn-etcd-sz latency is less than mn-etcd-mz but the difference is ~+/-5%.
• Compared to mn-etcd-sz, sn-etcd-sz latency is higher and gradually grows with more clients and larger value size.
• Compared to mn-etcd-mz, mn-etcd-sz latency is higher and gradually grows with more clients and larger value size.
• Compared to follower, leader latency is less, when benchmark tool with single client for all cases.
• Compared to follower, leader latency is high, when benchmark tool with multiple clients for all cases.

Sample commands:

# write to leader
benchmark put --target-leader --conns=1 --clients=1 --precise \
    --sequential-keys --key-starts 0 --val-size=256 --total=10000 \
    --endpoints=$ETCD_HOST 


# write to follower
benchmark put  --conns=1 --clients=1 --precise \
    --sequential-keys --key-starts 0 --val-size=256 --total=10000 \
    --endpoints=$ETCD_FOLLOWER_HOST

Latency analysis during PUT requests to etcd

• In this case benchmark tool tries to put key with random 256 bytes value.

  • Benchmark tool loads key/value to leader with single client .

    • sn-etcd-sz latency (~0.815ms) is ~50% lesser than mn-etcd-sz (~1.74ms ).
    • • mn-etcd-sz latency (~1.74ms ) is slightly lesser than mn-etcd-mz (~1.8ms) but the difference is negligible (within same ms).

    • Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      10000	256	1	1	leader	1220.0520	0.815ms	eu-west-1c	etcd-main-0	sn-etcd-sz
      10000	256	1	1	leader	586.545	1.74ms	eu-west-1a	etcd-main-1	mn-etcd-sz
      10000	256	1	1	leader	554.0155654442634	1.8ms	eu-west-1a	etcd-main-1	mn-etcd-mz

  • Benchmark tool loads key/value to follower with single client.

    • mn-etcd-sz latency(~2.2ms) is 20% to 30% lesser than mn-etcd-mz(~2.7ms).
    • Compare to follower, leader has lower latency.

    • Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      10000	256	1	1	follower-1	445.743	2.23ms	eu-west-1a	etcd-main-0	mn-etcd-sz
      10000	256	1	1	follower-1	378.9366747610789	2.63ms	eu-west-1c	etcd-main-0	mn-etcd-mz

      Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      10000	256	1	1	follower-2	457.967	2.17ms	eu-west-1a	etcd-main-2	mn-etcd-sz
      10000	256	1	1	follower-2	345.6586129825796	2.89ms	eu-west-1b	etcd-main-2	mn-etcd-mz

  • Benchmark tool loads key/value to leader with multiple clients.

    • sn-etcd-sz latency(~78.3ms) is ~10% greater than mn-etcd-sz(~71.81ms).
    • mn-etcd-sz latency(~71.81ms) is less than mn-etcd-mz(~72.5ms) but the difference is negligible.

    • Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      100000	256	100	1000	leader	12638.905	78.32ms	eu-west-1c	etcd-main-0	sn-etcd-sz
      100000	256	100	1000	leader	13789.248	71.81ms	eu-west-1a	etcd-main-1	mn-etcd-sz
      100000	256	100	1000	leader	13728.446436395223	72.5ms	eu-west-1a	etcd-main-1	mn-etcd-mz

  • Benchmark tool loads key/value to follower with multiple clients.

    • mn-etcd-sz latency(~69.8ms) is ~5% greater than mn-etcd-mz(~72.6ms).
    • Compare to leader, follower has lower latency.

    • Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      100000	256	100	1000	follower-1	14271.983	69.80ms	eu-west-1a	etcd-main-0	mn-etcd-sz
      100000	256	100	1000	follower-1	13695.98	72.62ms	eu-west-1a	etcd-main-1	mn-etcd-mz

      Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      100000	256	100	1000	follower-2	14325.436	69.47ms	eu-west-1a	etcd-main-2	mn-etcd-sz
      100000	256	100	1000	follower-2	15750.409490407475	63.3ms	eu-west-1b	etcd-main-2	mn-etcd-mz

• In this case benchmark tool tries to put key with random 1 MB value.

  • Benchmark tool loads key/value to leader with single client.

    • sn-etcd-sz latency(~16.35ms) is ~20% lesser than mn-etcd-sz(~20.64ms).
    • mn-etcd-sz latency(~20.64ms) is less than mn-etcd-mz(~21.08ms) but the difference is negligible..

    • Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	1	1	leader	61.117	16.35ms	eu-west-1c	etcd-main-0	sn-etcd-sz
      1000	1000000	1	1	leader	48.416	20.64ms	eu-west-1a	etcd-main-1	mn-etcd-sz
      1000	1000000	1	1	leader	45.7517341664802	21.08ms	eu-west-1a	etcd-main-1	mn-etcd-mz

  • Benchmark tool loads key/value withto follower single client.

    • mn-etcd-sz latency(~23.10ms) is ~10% greater than mn-etcd-mz(~21.8ms).
    • Compare to follower, leader has lower latency.

    • Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	1	1	follower-1	43.261	23.10ms	eu-west-1a	etcd-main-0	mn-etcd-sz
      1000	1000000	1	1	follower-1	45.7517341664802	21.8ms	eu-west-1c	etcd-main-0	mn-etcd-mz
      1000	1000000	1	1	follower-1	45.33	22.05ms	eu-west-1c	etcd-main-0	mn-etcd-mz

      Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	1	1	follower-2	40.0518	24.95ms	eu-west-1a	etcd-main-2	mn-etcd-sz
      1000	1000000	1	1	follower-2	43.28573155709838	23.09ms	eu-west-1b	etcd-main-2	mn-etcd-mz
      1000	1000000	1	1	follower-2	45.92	21.76ms	eu-west-1a	etcd-main-1	mn-etcd-mz
      1000	1000000	1	1	follower-2	35.5705	28.1ms	eu-west-1b	etcd-main-2	mn-etcd-mz

  • Benchmark tool loads key/value to leader with multiple clients.

    • sn-etcd-sz latency(~6.0375secs) is ~30% greater than mn-etcd-sz``~4.000secs).
    • mn-etcd-sz latency(~4.000secs) is less than mn-etcd-mz(~ 4.09secs) but the difference is negligible.

    • Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	100	300	leader	55.373	6.0375secs	eu-west-1c	etcd-main-0	sn-etcd-sz
      1000	1000000	100	300	leader	67.319	4.000secs	eu-west-1a	etcd-main-1	mn-etcd-sz
      1000	1000000	100	300	leader	65.91914167957594	4.09secs	eu-west-1a	etcd-main-1	mn-etcd-mz

  • Benchmark tool loads key/value to follower with multiple clients.

    • mn-etcd-sz latency(~4.04secs) is ~5% greater than mn-etcd-mz(~ 3.90secs).
    • Compare to leader, follower has lower latency.

    • Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	100	300	follower-1	66.528	4.0417secs	eu-west-1a	etcd-main-0	mn-etcd-sz
      1000	1000000	100	300	follower-1	70.6493461856332	3.90secs	eu-west-1c	etcd-main-0	mn-etcd-mz
      1000	1000000	100	300	follower-1	71.95	3.84secs	eu-west-1c	etcd-main-0	mn-etcd-mz

      Number of keys	Value size	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	100	300	follower-2	66.447	4.0164secs	eu-west-1a	etcd-main-2	mn-etcd-sz
      1000	1000000	100	300	follower-2	67.53038086369484	3.87secs	eu-west-1b	etcd-main-2	mn-etcd-mz
      1000	1000000	100	300	follower-2	68.46	3.92secs	eu-west-1a	etcd-main-1	mn-etcd-mz

Range Analysis

Sample commands are:

# Single connection read request with sequential keys
benchmark range 0 --target-leader --conns=1 --clients=1 --precise \
    --sequential-keys --key-starts 0  --total=10000 \
    --consistency=l \
    --endpoints=$ETCD_HOST 
# --consistency=s [Serializable]
benchmark range 0 --target-leader --conns=1 --clients=1 --precise \
    --sequential-keys --key-starts 0  --total=10000 \
    --consistency=s \
    --endpoints=$ETCD_HOST 
# Each read request with range query matches key 0 9999 and repeats for total number of requests.  
benchmark range 0 9999 --target-leader --conns=1 --clients=1 --precise \
    --total=10 \
    --consistency=s \
    --endpoints=https://etcd-main-client:2379
# Read requests with multiple connections
benchmark range 0 --target-leader --conns=100 --clients=1000 --precise \
    --sequential-keys --key-starts 0  --total=100000 \
    --consistency=l \
    --endpoints=$ETCD_HOST 
benchmark range 0 --target-leader --conns=100 --clients=1000 --precise \
    --sequential-keys --key-starts 0  --total=100000 \
    --consistency=s \
    --endpoints=$ETCD_HOST 

Latency analysis during Range requests to etcd

• In this case benchmark tool tries to get specific key with random 256 bytes value.

  • Benchmark tool range requests to leader with single client.

    • sn-etcd-sz latency(~1.24ms) is ~40% greater than mn-etcd-sz(~0.67ms).

    • mn-etcd-sz latency(~0.67ms) is ~20% lesser than mn-etcd-mz(~0.85ms).

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      10000	256	1	1	true	l	leader	800.272	1.24ms	eu-west-1c	etcd-main-0	sn-etcd-sz
      10000	256	1	1	true	l	leader	1173.9081	0.67ms	eu-west-1a	etcd-main-1	mn-etcd-sz
      10000	256	1	1	true	l	leader	999.3020189178693	0.85ms	eu-west-1a	etcd-main-1	mn-etcd-mz

    • Compare to consistency Linearizable, Serializable is ~40% less for all cases

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      10000	256	1	1	true	s	leader	1411.229	0.70ms	eu-west-1c	etcd-main-0	sn-etcd-sz
      10000	256	1	1	true	s	leader	2033.131	0.35ms	eu-west-1a	etcd-main-1	mn-etcd-sz
      10000	256	1	1	true	s	leader	2100.2426362012025	0.47ms	eu-west-1a	etcd-main-1	mn-etcd-mz

  • Benchmark tool range requests to follower with single client .

    • mn-etcd-sz latency(~1.3ms) is ~20% lesser than mn-etcd-mz(~1.6ms).
    • Compare to follower, leader read request latency is ~50% less for both mn-etcd-sz, mn-etcd-mz

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      10000	256	1	1	true	l	follower-1	765.325	1.3ms	eu-west-1a	etcd-main-0	mn-etcd-sz
      10000	256	1	1	true	l	follower-1	596.1	1.6ms	eu-west-1c	etcd-main-0	mn-etcd-mz

    • Compare to consistency Linearizable, Serializable is ~50% less for all cases

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      10000	256	1	1	true	s	follower-1	1823.631	0.54ms	eu-west-1a	etcd-main-0	mn-etcd-sz
      10000	256	1	1	true	s	follower-1	1442.6	0.69ms	eu-west-1c	etcd-main-0	mn-etcd-mz
      10000	256	1	1	true	s	follower-1	1416.39	0.70ms	eu-west-1c	etcd-main-0	mn-etcd-mz
      10000	256	1	1	true	s	follower-1	2077.449	0.47ms	eu-west-1a	etcd-main-1	mn-etcd-mz

  • Benchmark tool range requests to leader with multiple client.

    • sn-etcd-sz latency(~84.66ms) is ~20% greater than mn-etcd-sz(~73.95ms).

    • mn-etcd-sz latency(~73.95ms) is more or less equal to mn-etcd-mz(~ 73.8ms).

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      100000	256	100	1000	true	l	leader	11775.721	84.66ms	eu-west-1c	etcd-main-0	sn-etcd-sz
      100000	256	100	1000	true	l	leader	13446.9598	73.95ms	eu-west-1a	etcd-main-1	mn-etcd-sz
      100000	256	100	1000	true	l	leader	13527.19810605353	73.8ms	eu-west-1a	etcd-main-1	mn-etcd-mz

    • Compare to consistency Linearizable, Serializable is ~20% lesser for all cases

    • sn-etcd-sz latency(~69.37ms) is more or less equal to mn-etcd-sz(~69.89ms).

    • mn-etcd-sz latency(~69.89ms) is slightly higher than mn-etcd-mz(~67.63ms).

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      100000	256	100	1000	true	s	leader	14334.9027	69.37ms	eu-west-1c	etcd-main-0	sn-etcd-sz
      100000	256	100	1000	true	s	leader	14270.008	69.89ms	eu-west-1a	etcd-main-1	mn-etcd-sz
      100000	256	100	1000	true	s	leader	14715.287354023869	67.63ms	eu-west-1a	etcd-main-1	mn-etcd-mz

  • Benchmark tool range requests to follower with multiple client.

    • mn-etcd-sz latency(~60.69ms) is ~20% lesser than mn-etcd-mz(~70.76ms).

    • Compare to leader, follower has lower read request latency.

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      100000	256	100	1000	true	l	follower-1	11586.032	60.69ms	eu-west-1a	etcd-main-0	mn-etcd-sz
      100000	256	100	1000	true	l	follower-1	14050.5	70.76ms	eu-west-1c	etcd-main-0	mn-etcd-mz

    • mn-etcd-sz latency(~86.09ms) is ~20 higher than mn-etcd-mz(~64.6ms).

    • • Compare to mn-etcd-sz consistency Linearizable, Serializable is ~20% higher.*

    • Compare to mn-etcd-mz consistency Linearizable, Serializable is ~slightly less.

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      100000	256	100	1000	true	s	follower-1	11582.438	86.09ms	eu-west-1a	etcd-main-0	mn-etcd-sz
      100000	256	100	1000	true	s	follower-1	15422.2	64.6ms	eu-west-1c	etcd-main-0	mn-etcd-mz

  • Benchmark tool range requests to leader all keys.

    • sn-etcd-sz latency(~678.77ms) is ~5% slightly lesser than mn-etcd-sz(~697.29ms).

    • mn-etcd-sz latency(~697.29ms) is less than mn-etcd-mz(~701ms) but the difference is negligible.

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      20	256	2	5	false	l	leader	6.8875	678.77ms	eu-west-1c	etcd-main-0	sn-etcd-sz
      20	256	2	5	false	l	leader	6.720	697.29ms	eu-west-1a	etcd-main-1	mn-etcd-sz
      20	256	2	5	false	l	leader	6.7	701ms	eu-west-1a	etcd-main-1	mn-etcd-mz

    • • Compare to consistency Linearizable, Serializable is ~5% slightly higher for all cases

    • sn-etcd-sz latency(~687.36ms) is less than mn-etcd-sz(~692.68ms) but the difference is negligible.

    • mn-etcd-sz latency(~692.68ms) is ~5% slightly lesser than mn-etcd-mz(~735.7ms).

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      20	256	2	5	false	s	leader	6.76	687.36ms	eu-west-1c	etcd-main-0	sn-etcd-sz
      20	256	2	5	false	s	leader	6.635	692.68ms	eu-west-1a	etcd-main-1	mn-etcd-sz
      20	256	2	5	false	s	leader	6.3	735.7ms	eu-west-1a	etcd-main-1	mn-etcd-mz

  • Benchmark tool range requests to follower all keys

    • mn-etcd-sz(~737.68ms) latency is ~5% slightly higher than mn-etcd-mz(~713.7ms).

    • Compare to leader consistency Linearizableread request, follower is ~5% slightly higher.

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      20	256	2	5	false	l	follower-1	6.163	737.68ms	eu-west-1a	etcd-main-0	mn-etcd-sz
      20	256	2	5	false	l	follower-1	6.52	713.7ms	eu-west-1c	etcd-main-0	mn-etcd-mz

    • mn-etcd-sz latency(~757.73ms) is ~10% higher than mn-etcd-mz(~690.4ms).

    • Compare to follower consistency Linearizableread request, follower consistency Serializable is ~3% slightly higher for mn-etcd-sz.

    • Compare to follower consistency Linearizableread request, follower consistency Serializable is ~5% less for mn-etcd-mz.

    • *Compare to leader consistency Serializableread request, follower consistency Serializable is ~5% less for mn-etcd-mz. *

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      20	256	2	5	false	s	follower-1	6.0295	757.73ms	eu-west-1a	etcd-main-0	mn-etcd-sz
      20	256	2	5	false	s	follower-1	6.87	690.4ms	eu-west-1c	etcd-main-0	mn-etcd-mz

  ---

  
  
• In this case benchmark tool tries to get specific key with random `1MB` value.

  • Benchmark tool range requests to leader with single client.

    • sn-etcd-sz latency(~5.96ms) is ~5% lesser than mn-etcd-sz(~6.28ms).

    • mn-etcd-sz latency(~6.28ms) is ~10% higher than mn-etcd-mz(~5.3ms).

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	1	1	true	l	leader	167.381	5.96ms	eu-west-1c	etcd-main-0	sn-etcd-sz
      1000	1000000	1	1	true	l	leader	158.822	6.28ms	eu-west-1a	etcd-main-1	mn-etcd-sz
      1000	1000000	1	1	true	l	leader	187.94	5.3ms	eu-west-1a	etcd-main-1	mn-etcd-mz

    • Compare to consistency Linearizable, Serializable is ~15% less for sn-etcd-sz, mn-etcd-sz, mn-etcd-mz

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	1	1	true	s	leader	184.95	5.398ms	eu-west-1c	etcd-main-0	sn-etcd-sz
      1000	1000000	1	1	true	s	leader	176.901	5.64ms	eu-west-1a	etcd-main-1	mn-etcd-sz
      1000	1000000	1	1	true	s	leader	209.99	4.7ms	eu-west-1a	etcd-main-1	mn-etcd-mz

  • Benchmark tool range requests to follower with single client.

    • mn-etcd-sz latency(~6.66ms) is ~10% higher than mn-etcd-mz(~6.16ms).

    • Compare to leader, follower read request latency is ~10% high for mn-etcd-sz

    • Compare to leader, follower read request latency is ~20% high for mn-etcd-mz

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	1	1	true	l	follower-1	150.680	6.66ms	eu-west-1a	etcd-main-0	mn-etcd-sz
      1000	1000000	1	1	true	l	follower-1	162.072	6.16ms	eu-west-1c	etcd-main-0	mn-etcd-mz

    • Compare to consistency Linearizable, Serializable is ~15% less for mn-etcd-sz(~5.84ms), mn-etcd-mz(~5.01ms).

    • Compare to leader, follower read request latency is ~5% slightly high for mn-etcd-sz, mn-etcd-mz

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	1	1	true	s	follower-1	170.918	5.84ms	eu-west-1a	etcd-main-0	mn-etcd-sz
      1000	1000000	1	1	true	s	follower-1	199.01	5.01ms	eu-west-1c	etcd-main-0	mn-etcd-mz

  • Benchmark tool range requests to leader with multiple clients.

    • sn-etcd-sz latency(~1.593secs) is ~20% lesser than mn-etcd-sz(~1.974secs).

    • mn-etcd-sz latency(~1.974secs) is ~5% greater than mn-etcd-mz(~1.81secs).

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	100	500	true	l	leader	252.149	1.593secs	eu-west-1c	etcd-main-0	sn-etcd-sz
      1000	1000000	100	500	true	l	leader	205.589	1.974secs	eu-west-1a	etcd-main-1	mn-etcd-sz
      1000	1000000	100	500	true	l	leader	230.42	1.81secs	eu-west-1a	etcd-main-1	mn-etcd-mz

    • Compare to consistency Linearizable, Serializable is more or less same for sn-etcd-sz(~1.57961secs), mn-etcd-mz(~1.8secs) not a big difference

    • Compare to consistency Linearizable, Serializable is ~10% high for mn-etcd-sz(~ 2.277secs).

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	100	500	true	s	leader	252.406	1.57961secs	eu-west-1c	etcd-main-0	sn-etcd-sz
      1000	1000000	100	500	true	s	leader	181.905	2.277secs	eu-west-1a	etcd-main-1	mn-etcd-sz
      1000	1000000	100	500	true	s	leader	227.64	1.8secs	eu-west-1a	etcd-main-1	mn-etcd-mz

  • Benchmark tool range requests to follower with multiple client.

    • mn-etcd-sz latency is ~20% less than mn-etcd-mz.

    • Compare to leader consistency Linearizable, follower read request latency is ~15 less for mn-etcd-sz(~1.694secs).

    • Compare to leader consistency Linearizable, follower read request latency is ~10% higher for mn-etcd-sz(~1.977secs).

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	100	500	true	l	follower-1	248.489	1.694secs	eu-west-1a	etcd-main-0	mn-etcd-sz
      1000	1000000	100	500	true	l	follower-1	210.22	1.977secs	eu-west-1c	etcd-main-0	mn-etcd-mz

      Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	100	500	true	l	follower-2	205.765	1.967secs	eu-west-1a	etcd-main-2	mn-etcd-sz
      1000	1000000	100	500	true	l	follower-2	195.2	2.159secs	eu-west-1b	etcd-main-2	mn-etcd-mz

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	100	500	true	s	follower-1	231.458	1.7413secs	eu-west-1a	etcd-main-0	mn-etcd-sz
      1000	1000000	100	500	true	s	follower-1	214.80	1.907secs	eu-west-1c	etcd-main-0	mn-etcd-mz

      Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      1000	1000000	100	500	true	s	follower-2	183.320	2.2810secs	eu-west-1a	etcd-main-2	mn-etcd-sz
      1000	1000000	100	500	true	s	follower-2	195.40	2.164secs	eu-west-1b	etcd-main-2	mn-etcd-mz

  • Benchmark tool range requests to leader all keys.

    • sn-etcd-sz latency(~8.993secs) is ~3% slightly lower than mn-etcd-sz(~9.236secs).

    • mn-etcd-sz latency(~9.236secs) is ~2% slightly lower than mn-etcd-mz(~9.100secs).

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      20	1000000	2	5	false	l	leader	0.5139	8.993secs	eu-west-1c	etcd-main-0	sn-etcd-sz
      20	1000000	2	5	false	l	leader	0.506	9.236secs	eu-west-1a	etcd-main-1	mn-etcd-sz
      20	1000000	2	5	false	l	leader	0.508	9.100secs	eu-west-1a	etcd-main-1	mn-etcd-mz

    • Compare to consistency Linearizableread request, follower for sn-etcd-sz(~9.secs) is a slight difference 10ms.

    • Compare to consistency Linearizableread request, follower for mn-etcd-sz(~9.113secs) is ~1% less, not a big difference.

    • Compare to consistency Linearizableread request, follower for mn-etcd-mz(~8.799secs) is ~3% less, not a big difference.

    • sn-etcd-sz latency(~9.secs) is ~1% slightly less than mn-etcd-sz(~9.113secs).

    • mn-etcd-sz latency(~9.113secs) is ~3% slightly higher than mn-etcd-mz(~8.799secs).

      Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      20	1000000	2	5	false	s	leader	0.51125	9.0003secs	eu-west-1c	etcd-main-0	sn-etcd-sz
      20	1000000	2	5	false	s	leader	0.4993	9.113secs	eu-west-1a	etcd-main-1	mn-etcd-sz
      20	1000000	2	5	false	s	leader	0.522	8.799secs	eu-west-1a	etcd-main-1	mn-etcd-mz

  • Benchmark tool range requests to follower all keys

    • mn-etcd-sz latency(~9.065secs) is ~1% slightly higher than mn-etcd-mz(~9.007secs).

    • Compare to leader consistency Linearizableread request, follower is ~1% slightly higher for both cases mn-etcd-sz, mn-etcd-mz .

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      20	1000000	2	5	false	l	follower-1	0.512	9.065secs	eu-west-1a	etcd-main-0	mn-etcd-sz
      20	1000000	2	5	false	l	follower-1	0.533	9.007secs	eu-west-1c	etcd-main-0	mn-etcd-mz

    • Compare to consistency Linearizableread request, follower for mn-etcd-sz(~9.553secs) is ~5% high.

    • Compare to consistency Linearizableread request, follower for mn-etcd-mz(~7.7433secs) is ~15% less.

    • mn-etcd-sz(~9.553secs) latency is ~20% higher than mn-etcd-mz(~7.7433secs).

    • Number of requests	Value size	Number of connections	Number of clients	sequential-keys	Consistency	Target etcd server	Average write QPS	Average latency per request	zone	server name	Test name
      20	1000000	2	5	false	s	follower-1	0.4743	9.553secs	eu-west-1a	etcd-main-0	mn-etcd-sz
      20	1000000	2	5	false	s	follower-1	0.5500	7.7433secs	eu-west-1c	etcd-main-0	mn-etcd-mz

  ---