3 minute read
etcd - Key-Value Store for Kubernetes
etcd is a strongly consistent key-value store and the most prevalent choice for the Kubernetes
persistence layer. All API cluster objects like
Secrets, etc. are stored in
makes it an essential part of a Kubernetes control plane.
Shoot cluster persistence
Each shoot cluster gets its very own persistence for the control plane. It runs in the shoot namespace on the respective
seed cluster. Concretely, there are two etcd instances per shoot cluster which the
Kube-Apiserver is configured
to use in the following way:
A store that contains all “cluster critical” or “long-term” objects. These object kinds are typically considered for a backup to prevent any data loss.
A store that contains all
Event objects (
events.k8s.io) of a cluster.
Events have usually a short retention
period, occur frequently but are not essential for a disaster recovery.
The setup above prevents both, the critical
etcd-main is not flooded by Kubernetes
Events as well as backup space is
not occupied by non-critical data. This segmentation saves time and resources.
Configuring, maintaining and health-checking
etcd is outsourced to a dedicated operator called ETCD Druid.
When Gardenlet reconciles a
Shoot resource, it creates or updates an Etcd
resources in the seed cluster, containing necessary information (backup information, defragmentation schedule, resources, etc.)
needs to manage the lifecycle of the desired etcd instance (today
events). Likewise, when the shoot is deleted,
Gardenlet deletes the
Etcd resource and ETCD Druid takes care about cleaning up
all related objects, e.g. the backing
Gardenlet maintains HVPA
objects for etcd
StatefulSets if the corresponding feature gate is enabled. This enables
a vertical scaling for
etcd. Downscaling is handled more pessimistic to prevent many subsequent
etcd restarts. Thus,
infrastructure clusters downscaling is deactivated and for all other clusters lower advertised requests/limits are only
applied during a shoot’s maintenance time window.
Seeds specify backups for etcd (example),
then Gardener and the respective provider extensions are responsible for creating a bucket
on the cloud provider’s side (modelled through BackupBucket resource). The bucket stores
backups of shoots scheduled on that seed. Furthermore, Gardener creates a BackupEntry
which subdivides the bucket and thus makes it possible to store backups of multiple shoot clusters.
etcd-main instance itself is configured to run with a special backup-restore sidecar. It takes care about regularly
backing up etcd data and restoring it in case of data loss. More information can be found on the component’s GitHub
How long backups are stored in the bucket after a shoot has been deleted, depends on the configured retention period in the
Seed resource. Please see this example configuration for more information.
etcd maintenance tasks must be performed from time to time in order to re-gain database storage and to ensure the system’s reliability. The backup-restore sidecar takes care about this job as well. Gardener chooses a random time within the shoot’s maintenance time to schedule these tasks.
Was this page helpful?