This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Docs

1 - Gardener

The core component providing the extension API server of your Kubernetes cluster

Gardener

Gardener Logo

CI Build status Slack channel #gardener Go Report Card GoDoc CII Best Practices

Gardener implements the automated management and operation of Kubernetes clusters as a service and provides a fully validated extensibility framework that can be adjusted to any programmatic cloud or infrastructure provider.

Gardener is 100% Kubernetes-native and exposes its own Cluster API to create homogeneous clusters on all supported infrastructures. This API differs from SIG Cluster Lifecycle’s Cluster API that only harmonizes how to get to clusters, while Gardener’s Cluster API goes one step further and also harmonizes the make-up of the clusters themselves. That means, Gardener gives you homogeneous clusters with exactly the same bill of material, configuration and behavior on all supported infrastructures, which you can see further down below in the section on our K8s Conformance Test Coverage.

In 2020, SIG Cluster Lifecycle’s Cluster API made a huge step forward with v1alpha3 and the newly added support for declarative control plane management. This made it possible to integrate managed services like GKE or Gardener. We would be more than happy, if the community would be interested, to contribute a Gardener control plane provider. For more information on the relation between Gardener API and SIG Cluster Lifecycle’s Cluster API, please see here.

Gardener’s main principle is to leverage Kubernetes concepts for all of its tasks.

In essence, Gardener is an extension API server that comes along with a bundle of custom controllers. It introduces new API objects in an existing Kubernetes cluster (which is called garden cluster) in order to use them for the management of end-user Kubernetes clusters (which are called shoot clusters). These shoot clusters are described via declarative cluster specifications which are observed by the controllers. They will bring up the clusters, reconcile their state, perform automated updates and make sure they are always up and running.

To accomplish these tasks reliably and to offer a high quality of service, Gardener controls the main components of a Kubernetes cluster (etcd, API server, controller manager, scheduler). These so-called control plane components are hosted in Kubernetes clusters themselves (which are called seed clusters). This is the main difference compared to many other OSS cluster provisioning tools: The shoot clusters do not have dedicated master VMs. Instead, the control plane is deployed as a native Kubernetes workload into the seeds (the architecture is commonly referred to as kubeception or inception design). This does not only effectively reduce the total cost of ownership but also allows easier implementations for “day-2 operations” (like cluster updates or robustness) by relying on all the mature Kubernetes features and capabilities.

Gardener reuses the identical Kubernetes design to span a scalable multi-cloud and multi-cluster landscape. Such familiarity with known concepts has proven to quickly ease the initial learning curve and accelerate developer productivity:

  • Kubernetes API Server = Gardener API Server
  • Kubernetes Controller Manager = Gardener Controller Manager
  • Kubernetes Scheduler = Gardener Scheduler
  • Kubelet = Gardenlet
  • Node = Seed cluster
  • Pod = Shoot cluster

Please find more information regarding the concepts and a detailed description of the architecture in our Gardener Wiki and our blog posts on kubernetes.io: Gardener - the Kubernetes Botanist (17.5.2018) and Gardener Project Update (2.12.2019).


K8s Conformance Test Coverage certified kubernetes logo

Gardener takes part in the Certified Kubernetes Conformance Program to attest its compatibility with the K8s conformance testsuite. Currently Gardener is certified for K8s versions up to v1.24, see the conformance spreadsheet.

Continuous conformance test results of the latest stable Gardener release are uploaded regularly to the CNCF test grid:

Provider/K8sv1.25v1.24v1.23v1.22v1.21v1.20v1.19v1.18v1.17
AWSN/AGardener v1.24 Conformance TestsGardener v1.23 Conformance TestsGardener v1.22 Conformance TestsGardener v1.21 Conformance TestsGardener v1.20 Conformance TestsGardener v1.19 Conformance TestsGardener v1.18 Conformance TestsGardener v1.17 Conformance Tests
AzureN/AGardener v1.24 Conformance TestsGardener v1.23 Conformance TestsGardener v1.22 Conformance TestsGardener v1.21 Conformance TestsGardener v1.20 Conformance TestsGardener v1.19 Conformance TestsGardener v1.18 Conformance TestsGardener v1.17 Conformance Tests
GCPN/AGardener v1.24 Conformance TestsGardener v1.23 Conformance TestsGardener v1.22 Conformance TestsGardener v1.21 Conformance TestsGardener v1.20 Conformance TestsGardener v1.19 Conformance TestsGardener v1.18 Conformance TestsGardener v1.17 Conformance Tests
OpenStackN/AGardener v1.24 Conformance TestsGardener v1.23 Conformance TestsGardener v1.22 Conformance TestsGardener v1.21 Conformance TestsGardener v1.20 Conformance TestsGardener v1.19 Conformance TestsGardener v1.18 Conformance TestsGardener v1.17 Conformance Tests
AlicloudN/AGardener v1.24 Conformance TestsGardener v1.23 Conformance TestsGardener v1.22 Conformance TestsGardener v1.21 Conformance TestsGardener v1.20 Conformance TestsGardener v1.19 Conformance TestsGardener v1.18 Conformance TestsGardener v1.17 Conformance Tests
Equinix MetalN/AN/AN/AN/AN/AN/AN/AN/AN/A
vSphereN/AN/AN/AN/AN/AN/AN/AN/AN/A

Get an overview of the test results at testgrid.

Start using or developing the Gardener locally

See our documentation in the /docs repository, please find the index here.

Setting up your own Gardener landscape in the Cloud

The quickest way to test drive Gardener is to install it virtually onto an existing Kubernetes cluster, just like you would install any other Kubernetes-ready application. You can do this with our Gardener Helm Chart.

Alternatively you can use our garden setup project to create a fully configured Gardener landscape which also includes our Gardener Dashboard.

Feedback and Support

Feedback and contributions are always welcome!

All channels for getting in touch or learning about our project are listed under the community section. We are cordially inviting interested parties to join our bi-weekly meetings.

Please report bugs or suggestions about our Kubernetes clusters as such or the Gardener itself as GitHub issues or join our Slack channel #gardener (please invite yourself to the Kubernetes workspace here).

Learn More!

Please find further resources about our project here:

1.1.1 - Authentication

Packages:

authentication.gardener.cloud/v1alpha1

Package v1alpha1 is a version of the API.

Resource Types:

AdminKubeconfigRequest

AdminKubeconfigRequest can be used to request a kubeconfig with admin credentials for a Shoot cluster.

FieldDescription
apiVersion
string
authentication.gardener.cloud/v1alpha1
kind
string
AdminKubeconfigRequest
metadata
Kubernetes meta/v1.ObjectMeta

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
AdminKubeconfigRequestSpec

Spec is the specification of the AdminKubeconfigRequest.



expirationSeconds
int64
(Optional)

ExpirationSeconds is the requested validity duration of the credential. The credential issuer may return a credential with a different validity duration so a client needs to check the ‘expirationTimestamp’ field in a response. Defaults to 1 hour.

status
AdminKubeconfigRequestStatus

Status is the status of the AdminKubeconfigRequest.

AdminKubeconfigRequestSpec

(Appears on: AdminKubeconfigRequest)

AdminKubeconfigRequestSpec contains the expiration time of the kubeconfig.

FieldDescription
expirationSeconds
int64
(Optional)

ExpirationSeconds is the requested validity duration of the credential. The credential issuer may return a credential with a different validity duration so a client needs to check the ‘expirationTimestamp’ field in a response. Defaults to 1 hour.

AdminKubeconfigRequestStatus

(Appears on: AdminKubeconfigRequest)

AdminKubeconfigRequestStatus is the status of the AdminKubeconfigRequest containing the kubeconfig and expiration of the credential.

FieldDescription
kubeconfig
[]byte

Kubeconfig contains the kubeconfig with cluster-admin privileges for the shoot cluster.

expirationTimestamp
Kubernetes meta/v1.Time

ExpirationTimestamp is the expiration timestamp of the returned credential.


Generated with gen-crd-api-reference-docs

1.1.2 - Core

Packages:

core.gardener.cloud/v1beta1

Package v1beta1 is a version of the API.

Resource Types:

BackupBucket

BackupBucket holds details about backup bucket

FieldDescription
apiVersion
string
core.gardener.cloud/v1beta1
kind
string
BackupBucket
metadata
Kubernetes meta/v1.ObjectMeta

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
BackupBucketSpec

Specification of the Backup Bucket.



provider
BackupBucketProvider

Provider holds the details of cloud provider of the object store. This field is immutable.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig is the configuration passed to BackupBucket resource.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the credentials to access object store.

seedName
string
(Optional)

SeedName holds the name of the seed allocated to BackupBucket for running controller. This field is immutable.

status
BackupBucketStatus

Most recently observed status of the Backup Bucket.

BackupEntry

BackupEntry holds details about shoot backup.

FieldDescription
apiVersion
string
core.gardener.cloud/v1beta1
kind
string
BackupEntry
metadata
Kubernetes meta/v1.ObjectMeta

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
BackupEntrySpec
(Optional)

Spec contains the specification of the Backup Entry.



bucketName
string

BucketName is the name of backup bucket for this Backup Entry.

seedName
string
(Optional)

SeedName holds the name of the seed to which this BackupEntry is scheduled

status
BackupEntryStatus
(Optional)

Status contains the most recently observed status of the Backup Entry.

CloudProfile

CloudProfile represents certain properties about a provider environment.

FieldDescription
apiVersion
string
core.gardener.cloud/v1beta1
kind
string
CloudProfile
metadata
Kubernetes meta/v1.ObjectMeta
(Optional)

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
CloudProfileSpec
(Optional)

Spec defines the provider environment properties.



caBundle
string
(Optional)

CABundle is a certificate bundle which will be installed onto every host machine of shoot cluster targeting this profile.

kubernetes
KubernetesSettings

Kubernetes contains constraints regarding allowed values of the ‘kubernetes’ block in the Shoot specification.

machineImages
[]MachineImage

MachineImages contains constraints regarding allowed values for machine images in the Shoot specification.

machineTypes
[]MachineType

MachineTypes contains constraints regarding allowed values for machine types in the ‘workers’ block in the Shoot specification.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig contains provider-specific configuration for the profile.

regions
[]Region

Regions contains constraints regarding allowed values for regions and zones.

seedSelector
SeedSelector
(Optional)

SeedSelector contains an optional list of labels on Seed resources that marks those seeds whose shoots may use this provider profile. An empty list means that all seeds of the same provider type are supported. This is useful for environments that are of the same type (like openstack) but may have different “instances”/landscapes. Optionally a list of possible providers can be added to enable cross-provider scheduling. By default, the provider type of the seed must match the shoot’s provider.

type
string

Type is the name of the provider.

volumeTypes
[]VolumeType
(Optional)

VolumeTypes contains constraints regarding allowed values for volume types in the ‘workers’ block in the Shoot specification.

ControllerDeployment

ControllerDeployment contains information about how this controller is deployed.

FieldDescription
apiVersion
string
core.gardener.cloud/v1beta1
kind
string
ControllerDeployment
metadata
Kubernetes meta/v1.ObjectMeta

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
type
string

Type is the deployment type.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension

ProviderConfig contains type-specific configuration. It contains assets that deploy the controller.

ControllerInstallation

ControllerInstallation represents an installation request for an external controller.

FieldDescription
apiVersion
string
core.gardener.cloud/v1beta1
kind
string
ControllerInstallation
metadata
Kubernetes meta/v1.ObjectMeta

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ControllerInstallationSpec

Spec contains the specification of this installation. If the object’s deletion timestamp is set, this field is immutable.



registrationRef
Kubernetes core/v1.ObjectReference

RegistrationRef is used to reference a ControllerRegistration resource. The name field of the RegistrationRef is immutable.

seedRef
Kubernetes core/v1.ObjectReference

SeedRef is used to reference a Seed resource. The name field of the SeedRef is immutable.

deploymentRef
Kubernetes core/v1.ObjectReference
(Optional)

DeploymentRef is used to reference a ControllerDeployment resource.

status
ControllerInstallationStatus

Status contains the status of this installation.

ControllerRegistration

ControllerRegistration represents a registration of an external controller.

FieldDescription
apiVersion
string
core.gardener.cloud/v1beta1
kind
string
ControllerRegistration
metadata
Kubernetes meta/v1.ObjectMeta

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ControllerRegistrationSpec

Spec contains the specification of this registration. If the object’s deletion timestamp is set, this field is immutable.



resources
[]ControllerResource
(Optional)

Resources is a list of combinations of kinds (DNSProvider, Infrastructure, Generic, …) and their actual types (aws-route53, gcp, auditlog, …).

deployment
ControllerRegistrationDeployment
(Optional)

Deployment contains information for how this controller is deployed.

Project

Project holds certain properties about a Gardener project.

FieldDescription
apiVersion
string
core.gardener.cloud/v1beta1
kind
string
Project
metadata
Kubernetes meta/v1.ObjectMeta
(Optional)

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ProjectSpec
(Optional)

Spec defines the project properties.



createdBy
Kubernetes rbac/v1.Subject
(Optional)

CreatedBy is a subject representing a user name, an email address, or any other identifier of a user who created the project. This field is immutable.

description
string
(Optional)

Description is a human-readable description of what the project is used for.

owner
Kubernetes rbac/v1.Subject
(Optional)

Owner is a subject representing a user name, an email address, or any other identifier of a user owning the project. IMPORTANT: Be aware that this field will be removed in the v1 version of this API in favor of the owner role. The only way to change the owner will be by moving the owner role. In this API version the only way to change the owner is to use this field. TODO: Remove this field in favor of the owner role in v1.

purpose
string
(Optional)

Purpose is a human-readable explanation of the project’s purpose.

members
[]ProjectMember
(Optional)

Members is a list of subjects representing a user name, an email address, or any other identifier of a user, group, or service account that has a certain role.

namespace
string
(Optional)

Namespace is the name of the namespace that has been created for the Project object. A nil value means that Gardener will determine the name of the namespace. This field is immutable.

tolerations
ProjectTolerations
(Optional)

Tolerations contains the tolerations for taints on seed clusters.

status
ProjectStatus
(Optional)

Most recently observed status of the Project.

Quota

Quota represents a quota on resources consumed by shoot clusters either per project or per provider secret.

FieldDescription
apiVersion
string
core.gardener.cloud/v1beta1
kind
string
Quota
metadata
Kubernetes meta/v1.ObjectMeta
(Optional)

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
QuotaSpec
(Optional)

Spec defines the Quota constraints.



clusterLifetimeDays
int32
(Optional)

ClusterLifetimeDays is the lifetime of a Shoot cluster in days before it will be terminated automatically.

metrics
Kubernetes core/v1.ResourceList

Metrics is a list of resources which will be put under constraints.

scope
Kubernetes core/v1.ObjectReference

Scope is the scope of the Quota object, either ‘project’ or ‘secret’. This field is immutable.

SecretBinding

SecretBinding represents a binding to a secret in the same or another namespace.

FieldDescription
apiVersion
string
core.gardener.cloud/v1beta1
kind
string
SecretBinding
metadata
Kubernetes meta/v1.ObjectMeta
(Optional)

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret object in the same or another namespace. This field is immutable.

quotas
[]Kubernetes core/v1.ObjectReference
(Optional)

Quotas is a list of references to Quota objects in the same or another namespace. This field is immutable.

provider
SecretBindingProvider
(Optional)

Provider defines the provider type of the SecretBinding. This field is immutable.

Seed

Seed represents an installation request for an external controller.

FieldDescription
apiVersion
string
core.gardener.cloud/v1beta1
kind
string
Seed
metadata
Kubernetes meta/v1.ObjectMeta

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
SeedSpec

Spec contains the specification of this installation.



backup
SeedBackup
(Optional)

Backup holds the object store configuration for the backups of shoot (currently only etcd). If it is not specified, then there won’t be any backups taken for shoots associated with this seed. If backup field is present in seed, then backups of the etcd from shoot control plane will be stored under the configured object store.

dns
SeedDNS

DNS contains DNS-relevant information about this seed cluster.

networks
SeedNetworks

Networks defines the pod, service and worker network of the Seed cluster.

provider
SeedProvider

Provider defines the provider type and region for this Seed cluster.

secretRef
Kubernetes core/v1.SecretReference
(Optional)

SecretRef is a reference to a Secret object containing the Kubeconfig of the Kubernetes cluster to be registered as Seed.

taints
[]SeedTaint
(Optional)

Taints describes taints on the seed.

volume
SeedVolume
(Optional)

Volume contains settings for persistentvolumes created in the seed cluster.

settings
SeedSettings
(Optional)

Settings contains certain settings for this seed cluster.

ingress
Ingress
(Optional)

Ingress configures Ingress specific settings of the Seed cluster. This field is immutable.

status
SeedStatus

Status contains the status of this installation.

Shoot

Shoot represents a Shoot cluster created and managed by Gardener.

FieldDescription
apiVersion
string
core.gardener.cloud/v1beta1
kind
string
Shoot
metadata
Kubernetes meta/v1.ObjectMeta
(Optional)

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ShootSpec
(Optional)

Specification of the Shoot cluster. If the object’s deletion timestamp is set, this field is immutable.



addons
Addons
(Optional)

Addons contains information about enabled/disabled addons and their configuration.

cloudProfileName
string

CloudProfileName is a name of a CloudProfile object. This field is immutable.

dns
DNS
(Optional)

DNS contains information about the DNS settings of the Shoot.

extensions
[]Extension
(Optional)

Extensions contain type and provider information for Shoot extensions.

hibernation
Hibernation
(Optional)

Hibernation contains information whether the Shoot is suspended or not.

kubernetes
Kubernetes

Kubernetes contains the version and configuration settings of the control plane components.

networking
Networking

Networking contains information about cluster networking such as CNI Plugin type, CIDRs, …etc.

maintenance
Maintenance
(Optional)

Maintenance contains information about the time window for maintenance operations and which operations should be performed.

monitoring
Monitoring
(Optional)

Monitoring contains information about custom monitoring configurations for the shoot.

provider
Provider

Provider contains all provider-specific and provider-relevant information.

purpose
ShootPurpose
(Optional)

Purpose is the purpose class for this cluster.

region
string

Region is a name of a region. This field is immutable.

secretBindingName
string

SecretBindingName is the name of the a SecretBinding that has a reference to the provider secret. The credentials inside the provider secret will be used to create the shoot in the respective account. This field is immutable.

seedName
string
(Optional)

SeedName is the name of the seed cluster that runs the control plane of the Shoot. This field is immutable when the SeedChange feature gate is disabled.

seedSelector
SeedSelector
(Optional)

SeedSelector is an optional selector which must match a seed’s labels for the shoot to be scheduled on that seed.

resources
[]NamedResourceReference
(Optional)

Resources holds a list of named resource references that can be referred to in extension configs by their names.

tolerations
[]Toleration
(Optional)

Tolerations contains the tolerations for taints on seed clusters.

exposureClassName
string
(Optional)

ExposureClassName is the optional name of an exposure class to apply a control plane endpoint exposure strategy. This field is immutable.

systemComponents
SystemComponents
(Optional)

SystemComponents contains the settings of system components in the control or data plane of the Shoot cluster.

controlPlane
ControlPlane
(Optional)

ControlPlane contains general settings for the control plane of the shoot.

status
ShootStatus
(Optional)

Most recently observed status of the Shoot cluster.

Addon

(Appears on: KubernetesDashboard, NginxIngress)

Addon allows enabling or disabling a specific addon and is used to derive from.

FieldDescription
enabled
bool

Enabled indicates whether the addon is enabled or not.

Addons

(Appears on: ShootSpec)

Addons is a collection of configuration for specific addons which are managed by the Gardener.

FieldDescription
kubernetesDashboard
KubernetesDashboard
(Optional)

KubernetesDashboard holds configuration settings for the kubernetes dashboard addon.

nginxIngress
NginxIngress
(Optional)

NginxIngress holds configuration settings for the nginx-ingress addon.

AdmissionPlugin

(Appears on: KubeAPIServerConfig)

AdmissionPlugin contains information about a specific admission plugin and its corresponding configuration.

FieldDescription
name
string

Name is the name of the plugin.

config
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

Config is the configuration of the plugin.

disabled
bool
(Optional)

Disabled specifies whether this plugin should be disabled.

Alerting

(Appears on: Monitoring)

Alerting contains information about how alerting will be done (i.e. who will receive alerts and how).

FieldDescription
emailReceivers
[]string
(Optional)

MonitoringEmailReceivers is a list of recipients for alerts

AuditConfig

(Appears on: KubeAPIServerConfig)

AuditConfig contains settings for audit of the api server

FieldDescription
auditPolicy
AuditPolicy
(Optional)

AuditPolicy contains configuration settings for audit policy of the kube-apiserver.

AuditPolicy

(Appears on: AuditConfig)

AuditPolicy contains audit policy for kube-apiserver

FieldDescription
configMapRef
Kubernetes core/v1.ObjectReference
(Optional)

ConfigMapRef is a reference to a ConfigMap object in the same namespace, which contains the audit policy for the kube-apiserver.

AvailabilityZone

(Appears on: Region)

AvailabilityZone is an availability zone.

FieldDescription
name
string

Name is an an availability zone name.

unavailableMachineTypes
[]string
(Optional)

UnavailableMachineTypes is a list of machine type names that are not availability in this zone.

unavailableVolumeTypes
[]string
(Optional)

UnavailableVolumeTypes is a list of volume type names that are not availability in this zone.

BackupBucketProvider

(Appears on: BackupBucketSpec)

BackupBucketProvider holds the details of cloud provider of the object store.

FieldDescription
type
string

Type is the type of provider.

region
string

Region is the region of the bucket.

BackupBucketSpec

(Appears on: BackupBucket)

BackupBucketSpec is the specification of a Backup Bucket.

FieldDescription
provider
BackupBucketProvider

Provider holds the details of cloud provider of the object store. This field is immutable.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig is the configuration passed to BackupBucket resource.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the credentials to access object store.

seedName
string
(Optional)

SeedName holds the name of the seed allocated to BackupBucket for running controller. This field is immutable.

BackupBucketStatus

(Appears on: BackupBucket)

BackupBucketStatus holds the most recently observed status of the Backup Bucket.

FieldDescription
providerStatus
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderStatus is the configuration passed to BackupBucket resource.

lastOperation
LastOperation
(Optional)

LastOperation holds information about the last operation on the BackupBucket.

lastError
LastError
(Optional)

LastError holds information about the last occurred error during an operation.

observedGeneration
int64
(Optional)

ObservedGeneration is the most recent generation observed for this BackupBucket. It corresponds to the BackupBucket’s generation, which is updated on mutation by the API Server.

generatedSecretRef
Kubernetes core/v1.SecretReference
(Optional)

GeneratedSecretRef is reference to the secret generated by backup bucket, which will have object store specific credentials.

BackupEntrySpec

(Appears on: BackupEntry)

BackupEntrySpec is the specification of a Backup Entry.

FieldDescription
bucketName
string

BucketName is the name of backup bucket for this Backup Entry.

seedName
string
(Optional)

SeedName holds the name of the seed to which this BackupEntry is scheduled

BackupEntryStatus

(Appears on: BackupEntry)

BackupEntryStatus holds the most recently observed status of the Backup Entry.

FieldDescription
lastOperation
LastOperation
(Optional)

LastOperation holds information about the last operation on the BackupEntry.

lastError
LastError
(Optional)

LastError holds information about the last occurred error during an operation.

observedGeneration
int64
(Optional)

ObservedGeneration is the most recent generation observed for this BackupEntry. It corresponds to the BackupEntry’s generation, which is updated on mutation by the API Server.

seedName
string
(Optional)

SeedName is the name of the seed to which this BackupEntry is currently scheduled. This field is populated at the beginning of a create/reconcile operation. It is used when moving the BackupEntry between seeds.

migrationStartTime
Kubernetes meta/v1.Time
(Optional)

MigrationStartTime is the time when a migration to a different seed was initiated.

CRI

(Appears on: MachineImageVersion, Worker)

CRI contains information about the Container Runtimes.

FieldDescription
name
CRIName

The name of the CRI library. Supported values are docker and containerd.

containerRuntimes
[]ContainerRuntime
(Optional)

ContainerRuntimes is the list of the required container runtimes supported for a worker pool.

CRIName (string alias)

(Appears on: CRI)

CRIName is a type alias for the CRI name string.

CloudProfileSpec

(Appears on: CloudProfile)

CloudProfileSpec is the specification of a CloudProfile. It must contain exactly one of its defined keys.

FieldDescription
caBundle
string
(Optional)

CABundle is a certificate bundle which will be installed onto every host machine of shoot cluster targeting this profile.

kubernetes
KubernetesSettings

Kubernetes contains constraints regarding allowed values of the ‘kubernetes’ block in the Shoot specification.

machineImages
[]MachineImage

MachineImages contains constraints regarding allowed values for machine images in the Shoot specification.

machineTypes
[]MachineType

MachineTypes contains constraints regarding allowed values for machine types in the ‘workers’ block in the Shoot specification.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig contains provider-specific configuration for the profile.

regions
[]Region

Regions contains constraints regarding allowed values for regions and zones.

seedSelector
SeedSelector
(Optional)

SeedSelector contains an optional list of labels on Seed resources that marks those seeds whose shoots may use this provider profile. An empty list means that all seeds of the same provider type are supported. This is useful for environments that are of the same type (like openstack) but may have different “instances”/landscapes. Optionally a list of possible providers can be added to enable cross-provider scheduling. By default, the provider type of the seed must match the shoot’s provider.

type
string

Type is the name of the provider.

volumeTypes
[]VolumeType
(Optional)

VolumeTypes contains constraints regarding allowed values for volume types in the ‘workers’ block in the Shoot specification.

ClusterAutoscaler

(Appears on: Kubernetes)

ClusterAutoscaler contains the configuration flags for the Kubernetes cluster autoscaler.

FieldDescription
scaleDownDelayAfterAdd
Kubernetes meta/v1.Duration
(Optional)

ScaleDownDelayAfterAdd defines how long after scale up that scale down evaluation resumes (default: 1 hour).

scaleDownDelayAfterDelete
Kubernetes meta/v1.Duration
(Optional)

ScaleDownDelayAfterDelete how long after node deletion that scale down evaluation resumes, defaults to scanInterval (default: 0 secs).

scaleDownDelayAfterFailure
Kubernetes meta/v1.Duration
(Optional)

ScaleDownDelayAfterFailure how long after scale down failure that scale down evaluation resumes (default: 3 mins).

scaleDownUnneededTime
Kubernetes meta/v1.Duration
(Optional)

ScaleDownUnneededTime defines how long a node should be unneeded before it is eligible for scale down (default: 30 mins).

scaleDownUtilizationThreshold
float64
(Optional)

ScaleDownUtilizationThreshold defines the threshold in fraction (0.0 - 1.0) under which a node is being removed (default: 0.5).

scanInterval
Kubernetes meta/v1.Duration
(Optional)

ScanInterval how often cluster is reevaluated for scale up or down (default: 10 secs).

expander
ExpanderMode
(Optional)

Expander defines the algorithm to use during scale up (default: least-waste). See: https://github.com/gardener/autoscaler/blob/machine-controller-manager-provider/cluster-autoscaler/FAQ.md#what-are-expanders.

maxNodeProvisionTime
Kubernetes meta/v1.Duration
(Optional)

MaxNodeProvisionTime defines how long CA waits for node to be provisioned (default: 20 mins).

maxGracefulTerminationSeconds
int32
(Optional)

MaxGracefulTerminationSeconds is the number of seconds CA waits for pod termination when trying to scale down a node (default: 600).

ignoreTaints
[]string
(Optional)

IgnoreTaints specifies a list of taint keys to ignore in node templates when considering to scale a node group.

Condition

(Appears on: ControllerInstallationStatus, SeedStatus, ShootStatus)

Condition holds the information about the state of a resource.

FieldDescription
type
ConditionType

Type of the condition.

status
ConditionStatus

Status of the condition, one of True, False, Unknown.

lastTransitionTime
Kubernetes meta/v1.Time

Last time the condition transitioned from one status to another.

lastUpdateTime
Kubernetes meta/v1.Time

Last time the condition was updated.

reason
string

The reason for the condition’s last transition.

message
string

A human readable message indicating details about the transition.

codes
[]ErrorCode
(Optional)

Well-defined error codes in case the condition reports a problem.

ConditionStatus (string alias)

(Appears on: Condition)

ConditionStatus is the status of a condition.

ConditionType (string alias)

(Appears on: Condition)

ConditionType is a string alias.

ContainerRuntime

(Appears on: CRI)

ContainerRuntime contains information about worker’s available container runtime

FieldDescription
type
string

Type is the type of the Container Runtime.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig is the configuration passed to container runtime resource.

ControlPlane

(Appears on: ShootSpec)

ControlPlane holds information about the general settings for the control plane of a shoot.

FieldDescription
highAvailability
HighAvailability
(Optional)

HighAvailability holds the configuration settings for high availability of the control plane of a shoot.

ControllerDeploymentPolicy (string alias)

(Appears on: ControllerRegistrationDeployment)

ControllerDeploymentPolicy is a string alias.

ControllerInstallationSpec

(Appears on: ControllerInstallation)

ControllerInstallationSpec is the specification of a ControllerInstallation.

FieldDescription
registrationRef
Kubernetes core/v1.ObjectReference

RegistrationRef is used to reference a ControllerRegistration resource. The name field of the RegistrationRef is immutable.

seedRef
Kubernetes core/v1.ObjectReference

SeedRef is used to reference a Seed resource. The name field of the SeedRef is immutable.

deploymentRef
Kubernetes core/v1.ObjectReference
(Optional)

DeploymentRef is used to reference a ControllerDeployment resource.

ControllerInstallationStatus

(Appears on: ControllerInstallation)

ControllerInstallationStatus is the status of a ControllerInstallation.

FieldDescription
conditions
[]Condition
(Optional)

Conditions represents the latest available observations of a ControllerInstallations’s current state.

providerStatus
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderStatus contains type-specific status.

ControllerRegistrationDeployment

(Appears on: ControllerRegistrationSpec)

ControllerRegistrationDeployment contains information for how this controller is deployed.

FieldDescription
policy
ControllerDeploymentPolicy
(Optional)

Policy controls how the controller is deployed. It defaults to ‘OnDemand’.

seedSelector
Kubernetes meta/v1.LabelSelector
(Optional)

SeedSelector contains an optional label selector for seeds. Only if the labels match then this controller will be considered for a deployment. An empty list means that all seeds are selected.

deploymentRefs
[]DeploymentRef
(Optional)

DeploymentRefs holds references to ControllerDeployments. Only one element is support now.

ControllerRegistrationSpec

(Appears on: ControllerRegistration)

ControllerRegistrationSpec is the specification of a ControllerRegistration.

FieldDescription
resources
[]ControllerResource
(Optional)

Resources is a list of combinations of kinds (DNSProvider, Infrastructure, Generic, …) and their actual types (aws-route53, gcp, auditlog, …).

deployment
ControllerRegistrationDeployment
(Optional)

Deployment contains information for how this controller is deployed.

ControllerResource

(Appears on: ControllerRegistrationSpec)

ControllerResource is a combination of a kind (DNSProvider, Infrastructure, Generic, …) and the actual type for this kind (aws-route53, gcp, auditlog, …).

FieldDescription
kind
string

Kind is the resource kind, for example “OperatingSystemConfig”.

type
string

Type is the resource type, for example “coreos” or “ubuntu”.

globallyEnabled
bool
(Optional)

GloballyEnabled determines if this ControllerResource is required by all Shoot clusters.

reconcileTimeout
Kubernetes meta/v1.Duration
(Optional)

ReconcileTimeout defines how long Gardener should wait for the resource reconciliation.

primary
bool
(Optional)

Primary determines if the controller backed by this ControllerRegistration is responsible for the extension resource’s lifecycle. This field defaults to true. There must be exactly one primary controller for this kind/type combination. This field is immutable.

CoreDNS

(Appears on: SystemComponents)

CoreDNS contains the settings of the Core DNS components running in the data plane of the Shoot cluster.

FieldDescription
autoscaling
CoreDNSAutoscaling
(Optional)

Autoscaling contains the settings related to autoscaling of the Core DNS components running in the data plane of the Shoot cluster.

rewriting
CoreDNSRewriting
(Optional)

Rewriting contains the setting related to rewriting of requests, which are obviously incorrect due to the unnecessary application of the search path.

CoreDNSAutoscaling

(Appears on: CoreDNS)

CoreDNSAutoscaling contains the settings related to autoscaling of the Core DNS components running in the data plane of the Shoot cluster.

FieldDescription
mode
CoreDNSAutoscalingMode

The mode of the autoscaling to be used for the Core DNS components running in the data plane of the Shoot cluster. Supported values are horizontal and cluster-proportional.

CoreDNSAutoscalingMode (string alias)

(Appears on: CoreDNSAutoscaling)

CoreDNSAutoscalingMode is a type alias for the Core DNS autoscaling mode string.

CoreDNSRewriting

(Appears on: CoreDNS)

CoreDNSRewriting contains the setting related to rewriting requests, which are obviously incorrect due to the unnecessary application of the search path.

FieldDescription
commonSuffixes
[]string
(Optional)

CommonSuffixes are expected to be the suffix of a fully qualified domain name. Each suffix should contain at least one or two dots (‘.’) to prevent accidental clashes.

DNS

(Appears on: ShootSpec)

DNS holds information about the provider, the hosted zone id and the domain.

FieldDescription
domain
string
(Optional)

Domain is the external available domain of the Shoot cluster. This domain will be written into the kubeconfig that is handed out to end-users. This field is immutable.

providers
[]DNSProvider
(Optional)

Providers is a list of DNS providers that shall be enabled for this shoot cluster. Only relevant if not a default domain is used.

DNSIncludeExclude

(Appears on: DNSProvider, SeedDNSProvider)

DNSIncludeExclude contains information about which domains shall be included/excluded.

FieldDescription
include
[]string
(Optional)

Include is a list of domains that shall be included.

exclude
[]string
(Optional)

Exclude is a list of domains that shall be excluded.

DNSProvider

(Appears on: DNS)

DNSProvider contains information about a DNS provider.

FieldDescription
domains
DNSIncludeExclude
(Optional)

Domains contains information about which domains shall be included/excluded for this provider.

primary
bool
(Optional)

Primary indicates that this DNSProvider is used for shoot related domains.

secretName
string
(Optional)

SecretName is a name of a secret containing credentials for the stated domain and the provider. When not specified, the Gardener will use the cloud provider credentials referenced by the Shoot and try to find respective credentials there (primary provider only). Specifying this field may override this behavior, i.e. forcing the Gardener to only look into the given secret.

type
string
(Optional)

Type is the DNS provider type.

zones
DNSIncludeExclude
(Optional)

Zones contains information about which hosted zones shall be included/excluded for this provider.

DataVolume

(Appears on: Worker)

DataVolume contains information about a data volume.

FieldDescription
name
string

Name of the volume to make it referencable.

type
string
(Optional)

Type is the type of the volume.

size
string

VolumeSize is the size of the volume.

encrypted
bool
(Optional)

Encrypted determines if the volume should be encrypted.

DeploymentRef

(Appears on: ControllerRegistrationDeployment)

DeploymentRef contains information about ControllerDeployment references.

FieldDescription
name
string

Name is the name of the ControllerDeployment that is being referred to.

ErrorCode (string alias)

(Appears on: Condition, LastError)

ErrorCode is a string alias.

ExpanderMode (string alias)

(Appears on: ClusterAutoscaler)

ExpanderMode is type used for Expander values

ExpirableVersion

(Appears on: KubernetesSettings, MachineImageVersion)

ExpirableVersion contains a version and an expiration date.

FieldDescription
version
string

Version is the version identifier.

expirationDate
Kubernetes meta/v1.Time
(Optional)

ExpirationDate defines the time at which this version expires.

classification
VersionClassification
(Optional)

Classification defines the state of a version (preview, supported, deprecated)

Extension

(Appears on: ShootSpec)

Extension contains type and provider information for Shoot extensions.

FieldDescription
type
string

Type is the type of the extension resource.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig is the configuration passed to extension resource.

disabled
bool
(Optional)

Disabled allows to disable extensions that were marked as ‘globally enabled’ by Gardener administrators.

FailureTolerance

(Appears on: HighAvailability)

FailureTolerance describes information about failure tolerance level of a highly available resource.

FieldDescription
type
FailureToleranceType

Type specifies the type of failure that the highly available resource can tolerate

FailureToleranceType (string alias)

(Appears on: FailureTolerance)

FailureToleranceType specifies the type of failure that a highly available shoot control plane that can tolerate.

Gardener

(Appears on: SeedStatus, ShootStatus)

Gardener holds the information about the Gardener version that operated a resource.

FieldDescription
id
string

ID is the Docker container id of the Gardener which last acted on a resource.

name
string

Name is the hostname (pod name) of the Gardener which last acted on a resource.

version
string

Version is the version of the Gardener which last acted on a resource.

Hibernation

(Appears on: ShootSpec)

Hibernation contains information whether the Shoot is suspended or not.

FieldDescription
enabled
bool
(Optional)

Enabled specifies whether the Shoot needs to be hibernated or not. If it is true, the Shoot’s desired state is to be hibernated. If it is false or nil, the Shoot’s desired state is to be awakened.

schedules
[]HibernationSchedule
(Optional)

Schedules determine the hibernation schedules.

HibernationSchedule

(Appears on: Hibernation)

HibernationSchedule determines the hibernation schedule of a Shoot. A Shoot will be regularly hibernated at each start time and will be woken up at each end time. Start or End can be omitted, though at least one of each has to be specified.

FieldDescription
start
string
(Optional)

Start is a Cron spec at which time a Shoot will be hibernated.

end
string
(Optional)

End is a Cron spec at which time a Shoot will be woken up.

location
string
(Optional)

Location is the time location in which both start and and shall be evaluated.

HighAvailability

(Appears on: ControlPlane)

HighAvailability specifies the configuration settings for high availability for a resource. Typical usages could be to configure HA for shoot control plane or for seed system components.

FieldDescription
failureTolerance
FailureTolerance

FailureTolerance holds information about failure tolerance level of a highly available resource.

HorizontalPodAutoscalerConfig

(Appears on: KubeControllerManagerConfig)

HorizontalPodAutoscalerConfig contains horizontal pod autoscaler configuration settings for the kube-controller-manager. Note: Descriptions were taken from the Kubernetes documentation.

FieldDescription
cpuInitializationPeriod
Kubernetes meta/v1.Duration
(Optional)

The period after which a ready pod transition is considered to be the first.

downscaleStabilization
Kubernetes meta/v1.Duration
(Optional)

The configurable window at which the controller will choose the highest recommendation for autoscaling.

initialReadinessDelay
Kubernetes meta/v1.Duration
(Optional)

The configurable period at which the horizontal pod autoscaler considers a Pod “not yet ready” given that it’s unready and it has transitioned to unready during that time.

syncPeriod
Kubernetes meta/v1.Duration
(Optional)

The period for syncing the number of pods in horizontal pod autoscaler.

tolerance
float64
(Optional)

The minimum change (from 1.0) in the desired-to-actual metrics ratio for the horizontal pod autoscaler to consider scaling.

Ingress

(Appears on: SeedSpec)

Ingress configures the Ingress specific settings of the Seed cluster

FieldDescription
domain
string

Domain specifies the IngressDomain of the Seed cluster pointing to the ingress controller endpoint. It will be used to construct ingress URLs for system applications running in Shoot clusters. Once set this field is immutable.

controller
IngressController

Controller configures a Gardener managed Ingress Controller listening on the ingressDomain

IngressController

(Appears on: Ingress)

IngressController enables a Gardener managed Ingress Controller listening on the ingressDomain

FieldDescription
kind
string

Kind defines which kind of IngressController to use, for example nginx

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig specifies infrastructure specific configuration for the ingressController

KubeAPIServerConfig

(Appears on: Kubernetes)

KubeAPIServerConfig contains configuration settings for the kube-apiserver.

FieldDescription
KubernetesConfig
KubernetesConfig

(Members of KubernetesConfig are embedded into this type.)

admissionPlugins
[]AdmissionPlugin
(Optional)

AdmissionPlugins contains the list of user-defined admission plugins (additional to those managed by Gardener), and, if desired, the corresponding configuration.

apiAudiences
[]string
(Optional)

APIAudiences are the identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. Defaults to [“kubernetes”].

auditConfig
AuditConfig
(Optional)

AuditConfig contains configuration settings for the audit of the kube-apiserver.

enableBasicAuthentication
bool
(Optional)

EnableBasicAuthentication defines whether basic authentication should be enabled for this cluster or not.

oidcConfig
OIDCConfig
(Optional)

OIDCConfig contains configuration settings for the OIDC provider.

runtimeConfig
map[string]bool
(Optional)

RuntimeConfig contains information about enabled or disabled APIs.

serviceAccountConfig
ServiceAccountConfig
(Optional)

ServiceAccountConfig contains configuration settings for the service account handling of the kube-apiserver.

watchCacheSizes
WatchCacheSizes
(Optional)

WatchCacheSizes contains configuration of the API server’s watch cache sizes. Configuring these flags might be useful for large-scale Shoot clusters with a lot of parallel update requests and a lot of watching controllers (e.g. large ManagedSeed clusters). When the API server’s watch cache’s capacity is too small to cope with the amount of update requests and watchers for a particular resource, it might happen that controller watches are permanently stopped with too old resource version errors. Starting from kubernetes v1.19, the API server’s watch cache size is adapted dynamically and setting the watch cache size flags will have no effect, except when setting it to 0 (which disables the watch cache).

requests
KubeAPIServerRequests
(Optional)

Requests contains configuration for request-specific settings for the kube-apiserver.

enableAnonymousAuthentication
bool
(Optional)

EnableAnonymousAuthentication defines whether anonymous requests to the secure port of the API server should be allowed (flag --anonymous-auth). See: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

eventTTL
Kubernetes meta/v1.Duration
(Optional)

EventTTL controls the amount of time to retain events. Defaults to 1h.

KubeAPIServerRequests

(Appears on: KubeAPIServerConfig)

KubeAPIServerRequests contains configuration for request-specific settings for the kube-apiserver.

FieldDescription
maxNonMutatingInflight
int32
(Optional)

MaxNonMutatingInflight is the maximum number of non-mutating requests in flight at a given time. When the server exceeds this, it rejects requests.

maxMutatingInflight
int32
(Optional)

MaxMutatingInflight is the maximum number of mutating requests in flight at a given time. When the server exceeds this, it rejects requests.

KubeControllerManagerConfig

(Appears on: Kubernetes)

KubeControllerManagerConfig contains configuration settings for the kube-controller-manager.

FieldDescription
KubernetesConfig
KubernetesConfig

(Members of KubernetesConfig are embedded into this type.)

horizontalPodAutoscaler
HorizontalPodAutoscalerConfig
(Optional)

HorizontalPodAutoscalerConfig contains horizontal pod autoscaler configuration settings for the kube-controller-manager.

nodeCIDRMaskSize
int32
(Optional)

NodeCIDRMaskSize defines the mask size for node cidr in cluster (default is 24). This field is immutable.

podEvictionTimeout
Kubernetes meta/v1.Duration
(Optional)

PodEvictionTimeout defines the grace period for deleting pods on failed nodes. Defaults to 2m.

nodeMonitorGracePeriod
Kubernetes meta/v1.Duration
(Optional)

NodeMonitorGracePeriod defines the grace period before an unresponsive node is marked unhealthy.

KubeProxyConfig

(Appears on: Kubernetes)

KubeProxyConfig contains configuration settings for the kube-proxy.

FieldDescription
KubernetesConfig
KubernetesConfig

(Members of KubernetesConfig are embedded into this type.)

mode
ProxyMode
(Optional)

Mode specifies which proxy mode to use. defaults to IPTables.

enabled
bool
(Optional)

Enabled indicates whether kube-proxy should be deployed or not. Depending on the networking extensions switching kube-proxy off might be rejected. Consulting the respective documentation of the used networking extension is recommended before using this field. defaults to true if not specified.

KubeSchedulerConfig

(Appears on: Kubernetes)

KubeSchedulerConfig contains configuration settings for the kube-scheduler.

FieldDescription
KubernetesConfig
KubernetesConfig

(Members of KubernetesConfig are embedded into this type.)

kubeMaxPDVols
string
(Optional)

KubeMaxPDVols allows to configure the KUBE_MAX_PD_VOLS environment variable for the kube-scheduler. Please find more information here: https://kubernetes.io/docs/concepts/storage/storage-limits/#custom-limits Note that using this field is considered alpha-/experimental-level and is on your own risk. You should be aware of all the side-effects and consequences when changing it.

profile
SchedulingProfile
(Optional)

Profile configures the scheduling profile for the cluster. If not specified, the used profile is “balanced” (provides the default kube-scheduler behavior).

KubeletConfig

(Appears on: Kubernetes, WorkerKubernetes)

KubeletConfig contains configuration settings for the kubelet.

FieldDescription
KubernetesConfig
KubernetesConfig

(Members of KubernetesConfig are embedded into this type.)

cpuCFSQuota
bool
(Optional)

CPUCFSQuota allows you to disable/enable CPU throttling for Pods.

cpuManagerPolicy
string
(Optional)

CPUManagerPolicy allows to set alternative CPU management policies (default: none).

evictionHard
KubeletConfigEviction
(Optional)

EvictionHard describes a set of eviction thresholds (e.g. memory.available<1Gi) that if met would trigger a Pod eviction. Default: memory.available: “100Mi/1Gi/5%” nodefs.available: “5%” nodefs.inodesFree: “5%” imagefs.available: “5%” imagefs.inodesFree: “5%”

evictionMaxPodGracePeriod
int32
(Optional)

EvictionMaxPodGracePeriod describes the maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. Default: 90

evictionMinimumReclaim
KubeletConfigEvictionMinimumReclaim
(Optional)

EvictionMinimumReclaim configures the amount of resources below the configured eviction threshold that the kubelet attempts to reclaim whenever the kubelet observes resource pressure. Default: 0 for each resource

evictionPressureTransitionPeriod
Kubernetes meta/v1.Duration
(Optional)

EvictionPressureTransitionPeriod is the duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. Default: 4m0s

evictionSoft
KubeletConfigEviction
(Optional)

EvictionSoft describes a set of eviction thresholds (e.g. memory.available<1.5Gi) that if met over a corresponding grace period would trigger a Pod eviction. Default: memory.available: “200Mi/1.5Gi/10%” nodefs.available: “10%” nodefs.inodesFree: “10%” imagefs.available: “10%” imagefs.inodesFree: “10%”

evictionSoftGracePeriod
KubeletConfigEvictionSoftGracePeriod
(Optional)

EvictionSoftGracePeriod describes a set of eviction grace periods (e.g. memory.available=1m30s) that correspond to how long a soft eviction threshold must hold before triggering a Pod eviction. Default: memory.available: 1m30s nodefs.available: 1m30s nodefs.inodesFree: 1m30s imagefs.available: 1m30s imagefs.inodesFree: 1m30s

maxPods
int32
(Optional)

MaxPods is the maximum number of Pods that are allowed by the Kubelet. Default: 110

podPidsLimit
int64
(Optional)

PodPIDsLimit is the maximum number of process IDs per pod allowed by the kubelet.

imagePullProgressDeadline
Kubernetes meta/v1.Duration
(Optional)

ImagePullProgressDeadline describes the time limit under which if no pulling progress is made, the image pulling will be cancelled. Default: 1m

failSwapOn
bool
(Optional)

FailSwapOn makes the Kubelet fail to start if swap is enabled on the node. (default true).

kubeReserved
KubeletConfigReserved
(Optional)

KubeReserved is the configuration for resources reserved for kubernetes node components (mainly kubelet and container runtime). When updating these values, be aware that cgroup resizes may not succeed on active worker nodes. Look for the NodeAllocatableEnforced event to determine if the configuration was applied. Default: cpu=80m,memory=1Gi,pid=20k

systemReserved
KubeletConfigReserved
(Optional)

SystemReserved is the configuration for resources reserved for system processes not managed by kubernetes (e.g. journald). When updating these values, be aware that cgroup resizes may not succeed on active worker nodes. Look for the NodeAllocatableEnforced event to determine if the configuration was applied.

imageGCHighThresholdPercent
int32
(Optional)

ImageGCHighThresholdPercent describes the percent of the disk usage which triggers image garbage collection. Default: 50

imageGCLowThresholdPercent
int32
(Optional)

ImageGCLowThresholdPercent describes the percent of the disk to which garbage collection attempts to free. Default: 40

serializeImagePulls
bool
(Optional)

SerializeImagePulls describes whether the images are pulled one at a time. Default: true

registryPullQPS
int32
(Optional)

RegistryPullQPS is the limit of registry pulls per second. The value must not be a negative number. Setting it to 0 means no limit. Default: 5

registryBurst
int32
(Optional)

RegistryBurst is the maximum size of bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registryPullQPS. The value must not be a negative number. Only used if registryPullQPS is greater than 0. Default: 10

KubeletConfigEviction

(Appears on: KubeletConfig)

KubeletConfigEviction contains kubelet eviction thresholds supporting either a resource.Quantity or a percentage based value.

FieldDescription
memoryAvailable
string
(Optional)

MemoryAvailable is the threshold for the free memory on the host server.

imageFSAvailable
string
(Optional)

ImageFSAvailable is the threshold for the free disk space in the imagefs filesystem (docker images and container writable layers).

imageFSInodesFree
string
(Optional)

ImageFSInodesFree is the threshold for the available inodes in the imagefs filesystem.

nodeFSAvailable
string
(Optional)

NodeFSAvailable is the threshold for the free disk space in the nodefs filesystem (docker volumes, logs, etc).

nodeFSInodesFree
string
(Optional)

NodeFSInodesFree is the threshold for the available inodes in the nodefs filesystem.

KubeletConfigEvictionMinimumReclaim

(Appears on: KubeletConfig)

KubeletConfigEvictionMinimumReclaim contains configuration for the kubelet eviction minimum reclaim.

FieldDescription
memoryAvailable
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

MemoryAvailable is the threshold for the memory reclaim on the host server.

imageFSAvailable
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

ImageFSAvailable is the threshold for the disk space reclaim in the imagefs filesystem (docker images and container writable layers).

imageFSInodesFree
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

ImageFSInodesFree is the threshold for the inodes reclaim in the imagefs filesystem.

nodeFSAvailable
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

NodeFSAvailable is the threshold for the disk space reclaim in the nodefs filesystem (docker volumes, logs, etc).

nodeFSInodesFree
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

NodeFSInodesFree is the threshold for the inodes reclaim in the nodefs filesystem.

KubeletConfigEvictionSoftGracePeriod

(Appears on: KubeletConfig)

KubeletConfigEvictionSoftGracePeriod contains grace periods for kubelet eviction thresholds.

FieldDescription
memoryAvailable
Kubernetes meta/v1.Duration
(Optional)

MemoryAvailable is the grace period for the MemoryAvailable eviction threshold.

imageFSAvailable
Kubernetes meta/v1.Duration
(Optional)

ImageFSAvailable is the grace period for the ImageFSAvailable eviction threshold.

imageFSInodesFree
Kubernetes meta/v1.Duration
(Optional)

ImageFSInodesFree is the grace period for the ImageFSInodesFree eviction threshold.

nodeFSAvailable
Kubernetes meta/v1.Duration
(Optional)

NodeFSAvailable is the grace period for the NodeFSAvailable eviction threshold.

nodeFSInodesFree
Kubernetes meta/v1.Duration
(Optional)

NodeFSInodesFree is the grace period for the NodeFSInodesFree eviction threshold.

KubeletConfigReserved

(Appears on: KubeletConfig)

KubeletConfigReserved contains reserved resources for daemons

FieldDescription
cpu
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

CPU is the reserved cpu.

memory
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

Memory is the reserved memory.

ephemeralStorage
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

EphemeralStorage is the reserved ephemeral-storage.

pid
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

PID is the reserved process-ids.

Kubernetes

(Appears on: ShootSpec)

Kubernetes contains the version and configuration variables for the Shoot control plane.

FieldDescription
allowPrivilegedContainers
bool
(Optional)

AllowPrivilegedContainers indicates whether privileged containers are allowed in the Shoot. Defaults to true for Kubernetes versions below v1.25. Unusable for Kubernetes versions v1.25 and higher.

clusterAutoscaler
ClusterAutoscaler
(Optional)

ClusterAutoscaler contains the configuration flags for the Kubernetes cluster autoscaler.

kubeAPIServer
KubeAPIServerConfig
(Optional)

KubeAPIServer contains configuration settings for the kube-apiserver.

kubeControllerManager
KubeControllerManagerConfig
(Optional)

KubeControllerManager contains configuration settings for the kube-controller-manager.

kubeScheduler
KubeSchedulerConfig
(Optional)

KubeScheduler contains configuration settings for the kube-scheduler.

kubeProxy
KubeProxyConfig
(Optional)

KubeProxy contains configuration settings for the kube-proxy.

kubelet
KubeletConfig
(Optional)

Kubelet contains configuration settings for the kubelet.

version
string

Version is the semantic Kubernetes version to use for the Shoot cluster.

verticalPodAutoscaler
VerticalPodAutoscaler
(Optional)

VerticalPodAutoscaler contains the configuration flags for the Kubernetes vertical pod autoscaler.

enableStaticTokenKubeconfig
bool
(Optional)

EnableStaticTokenKubeconfig indicates whether static token kubeconfig secret should be present in garden cluster (default: true).

KubernetesConfig

(Appears on: KubeAPIServerConfig, KubeControllerManagerConfig, KubeProxyConfig, KubeSchedulerConfig, KubeletConfig)

KubernetesConfig contains common configuration fields for the control plane components.

FieldDescription
featureGates
map[string]bool
(Optional)

FeatureGates contains information about enabled feature gates.

KubernetesDashboard

(Appears on: Addons)

KubernetesDashboard describes configuration values for the kubernetes-dashboard addon.

FieldDescription
Addon
Addon

(Members of Addon are embedded into this type.)

authenticationMode
string
(Optional)

AuthenticationMode defines the authentication mode for the kubernetes-dashboard.

KubernetesSettings

(Appears on: CloudProfileSpec)

KubernetesSettings contains constraints regarding allowed values of the ‘kubernetes’ block in the Shoot specification.

FieldDescription
versions
[]ExpirableVersion
(Optional)

Versions is the list of allowed Kubernetes versions with optional expiration dates for Shoot clusters.

LastError

(Appears on: BackupBucketStatus, BackupEntryStatus, ShootStatus)

LastError indicates the last occurred error for an operation on a resource.

FieldDescription
description
string

A human readable message indicating details about the last error.

taskID
string
(Optional)

ID of the task which caused this last error

codes
[]ErrorCode
(Optional)

Well-defined error codes of the last error(s).

lastUpdateTime
Kubernetes meta/v1.Time
(Optional)

Last time the error was reported

LastOperation

(Appears on: BackupBucketStatus, BackupEntryStatus, ShootStatus)

LastOperation indicates the type and the state of the last operation, along with a description message and a progress indicator.

FieldDescription
description
string

A human readable message indicating details about the last operation.

lastUpdateTime
Kubernetes meta/v1.Time

Last time the operation state transitioned from one to another.

progress
int32

The progress in percentage (0-100) of the last operation.

state
LastOperationState

Status of the last operation, one of Aborted, Processing, Succeeded, Error, Failed.

type
LastOperationType

Type of the last operation, one of Create, Reconcile, Delete.

LastOperationState (string alias)

(Appears on: LastOperation)

LastOperationState is a string alias.

LastOperationType (string alias)

(Appears on: LastOperation)

LastOperationType is a string alias.

Machine

(Appears on: Worker)

Machine contains information about the machine type and image.

FieldDescription
type
string

Type is the machine type of the worker group.

image
ShootMachineImage
(Optional)

Image holds information about the machine image to use for all nodes of this pool. It will default to the latest version of the first image stated in the referenced CloudProfile if no value has been provided.

architecture
string
(Optional)

Architecture is CPU architecture of machines in this worker pool.

MachineControllerManagerSettings

(Appears on: Worker)

MachineControllerManagerSettings contains configurations for different worker-pools. Eg. MachineDrainTimeout, MachineHealthTimeout.

FieldDescription
machineDrainTimeout
Kubernetes meta/v1.Duration
(Optional)

MachineDrainTimeout is the period after which machine is forcefully deleted.

machineHealthTimeout
Kubernetes meta/v1.Duration
(Optional)

MachineHealthTimeout is the period after which machine is declared failed.

machineCreationTimeout
Kubernetes meta/v1.Duration
(Optional)

MachineCreationTimeout is the period after which creation of the machine is declared failed.

maxEvictRetries
int32
(Optional)

MaxEvictRetries are the number of eviction retries on a pod after which drain is declared failed, and forceful deletion is triggered.

nodeConditions
[]string
(Optional)

NodeConditions are the set of conditions if set to true for the period of MachineHealthTimeout, machine will be declared failed.

MachineImage

(Appears on: CloudProfileSpec)

MachineImage defines the name and multiple versions of the machine image in any environment.

FieldDescription
name
string

Name is the name of the image.

versions
[]MachineImageVersion

Versions contains versions, expiration dates and container runtimes of the machine image

MachineImageVersion

(Appears on: MachineImage)

MachineImageVersion is an expirable version with list of supported container runtimes and interfaces

FieldDescription
ExpirableVersion
ExpirableVersion

(Members of ExpirableVersion are embedded into this type.)

cri
[]CRI
(Optional)

CRI list of supported container runtime and interfaces supported by this version

architectures
[]string
(Optional)

Architectures is the list of CPU architectures of the machine image in this version.

MachineType

(Appears on: CloudProfileSpec)

MachineType contains certain properties of a machine type.

FieldDescription
cpu
k8s.io/apimachinery/pkg/api/resource.Quantity

CPU is the number of CPUs for this machine type.

gpu
k8s.io/apimachinery/pkg/api/resource.Quantity

GPU is the number of GPUs for this machine type.

memory
k8s.io/apimachinery/pkg/api/resource.Quantity

Memory is the amount of memory for this machine type.

name
string

Name is the name of the machine type.

storage
MachineTypeStorage
(Optional)

Storage is the amount of storage associated with the root volume of this machine type.

usable
bool
(Optional)

Usable defines if the machine type can be used for shoot clusters.

architecture
string
(Optional)

Architecture is the CPU architecture of this machine type.

MachineTypeStorage

(Appears on: MachineType)

MachineTypeStorage is the amount of storage associated with the root volume of this machine type.

FieldDescription
class
string

Class is the class of the storage type.

size
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

StorageSize is the storage size.

type
string

Type is the type of the storage.

minSize
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

MinSize is the minimal supported storage size. This overrides any other common minimum size configuration from spec.volumeTypes[*].minSize.

Maintenance

(Appears on: ShootSpec)

Maintenance contains information about the time window for maintenance operations and which operations should be performed.

FieldDescription
autoUpdate
MaintenanceAutoUpdate
(Optional)

AutoUpdate contains information about which constraints should be automatically updated.

timeWindow
MaintenanceTimeWindow
(Optional)

TimeWindow contains information about the time window for maintenance operations.

confineSpecUpdateRollout
bool
(Optional)

ConfineSpecUpdateRollout prevents that changes/updates to the shoot specification will be rolled out immediately. Instead, they are rolled out during the shoot’s maintenance time window. There is one exception that will trigger an immediate roll out which is changes to the Spec.Hibernation.Enabled field.

MaintenanceAutoUpdate

(Appears on: Maintenance)

MaintenanceAutoUpdate contains information about which constraints should be automatically updated.

FieldDescription
kubernetesVersion
bool

KubernetesVersion indicates whether the patch Kubernetes version may be automatically updated (default: true).

machineImageVersion
bool

MachineImageVersion indicates whether the machine image version may be automatically updated (default: true).

MaintenanceTimeWindow

(Appears on: Maintenance)

MaintenanceTimeWindow contains information about the time window for maintenance operations.

FieldDescription
begin
string

Begin is the beginning of the time window in the format HHMMSS+ZONE, e.g. “220000+0100”. If not present, a random value will be computed.

end
string

End is the end of the time window in the format HHMMSS+ZONE, e.g. “220000+0100”. If not present, the value will be computed based on the “Begin” value.

Monitoring

(Appears on: ShootSpec)

Monitoring contains information about the monitoring configuration for the shoot.

FieldDescription
alerting
Alerting
(Optional)

Alerting contains information about the alerting configuration for the shoot cluster.

NamedResourceReference

(Appears on: ShootSpec)

NamedResourceReference is a named reference to a resource.

FieldDescription
name
string

Name of the resource reference.

resourceRef
Kubernetes autoscaling/v1.CrossVersionObjectReference

ResourceRef is a reference to a resource.

Networking

(Appears on: ShootSpec)

Networking defines networking parameters for the shoot cluster.

FieldDescription
type
string

Type identifies the type of the networking plugin. This field is immutable.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig is the configuration passed to network resource.

pods
string
(Optional)

Pods is the CIDR of the pod network. This field is immutable.

nodes
string
(Optional)

Nodes is the CIDR of the entire node network. This field is immutable.

services
string
(Optional)

Services is the CIDR of the service network. This field is immutable.

NginxIngress

(Appears on: Addons)

NginxIngress describes configuration values for the nginx-ingress addon.

FieldDescription
Addon
Addon

(Members of Addon are embedded into this type.)

loadBalancerSourceRanges
[]string
(Optional)

LoadBalancerSourceRanges is list of allowed IP sources for NginxIngress

config
map[string]string
(Optional)

Config contains custom configuration for the nginx-ingress-controller configuration. See https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/configmap.md#configuration-options

externalTrafficPolicy
Kubernetes core/v1.ServiceExternalTrafficPolicyType
(Optional)

ExternalTrafficPolicy controls the .spec.externalTrafficPolicy value of the load balancer Service exposing the nginx-ingress. Defaults to Cluster.

NodeLocalDNS

(Appears on: SystemComponents)

NodeLocalDNS contains the settings of the node local DNS components running in the data plane of the Shoot cluster.

FieldDescription
enabled
bool

Enabled indicates whether node local DNS is enabled or not.

forceTCPToClusterDNS
bool
(Optional)

ForceTCPToClusterDNS indicates whether the connection from the node local DNS to the cluster DNS (Core DNS) will be forced to TCP or not. Default, if unspecified, is to enforce TCP.

forceTCPToUpstreamDNS
bool
(Optional)

ForceTCPToUpstreamDNS indicates whether the connection from the node local DNS to the upstream DNS (infrastructure DNS) will be forced to TCP or not. Default, if unspecified, is to enforce TCP.

OIDCConfig

(Appears on: KubeAPIServerConfig)

OIDCConfig contains configuration settings for the OIDC provider. Note: Descriptions were taken from the Kubernetes documentation.

FieldDescription
caBundle
string
(Optional)

If set, the OpenID server’s certificate will be verified by one of the authorities in the oidc-ca-file, otherwise the host’s root CA set will be used.

clientAuthentication
OpenIDConnectClientAuthentication
(Optional)

ClientAuthentication can optionally contain client configuration used for kubeconfig generation.

clientID
string
(Optional)

The client ID for the OpenID Connect client, must be set if oidc-issuer-url is set.

groupsClaim
string
(Optional)

If provided, the name of a custom OpenID Connect claim for specifying user groups. The claim value is expected to be a string or array of strings. This flag is experimental, please see the authentication documentation for further details.

groupsPrefix
string
(Optional)

If provided, all groups will be prefixed with this value to prevent conflicts with other authentication strategies.

issuerURL
string
(Optional)

The URL of the OpenID issuer, only HTTPS scheme will be accepted. If set, it will be used to verify the OIDC JSON Web Token (JWT).

requiredClaims
map[string]string
(Optional)

key=value pairs that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value.

signingAlgs
[]string
(Optional)

List of allowed JOSE asymmetric signing algorithms. JWTs with a ‘alg’ header value not in this list will be rejected. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1

usernameClaim
string
(Optional)

The OpenID claim to use as the user name. Note that claims other than the default (‘sub’) is not guaranteed to be unique and immutable. This flag is experimental, please see the authentication documentation for further details. (default “sub”)

usernamePrefix
string
(Optional)

If provided, all usernames will be prefixed with this value. If not provided, username claims other than ‘email’ are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value ‘-’.

OpenIDConnectClientAuthentication

(Appears on: OIDCConfig)

OpenIDConnectClientAuthentication contains configuration for OIDC clients.

FieldDescription
extraConfig
map[string]string
(Optional)

Extra configuration added to kubeconfig’s auth-provider. Must not be any of idp-issuer-url, client-id, client-secret, idp-certificate-authority, idp-certificate-authority-data, id-token or refresh-token

secret
string
(Optional)

The client Secret for the OpenID Connect client.

ProjectMember

(Appears on: ProjectSpec)

ProjectMember is a member of a project.

FieldDescription
Subject
Kubernetes rbac/v1.Subject

(Members of Subject are embedded into this type.)

Subject is representing a user name, an email address, or any other identifier of a user, group, or service account that has a certain role.

role
string

Role represents the role of this member. IMPORTANT: Be aware that this field will be removed in the v1 version of this API in favor of the roles list. TODO: Remove this field in favor of the roles list in v1.

roles
[]string
(Optional)

Roles represents the list of roles of this member.

ProjectPhase (string alias)

(Appears on: ProjectStatus)

ProjectPhase is a label for the condition of a project at the current time.

ProjectSpec

(Appears on: Project)

ProjectSpec is the specification of a Project.

FieldDescription
createdBy
Kubernetes rbac/v1.Subject
(Optional)

CreatedBy is a subject representing a user name, an email address, or any other identifier of a user who created the project. This field is immutable.

description
string
(Optional)

Description is a human-readable description of what the project is used for.

owner
Kubernetes rbac/v1.Subject
(Optional)

Owner is a subject representing a user name, an email address, or any other identifier of a user owning the project. IMPORTANT: Be aware that this field will be removed in the v1 version of this API in favor of the owner role. The only way to change the owner will be by moving the owner role. In this API version the only way to change the owner is to use this field. TODO: Remove this field in favor of the owner role in v1.

purpose
string
(Optional)

Purpose is a human-readable explanation of the project’s purpose.

members
[]ProjectMember
(Optional)

Members is a list of subjects representing a user name, an email address, or any other identifier of a user, group, or service account that has a certain role.

namespace
string
(Optional)

Namespace is the name of the namespace that has been created for the Project object. A nil value means that Gardener will determine the name of the namespace. This field is immutable.

tolerations
ProjectTolerations
(Optional)

Tolerations contains the tolerations for taints on seed clusters.

ProjectStatus

(Appears on: Project)

ProjectStatus holds the most recently observed status of the project.

FieldDescription
observedGeneration
int64
(Optional)

ObservedGeneration is the most recent generation observed for this project.

phase
ProjectPhase

Phase is the current phase of the project.

staleSinceTimestamp
Kubernetes meta/v1.Time
(Optional)

StaleSinceTimestamp contains the timestamp when the project was first discovered to be stale/unused.

staleAutoDeleteTimestamp
Kubernetes meta/v1.Time
(Optional)

StaleAutoDeleteTimestamp contains the timestamp when the project will be garbage-collected/automatically deleted because it’s stale/unused.

lastActivityTimestamp
Kubernetes meta/v1.Time
(Optional)

LastActivityTimestamp contains the timestamp from the last activity performed in this project.

ProjectTolerations

(Appears on: ProjectSpec)

ProjectTolerations contains the tolerations for taints on seed clusters.

FieldDescription
defaults
[]Toleration
(Optional)

Defaults contains a list of tolerations that are added to the shoots in this project by default.

whitelist
[]Toleration
(Optional)

Whitelist contains a list of tolerations that are allowed to be added to the shoots in this project. Please note that this list may only be added by users having the spec-tolerations-whitelist verb for project resources.

Provider

(Appears on: ShootSpec)

Provider contains provider-specific information that are handed-over to the provider-specific extension controller.

FieldDescription
type
string

Type is the type of the provider. This field is immutable.

controlPlaneConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ControlPlaneConfig contains the provider-specific control plane config blob. Please look up the concrete definition in the documentation of your provider extension.

infrastructureConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

InfrastructureConfig contains the provider-specific infrastructure config blob. Please look up the concrete definition in the documentation of your provider extension.

workers
[]Worker

Workers is a list of worker groups.

ProxyMode (string alias)

(Appears on: KubeProxyConfig)

ProxyMode available in Linux platform: ‘userspace’ (older, going to be EOL), ‘iptables’ (newer, faster), ‘ipvs’ (newest, better in performance and scalability). As of now only ‘iptables’ and ‘ipvs’ is supported by Gardener. In Linux platform, if the iptables proxy is selected, regardless of how, but the system’s kernel or iptables versions are insufficient, this always falls back to the userspace proxy. IPVS mode will be enabled when proxy mode is set to ‘ipvs’, and the fall back path is firstly iptables and then userspace.

QuotaSpec

(Appears on: Quota)

QuotaSpec is the specification of a Quota.

FieldDescription
clusterLifetimeDays
int32
(Optional)

ClusterLifetimeDays is the lifetime of a Shoot cluster in days before it will be terminated automatically.

metrics
Kubernetes core/v1.ResourceList

Metrics is a list of resources which will be put under constraints.

scope
Kubernetes core/v1.ObjectReference

Scope is the scope of the Quota object, either ‘project’ or ‘secret’. This field is immutable.

Region

(Appears on: CloudProfileSpec)

Region contains certain properties of a region.

FieldDescription
name
string

Name is a region name.

zones
[]AvailabilityZone
(Optional)

Zones is a list of availability zones in this region.

labels
map[string]string
(Optional)

Labels is an optional set of key-value pairs that contain certain administrator-controlled labels for this region. It can be used by Gardener administrators/operators to provide additional information about a region, e.g. wrt quality, reliability, access restrictions, etc.

ResourceWatchCacheSize

(Appears on: WatchCacheSizes)

ResourceWatchCacheSize contains configuration of the API server’s watch cache size for one specific resource.

FieldDescription
apiGroup
string
(Optional)

APIGroup is the API group of the resource for which the watch cache size should be configured. An unset value is used to specify the legacy core API (e.g. for secrets).

resource
string

Resource is the name of the resource for which the watch cache size should be configured (in lowercase plural form, e.g. secrets).

size
int32

CacheSize specifies the watch cache size that should be configured for the specified resource.

SchedulingProfile (string alias)

(Appears on: KubeSchedulerConfig)

SchedulingProfile is a string alias used for scheduling profile values.

SecretBindingProvider

(Appears on: SecretBinding)

SecretBindingProvider defines the provider type of the SecretBinding.

FieldDescription
type
string

Type is the type of the provider.

For backwards compatibility, the field can contain multiple providers separated by a comma. However the usage of single SecretBinding (hence Secret) for different cloud providers is strongly discouraged.

SeedBackup

(Appears on: SeedSpec)

SeedBackup contains the object store configuration for backups for shoot (currently only etcd).

FieldDescription
provider
string

Provider is a provider name. This field is immutable.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig is the configuration passed to BackupBucket resource.

region
string
(Optional)

Region is a region name. This field is immutable.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a Secret object containing the cloud provider credentials for the object store where backups should be stored. It should have enough privileges to manipulate the objects as well as buckets.

SeedDNS

(Appears on: SeedSpec)

SeedDNS contains DNS-relevant information about this seed cluster.

FieldDescription
ingressDomain
string
(Optional)

IngressDomain is the domain of the Seed cluster pointing to the ingress controller endpoint. It will be used to construct ingress URLs for system applications running in Shoot clusters. This field is immutable. This will be removed in the next API version and replaced by spec.ingress.domain.

provider
SeedDNSProvider
(Optional)

Provider configures a DNSProvider

SeedDNSProvider

(Appears on: SeedDNS)

SeedDNSProvider configures a DNSProvider for Seeds

FieldDescription
type
string

Type describes the type of the dns-provider, for example aws-route53

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a Secret object containing cloud provider credentials used for registering external domains.

domains
DNSIncludeExclude
(Optional)

Domains contains information about which domains shall be included/excluded for this provider.

zones
DNSIncludeExclude
(Optional)

Zones contains information about which hosted zones shall be included/excluded for this provider.

SeedNetworks

(Appears on: SeedSpec)

SeedNetworks contains CIDRs for the pod, service and node networks of a Kubernetes cluster.

FieldDescription
nodes
string
(Optional)

Nodes is the CIDR of the node network. This field is immutable.

pods
string

Pods is the CIDR of the pod network. This field is immutable.

services
string

Services is the CIDR of the service network. This field is immutable.

shootDefaults
ShootNetworks
(Optional)

ShootDefaults contains the default networks CIDRs for shoots.

blockCIDRs
[]string
(Optional)

BlockCIDRs is a list of network addresses that should be blocked for shoot control plane components running in the seed cluster.

SeedProvider

(Appears on: SeedSpec)

SeedProvider defines the provider type and region for this Seed cluster.

FieldDescription
type
string

Type is the name of the provider.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig is the configuration passed to Seed resource.

region
string

Region is a name of a region.

SeedSelector

(Appears on: CloudProfileSpec, ShootSpec)

SeedSelector contains constraints for selecting seed to be usable for shoots using a profile

FieldDescription
LabelSelector
Kubernetes meta/v1.LabelSelector

(Members of LabelSelector are embedded into this type.)

(Optional)

LabelSelector is optional and can be used to select seeds by their label settings

providerTypes
[]string
(Optional)

Providers is optional and can be used by restricting seeds by their provider type. ‘*’ can be used to enable seeds regardless of their provider type.

SeedSettingDependencyWatchdog

(Appears on: SeedSettings)

SeedSettingDependencyWatchdog controls the dependency-watchdog settings for the seed.

FieldDescription
endpoint
SeedSettingDependencyWatchdogEndpoint
(Optional)

Endpoint controls the endpoint settings for the dependency-watchdog for the seed.

probe
SeedSettingDependencyWatchdogProbe
(Optional)

Probe controls the probe settings for the dependency-watchdog for the seed.

SeedSettingDependencyWatchdogEndpoint

(Appears on: SeedSettingDependencyWatchdog)

SeedSettingDependencyWatchdogEndpoint controls the endpoint settings for the dependency-watchdog for the seed.

FieldDescription
enabled
bool

Enabled controls whether the endpoint controller of the dependency-watchdog should be enabled. This controller helps to alleviate the delay where control plane components remain unavailable by finding the respective pods in CrashLoopBackoff status and restarting them once their dependants become ready and available again.

SeedSettingDependencyWatchdogProbe

(Appears on: SeedSettingDependencyWatchdog)

SeedSettingDependencyWatchdogProbe controls the probe settings for the dependency-watchdog for the seed.

FieldDescription
enabled
bool

Enabled controls whether the probe controller of the dependency-watchdog should be enabled. This controller scales down the kube-controller-manager of shoot clusters in case their respective kube-apiserver is not reachable via its external ingress in order to avoid melt-down situations.

SeedSettingExcessCapacityReservation

(Appears on: SeedSettings)

SeedSettingExcessCapacityReservation controls the excess capacity reservation for shoot control planes in the seed.

FieldDescription
enabled
bool

Enabled controls whether the excess capacity reservation should be enabled.

SeedSettingLoadBalancerServices

(Appears on: SeedSettings)

SeedSettingLoadBalancerServices controls certain settings for services of type load balancer that are created in the seed.

FieldDescription
annotations
map[string]string
(Optional)

Annotations is a map of annotations that will be injected/merged into every load balancer service object.

SeedSettingOwnerChecks

(Appears on: SeedSettings)

SeedSettingOwnerChecks controls certain owner checks settings for shoots scheduled on this seed.

FieldDescription
enabled
bool

Enabled controls whether owner checks are enabled for shoots scheduled on this seed. It is enabled by default because it is a prerequisite for control plane migration.

SeedSettingScheduling

(Appears on: SeedSettings)

SeedSettingScheduling controls settings for scheduling decisions for the seed.

FieldDescription
visible
bool

Visible controls whether the gardener-scheduler shall consider this seed when scheduling shoots. Invisible seeds are not considered by the scheduler.

SeedSettingShootDNS

(Appears on: SeedSettings)

SeedSettingShootDNS controls the shoot DNS settings for the seed.

FieldDescription
enabled
bool

Enabled controls whether the DNS for shoot clusters should be enabled. When disabled then all shoots using the seed won’t get any DNS providers, DNS records, and no DNS extension controller is required to be installed here. This is useful for environments where DNS is not required.

SeedSettingVerticalPodAutoscaler

(Appears on: SeedSettings)

SeedSettingVerticalPodAutoscaler controls certain settings for the vertical pod autoscaler components deployed in the seed.

FieldDescription
enabled
bool

Enabled controls whether the VPA components shall be deployed into the garden namespace in the seed cluster. It is enabled by default because Gardener heavily relies on a VPA being deployed. You should only disable this if your seed cluster already has another, manually/custom managed VPA deployment.

SeedSettings

(Appears on: SeedSpec)

SeedSettings contains certain settings for this seed cluster.

FieldDescription
excessCapacityReservation
SeedSettingExcessCapacityReservation
(Optional)

ExcessCapacityReservation controls the excess capacity reservation for shoot control planes in the seed.

scheduling
SeedSettingScheduling
(Optional)

Scheduling controls settings for scheduling decisions for the seed.

shootDNS
SeedSettingShootDNS
(Optional)

ShootDNS controls the shoot DNS settings for the seed. Deprecated: This field is deprecated and will be removed in a future version of Gardener. Do not use it.

loadBalancerServices
SeedSettingLoadBalancerServices
(Optional)

LoadBalancerServices controls certain settings for services of type load balancer that are created in the seed.

verticalPodAutoscaler
SeedSettingVerticalPodAutoscaler
(Optional)

VerticalPodAutoscaler controls certain settings for the vertical pod autoscaler components deployed in the seed.

ownerChecks
SeedSettingOwnerChecks
(Optional)

SeedSettingOwnerChecks controls certain owner checks settings for shoots scheduled on this seed.

dependencyWatchdog
SeedSettingDependencyWatchdog
(Optional)

DependencyWatchdog controls certain settings for the dependency-watchdog components deployed in the seed.

SeedSpec

(Appears on: Seed, SeedTemplate)

SeedSpec is the specification of a Seed.

FieldDescription
backup
SeedBackup
(Optional)

Backup holds the object store configuration for the backups of shoot (currently only etcd). If it is not specified, then there won’t be any backups taken for shoots associated with this seed. If backup field is present in seed, then backups of the etcd from shoot control plane will be stored under the configured object store.

dns
SeedDNS

DNS contains DNS-relevant information about this seed cluster.

networks
SeedNetworks

Networks defines the pod, service and worker network of the Seed cluster.

provider
SeedProvider

Provider defines the provider type and region for this Seed cluster.

secretRef
Kubernetes core/v1.SecretReference
(Optional)

SecretRef is a reference to a Secret object containing the Kubeconfig of the Kubernetes cluster to be registered as Seed.

taints
[]SeedTaint
(Optional)

Taints describes taints on the seed.

volume
SeedVolume
(Optional)

Volume contains settings for persistentvolumes created in the seed cluster.

settings
SeedSettings
(Optional)

Settings contains certain settings for this seed cluster.

ingress
Ingress
(Optional)

Ingress configures Ingress specific settings of the Seed cluster. This field is immutable.

SeedStatus

(Appears on: Seed)

SeedStatus is the status of a Seed.

FieldDescription
gardener
Gardener
(Optional)

Gardener holds information about the Gardener which last acted on the Shoot.

kubernetesVersion
string
(Optional)

KubernetesVersion is the Kubernetes version of the seed cluster.

conditions
[]Condition
(Optional)

Conditions represents the latest available observations of a Seed’s current state.

observedGeneration
int64
(Optional)

ObservedGeneration is the most recent generation observed for this Seed. It corresponds to the Seed’s generation, which is updated on mutation by the API Server.

clusterIdentity
string
(Optional)

ClusterIdentity is the identity of the Seed cluster. This field is immutable.

capacity
Kubernetes core/v1.ResourceList
(Optional)

Capacity represents the total resources of a seed.

allocatable
Kubernetes core/v1.ResourceList
(Optional)

Allocatable represents the resources of a seed that are available for scheduling. Defaults to Capacity.

clientCertificateExpirationTimestamp
Kubernetes meta/v1.Time
(Optional)

ClientCertificateExpirationTimestamp is the timestamp at which gardenlet’s client certificate expires.

SeedTaint

(Appears on: SeedSpec)

SeedTaint describes a taint on a seed.

FieldDescription
key
string

Key is the taint key to be applied to a seed.

value
string
(Optional)

Value is the taint value corresponding to the taint key.

SeedTemplate

SeedTemplate is a template for creating a Seed object.

FieldDescription
metadata
Kubernetes meta/v1.ObjectMeta
(Optional)

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
SeedSpec
(Optional)

Specification of the desired behavior of the Seed.



backup
SeedBackup
(Optional)

Backup holds the object store configuration for the backups of shoot (currently only etcd). If it is not specified, then there won’t be any backups taken for shoots associated with this seed. If backup field is present in seed, then backups of the etcd from shoot control plane will be stored under the configured object store.

dns
SeedDNS

DNS contains DNS-relevant information about this seed cluster.

networks
SeedNetworks

Networks defines the pod, service and worker network of the Seed cluster.

provider
SeedProvider

Provider defines the provider type and region for this Seed cluster.

secretRef
Kubernetes core/v1.SecretReference
(Optional)

SecretRef is a reference to a Secret object containing the Kubeconfig of the Kubernetes cluster to be registered as Seed.

taints
[]SeedTaint
(Optional)

Taints describes taints on the seed.

volume
SeedVolume
(Optional)

Volume contains settings for persistentvolumes created in the seed cluster.

settings
SeedSettings
(Optional)

Settings contains certain settings for this seed cluster.

ingress
Ingress
(Optional)

Ingress configures Ingress specific settings of the Seed cluster. This field is immutable.

SeedVolume

(Appears on: SeedSpec)

SeedVolume contains settings for persistentvolumes created in the seed cluster.

FieldDescription
minimumSize
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

MinimumSize defines the minimum size that should be used for PVCs in the seed.

providers
[]SeedVolumeProvider
(Optional)

Providers is a list of storage class provisioner types for the seed.

SeedVolumeProvider

(Appears on: SeedVolume)

SeedVolumeProvider is a storage class provisioner type.

FieldDescription
purpose
string

Purpose is the purpose of this provider.

name
string

Name is the name of the storage class provisioner type.

ServiceAccountConfig

(Appears on: KubeAPIServerConfig)

ServiceAccountConfig is the kube-apiserver configuration for service accounts.

FieldDescription
issuer
string
(Optional)

Issuer is the identifier of the service account token issuer. The issuer will assert this identifier in “iss” claim of issued tokens. This value is used to generate new service account tokens. This value is a string or URI. Defaults to URI of the API server.

signingKeySecretName
Kubernetes core/v1.LocalObjectReference
(Optional)

SigningKeySecret is a reference to a secret that contains an optional private key of the service account token issuer. The issuer will sign issued ID tokens with this private key. Only useful if service account tokens are also issued by another external system. Deprecated: This field is deprecated and will be removed in a future version of Gardener. Do not use it.

extendTokenExpiration
bool
(Optional)

ExtendTokenExpiration turns on projected service account expiration extension during token generation, which helps safe transition from legacy token to bound service account token feature. If this flag is enabled, admission injected tokens would be extended up to 1 year to prevent unexpected failure during transition, ignoring value of service-account-max-token-expiration.

maxTokenExpiration
Kubernetes meta/v1.Duration
(Optional)

MaxTokenExpiration is the maximum validity duration of a token created by the service account token issuer. If an otherwise valid TokenRequest with a validity duration larger than this value is requested, a token will be issued with a validity duration of this value. This field must be within [30d,90d].

acceptedIssuers
[]string
(Optional)

AcceptedIssuers is an additional set of issuers that are used to determine which service account tokens are accepted. These values are not used to generate new service account tokens. Only useful when service account tokens are also issued by another external system or a change of the current issuer that is used for generating tokens is being performed. This field is only available for Kubernetes v1.22 or later.

ShootAdvertisedAddress

(Appears on: ShootStatus)

ShootAdvertisedAddress contains information for the shoot’s Kube API server.

FieldDescription
name
string

Name of the advertised address. e.g. external

url
string

The URL of the API Server. e.g. https://api.foo.bar or https://1.2.3.4

ShootCARotation

(Appears on: ShootCredentialsRotation)

ShootCARotation contains information about the certificate authority credential rotation.

FieldDescription
phase
ShootCredentialsRotationPhase

Phase describes the phase of the certificate authority credential rotation.

lastInitiationTime
Kubernetes meta/v1.Time
(Optional)

LastInitiationTime is the most recent time when the certificate authority credential rotation was initiated.

lastCompletionTime
Kubernetes meta/v1.Time
(Optional)

LastCompletionTime is the most recent time when the certificate authority credential rotation was successfully completed.

ShootCredentials

(Appears on: ShootStatus)

ShootCredentials contains information about the shoot credentials.

FieldDescription
rotation
ShootCredentialsRotation
(Optional)

Rotation contains information about the credential rotations.

ShootCredentialsRotation

(Appears on: ShootCredentials)

ShootCredentialsRotation contains information about the rotation of credentials.

FieldDescription
certificateAuthorities
ShootCARotation
(Optional)

CertificateAuthorities contains information about the certificate authority credential rotation.

kubeconfig
ShootKubeconfigRotation
(Optional)

Kubeconfig contains information about the kubeconfig credential rotation.

sshKeypair
ShootSSHKeypairRotation
(Optional)

SSHKeypair contains information about the ssh-keypair credential rotation.

observability
ShootObservabilityRotation
(Optional)

Observability contains information about the observability credential rotation.

serviceAccountKey
ShootServiceAccountKeyRotation
(Optional)

ServiceAccountKey contains information about the service account key credential rotation.

etcdEncryptionKey
ShootETCDEncryptionKeyRotation
(Optional)

ETCDEncryptionKey contains information about the ETCD encryption key credential rotation.

ShootCredentialsRotationPhase (string alias)

(Appears on: ShootCARotation, ShootETCDEncryptionKeyRotation, ShootServiceAccountKeyRotation)

ShootCredentialsRotationPhase is a string alias.

ShootETCDEncryptionKeyRotation

(Appears on: ShootCredentialsRotation)

ShootETCDEncryptionKeyRotation contains information about the ETCD encryption key credential rotation.

FieldDescription
phase
ShootCredentialsRotationPhase

Phase describes the phase of the ETCD encryption key credential rotation.

lastInitiationTime
Kubernetes meta/v1.Time
(Optional)

LastInitiationTime is the most recent time when the ETCD encryption key credential rotation was initiated.

lastCompletionTime
Kubernetes meta/v1.Time
(Optional)

LastCompletionTime is the most recent time when the ETCD encryption key credential rotation was successfully completed.

ShootKubeconfigRotation

(Appears on: ShootCredentialsRotation)

ShootKubeconfigRotation contains information about the kubeconfig credential rotation.

FieldDescription
lastInitiationTime
Kubernetes meta/v1.Time
(Optional)

LastInitiationTime is the most recent time when the kubeconfig credential rotation was initiated.

lastCompletionTime
Kubernetes meta/v1.Time
(Optional)

LastCompletionTime is the most recent time when the kubeconfig credential rotation was successfully completed.

ShootMachineImage

(Appears on: Machine)

ShootMachineImage defines the name and the version of the shoot’s machine image in any environment. Has to be defined in the respective CloudProfile.

FieldDescription
name
string

Name is the name of the image.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig is the shoot’s individual configuration passed to an extension resource.

version
string
(Optional)

Version is the version of the shoot’s image. If version is not provided, it will be defaulted to the latest version from the CloudProfile.

ShootNetworks

(Appears on: SeedNetworks)

ShootNetworks contains the default networks CIDRs for shoots.

FieldDescription
pods
string
(Optional)

Pods is the CIDR of the pod network.

services
string
(Optional)

Services is the CIDR of the service network.

ShootObservabilityRotation

(Appears on: ShootCredentialsRotation)

ShootObservabilityRotation contains information about the observability credential rotation.

FieldDescription
lastInitiationTime
Kubernetes meta/v1.Time
(Optional)

LastInitiationTime is the most recent time when the observability credential rotation was initiated.

lastCompletionTime
Kubernetes meta/v1.Time
(Optional)

LastCompletionTime is the most recent time when the observability credential rotation was successfully completed.

ShootPurpose (string alias)

(Appears on: ShootSpec)

ShootPurpose is a type alias for string.

ShootSSHKeypairRotation

(Appears on: ShootCredentialsRotation)

ShootSSHKeypairRotation contains information about the ssh-keypair credential rotation.

FieldDescription
lastInitiationTime
Kubernetes meta/v1.Time
(Optional)

LastInitiationTime is the most recent time when the ssh-keypair credential rotation was initiated.

lastCompletionTime
Kubernetes meta/v1.Time
(Optional)

LastCompletionTime is the most recent time when the ssh-keypair credential rotation was successfully completed.

ShootServiceAccountKeyRotation

(Appears on: ShootCredentialsRotation)

ShootServiceAccountKeyRotation contains information about the service account key credential rotation.

FieldDescription
phase
ShootCredentialsRotationPhase

Phase describes the phase of the service account key credential rotation.

lastInitiationTime
Kubernetes meta/v1.Time
(Optional)

LastInitiationTime is the most recent time when the service account key credential rotation was initiated.

lastCompletionTime
Kubernetes meta/v1.Time
(Optional)

LastCompletionTime is the most recent time when the service account key credential rotation was successfully completed.

ShootSpec

(Appears on: Shoot, ShootTemplate)

ShootSpec is the specification of a Shoot.

FieldDescription
addons
Addons
(Optional)

Addons contains information about enabled/disabled addons and their configuration.

cloudProfileName
string

CloudProfileName is a name of a CloudProfile object. This field is immutable.

dns
DNS
(Optional)

DNS contains information about the DNS settings of the Shoot.

extensions
[]Extension
(Optional)

Extensions contain type and provider information for Shoot extensions.

hibernation
Hibernation
(Optional)

Hibernation contains information whether the Shoot is suspended or not.

kubernetes
Kubernetes

Kubernetes contains the version and configuration settings of the control plane components.

networking
Networking

Networking contains information about cluster networking such as CNI Plugin type, CIDRs, …etc.

maintenance
Maintenance
(Optional)

Maintenance contains information about the time window for maintenance operations and which operations should be performed.

monitoring
Monitoring
(Optional)

Monitoring contains information about custom monitoring configurations for the shoot.

provider
Provider

Provider contains all provider-specific and provider-relevant information.

purpose
ShootPurpose
(Optional)

Purpose is the purpose class for this cluster.

region
string

Region is a name of a region. This field is immutable.

secretBindingName
string

SecretBindingName is the name of the a SecretBinding that has a reference to the provider secret. The credentials inside the provider secret will be used to create the shoot in the respective account. This field is immutable.

seedName
string
(Optional)

SeedName is the name of the seed cluster that runs the control plane of the Shoot. This field is immutable when the SeedChange feature gate is disabled.

seedSelector
SeedSelector
(Optional)

SeedSelector is an optional selector which must match a seed’s labels for the shoot to be scheduled on that seed.

resources
[]NamedResourceReference
(Optional)

Resources holds a list of named resource references that can be referred to in extension configs by their names.

tolerations
[]Toleration
(Optional)

Tolerations contains the tolerations for taints on seed clusters.

exposureClassName
string
(Optional)

ExposureClassName is the optional name of an exposure class to apply a control plane endpoint exposure strategy. This field is immutable.

systemComponents
SystemComponents
(Optional)

SystemComponents contains the settings of system components in the control or data plane of the Shoot cluster.

controlPlane
ControlPlane
(Optional)

ControlPlane contains general settings for the control plane of the shoot.

ShootStatus

(Appears on: Shoot)

ShootStatus holds the most recently observed status of the Shoot cluster.

FieldDescription
conditions
[]Condition
(Optional)

Conditions represents the latest available observations of a Shoots’s current state.

constraints
[]Condition
(Optional)

Constraints represents conditions of a Shoot’s current state that constraint some operations on it.

gardener
Gardener

Gardener holds information about the Gardener which last acted on the Shoot.

hibernated
bool

IsHibernated indicates whether the Shoot is currently hibernated.

lastOperation
LastOperation
(Optional)

LastOperation holds information about the last operation on the Shoot.

lastErrors
[]LastError
(Optional)

LastErrors holds information about the last occurred error(s) during an operation.

observedGeneration
int64
(Optional)

ObservedGeneration is the most recent generation observed for this Shoot. It corresponds to the Shoot’s generation, which is updated on mutation by the API Server.

retryCycleStartTime
Kubernetes meta/v1.Time
(Optional)

RetryCycleStartTime is the start time of the last retry cycle (used to determine how often an operation must be retried until we give up).

seedName
string
(Optional)

SeedName is the name of the seed cluster that runs the control plane of the Shoot. This value is only written after a successful create/reconcile operation. It will be used when control planes are moved between Seeds.

technicalID
string

TechnicalID is the name that is used for creating the Seed namespace, the infrastructure resources, and basically everything that is related to this particular Shoot. This field is immutable.

uid
k8s.io/apimachinery/pkg/types.UID

UID is a unique identifier for the Shoot cluster to avoid portability between Kubernetes clusters. It is used to compute unique hashes. This field is immutable.

clusterIdentity
string
(Optional)

ClusterIdentity is the identity of the Shoot cluster. This field is immutable.

advertisedAddresses
[]ShootAdvertisedAddress
(Optional)

List of addresses on which the Kube API server can be reached.

migrationStartTime
Kubernetes meta/v1.Time
(Optional)

MigrationStartTime is the time when a migration to a different seed was initiated.

credentials
ShootCredentials
(Optional)

Credentials contains information about the shoot credentials.

lastHibernationTriggerTime
Kubernetes meta/v1.Time
(Optional)

LastHibernationTriggerTime indicates the last time when the hibernation controller managed to change the hibernation settings of the cluster

ShootTemplate

ShootTemplate is a template for creating a Shoot object.

FieldDescription
metadata
Kubernetes meta/v1.ObjectMeta
(Optional)

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ShootSpec
(Optional)

Specification of the desired behavior of the Shoot.



addons
Addons
(Optional)

Addons contains information about enabled/disabled addons and their configuration.

cloudProfileName
string

CloudProfileName is a name of a CloudProfile object. This field is immutable.

dns
DNS
(Optional)

DNS contains information about the DNS settings of the Shoot.

extensions
[]Extension
(Optional)

Extensions contain type and provider information for Shoot extensions.

hibernation
Hibernation
(Optional)

Hibernation contains information whether the Shoot is suspended or not.

kubernetes
Kubernetes

Kubernetes contains the version and configuration settings of the control plane components.

networking
Networking

Networking contains information about cluster networking such as CNI Plugin type, CIDRs, …etc.

maintenance
Maintenance
(Optional)

Maintenance contains information about the time window for maintenance operations and which operations should be performed.

monitoring
Monitoring
(Optional)

Monitoring contains information about custom monitoring configurations for the shoot.

provider
Provider

Provider contains all provider-specific and provider-relevant information.

purpose
ShootPurpose
(Optional)

Purpose is the purpose class for this cluster.

region
string

Region is a name of a region. This field is immutable.

secretBindingName
string

SecretBindingName is the name of the a SecretBinding that has a reference to the provider secret. The credentials inside the provider secret will be used to create the shoot in the respective account. This field is immutable.

seedName
string
(Optional)

SeedName is the name of the seed cluster that runs the control plane of the Shoot. This field is immutable when the SeedChange feature gate is disabled.

seedSelector
SeedSelector
(Optional)

SeedSelector is an optional selector which must match a seed’s labels for the shoot to be scheduled on that seed.

resources
[]NamedResourceReference
(Optional)

Resources holds a list of named resource references that can be referred to in extension configs by their names.

tolerations
[]Toleration
(Optional)

Tolerations contains the tolerations for taints on seed clusters.

exposureClassName
string
(Optional)

ExposureClassName is the optional name of an exposure class to apply a control plane endpoint exposure strategy. This field is immutable.

systemComponents
SystemComponents
(Optional)

SystemComponents contains the settings of system components in the control or data plane of the Shoot cluster.

controlPlane
ControlPlane
(Optional)

ControlPlane contains general settings for the control plane of the shoot.

SystemComponents

(Appears on: ShootSpec)

SystemComponents contains the settings of system components in the control or data plane of the Shoot cluster.

FieldDescription
coreDNS
CoreDNS
(Optional)

CoreDNS contains the settings of the Core DNS components running in the data plane of the Shoot cluster.

nodeLocalDNS
NodeLocalDNS
(Optional)

NodeLocalDNS contains the settings of the node local DNS components running in the data plane of the Shoot cluster.

Toleration

(Appears on: ProjectTolerations, ShootSpec)

Toleration is a toleration for a seed taint.

FieldDescription
key
string

Key is the toleration key to be applied to a project or shoot.

value
string
(Optional)

Value is the toleration value corresponding to the toleration key.

VersionClassification (string alias)

(Appears on: ExpirableVersion)

VersionClassification is the logical state of a version.

VerticalPodAutoscaler

(Appears on: Kubernetes)

VerticalPodAutoscaler contains the configuration flags for the Kubernetes vertical pod autoscaler.

FieldDescription
enabled
bool

Enabled specifies whether the Kubernetes VPA shall be enabled for the shoot cluster.

evictAfterOOMThreshold
Kubernetes meta/v1.Duration
(Optional)

EvictAfterOOMThreshold defines the threshold that will lead to pod eviction in case it OOMed in less than the given threshold since its start and if it has only one container (default: 10m0s).

evictionRateBurst
int32
(Optional)

EvictionRateBurst defines the burst of pods that can be evicted (default: 1)

evictionRateLimit
float64
(Optional)

EvictionRateLimit defines the number of pods that can be evicted per second. A rate limit set to 0 or -1 will disable the rate limiter (default: -1).

evictionTolerance
float64
(Optional)

EvictionTolerance defines the fraction of replica count that can be evicted for update in case more than one pod can be evicted (default: 0.5).

recommendationMarginFraction
float64
(Optional)

RecommendationMarginFraction is the fraction of usage added as the safety margin to the recommended request (default: 0.15).

updaterInterval
Kubernetes meta/v1.Duration
(Optional)

UpdaterInterval is the interval how often the updater should run (default: 1m0s).

recommenderInterval
Kubernetes meta/v1.Duration
(Optional)

RecommenderInterval is the interval how often metrics should be fetched (default: 1m0s).

Volume

(Appears on: Worker)

Volume contains information about the volume type, size, and encryption.

FieldDescription
name
string
(Optional)

Name of the volume to make it referencable.

type
string
(Optional)

Type is the type of the volume.

size
string

VolumeSize is the size of the volume.

encrypted
bool
(Optional)

Encrypted determines if the volume should be encrypted.

VolumeType

(Appears on: CloudProfileSpec)

VolumeType contains certain properties of a volume type.

FieldDescription
class
string

Class is the class of the volume type.

name
string

Name is the name of the volume type.

usable
bool
(Optional)

Usable defines if the volume type can be used for shoot clusters.

minSize
k8s.io/apimachinery/pkg/api/resource.Quantity
(Optional)

MinSize is the minimal supported storage size.

WatchCacheSizes

(Appears on: KubeAPIServerConfig)

WatchCacheSizes contains configuration of the API server’s watch cache sizes.

FieldDescription
default
int32
(Optional)

Default configures the default watch cache size of the kube-apiserver (flag --default-watch-cache-size, defaults to 100). See: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

resources
[]ResourceWatchCacheSize
(Optional)

Resources configures the watch cache size of the kube-apiserver per resource (flag --watch-cache-sizes). See: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

Worker

(Appears on: Provider)

Worker is the base definition of a worker group.

FieldDescription
annotations
map[string]string
(Optional)

Annotations is a map of key/value pairs for annotations for all the Node objects in this worker pool.

caBundle
string
(Optional)

CABundle is a certificate bundle which will be installed onto every machine of this worker pool.

cri
CRI
(Optional)

CRI contains configurations of CRI support of every machine in the worker pool. Defaults to a CRI with name containerd when the Kubernetes version of the Shoot is >= 1.22.

kubernetes
WorkerKubernetes
(Optional)

Kubernetes contains configuration for Kubernetes components related to this worker pool.

labels
map[string]string
(Optional)

Labels is a map of key/value pairs for labels for all the Node objects in this worker pool.

name
string

Name is the name of the worker group.

machine
Machine

Machine contains information about the machine type and image.

maximum
int32

Maximum is the maximum number of VMs to create.

minimum
int32

Minimum is the minimum number of VMs to create.

maxSurge
k8s.io/apimachinery/pkg/util/intstr.IntOrString
(Optional)

MaxSurge is maximum number of VMs that are created during an update.

maxUnavailable
k8s.io/apimachinery/pkg/util/intstr.IntOrString
(Optional)

MaxUnavailable is the maximum number of VMs that can be unavailable during an update.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig is the provider-specific configuration for this worker pool.

taints
[]Kubernetes core/v1.Taint
(Optional)

Taints is a list of taints for all the Node objects in this worker pool.

volume
Volume
(Optional)

Volume contains information about the volume type and size.

dataVolumes
[]DataVolume
(Optional)

DataVolumes contains a list of additional worker volumes.

kubeletDataVolumeName
string
(Optional)

KubeletDataVolumeName contains the name of a dataVolume that should be used for storing kubelet state.

zones
[]string
(Optional)

Zones is a list of availability zones that are used to evenly distribute this worker pool. Optional as not every provider may support availability zones.

systemComponents
WorkerSystemComponents
(Optional)

SystemComponents contains configuration for system components related to this worker pool

machineControllerManager
MachineControllerManagerSettings
(Optional)

MachineControllerManagerSettings contains configurations for different worker-pools. Eg. MachineDrainTimeout, MachineHealthTimeout.

WorkerKubernetes

(Appears on: Worker)

WorkerKubernetes contains configuration for Kubernetes components related to this worker pool.

FieldDescription
kubelet
KubeletConfig
(Optional)

Kubelet contains configuration settings for all kubelets of this worker pool. If set, all spec.kubernetes.kubelet settings will be overwritten for this worker pool (no merge of settings).

version
string
(Optional)

Version is the semantic Kubernetes version to use for the Kubelet in this Worker Group. If not specified the kubelet version is derived from the global shoot cluster kubernetes version. version must be equal or lower than the version of the shoot kubernetes version. Only one minor version difference to other worker groups and global kubernetes version is allowed.

WorkerSystemComponents

(Appears on: Worker)

WorkerSystemComponents contains configuration for system components related to this worker pool

FieldDescription
allow
bool

Allow determines whether the pool should be allowed to host system components or not (defaults to true)


Generated with gen-crd-api-reference-docs

1.1.3 - Extensions

Packages:

extensions.gardener.cloud/v1alpha1

Package v1alpha1 is the v1alpha1 version of the API.

Resource Types:

BackupBucket

BackupBucket is a specification for backup bucket.

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
BackupBucket
metadata
Kubernetes meta/v1.ObjectMeta
(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
BackupBucketSpec

Specification of the BackupBucket. If the object’s deletion timestamp is set, this field is immutable.



DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

region
string

Region is the region of this bucket. This field is immutable.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the credentials to access object store.

status
BackupBucketStatus
(Optional)

BackupEntry

BackupEntry is a specification for backup Entry.

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
BackupEntry
metadata
Kubernetes meta/v1.ObjectMeta
(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
BackupEntrySpec

Specification of the BackupEntry. If the object’s deletion timestamp is set, this field is immutable.



DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

backupBucketProviderStatus
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

BackupBucketProviderStatus contains the provider status that has been generated by the controller responsible for the BackupBucket resource.

region
string

Region is the region of this Entry. This field is immutable.

bucketName
string

BucketName is the name of backup bucket for this Backup Entry.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the credentials to access object store.

status
BackupEntryStatus
(Optional)

Bastion

Bastion is a bastion or jump host that is dynamically created to provide SSH access to shoot nodes.

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
Bastion
metadata
Kubernetes meta/v1.ObjectMeta
(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
BastionSpec

Spec is the specification of this Bastion. If the object’s deletion timestamp is set, this field is immutable.



DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

userData
[]byte

UserData is the base64-encoded user data for the bastion instance. This should contain code to provision the SSH key on the bastion instance. This field is immutable.

ingress
[]BastionIngressPolicy

Ingress controls from where the created bastion host should be reachable.

status
BastionStatus
(Optional)

Status is the bastion’s status.

Cluster

Cluster is a specification for a Cluster resource.

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
Cluster
metadata
Kubernetes meta/v1.ObjectMeta
Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ClusterSpec


cloudProfile
k8s.io/apimachinery/pkg/runtime.RawExtension

CloudProfile is a raw extension field that contains the cloudprofile resource referenced by the shoot that has to be reconciled.

seed
k8s.io/apimachinery/pkg/runtime.RawExtension

Seed is a raw extension field that contains the seed resource referenced by the shoot that has to be reconciled.

shoot
k8s.io/apimachinery/pkg/runtime.RawExtension

Shoot is a raw extension field that contains the shoot resource that has to be reconciled.

ContainerRuntime

ContainerRuntime is a specification for a container runtime resource.

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
ContainerRuntime
metadata
Kubernetes meta/v1.ObjectMeta
(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ContainerRuntimeSpec

Specification of the ContainerRuntime. If the object’s deletion timestamp is set, this field is immutable.



binaryPath
string

BinaryPath is the Worker’s machine path where container runtime extensions should copy the binaries to.

workerPool
ContainerRuntimeWorkerPool

WorkerPool identifies the worker pool of the Shoot. For each worker pool and type, Gardener deploys a ContainerRuntime CRD.

DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

status
ContainerRuntimeStatus
(Optional)

ControlPlane

ControlPlane is a specification for a ControlPlane resource.

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
ControlPlane
metadata
Kubernetes meta/v1.ObjectMeta
Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ControlPlaneSpec

Specification of the ControlPlane. If the object’s deletion timestamp is set, this field is immutable.



DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

purpose
Purpose
(Optional)

Purpose contains the data if a cloud provider needs additional components in order to expose the control plane. This field is immutable.

infrastructureProviderStatus
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

InfrastructureProviderStatus contains the provider status that has been generated by the controller responsible for the Infrastructure resource.

region
string

Region is the region of this control plane. This field is immutable.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the cloud provider specific credentials.

status
ControlPlaneStatus
(Optional)

DNSRecord

DNSRecord is a specification for a DNSRecord resource.

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
DNSRecord
metadata
Kubernetes meta/v1.ObjectMeta
Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
DNSRecordSpec

Specification of the DNSRecord. If the object’s deletion timestamp is set, this field is immutable.



DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the cloud provider specific credentials.

region
string
(Optional)

Region is the region of this DNS record. If not specified, the region specified in SecretRef will be used. If that is also not specified, the extension controller will use its default region.

zone
string
(Optional)

Zone is the DNS hosted zone of this DNS record. If not specified, it will be determined automatically by getting all hosted zones of the account and searching for the longest zone name that is a suffix of Name.

name
string

Name is the fully qualified domain name, e.g. “api.”. This field is immutable.

recordType
DNSRecordType

RecordType is the DNS record type. Only A, CNAME, and TXT records are currently supported. This field is immutable.

values
[]string

Values is a list of IP addresses for A records, a single hostname for CNAME records, or a list of texts for TXT records.

ttl
int64
(Optional)

TTL is the time to live in seconds. Defaults to 120.

status
DNSRecordStatus
(Optional)

Extension

Extension is a specification for a Extension resource.

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
Extension
metadata
Kubernetes meta/v1.ObjectMeta
(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ExtensionSpec

Specification of the Extension. If the object’s deletion timestamp is set, this field is immutable.



DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

status
ExtensionStatus
(Optional)

Infrastructure

Infrastructure is a specification for cloud provider infrastructure.

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
Infrastructure
metadata
Kubernetes meta/v1.ObjectMeta
(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
InfrastructureSpec

Specification of the Infrastructure. If the object’s deletion timestamp is set, this field is immutable.



DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

region
string

Region is the region of this infrastructure. This field is immutable.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the actual result of the generated cloud config.

sshPublicKey
[]byte
(Optional)

SSHPublicKey is the public SSH key that should be used with this infrastructure.

status
InfrastructureStatus
(Optional)

Network

Network is the specification for cluster networking.

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
Network
metadata
Kubernetes meta/v1.ObjectMeta
(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
NetworkSpec

Specification of the Network. If the object’s deletion timestamp is set, this field is immutable.



DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

podCIDR
string

PodCIDR defines the CIDR that will be used for pods. This field is immutable.

serviceCIDR
string

ServiceCIDR defines the CIDR that will be used for services. This field is immutable.

status
NetworkStatus
(Optional)

OperatingSystemConfig

OperatingSystemConfig is a specification for a OperatingSystemConfig resource

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
OperatingSystemConfig
metadata
Kubernetes meta/v1.ObjectMeta
(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
OperatingSystemConfigSpec

Specification of the OperatingSystemConfig. If the object’s deletion timestamp is set, this field is immutable.



criConfig
CRIConfig
(Optional)

CRI config is a structure contains configurations of the CRI library

DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

purpose
OperatingSystemConfigPurpose

Purpose describes how the result of this OperatingSystemConfig is used by Gardener. Either it gets sent to the Worker extension controller to bootstrap a VM, or it is downloaded by the cloud-config-downloader script already running on a bootstrapped VM. This field is immutable.

reloadConfigFilePath
string
(Optional)

ReloadConfigFilePath is the path to the generated operating system configuration. If set, controllers are asked to use it when determining the .status.command of this resource. For example, if for CoreOS the reload-path might be “/var/lib/config”; then the controller shall set .status.command to “/usr/bin/coreos-cloudinit –from-file=/var/lib/config”.

units
[]Unit
(Optional)

Units is a list of unit for the operating system configuration (usually, a systemd unit).

files
[]File
(Optional)

Files is a list of files that should get written to the host’s file system.

status
OperatingSystemConfigStatus
(Optional)

Worker

Worker is a specification for a Worker resource.

FieldDescription
apiVersion
string
extensions.gardener.cloud/v1alpha1
kind
string
Worker
metadata
Kubernetes meta/v1.ObjectMeta
(Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
WorkerSpec

Specification of the Worker. If the object’s deletion timestamp is set, this field is immutable.



DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

infrastructureProviderStatus
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

InfrastructureProviderStatus is a raw extension field that contains the provider status that has been generated by the controller responsible for the Infrastructure resource.

region
string

Region is the name of the region where the worker pool should be deployed to. This field is immutable.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the cloud provider specific credentials.

sshPublicKey
[]byte
(Optional)

SSHPublicKey is the public SSH key that should be used with these workers.

pools
[]WorkerPool

Pools is a list of worker pools.

status
WorkerStatus
(Optional)

BackupBucketSpec

(Appears on: BackupBucket)

BackupBucketSpec is the spec for an BackupBucket resource.

FieldDescription
DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

region
string

Region is the region of this bucket. This field is immutable.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the credentials to access object store.

BackupBucketStatus

(Appears on: BackupBucket)

BackupBucketStatus is the status for an BackupBucket resource.

FieldDescription
DefaultStatus
DefaultStatus

(Members of DefaultStatus are embedded into this type.)

DefaultStatus is a structure containing common fields used by all extension resources.

generatedSecretRef
Kubernetes core/v1.SecretReference
(Optional)

GeneratedSecretRef is reference to the secret generated by backup bucket, which will have object store specific credentials.

BackupEntrySpec

(Appears on: BackupEntry)

BackupEntrySpec is the spec for an BackupEntry resource.

FieldDescription
DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

backupBucketProviderStatus
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

BackupBucketProviderStatus contains the provider status that has been generated by the controller responsible for the BackupBucket resource.

region
string

Region is the region of this Entry. This field is immutable.

bucketName
string

BucketName is the name of backup bucket for this Backup Entry.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the credentials to access object store.

BackupEntryStatus

(Appears on: BackupEntry)

BackupEntryStatus is the status for an BackupEntry resource.

FieldDescription
DefaultStatus
DefaultStatus

(Members of DefaultStatus are embedded into this type.)

DefaultStatus is a structure containing common fields used by all extension resources.

BastionIngressPolicy

(Appears on: BastionSpec)

BastionIngressPolicy represents an ingress policy for SSH bastion hosts.

FieldDescription
ipBlock
Kubernetes networking/v1.IPBlock

IPBlock defines an IP block that is allowed to access the bastion.

BastionSpec

(Appears on: Bastion)

BastionSpec contains the specification for an SSH bastion host.

FieldDescription
DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

userData
[]byte

UserData is the base64-encoded user data for the bastion instance. This should contain code to provision the SSH key on the bastion instance. This field is immutable.

ingress
[]BastionIngressPolicy

Ingress controls from where the created bastion host should be reachable.

BastionStatus

(Appears on: Bastion)

BastionStatus holds the most recently observed status of the Bastion.

FieldDescription
DefaultStatus
DefaultStatus

(Members of DefaultStatus are embedded into this type.)

DefaultStatus is a structure containing common fields used by all extension resources.

ingress
Kubernetes core/v1.LoadBalancerIngress
(Optional)

Ingress is the external IP and/or hostname of the bastion host.

CRIConfig

(Appears on: OperatingSystemConfigSpec)

CRIConfig contains configurations of the CRI library.

FieldDescription
name
CRIName

Name is a mandatory string containing the name of the CRI library. Supported values are docker and containerd.

CRIName (string alias)

(Appears on: CRIConfig)

CRIName is a type alias for the CRI name string.

CloudConfig

(Appears on: OperatingSystemConfigStatus)

CloudConfig contains the generated output for the given operating system config spec. It contains a reference to a secret as the result may contain confidential data.

FieldDescription
secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the actual result of the generated cloud config.

ClusterSpec

(Appears on: Cluster)

ClusterSpec is the spec for a Cluster resource.

FieldDescription
cloudProfile
k8s.io/apimachinery/pkg/runtime.RawExtension

CloudProfile is a raw extension field that contains the cloudprofile resource referenced by the shoot that has to be reconciled.

seed
k8s.io/apimachinery/pkg/runtime.RawExtension

Seed is a raw extension field that contains the seed resource referenced by the shoot that has to be reconciled.

shoot
k8s.io/apimachinery/pkg/runtime.RawExtension

Shoot is a raw extension field that contains the shoot resource that has to be reconciled.

ContainerRuntimeSpec

(Appears on: ContainerRuntime)

ContainerRuntimeSpec is the spec for a ContainerRuntime resource.

FieldDescription
binaryPath
string

BinaryPath is the Worker’s machine path where container runtime extensions should copy the binaries to.

workerPool
ContainerRuntimeWorkerPool

WorkerPool identifies the worker pool of the Shoot. For each worker pool and type, Gardener deploys a ContainerRuntime CRD.

DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

ContainerRuntimeStatus

(Appears on: ContainerRuntime)

ContainerRuntimeStatus is the status for a ContainerRuntime resource.

FieldDescription
DefaultStatus
DefaultStatus

(Members of DefaultStatus are embedded into this type.)

DefaultStatus is a structure containing common fields used by all extension resources.

ContainerRuntimeWorkerPool

(Appears on: ContainerRuntimeSpec)

ContainerRuntimeWorkerPool identifies a Shoot worker pool by its name and selector.

FieldDescription
name
string

Name specifies the name of the worker pool the container runtime should be available for. This field is immutable.

selector
Kubernetes meta/v1.LabelSelector

Selector is the label selector used by the extension to match the nodes belonging to the worker pool.

ControlPlaneSpec

(Appears on: ControlPlane)

ControlPlaneSpec is the spec of a ControlPlane resource.

FieldDescription
DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

purpose
Purpose
(Optional)

Purpose contains the data if a cloud provider needs additional components in order to expose the control plane. This field is immutable.

infrastructureProviderStatus
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

InfrastructureProviderStatus contains the provider status that has been generated by the controller responsible for the Infrastructure resource.

region
string

Region is the region of this control plane. This field is immutable.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the cloud provider specific credentials.

ControlPlaneStatus

(Appears on: ControlPlane)

ControlPlaneStatus is the status of a ControlPlane resource.

FieldDescription
DefaultStatus
DefaultStatus

(Members of DefaultStatus are embedded into this type.)

DefaultStatus is a structure containing common fields used by all extension resources.

DNSRecordSpec

(Appears on: DNSRecord)

DNSRecordSpec is the spec of a DNSRecord resource.

FieldDescription
DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the cloud provider specific credentials.

region
string
(Optional)

Region is the region of this DNS record. If not specified, the region specified in SecretRef will be used. If that is also not specified, the extension controller will use its default region.

zone
string
(Optional)

Zone is the DNS hosted zone of this DNS record. If not specified, it will be determined automatically by getting all hosted zones of the account and searching for the longest zone name that is a suffix of Name.

name
string

Name is the fully qualified domain name, e.g. “api.”. This field is immutable.

recordType
DNSRecordType

RecordType is the DNS record type. Only A, CNAME, and TXT records are currently supported. This field is immutable.

values
[]string

Values is a list of IP addresses for A records, a single hostname for CNAME records, or a list of texts for TXT records.

ttl
int64
(Optional)

TTL is the time to live in seconds. Defaults to 120.

DNSRecordStatus

(Appears on: DNSRecord)

DNSRecordStatus is the status of a DNSRecord resource.

FieldDescription
DefaultStatus
DefaultStatus

(Members of DefaultStatus are embedded into this type.)

DefaultStatus is a structure containing common fields used by all extension resources.

zone
string
(Optional)

Zone is the DNS hosted zone of this DNS record.

DNSRecordType (string alias)

(Appears on: DNSRecordSpec)

DNSRecordType is a string alias.

DataVolume

(Appears on: WorkerPool)

DataVolume contains information about a data volume.

FieldDescription
name
string

Name of the volume to make it referencable.

type
string
(Optional)

Type is the type of the volume.

size
string

Size is the of the root volume.

encrypted
bool
(Optional)

Encrypted determines if the volume should be encrypted.

DefaultSpec

(Appears on: BackupBucketSpec, BackupEntrySpec, BastionSpec, ContainerRuntimeSpec, ControlPlaneSpec, DNSRecordSpec, ExtensionSpec, InfrastructureSpec, NetworkSpec, OperatingSystemConfigSpec, WorkerSpec)

DefaultSpec contains common status fields for every extension resource.

FieldDescription
type
string

Type contains the instance of the resource’s kind.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig is the provider specific configuration.

DefaultStatus

(Appears on: BackupBucketStatus, BackupEntryStatus, BastionStatus, ContainerRuntimeStatus, ControlPlaneStatus, DNSRecordStatus, ExtensionStatus, InfrastructureStatus, NetworkStatus, OperatingSystemConfigStatus, WorkerStatus)

DefaultStatus contains common status fields for every extension resource.

FieldDescription
providerStatus
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderStatus contains provider-specific status.

conditions
[]github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition
(Optional)

Conditions represents the latest available observations of a Seed’s current state.

lastError
github.com/gardener/gardener/pkg/apis/core/v1beta1.LastError
(Optional)

LastError holds information about the last occurred error during an operation.

lastOperation
github.com/gardener/gardener/pkg/apis/core/v1beta1.LastOperation
(Optional)

LastOperation holds information about the last operation on the resource.

observedGeneration
int64

ObservedGeneration is the most recent generation observed for this resource.

state
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

State can be filled by the operating controller with what ever data it needs.

resources
[]github.com/gardener/gardener/pkg/apis/core/v1beta1.NamedResourceReference
(Optional)

Resources holds a list of named resource references that can be referred to in the state by their names.

DropIn

(Appears on: Unit)

DropIn is a drop-in configuration for a systemd unit.

FieldDescription
name
string

Name is the name of the drop-in.

content
string

Content is the content of the drop-in.

ExtensionSpec

(Appears on: Extension)

ExtensionSpec is the spec for a Extension resource.

FieldDescription
DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

ExtensionStatus

(Appears on: Extension)

ExtensionStatus is the status for a Extension resource.

FieldDescription
DefaultStatus
DefaultStatus

(Members of DefaultStatus are embedded into this type.)

DefaultStatus is a structure containing common fields used by all extension resources.

File

(Appears on: OperatingSystemConfigSpec)

File is a file that should get written to the host’s file system. The content can either be inlined or referenced from a secret in the same namespace.

FieldDescription
path
string

Path is the path of the file system where the file should get written to.

permissions
int32
(Optional)

Permissions describes with which permissions the file should get written to the file system. Should be defaulted to octal 0644.

content
FileContent

Content describe the file’s content.

FileCodecID (string alias)

FileCodecID is the id of a FileCodec for cloud-init scripts.

FileContent

(Appears on: File)

FileContent can either reference a secret or contain inline configuration.

FieldDescription
secretRef
FileContentSecretRef
(Optional)

SecretRef is a struct that contains information about the referenced secret.

inline
FileContentInline
(Optional)

Inline is a struct that contains information about the inlined data.

transmitUnencoded
bool
(Optional)

TransmitUnencoded set to true will ensure that the os-extension does not encode the file content when sent to the node. This for example can be used to manipulate the clear-text content before it reaches the node.

FileContentInline

(Appears on: FileContent)

FileContentInline contains keys for inlining a file content’s data and encoding.

FieldDescription
encoding
string

Encoding is the file’s encoding (e.g. base64).

data
string

Data is the file’s data.

FileContentSecretRef

(Appears on: FileContent)

FileContentSecretRef contains keys for referencing a file content’s data from a secret in the same namespace.

FieldDescription
name
string

Name is the name of the secret.

dataKey
string

DataKey is the key in the secret’s .data field that should be read.

InfrastructureSpec

(Appears on: Infrastructure)

InfrastructureSpec is the spec for an Infrastructure resource.

FieldDescription
DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

region
string

Region is the region of this infrastructure. This field is immutable.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the actual result of the generated cloud config.

sshPublicKey
[]byte
(Optional)

SSHPublicKey is the public SSH key that should be used with this infrastructure.

InfrastructureStatus

(Appears on: Infrastructure)

InfrastructureStatus is the status for an Infrastructure resource.

FieldDescription
DefaultStatus
DefaultStatus

(Members of DefaultStatus are embedded into this type.)

DefaultStatus is a structure containing common fields used by all extension resources.

nodesCIDR
string
(Optional)

NodesCIDR is the CIDR of the node network that was optionally created by the acting extension controller. This might be needed in environments in which the CIDR for the network for the shoot worker node cannot be statically defined in the Shoot resource but must be computed dynamically.

MachineDeployment

(Appears on: WorkerStatus)

MachineDeployment is a created machine deployment.

FieldDescription
name
string

Name is the name of the MachineDeployment resource.

minimum
int32

Minimum is the minimum number for this machine deployment.

maximum
int32

Maximum is the maximum number for this machine deployment.

MachineImage

(Appears on: WorkerPool)

MachineImage contains logical information about the name and the version of the machie image that should be used. The logical information must be mapped to the provider-specific information (e.g., AMIs, …) by the provider itself.

FieldDescription
name
string

Name is the logical name of the machine image.

version
string

Version is the version of the machine image.

NetworkSpec

(Appears on: Network)

NetworkSpec is the spec for an Network resource.

FieldDescription
DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

podCIDR
string

PodCIDR defines the CIDR that will be used for pods. This field is immutable.

serviceCIDR
string

ServiceCIDR defines the CIDR that will be used for services. This field is immutable.

NetworkStatus

(Appears on: Network)

NetworkStatus is the status for an Network resource.

FieldDescription
DefaultStatus
DefaultStatus

(Members of DefaultStatus are embedded into this type.)

DefaultStatus is a structure containing common fields used by all extension resources.

NodeTemplate

(Appears on: WorkerPool)

NodeTemplate contains information about the expected node properties.

FieldDescription
capacity
Kubernetes core/v1.ResourceList

Capacity represents the expected Node capacity.

Object

Object is an extension object resource.

OperatingSystemConfigPurpose (string alias)

(Appears on: OperatingSystemConfigSpec)

OperatingSystemConfigPurpose is a string alias.

OperatingSystemConfigSpec

(Appears on: OperatingSystemConfig)

OperatingSystemConfigSpec is the spec for a OperatingSystemConfig resource.

FieldDescription
criConfig
CRIConfig
(Optional)

CRI config is a structure contains configurations of the CRI library

DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

purpose
OperatingSystemConfigPurpose

Purpose describes how the result of this OperatingSystemConfig is used by Gardener. Either it gets sent to the Worker extension controller to bootstrap a VM, or it is downloaded by the cloud-config-downloader script already running on a bootstrapped VM. This field is immutable.

reloadConfigFilePath
string
(Optional)

ReloadConfigFilePath is the path to the generated operating system configuration. If set, controllers are asked to use it when determining the .status.command of this resource. For example, if for CoreOS the reload-path might be “/var/lib/config”; then the controller shall set .status.command to “/usr/bin/coreos-cloudinit –from-file=/var/lib/config”.

units
[]Unit
(Optional)

Units is a list of unit for the operating system configuration (usually, a systemd unit).

files
[]File
(Optional)

Files is a list of files that should get written to the host’s file system.

OperatingSystemConfigStatus

(Appears on: OperatingSystemConfig)

OperatingSystemConfigStatus is the status for a OperatingSystemConfig resource.

FieldDescription
DefaultStatus
DefaultStatus

(Members of DefaultStatus are embedded into this type.)

DefaultStatus is a structure containing common fields used by all extension resources.

cloudConfig
CloudConfig
(Optional)

CloudConfig is a structure for containing the generated output for the given operating system config spec. It contains a reference to a secret as the result may contain confidential data.

command
string
(Optional)

Command is the command whose execution renews/reloads the cloud config on an existing VM, e.g. “/usr/bin/reload-cloud-config -from-file=”. The is optionally provided by Gardener in the .spec.reloadConfigFilePath field.

units
[]string
(Optional)

Units is a list of systemd unit names that are part of the generated Cloud Config and shall be restarted when a new version has been downloaded.

Purpose (string alias)

(Appears on: ControlPlaneSpec)

Purpose is a string alias.

Spec

Spec is the spec section of an Object.

Status

Status is the status of an Object.

Unit

(Appears on: OperatingSystemConfigSpec)

Unit is a unit for the operating system configuration (usually, a systemd unit).

FieldDescription
name
string

Name is the name of a unit.

command
string
(Optional)

Command is the unit’s command.

enable
bool
(Optional)

Enable describes whether the unit is enabled or not.

content
string
(Optional)

Content is the unit’s content.

dropIns
[]DropIn
(Optional)

DropIns is a list of drop-ins for this unit.

Volume

(Appears on: WorkerPool)

Volume contains information about the root disks that should be used for worker pools.

FieldDescription
name
string
(Optional)

Name of the volume to make it referencable.

type
string
(Optional)

Type is the type of the volume.

size
string

Size is the of the root volume.

encrypted
bool
(Optional)

Encrypted determines if the volume should be encrypted.

WorkerPool

(Appears on: WorkerSpec)

WorkerPool is the definition of a specific worker pool.

FieldDescription
machineType
string

MachineType contains information about the machine type that should be used for this worker pool.

maximum
int32

Maximum is the maximum size of the worker pool.

maxSurge
k8s.io/apimachinery/pkg/util/intstr.IntOrString

MaxSurge is maximum number of VMs that are created during an update.

maxUnavailable
k8s.io/apimachinery/pkg/util/intstr.IntOrString

MaxUnavailable is the maximum number of VMs that can be unavailable during an update.

annotations
map[string]string
(Optional)

Annotations is a map of key/value pairs for annotations for all the Node objects in this worker pool.

labels
map[string]string
(Optional)

Labels is a map of key/value pairs for labels for all the Node objects in this worker pool.

taints
[]Kubernetes core/v1.Taint
(Optional)

Taints is a list of taints for all the Node objects in this worker pool.

machineImage
MachineImage

MachineImage contains logical information about the name and the version of the machie image that should be used. The logical information must be mapped to the provider-specific information (e.g., AMIs, …) by the provider itself.

minimum
int32

Minimum is the minimum size of the worker pool.

name
string

Name is the name of this worker pool.

providerConfig
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

ProviderConfig is a provider specific configuration for the worker pool.

userData
[]byte

UserData is a base64-encoded string that contains the data that is sent to the provider’s APIs when a new machine/VM that is part of this worker pool shall be spawned.

volume
Volume
(Optional)

Volume contains information about the root disks that should be used for this worker pool.

dataVolumes
[]DataVolume
(Optional)

DataVolumes contains a list of additional worker volumes.

kubeletDataVolumeName
string
(Optional)

KubeletDataVolumeName contains the name of a dataVolume that should be used for storing kubelet state.

zones
[]string
(Optional)

Zones contains information about availability zones for this worker pool.

machineControllerManager
github.com/gardener/gardener/pkg/apis/core/v1beta1.MachineControllerManagerSettings
(Optional)

MachineControllerManagerSettings contains configurations for different worker-pools. Eg. MachineDrainTimeout, MachineHealthTimeout.

kubernetesVersion
string
(Optional)

KubernetesVersion is the kubernetes version in this worker pool

nodeTemplate
NodeTemplate
(Optional)

NodeTemplate contains resource information of the machine which is used by Cluster Autoscaler to generate nodeTemplate during scaling a nodeGroup from zero

architecture
string
(Optional)

Architecture is the CPU architecture of the worker pool machines and machine image.

WorkerSpec

(Appears on: Worker)

WorkerSpec is the spec for a Worker resource.

FieldDescription
DefaultSpec
DefaultSpec

(Members of DefaultSpec are embedded into this type.)

DefaultSpec is a structure containing common fields used by all extension resources.

infrastructureProviderStatus
k8s.io/apimachinery/pkg/runtime.RawExtension
(Optional)

InfrastructureProviderStatus is a raw extension field that contains the provider status that has been generated by the controller responsible for the Infrastructure resource.

region
string

Region is the name of the region where the worker pool should be deployed to. This field is immutable.

secretRef
Kubernetes core/v1.SecretReference

SecretRef is a reference to a secret that contains the cloud provider specific credentials.

sshPublicKey
[]byte
(Optional)

SSHPublicKey is the public SSH key that should be used with these workers.

pools
[]WorkerPool

Pools is a list of worker pools.

WorkerStatus

(Appears on: Worker)

WorkerStatus is the status for a Worker resource.

FieldDescription
DefaultStatus
DefaultStatus

(Members of DefaultStatus are embedded into this type.)

DefaultStatus is a structure containing common fields used by all extension resources.

machineDeployments
[]MachineDeployment

MachineDeployments is a list of created machine deployments. It will be used to e.g. configure the cluster-autoscaler properly.


Generated with gen-crd-api-reference-docs

1.1.4 - Operations

Packages:

operations.gardener.cloud/v1alpha1

Package v1alpha1 is a version of the API.

Resource Types:

Bastion

Bastion holds details about an SSH bastion for a shoot cluster.

FieldDescription
apiVersion
string
operations.gardener.cloud/v1alpha1
kind
string
Bastion
metadata
Kubernetes meta/v1.ObjectMeta

Standard object metadata.

Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
BastionSpec

Specification of the Bastion.



shootRef
Kubernetes core/v1.LocalObjectReference

ShootRef defines the target shoot for a Bastion. The name field of the ShootRef is immutable.

seedName
string
(Optional)

SeedName is the name of the seed to which this Bastion is currently scheduled. This field is populated at the beginning of a create/reconcile operation.

providerType
string
(Optional)

ProviderType is cloud provider used by the referenced Shoot.

sshPublicKey
string

SSHPublicKey is the user’s public key. This field is immutable.

ingress
[]BastionIngressPolicy

Ingress controls from where the created bastion host should be reachable.

status
BastionStatus
(Optional)

Most recently observed status of the Bastion.

BastionIngressPolicy

(Appears on: BastionSpec)

BastionIngressPolicy represents an ingress policy for SSH bastion hosts.

FieldDescription
ipBlock
Kubernetes networking/v1.IPBlock

IPBlock defines an IP block that is allowed to access the bastion.

BastionSpec

(Appears on: Bastion)

BastionSpec is the specification of a Bastion.

FieldDescription
shootRef
Kubernetes core/v1.LocalObjectReference

ShootRef defines the target shoot for a Bastion. The name field of the ShootRef is immutable.

seedName
string
(Optional)

SeedName is the name of the seed to which this Bastion is currently scheduled. This field is populated at the beginning of a create/reconcile operation.

providerType
string
(Optional)

ProviderType is cloud provider used by the referenced Shoot.

sshPublicKey
string

SSHPublicKey is the user’s public key. This field is immutable.

ingress
[]BastionIngressPolicy

Ingress controls from where the created bastion host should be reachable.

BastionStatus

(Appears on: Bastion)

BastionStatus holds the most recently observed status of the Bastion.

FieldDescription
ingress
Kubernetes core/v1.LoadBalancerIngress
(Optional)

Ingress holds the public IP and/or hostname of the bastion instance.

conditions
[]github.com/gardener/gardener/pkg/apis/core/v1alpha1.Condition
(Optional)

Conditions represents the latest available observations of a Bastion’s current state.

lastHeartbeatTimestamp
Kubernetes meta/v1.Time
(Optional)

LastHeartbeatTimestamp is the time when the bastion was last marked as not to be deleted. When this is set, the ExpirationTimestamp is advanced as well.

expirationTimestamp
Kubernetes meta/v1.Time
(Optional)

ExpirationTimestamp is the time after which a Bastion is supposed to be garbage collected.

observedGeneration
int64
(Optional)

ObservedGeneration is the most recent generation observed for this Bastion. It corresponds to the Bastion’s generation, which is updated on mutation by the API Server.


Generated with gen-crd-api-reference-docs

1.1.5 - Provider Local

Packages:

local.provider.extensions.gardener.cloud/v1alpha1

Package v1alpha1 contains the local provider API resources.

Resource Types:

CloudProfileConfig

CloudProfileConfig contains provider-specific configuration that is embedded into Gardener’s CloudProfile resource.

FieldDescription
apiVersion
string
local.provider.extensions.gardener.cloud/v1alpha1
kind
string
CloudProfileConfig
machineImages
[]MachineImages

MachineImages is the list of machine images that are understood by the controller. It maps logical names and versions to provider-specific identifiers.

WorkerStatus

WorkerStatus contains information about created worker resources.

FieldDescription
apiVersion
string
local.provider.extensions.gardener.cloud/v1alpha1
kind
string
WorkerStatus
machineImages
[]MachineImage
(Optional)

MachineImages is a list of machine images that have been used in this worker. Usually, the extension controller gets the mapping from name/version to the provider-specific machine image data from the CloudProfile. However, if a version that is still in use gets removed from this componentconfig it cannot reconcile anymore existing Worker resources that are still using this version. Hence, it stores the used versions in the provider status to ensure reconciliation is possible.

MachineImage

(Appears on: WorkerStatus)

MachineImage is a mapping from logical names and versions to provider-specific machine image data.

FieldDescription
name
string

Name is the logical name of the machine image.

version
string

Version is the logical version of the machine image.

image
string

Image is the image for the machine image.

MachineImageVersion

(Appears on: MachineImages)

MachineImageVersion contains a version and a provider-specific identifier.

FieldDescription
version
string

Version is the version of the image.

image
string

Image is the image for the machine image.

MachineImages

(Appears on: CloudProfileConfig)

MachineImages is a mapping from logical names and versions to provider-specific identifiers.

FieldDescription
name
string

Name is the logical name of the machine image.

versions
[]MachineImageVersion

Versions contains versions and a provider-specific identifier.


Generated with gen-crd-api-reference-docs

1.1.6 - Resources

Packages:

resources.gardener.cloud/v1alpha1

Package v1alpha1 contains the configuration of the Gardener Resource Manager.

Resource Types:

    ManagedResource

    ManagedResource describes a list of managed resources.

    FieldDescription
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ManagedResourceSpec

    Spec contains the specification of this managed resource.



    class
    string
    (Optional)

    Class holds the resource class used to control the responsibility for multiple resource manager instances

    secretRefs
    []Kubernetes core/v1.LocalObjectReference

    SecretRefs is a list of secret references.

    injectLabels
    map[string]string
    (Optional)

    InjectLabels injects the provided labels into every resource that is part of the referenced secrets.

    forceOverwriteLabels
    bool
    (Optional)

    ForceOverwriteLabels specifies that all existing labels should be overwritten. Defaults to false.

    forceOverwriteAnnotations
    bool
    (Optional)

    ForceOverwriteAnnotations specifies that all existing annotations should be overwritten. Defaults to false.

    keepObjects
    bool
    (Optional)

    KeepObjects specifies whether the objects should be kept although the managed resource has already been deleted. Defaults to false.

    equivalences
    [][]k8s.io/apimachinery/pkg/apis/meta/v1.GroupKind
    (Optional)

    Equivalences specifies possible group/kind equivalences for objects.

    deletePersistentVolumeClaims
    bool
    (Optional)

    DeletePersistentVolumeClaims specifies if PersistentVolumeClaims created by StatefulSets, which are managed by this resource, should also be deleted when the corresponding StatefulSet is deleted (defaults to false).

    status
    ManagedResourceStatus

    Status contains the status of this managed resource.

    ManagedResourceSpec

    (Appears on: ManagedResource)

    ManagedResourceSpec contains the specification of this managed resource.

    FieldDescription
    class
    string
    (Optional)

    Class holds the resource class used to control the responsibility for multiple resource manager instances

    secretRefs
    []Kubernetes core/v1.LocalObjectReference

    SecretRefs is a list of secret references.

    injectLabels
    map[string]string
    (Optional)

    InjectLabels injects the provided labels into every resource that is part of the referenced secrets.

    forceOverwriteLabels
    bool
    (Optional)

    ForceOverwriteLabels specifies that all existing labels should be overwritten. Defaults to false.

    forceOverwriteAnnotations
    bool
    (Optional)

    ForceOverwriteAnnotations specifies that all existing annotations should be overwritten. Defaults to false.

    keepObjects
    bool
    (Optional)

    KeepObjects specifies whether the objects should be kept although the managed resource has already been deleted. Defaults to false.

    equivalences
    [][]k8s.io/apimachinery/pkg/apis/meta/v1.GroupKind
    (Optional)

    Equivalences specifies possible group/kind equivalences for objects.

    deletePersistentVolumeClaims
    bool
    (Optional)

    DeletePersistentVolumeClaims specifies if PersistentVolumeClaims created by StatefulSets, which are managed by this resource, should also be deleted when the corresponding StatefulSet is deleted (defaults to false).

    ManagedResourceStatus

    (Appears on: ManagedResource)

    ManagedResourceStatus is the status of a managed resource.

    FieldDescription
    conditions
    []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition
    observedGeneration
    int64

    ObservedGeneration is the most recent generation observed for this resource.

    resources
    []ObjectReference
    (Optional)

    Resources is a list of objects that have been created.

    secretsDataChecksum
    string
    (Optional)

    SecretsDataChecksum is the checksum of referenced secrets data.

    ObjectReference

    (Appears on: ManagedResourceStatus)

    ObjectReference is a reference to another object.

    FieldDescription
    ObjectReference
    Kubernetes core/v1.ObjectReference

    (Members of ObjectReference are embedded into this type.)

    labels
    map[string]string

    Labels is a map of labels that were used during last update of the resource.

    annotations
    map[string]string

    Annotations is a map of annotations that were used during last update of the resource.


    Generated with gen-crd-api-reference-docs

    1.1.7 - Seedmanagement

    Packages:

    seedmanagement.gardener.cloud/v1alpha1

    Package v1alpha1 is a version of the API.

    Resource Types:

    ManagedSeed

    ManagedSeed represents a Shoot that is registered as Seed.

    FieldDescription
    apiVersion
    string
    seedmanagement.gardener.cloud/v1alpha1
    kind
    string
    ManagedSeed
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ManagedSeedSpec
    (Optional)

    Specification of the ManagedSeed.



    shoot
    Shoot
    (Optional)

    Shoot references a Shoot that should be registered as Seed. This field is immutable.

    seedTemplate
    github.com/gardener/gardener/pkg/apis/core/v1beta1.SeedTemplate
    (Optional)

    SeedTemplate is a template for a Seed object, that should be used to register a given cluster as a Seed. Either SeedTemplate or Gardenlet must be specified. When Seed is specified, the ManagedSeed controller will not deploy a gardenlet into the cluster and an existing gardenlet reconciling the new Seed is required.

    gardenlet
    Gardenlet
    (Optional)

    Gardenlet specifies that the ManagedSeed controller should deploy a gardenlet into the cluster with the given deployment parameters and GardenletConfiguration.

    status
    ManagedSeedStatus
    (Optional)

    Most recently observed status of the ManagedSeed.

    ManagedSeedSet

    ManagedSeedSet represents a set of identical ManagedSeeds.

    FieldDescription
    apiVersion
    string
    seedmanagement.gardener.cloud/v1alpha1
    kind
    string
    ManagedSeedSet
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ManagedSeedSetSpec
    (Optional)

    Spec defines the desired identities of ManagedSeeds and Shoots in this set.



    replicas
    int32
    (Optional)

    Replicas is the desired number of replicas of the given Template. Defaults to 1.

    selector
    Kubernetes meta/v1.LabelSelector

    Selector is a label query over ManagedSeeds and Shoots that should match the replica count. It must match the ManagedSeeds and Shoots template’s labels. This field is immutable.

    template
    ManagedSeedTemplate

    Template describes the ManagedSeed that will be created if insufficient replicas are detected. Each ManagedSeed created / updated by the ManagedSeedSet will fulfill this template.

    shootTemplate
    github.com/gardener/gardener/pkg/apis/core/v1beta1.ShootTemplate

    ShootTemplate describes the Shoot that will be created if insufficient replicas are detected for hosting the corresponding ManagedSeed. Each Shoot created / updated by the ManagedSeedSet will fulfill this template.

    updateStrategy
    UpdateStrategy
    (Optional)

    UpdateStrategy specifies the UpdateStrategy that will be employed to update ManagedSeeds / Shoots in the ManagedSeedSet when a revision is made to Template / ShootTemplate.

    revisionHistoryLimit
    int32
    (Optional)

    RevisionHistoryLimit is the maximum number of revisions that will be maintained in the ManagedSeedSet’s revision history. Defaults to 10. This field is immutable.

    status
    ManagedSeedSetStatus
    (Optional)

    Status is the current status of ManagedSeeds and Shoots in this ManagedSeedSet.

    Bootstrap (string alias)

    (Appears on: Gardenlet)

    Bootstrap describes a mechanism for bootstrapping gardenlet connection to the Garden cluster.

    Gardenlet

    (Appears on: ManagedSeedSpec)

    Gardenlet specifies gardenlet deployment parameters and the GardenletConfiguration used to configure gardenlet.

    FieldDescription
    deployment
    GardenletDeployment
    (Optional)

    Deployment specifies certain gardenlet deployment parameters, such as the number of replicas, the image, etc.

    config
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    Config is the GardenletConfiguration used to configure gardenlet.

    bootstrap
    Bootstrap
    (Optional)

    Bootstrap is the mechanism that should be used for bootstrapping gardenlet connection to the Garden cluster. One of ServiceAccount, BootstrapToken, None. If set to ServiceAccount or BootstrapToken, a service account or a bootstrap token will be created in the garden cluster and used to compute the bootstrap kubeconfig. If set to None, the gardenClientConnection.kubeconfig field will be used to connect to the Garden cluster. Defaults to BootstrapToken. This field is immutable.

    mergeWithParent
    bool
    (Optional)

    MergeWithParent specifies whether the GardenletConfiguration of the parent gardenlet should be merged with the specified GardenletConfiguration. Defaults to true. This field is immutable.

    GardenletDeployment

    (Appears on: Gardenlet)

    GardenletDeployment specifies certain gardenlet deployment parameters, such as the number of replicas, the image, etc.

    FieldDescription
    replicaCount
    int32
    (Optional)

    ReplicaCount is the number of gardenlet replicas. Defaults to 1.

    revisionHistoryLimit
    int32
    (Optional)

    RevisionHistoryLimit is the number of old gardenlet ReplicaSets to retain to allow rollback. Defaults to 10.

    serviceAccountName
    string
    (Optional)

    ServiceAccountName is the name of the ServiceAccount to use to run gardenlet pods.

    image
    Image
    (Optional)

    Image is the gardenlet container image.

    resources
    Kubernetes core/v1.ResourceRequirements
    (Optional)

    Resources are the compute resources required by the gardenlet container.

    podLabels
    map[string]string
    (Optional)

    PodLabels are the labels on gardenlet pods.

    podAnnotations
    map[string]string
    (Optional)

    PodAnnotations are the annotations on gardenlet pods.

    additionalVolumes
    []Kubernetes core/v1.Volume
    (Optional)

    AdditionalVolumes is the list of additional volumes that should be mounted by gardenlet containers.

    additionalVolumeMounts
    []Kubernetes core/v1.VolumeMount
    (Optional)

    AdditionalVolumeMounts is the list of additional pod volumes to mount into the gardenlet container’s filesystem.

    env
    []Kubernetes core/v1.EnvVar
    (Optional)

    Env is the list of environment variables to set in the gardenlet container.

    vpa
    bool
    (Optional)

    VPA specifies whether to enable VPA for gardenlet. Defaults to true.

    Image

    (Appears on: GardenletDeployment)

    Image specifies container image parameters.

    FieldDescription
    repository
    string
    (Optional)

    Repository is the image repository.

    tag
    string
    (Optional)

    Tag is the image tag.

    pullPolicy
    Kubernetes core/v1.PullPolicy
    (Optional)

    PullPolicy is the image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if latest tag is specified, or IfNotPresent otherwise.

    ManagedSeedSetSpec

    (Appears on: ManagedSeedSet)

    ManagedSeedSetSpec is the specification of a ManagedSeedSet.

    FieldDescription
    replicas
    int32
    (Optional)

    Replicas is the desired number of replicas of the given Template. Defaults to 1.

    selector
    Kubernetes meta/v1.LabelSelector

    Selector is a label query over ManagedSeeds and Shoots that should match the replica count. It must match the ManagedSeeds and Shoots template’s labels. This field is immutable.

    template
    ManagedSeedTemplate

    Template describes the ManagedSeed that will be created if insufficient replicas are detected. Each ManagedSeed created / updated by the ManagedSeedSet will fulfill this template.

    shootTemplate
    github.com/gardener/gardener/pkg/apis/core/v1beta1.ShootTemplate

    ShootTemplate describes the Shoot that will be created if insufficient replicas are detected for hosting the corresponding ManagedSeed. Each Shoot created / updated by the ManagedSeedSet will fulfill this template.

    updateStrategy
    UpdateStrategy
    (Optional)

    UpdateStrategy specifies the UpdateStrategy that will be employed to update ManagedSeeds / Shoots in the ManagedSeedSet when a revision is made to Template / ShootTemplate.

    revisionHistoryLimit
    int32
    (Optional)

    RevisionHistoryLimit is the maximum number of revisions that will be maintained in the ManagedSeedSet’s revision history. Defaults to 10. This field is immutable.

    ManagedSeedSetStatus

    (Appears on: ManagedSeedSet)

    ManagedSeedSetStatus represents the current state of a ManagedSeedSet.

    FieldDescription
    observedGeneration
    int64

    ObservedGeneration is the most recent generation observed for this ManagedSeedSet. It corresponds to the ManagedSeedSet’s generation, which is updated on mutation by the API Server.

    replicas
    int32

    Replicas is the number of replicas (ManagedSeeds and their corresponding Shoots) created by the ManagedSeedSet controller.

    readyReplicas
    int32

    ReadyReplicas is the number of ManagedSeeds created by the ManagedSeedSet controller that have a Ready Condition.

    nextReplicaNumber
    int32

    NextReplicaNumber is the ordinal number that will be assigned to the next replica of the ManagedSeedSet.

    currentReplicas
    int32

    CurrentReplicas is the number of ManagedSeeds created by the ManagedSeedSet controller from the ManagedSeedSet version indicated by CurrentRevision.

    updatedReplicas
    int32

    UpdatedReplicas is the number of ManagedSeeds created by the ManagedSeedSet controller from the ManagedSeedSet version indicated by UpdateRevision.

    currentRevision
    string

    CurrentRevision, if not empty, indicates the version of the ManagedSeedSet used to generate ManagedSeeds with smaller ordinal numbers during updates.

    updateRevision
    string

    UpdateRevision, if not empty, indicates the version of the ManagedSeedSet used to generate ManagedSeeds with larger ordinal numbers during updates

    collisionCount
    int32
    (Optional)

    CollisionCount is the count of hash collisions for the ManagedSeedSet. The ManagedSeedSet controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest ControllerRevision.

    conditions
    []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition
    (Optional)

    Conditions represents the latest available observations of a ManagedSeedSet’s current state.

    pendingReplica
    PendingReplica
    (Optional)

    PendingReplica, if not empty, indicates the replica that is currently pending creation, update, or deletion. This replica is in a state that requires the controller to wait for it to change before advancing to the next replica.

    ManagedSeedSpec

    (Appears on: ManagedSeed, ManagedSeedTemplate)

    ManagedSeedSpec is the specification of a ManagedSeed.

    FieldDescription
    shoot
    Shoot
    (Optional)

    Shoot references a Shoot that should be registered as Seed. This field is immutable.

    seedTemplate
    github.com/gardener/gardener/pkg/apis/core/v1beta1.SeedTemplate
    (Optional)

    SeedTemplate is a template for a Seed object, that should be used to register a given cluster as a Seed. Either SeedTemplate or Gardenlet must be specified. When Seed is specified, the ManagedSeed controller will not deploy a gardenlet into the cluster and an existing gardenlet reconciling the new Seed is required.

    gardenlet
    Gardenlet
    (Optional)

    Gardenlet specifies that the ManagedSeed controller should deploy a gardenlet into the cluster with the given deployment parameters and GardenletConfiguration.

    ManagedSeedStatus

    (Appears on: ManagedSeed)

    ManagedSeedStatus is the status of a ManagedSeed.

    FieldDescription
    conditions
    []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition
    (Optional)

    Conditions represents the latest available observations of a ManagedSeed’s current state.

    observedGeneration
    int64

    ObservedGeneration is the most recent generation observed for this ManagedSeed. It corresponds to the ManagedSeed’s generation, which is updated on mutation by the API Server.

    ManagedSeedTemplate

    (Appears on: ManagedSeedSetSpec)

    ManagedSeedTemplate is a template for creating a ManagedSeed object.

    FieldDescription
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ManagedSeedSpec
    (Optional)

    Specification of the desired behavior of the ManagedSeed.



    shoot
    Shoot
    (Optional)

    Shoot references a Shoot that should be registered as Seed. This field is immutable.

    seedTemplate
    github.com/gardener/gardener/pkg/apis/core/v1beta1.SeedTemplate
    (Optional)

    SeedTemplate is a template for a Seed object, that should be used to register a given cluster as a Seed. Either SeedTemplate or Gardenlet must be specified. When Seed is specified, the ManagedSeed controller will not deploy a gardenlet into the cluster and an existing gardenlet reconciling the new Seed is required.

    gardenlet
    Gardenlet
    (Optional)

    Gardenlet specifies that the ManagedSeed controller should deploy a gardenlet into the cluster with the given deployment parameters and GardenletConfiguration.

    PendingReplica

    (Appears on: ManagedSeedSetStatus)

    PendingReplica contains information about a replica that is currently pending creation, update, or deletion.

    FieldDescription
    name
    string

    Name is the replica name.

    reason
    PendingReplicaReason

    Reason is the reason for the replica to be pending.

    since
    Kubernetes meta/v1.Time

    Since is the moment in time since the replica is pending with the specified reason.

    retries
    int32
    (Optional)

    Retries is the number of times the shoot operation (reconcile or delete) has been retried after having failed. Only applicable if Reason is ShootReconciling or ShootDeleting.

    PendingReplicaReason (string alias)

    (Appears on: PendingReplica)

    PendingReplicaReason is a string enumeration type that enumerates all possible reasons for a replica to be pending.

    RollingUpdateStrategy

    (Appears on: UpdateStrategy)

    RollingUpdateStrategy is used to communicate parameters for RollingUpdateStrategyType.

    FieldDescription
    partition
    int32
    (Optional)

    Partition indicates the ordinal at which the ManagedSeedSet should be partitioned. Defaults to 0.

    Shoot

    (Appears on: ManagedSeedSpec)

    Shoot identifies the Shoot that should be registered as Seed.

    FieldDescription
    name
    string

    Name is the name of the Shoot that will be registered as Seed.

    UpdateStrategy

    (Appears on: ManagedSeedSetSpec)

    UpdateStrategy specifies the strategy that the ManagedSeedSet controller will use to perform updates. It includes any additional parameters necessary to perform the update for the indicated strategy.

    FieldDescription
    type
    UpdateStrategyType
    (Optional)

    Type indicates the type of the UpdateStrategy. Defaults to RollingUpdate.

    rollingUpdate
    RollingUpdateStrategy
    (Optional)

    RollingUpdate is used to communicate parameters when Type is RollingUpdateStrategyType.

    UpdateStrategyType (string alias)

    (Appears on: UpdateStrategy)

    UpdateStrategyType is a string enumeration type that enumerates all possible update strategies for the ManagedSeedSet controller.


    Generated with gen-crd-api-reference-docs

    1.1.8 - Settings

    Packages:

    settings.gardener.cloud/v1alpha1

    Package v1alpha1 is a version of the API.

    Resource Types:

    ClusterOpenIDConnectPreset

    ClusterOpenIDConnectPreset is a OpenID Connect configuration that is applied to a Shoot objects cluster-wide.

    FieldDescription
    apiVersion
    string
    settings.gardener.cloud/v1alpha1
    kind
    string
    ClusterOpenIDConnectPreset
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ClusterOpenIDConnectPresetSpec

    Spec is the specification of this OpenIDConnect preset.



    OpenIDConnectPresetSpec
    OpenIDConnectPresetSpec

    (Members of OpenIDConnectPresetSpec are embedded into this type.)

    projectSelector
    Kubernetes meta/v1.LabelSelector
    (Optional)

    Project decides whether to apply the configuration if the Shoot is in a specific Project matching the label selector. Use the selector only if the OIDC Preset is opt-in, because end users may skip the admission by setting the labels. Defaults to the empty LabelSelector, which matches everything.

    OpenIDConnectPreset

    OpenIDConnectPreset is a OpenID Connect configuration that is applied to a Shoot in a namespace.

    FieldDescription
    apiVersion
    string
    settings.gardener.cloud/v1alpha1
    kind
    string
    OpenIDConnectPreset
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    OpenIDConnectPresetSpec

    Spec is the specification of this OpenIDConnect preset.



    server
    KubeAPIServerOpenIDConnect

    Server contains the kube-apiserver’s OpenID Connect configuration. This configuration is not overwritting any existing OpenID Connect configuration already set on the Shoot object.

    client
    OpenIDConnectClientAuthentication
    (Optional)

    Client contains the configuration used for client OIDC authentication of Shoot clusters. This configuration is not overwritting any existing OpenID Connect client authentication already set on the Shoot object.

    shootSelector
    Kubernetes meta/v1.LabelSelector
    (Optional)

    ShootSelector decides whether to apply the configuration if the Shoot has matching labels. Use the selector only if the OIDC Preset is opt-in, because end users may skip the admission by setting the labels. Default to the empty LabelSelector, which matches everything.

    weight
    int32

    Weight associated with matching the corresponding preset, in the range 1-100. Required.

    ClusterOpenIDConnectPresetSpec

    (Appears on: ClusterOpenIDConnectPreset)

    ClusterOpenIDConnectPresetSpec contains the OpenIDConnect specification and project selector matching Shoots in Projects.

    FieldDescription
    OpenIDConnectPresetSpec
    OpenIDConnectPresetSpec

    (Members of OpenIDConnectPresetSpec are embedded into this type.)

    projectSelector
    Kubernetes meta/v1.LabelSelector
    (Optional)

    Project decides whether to apply the configuration if the Shoot is in a specific Project matching the label selector. Use the selector only if the OIDC Preset is opt-in, because end users may skip the admission by setting the labels. Defaults to the empty LabelSelector, which matches everything.

    KubeAPIServerOpenIDConnect

    (Appears on: OpenIDConnectPresetSpec)

    KubeAPIServerOpenIDConnect contains configuration settings for the OIDC provider. Note: Descriptions were taken from the Kubernetes documentation.

    FieldDescription
    caBundle
    string
    (Optional)

    If set, the OpenID server’s certificate will be verified by one of the authorities in the oidc-ca-file, otherwise the host’s root CA set will be used.

    clientID
    string

    The client ID for the OpenID Connect client. Required.

    groupsClaim
    string
    (Optional)

    If provided, the name of a custom OpenID Connect claim for specifying user groups. The claim value is expected to be a string or array of strings. This field is experimental, please see the authentication documentation for further details.

    groupsPrefix
    string
    (Optional)

    If provided, all groups will be prefixed with this value to prevent conflicts with other authentication strategies.

    issuerURL
    string

    The URL of the OpenID issuer, only HTTPS scheme will be accepted. If set, it will be used to verify the OIDC JSON Web Token (JWT). Required.

    requiredClaims
    map[string]string
    (Optional)

    key=value pairs that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value.

    signingAlgs
    []string
    (Optional)

    List of allowed JOSE asymmetric signing algorithms. JWTs with a ‘alg’ header value not in this list will be rejected. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1 Defaults to [RS256]

    usernameClaim
    string
    (Optional)

    The OpenID claim to use as the user name. Note that claims other than the default (‘sub’) is not guaranteed to be unique and immutable. This field is experimental, please see the authentication documentation for further details. Defaults to “sub”.

    usernamePrefix
    string
    (Optional)

    If provided, all usernames will be prefixed with this value. If not provided, username claims other than ‘email’ are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value ‘-’.

    OpenIDConnectClientAuthentication

    (Appears on: OpenIDConnectPresetSpec)

    OpenIDConnectClientAuthentication contains configuration for OIDC clients.

    FieldDescription
    secret
    string
    (Optional)

    The client Secret for the OpenID Connect client.

    extraConfig
    map[string]string
    (Optional)

    Extra configuration added to kubeconfig’s auth-provider. Must not be any of idp-issuer-url, client-id, client-secret, idp-certificate-authority, idp-certificate-authority-data, id-token or refresh-token

    OpenIDConnectPresetSpec

    (Appears on: OpenIDConnectPreset, ClusterOpenIDConnectPresetSpec)

    OpenIDConnectPresetSpec contains the Shoot selector for which a specific OpenID Connect configuration is applied.

    FieldDescription
    server
    KubeAPIServerOpenIDConnect

    Server contains the kube-apiserver’s OpenID Connect configuration. This configuration is not overwritting any existing OpenID Connect configuration already set on the Shoot object.

    client
    OpenIDConnectClientAuthentication
    (Optional)

    Client contains the configuration used for client OIDC authentication of Shoot clusters. This configuration is not overwritting any existing OpenID Connect client authentication already set on the Shoot object.

    shootSelector
    Kubernetes meta/v1.LabelSelector
    (Optional)

    ShootSelector decides whether to apply the configuration if the Shoot has matching labels. Use the selector only if the OIDC Preset is opt-in, because end users may skip the admission by setting the labels. Default to the empty LabelSelector, which matches everything.

    weight
    int32

    Weight associated with matching the corresponding preset, in the range 1-100. Required.


    Generated with gen-crd-api-reference-docs

    1.2 - Concepts

    1.2.1 - Admission Controller

    Gardener Admission Controller

    While the Gardener API server works with admission plugins to validate and mutate resources belonging to Gardener related API groups, e.g. core.gardener.cloud, the same is needed for resources belonging to non-Gardener API groups as well, e.g. Secrets in the core API group. Therefore, the Gardener Admission Controller runs a http(s) server with the following handlers which serve as validating/mutating endpoints for admission webhooks. It is also used to serve http(s) handlers for authorization webhooks.

    Admission Webhook Handlers

    This section describes the admission webhook handlers that are currently served.

    Kubeconfig Secret Validator

    Malicious Kubeconfigs applied by end users may cause a leakage of sensitive data. This handler checks if the incoming request contains a Kubernetes secret with a .data.kubeconfig field and denies the request if the Kubeconfig structure violates Gardener’s security standards.

    Namespace Validator

    Namespaces are the backing entities of Gardener projects in which shoot clusters objects reside. This validation handler protects active namespaces against premature deletion requests. Therefore, it denies deletion requests if a namespace still contains shoot clusters or if it belongs to a non-deleting Gardener project (w/o .metadata.deletionTimestamp).

    Resource Size Validator

    Since users directly apply Kubernetes native objects to the Garden cluster, it also involves the risk of being vulnerable to DoS attacks because these resources are read continuously watched and read by controllers. One example is the creation of Shoot resources with large annotation values (up to 256 kB per value) which can cause severe out-of-memory issues for the Gardenlet component. Vertical autoscaling can help to mitigate such situations, but we cannot expect to scale infinitely, and thus need means to block the attack itself.

    The Resource Size Validator checks arbitrary incoming admission requests against a configured maximum size for the resource’s group-version-kind combination and denies the request if the contained object exceeds the quota.

    Example for Gardener Admission Controller configuration:

    server:
      resourceAdmissionConfiguration:
        limits:
        - apiGroups: ["core.gardener.cloud"]
          apiVersions: ["*"]
          resources: ["shoots"]
          size: 100k
        - apiGroups: [""]
          apiVersions: ["v1"]
          resources: ["secrets"]
          size: 100k
        unrestrictedSubjects:
        - kind: Group
          name: gardener.cloud:system:seeds
          apiGroup: rbac.authorization.k8s.io
     #  - kind: User
     #    name: admin
     #    apiGroup: rbac.authorization.k8s.io
     #  - kind: ServiceAccount
     #    name: "*"
     #    namespace: garden
     #    apiGroup: ""
        operationMode: block #log
    

    With the configuration above, the Resource Size Validator denies requests for shoots with Gardener’s core API group which exceed a size of 100 kB. The same is done for Kubernetes secrets.

    As this feature is meant to protect the system from malicious requests sent by users, it is recommended to exclude trusted groups, users or service accounts from the size restriction via resourceAdmissionConfiguration.unrestrictedSubjects. For example, the backing user for the Gardenlet should always be capable of changing the shoot resource instead of being blocked due to size restrictions. This is because the Gardenlet itself occasionally changes the shoot specification, labels or annotations, and might violate the quota if the existing resource is already close to the quota boundary. Also, operators are supposed to be trusted users and subjecting them to a size limitation can inhibit important operational tasks. Wildcard ("*") in subject name is supported.

    Size limitations depend on the individual Gardener setup and choosing the wrong values can affect the availability of your Gardener service. resourceAdmissionConfiguration.operationMode allows to control if a violating request is actually denied (default) or only logged. It’s recommended to start with log, check the logs for exceeding requests, adjust the limits if necessary and finally switch to block.

    SeedRestriction

    Please refer to this document for more information.

    Authorization Webhook Handlers

    This section describes the authorization webhook handlers that are currently served.

    SeedAuthorization

    Please refer to this document for more information.

    1.2.2 - API Server

    Gardener API server

    The Gardener API server is a Kubernetes-native extension based on its aggregation layer. It is registered via an APIService object and designed to run inside a Kubernetes cluster whose API it wants to extend.

    After registration, it exposes the following resources:

    CloudProfiles

    CloudProfiles are resources that describe a specific environment of an underlying infrastructure provider, e.g. AWS, Azure, etc. Each shoot has to reference a CloudProfile to declare the environment it should be created in. In a CloudProfile the gardener operator specifies certain constraints like available machine types, regions, which Kubernetes versions they want to offer, etc. End-users can read CloudProfiles to see these values, but only operators can change the content or create/delete them. When a shoot is created or updated then an admission plugin checks that only values are used that are allowed via the referenced CloudProfile.

    Additionally, a CloudProfile may contain a providerConfig which is a special configuration dedicated for the infrastructure provider. Gardener does not evaluate or understand this config, but extension controllers might need for declaration of provider-specific constraints, or global settings.

    Please see this example manifest and consult the documentation of your provider extension controller to get information about its providerConfig.

    Seeds

    Seeds are resources that represent seed clusters. Gardener does not care about how a seed cluster got created - the only requirement is that it is of at least Kubernetes v1.17 and passes the Kubernetes conformance tests. The Gardener operator has to either deploy the Gardenlet into the cluster they want to use as seed (recommended, then the Gardenlet will create the Seed object itself after bootstrapping), or they provide the kubeconfig to the cluster inside a secret (that is referenced by the Seed resource) and create the Seed resource themselves.

    Please see this, this(, and optionally this) example manifests.

    ShootQuotas

    In order to allow end-users not having their own dedicated infrastructure account to try out Gardener the operator can register an account owned by them that they allow to be used for trial clusters. Trial clusters can be put under quota such that they don’t consume too many resources (resulting in costs), and so that one user cannot consume all resources on their own. These clusters are automatically terminated after a specified time, but end-users may extend the lifetime manually if needed.

    Please see this example manifest.

    Projects

    The first thing before creating a shoot cluster is to create a Project. A project is used to group multiple shoot clusters together. End-users can invite colleagues to the project to enable collaboration, and they can either make them admin or viewer. After an end-user has created a project they will get a dedicated namespace in the garden cluster for all their shoots.

    Please see this example manifest.

    SecretBindings

    Now that the end-user has a namespace the next step is registering their infrastructure provider account.

    Please see this example manifest and consult the documentation of the extension controller for the respective infrastructure provider to get information about which keys are required in this secret.

    After the secret has been created the end-user has to create a special SecretBinding resource that binds this secret. Later when creating shoot clusters they will reference such a binding.

    Please see this example manifest.

    Shoots

    Shoot cluster contain various settings that influence how end-user Kubernetes clusters will look like in the end. As Gardener heavily relies on extension controllers for operating system configuration, networking, and infrastructure specifics, the end-user has the possibility (and responsibility) to provide these provider-specific configurations as well. Such configurations are not evaluated by Gardener (because it doesn’t know/understand them), but they are only transported to the respective extension controller.

    ⚠️ This means that any configuration issues/mistake on the end-user side that relates to a provider-specific flag or setting cannot be caught during the update request itself but only later during the reconciliation (unless a validator webhook has been registered in the garden cluster by an operator).

    Please see this example manifest and consult the documentation of the provider extension controller to get information about its spec.provider.controlPlaneConfig, .spec.provider.infrastructureConfig, and .spec.provider.workers[].providerConfig.

    (Cluster)OpenIDConnectPresets

    Please see this separate documentation file.

    Overview Data Model

    Gardener Overview Data Model

    1.2.3 - APIServer Admission Plugins

    Admission Plugins

    Similar to the kube-apiserver, the gardener-apiserver comes with a few in-tree managed admission plugins. If you want to get an overview of the what and why of admission plugins then this document might be a good start.

    This document lists all existing admission plugins with a short explanation of what it is responsible for.

    ClusterOpenIDConnectPreset, OpenIDConnectPreset

    (both enabled by default)

    These admission controllers react on CREATE operations for Shoots. If the Shoot does not specify any OIDC configuration (.spec.kubernetes.kubeAPIServer.oidcConfig=nil) then it tries to find a matching ClusterOpenIDConnectPreset or OpenIDConnectPreset, respectively. If there are multiples that match then the one with the highest weight “wins”. In this case, the admission controller will default the OIDC configuration in the Shoot.

    ControllerRegistrationResources

    (enabled by default)

    This admission controller reacts on CREATE and UPDATE operations for ControllerRegistrations. It validates that there exists only one ControllerRegistration in the system that is primarily responsible for a given kind/type resource combination. This prevents misconfiguration by the Gardener administrator/operator.

    CustomVerbAuthorizer

    (enabled by default)

    This admission controller reacts on CREATE and UPDATE operations for Projects. It validates whether the user is bound to a RBAC role with the modify-spec-tolerations-whitelist verb in case the user tries to change the .spec.tolerations.whitelist field of the respective Project resource. Usually, regular project members are not bound to this custom verb, allowing the Gardener administrator to manage certain toleration whitelists on Project basis.

    DeletionConfirmation

    (enabled by default)

    This admission controller reacts on DELETE operations for Projects and Shoots and ShootStates. It validates that the respective resource is annotated with a deletion confirmation annotation, namely confirmation.gardener.cloud/deletion=true. Only if this annotation is present it allows the DELETE operation to pass. This prevents users from accidental/undesired deletions.

    ExposureClass

    (enabled by default)

    This admission controller reacts on Create operations for Shootss. It mutates Shoot resources which has an ExposureClass referenced by merging their both shootSelectors and/or tolerations into the Shoot resource.

    ExtensionValidator

    (enabled by default)

    This admission controller reacts on CREATE and UPDATE operations for BackupEntrys, BackupBuckets, Seeds, and Shoots. For all the various extension types in the specifications of these objects, it validates whether there exists a ControllerRegistration in the system that is primarily responsible for the stated extension type(s). This prevents misconfigurations that would otherwise allow users to create such resources with extension types that don’t exist in the cluster, effectively leading to failing reconciliation loops.

    ExtensionLabels

    (enabled by default)

    This admission controller reacts on CREATE and UPDATE operations for BackupBuckets, BackupEntrys, CloudProfiles, Seeds, SecretBindings and Shoots. For all the various extension types in the specifications of these objects, it adds a corresponding label in the resource. This would allow extension admission webhooks to filter out the resources they are responsible for and ignore all others. This label is of the form <extension-type>.extensions.gardener.cloud/<extension-name> : "true". For example, an extension label for provider extension type aws, looks like provider.extensions.gardener.cloud/aws : "true".

    ProjectValidator

    (enabled by default)

    This admission controller reacts on CREATE operations for Projects. It prevents creating Projects with a non-empty .spec.namespace if the value in .spec.namespace does not start with garden-.

    ⚠️ This admission plugin will be removed in a future release and its business logic will be incorporated into the static validation of the gardener-apiserver.

    ResourceQuota

    (enabled by default)

    This admission controller enables object count ResourceQuotas for Gardener resources, e.g. Shoots, SecretBindings, Projects, etc..

    ⚠️ In addition to this admission plugin, the ResourceQuota controller must be enabled for the Kube-Controller-Manager of your Garden cluster.

    ResourceReferenceManager

    (enabled by default)

    This admission controller reacts on CREATE and UPDATE operations for CloudProfiles, Projects, SecretBindings, Seeds, and Shoots. Generally, it checks whether referred resources stated in the specifications of these objects exist in the system (e.g., if a referenced Secret exists). However, it also has some special behaviours for certain resources:

    • CloudProfiles: It rejects removing Kubernetes or machine image versions if there is at least one Shoot that refers to them.
    • Projects: It sets the .spec.createdBy field for newly created Project resources, and defaults the .spec.owner field in case it is empty (to the same value of .spec.createdBy).
    • Seeds: It rejects changing the .spec.settings.shootDNS.enabled value if there is at least one Shoot that refers to this seed.
    • Shoots: It sets the gardener.cloud/created-by=<username> annotation for newly created Shoot resources.

    SeedValidator

    (enabled by default)

    This admission controller reacts on DELETE operations for Seeds. Rejects the deletion if Shoot(s) reference the seed cluster.

    ShootBinding

    (enabled by default)

    This admission controller reacts on UPDATE operation for Binding subresource of Shoots. It checks for various scheduling constraints for the Seed in the spec.seedName. It rejects the update if SeedChange feature gate is disabled and the Shoot is already assigned to a Seed.

    ShootDNS

    (enabled by default)

    This admission controller reacts on CREATE and UPDATE operations for Shoots. It tries to assign a default domain to the Shoot if it gets scheduled to a seed that enables DNS for shoots (.spec.settings.shootDNS.enabled=true). It also validates that the DNS configuration (.spec.dns) is not set if the seed disables DNS for shoots.

    ShootNodeLocalDNSEnabledByDefault

    (disabled by default)

    This admission controller reacts on CREATE operations for Shoots. If enabled, it will enable node local dns within the shoot cluster (see this doc) by setting spec.systemComponents.nodeLocalDNS.enabled=true for newly created Shoots. Already existing Shoots and new Shoots that explicitly disable node local dns (spec.systemComponents.nodeLocalDNS.enabled=false) will not be affected by this admission plugin.

    ShootQuotaValidator

    (enabled by default)

    This admission controller reacts on CREATE and UPDATE operations for Shoots. It validates the resource consumption declared in the specification against applicable Quota resources. Only if the applicable Quota resources admit the configured resources in the Shoot then it allows the request. Applicable Quotas are referred in the SecretBinding that is used by the Shoot.

    ShootVPAEnabledByDefault

    (disabled by default)

    This admission controller reacts on CREATE operations for Shoots. If enabled, it will enable the managed VerticalPodAutoscaler components (see this doc) by setting spec.kubernetes.verticalPodAutoscaler.enabled=true for newly created Shoots. Already existing Shoots and new Shoots that explicitly disable VPA (spec.kubernetes.verticalPodAutoscaler.enabled=false) will not be affected by this admission plugin.

    ShootTolerationRestriction

    (enabled by default)

    This admission controller reacts on CREATE and UPDATE operations for Shoots. It validates the .spec.tolerations used in Shoots against the whitelist of its Project, or against the whitelist configured in the admission controller’s configuration, respectively. Additionally, it defaults the .spec.tolerations in Shoots with those configured in its Project, and those configured in the admission controller’s configuration, respectively.

    ShootValidator

    (enabled by default)

    This admission controller reacts on CREATE, UPDATE and DELETE operations for Shoots. It validates certain configurations in the specification against the referred CloudProfile (e.g., machine images, machine types, used Kubernetes version, …). Generally, it performs validations that cannot be handled by the static API validation due to their dynamic nature (e.g., when something needs to be checked against referred resources). Additionally, it takes over certain defaulting tasks (e.g., default machine image for worker pools).

    ShootManagedSeed

    (enabled by default)

    This admission controller reacts on UPDATE and DELETE operations for Shoots. It validates certain configuration values in the specification that are specific to ManagedSeeds (e.g. the nginx-addon of the Shoot has to be disabled, the Shoot VPA has to be enabled). It rejects the deletion if the Shoot is referred to by a ManagedSeed.

    ManagedSeedValidator

    (enabled by default)

    This admission controller reacts on CREATE and UPDATE operations for ManagedSeedss. It validates certain configuration values in the specification against the referred Shoot, for example Seed provider, network ranges, DNS domain, etc. Similarly to ShootValidator, it performs validations that cannot be handled by the static API validation due to their dynamic nature. Additionally, it performs certain defaulting tasks, making sure that configuration values that are not specified are defaulted to the values of the referred Shoot, for example Seed provider, network ranges, DNS domain, etc.

    ManagedSeedShoot

    (enabled by default)

    This admission controller reacts on DELETE operations for ManagedSeeds. It rejects the deletion if there are Shoots that are scheduled onto the Seed that is registered by the ManagedSeed.

    ShootDNSRewriting

    (disabled by default)

    This admission controller reacts on CREATE operations for Shoots. If enabled, it adds a set of common suffixes configured in its admission plugin configuration to the Shoot (spec.systemComponents.coreDNS.rewriting.commonSuffixes) (see this doc). Already existing Shoots will not be affected by this admission plugin.

    1.2.4 - Architecture

    Official Definition - What is Kubernetes?

    “Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.”

    Introduction - Basic Principle

    The foundation of the Gardener (providing Kubernetes Clusters as a Service) is Kubernetes itself, because Kubernetes is the go-to solution to manage software in the Cloud, even when it’s Kubernetes itself (see also OpenStack which is provisioned more and more on top of Kubernetes as well).

    While self-hosting, meaning to run Kubernetes components inside Kubernetes, is a popular topic in the community, we apply a special pattern catering to the needs of our cloud platform to provision hundreds or even thousands of clusters. We take a so-called “seed” cluster and seed the control plane (such as the API server, scheduler, controllers, etcd persistence and others) of an end-user cluster, which we call “shoot” cluster, as pods into the “seed” cluster. That means one “seed” cluster, of which we will have one per IaaS and region, hosts the control planes of multiple “shoot” clusters. That allows us to avoid dedicated hardware/virtual machines for the “shoot” cluster control planes. We simply put the control plane into pods/containers and since the “seed” cluster watches them, they can be deployed with a replica count of 1 and only need to be scaled out when the control plane gets under pressure, but no longer for HA reasons. At the same time, the deployments get simpler (standard Kubernetes deployment) and easier to update (standard Kubernetes rolling update). The actual “shoot” cluster consists only out of the worker nodes (no control plane) and therefore the users may get full administrative access to their clusters.

    Setting The Scene - Components and Procedure

    We provide a central operator UI, which we call the “Gardener Dashboard”. It talks to a dedicated cluster, which we call the “Garden” cluster and uses custom resources managed by an aggregated API server, one of the general extension concepts of Kubernetes) to represent “shoot” clusters. In this “Garden” cluster runs the “Gardener”, which is basically a Kubernetes controller that watches the custom resources and acts upon them, i.e. creates, updates/modifies, or deletes “shoot” clusters. The creation follows basically these steps:

    • Create a namespace in the “seed” cluster for the “shoot” cluster which will host the “shoot” cluster control plane
    • Generate secrets and credentials which the worker nodes will need to talk to the control plane
    • Create the infrastructure (using Terraform), which basically consists out of the network setup)
    • Deploy the “shoot” cluster control plane into the “shoot” namespace in the “seed” cluster, containing the “machine-controller-manager” pod
    • Create machine CRDs in the “seed” cluster, describing the configuration and the number of worker machines for the “shoot” (the machine-controller-manager watches the CRDs and creates virtual machines out of it)
    • Wait for the “shoot” cluster API server to become responsive (pods will be scheduled, persistent volumes and load balancers are created by Kubernetes via the respective cloud provider)
    • Finally we deploy kube-system daemons like kube-proxy and further add-ons like the dashboard into the “shoot” cluster and the cluster becomes active

    Overview Architecture Diagram

    Gardener Overview Architecture Diagram

    Detailed Architecture Diagram

    Gardener Detailed Architecture Diagram

    Note: The kubelet as well as the pods inside the “shoot” cluster talk through the front-door (load balancer IP; public Internet) to its “shoot” cluster API server running in the “seed” cluster. The reverse communication from the API server to the pod, service, and node networks happens through a VPN connection that we deploy into “seed” and “shoot” clusters.

    1.2.5 - Backup Restore

    Backup and restore

    Kubernetes uses Etcd as the key-value store for its resource definitions. Gardener supports the backup and restore of etcd. It is the responsibility of the shoot owners to backup the workload data.

    Gardener uses etcd-backup-restore component to backup the etcd backing the Shoot cluster regularly and restore in case of disaster. It is deployed as sidecar via etcd-druid. This doc mainly focuses on the backup and restore configuration used by Gardener when deploying these components. For more details on the design and internal implementation details, please refer GEP-06 and documentation on individual repository.

    Bucket provisioning

    Refer the backup bucket extension document to know details about configuring backup bucket.

    Backup Policy

    etcd-backup-restore supports full snapshot and delta snapshots over full snapshot. In Gardener, this configuration is currently hard-coded to following parameters:

    • Full Snapshot Schedule:
      • Daily, 24hr interval.
      • For each Shoot, the schedule time in a day is randomized based on the configured Shoot maintenance window.
    • Delta Snapshot schedule:
      • At 5min interval.
      • If aggregated events size since last snapshot goes beyond 100Mib.
    • Backup History / Garbage backup deletion policy:
      • Gardener configure backup restore to have Exponential garbage collection policy.
      • As per policy, following backups are retained.
      • All full backups and delta backups for the previous hour.
      • Latest full snapshot of each previous hour for the day.
      • Latest full snapshot of each previous day for 7 days.
      • Latest full snapshot of the previous 4 weeks.
      • Garbage Collection is configured at 12hr interval.
    • Listing:
      • Gardener don’t have any API to list out the backups.
      • To find the backup list, admin can checkout the BackupEntry resource associated with Shoot which holds the bucket and prefix details on object store.

    Restoration

    Restoration process of etcd is automated through the etcd-backup-restore component from latest snapshot. Gardener dosen’t support Point-In-Time-Recovery (PITR) of etcd. In case of etcd disaster, the etcd is recovered from latest backup automatically. For further details, please refer the doc. Post restoration of etcd, the Shoot reconciliation loop brings back the cluster to same state.

    Again, Shoot owner is responsible for maintaining the backup/restore of his workload. Gardener does only take care of the cluster’s etcd.

    1.2.6 - Cluster API

    Relation between Gardener API and Cluster API (SIG Cluster Lifecycle)

    In essence, the Cluster API harmonizes how to get to clusters, while Gardener goes one step further and also harmonizes the clusters themselves. The Cluster API delegates the specifics to so-called providers for infrastructures or control planes via specific CR(D)s while Gardener only has one cluster CR(D). Different Cluster API providers, e.g. for AWS, Azure, GCP, etc. give you vastly different Kubernetes clusters. In contrast, Gardener gives you the exact same clusters with the exact same K8s version, operating system, control plane configuration like for API server or kubelet, add-ons like overlay network, HPA/VPA, DNS and certificate controllers, ingress and network policy controllers, control plane monitoring and logging stacks, down to the behavior of update procedures, auto-scaling, self-healing, etc. on all supported infrastructures. These homogeneous clusters are an essential goal for Gardener as its main purpose is to simplify operations for teams that need to develop and ship software on Kubernetes clusters on a plethora of infrastructures (a.k.a. multi-cloud).

    Incidentally, Gardener influenced the Machine API in the Cluster API with its Machine Controller Manager and was the first to adopt it, see also joint SIG Cluster Lifecycle KubeCon talk where @hardikdr from our Gardener team in India spoke.

    That means, we follow the Cluster API with great interest and are active members. It was completely overhauled from v1alpha1 to v1alpha2. But because v1alpha2 made too many assumptions about the bring-up of masters and was enforcing master machine operations (see here: “As of v1alpha2, Machine-Based is the only control plane type that Cluster API supports”), services that managed their control planes differently like GKE or Gardener couldn’t adopt it (e.g. Google only supports v1alpha1). In 2020 v1alpha3 was introduced and made it possible (again) to integrate managed services like GKE or Gardener. The mapping from the Gardener API to the Cluster API is mostly syntactic.

    To wrap it up, while the Cluster API knows about clusters, it doesn’t know about their make-up. With Gardener, we wanted to go beyond that and harmonize the make-up of the clusters themselves and make them homogeneous across all supported infrastructures. Gardener can therefore deliver homogeneous clusters with exactly the same configuration and behavior on all infrastructures (see also Gardener’s coverage in the official conformance test grid).

    With Cluster API v1alpha3 and the support for declarative control plane management, it became now possible (again) to enable Kubernetes managed services like GKE or Gardener. We would be more than happy, if the community would be interested, to contribute a Gardener control plane provider.

    1.2.7 - Controller Manager

    Gardener Controller Manager

    The gardener-controller-manager (often refered to as “GCM”) is a component that runs next to the Gardener API server, similar to the Kubernetes Controller Manager. It runs several control loops that do not require talking to any seed or shoot cluster. Also, as of today it exposes an HTTP server that is serving several health check endpoints and metrics.

    This document explains the various functionalities of the gardener-controller-manager and their purpose.

    Control Loops

    Bastion Controller

    Bastion resources have a limited lifetime which can be extended up to a certain amount by performing a heartbeat on them. The Bastion controller is responsible for deleting expired or rotten Bastions.

    • “expired” means a Bastion has exceeded its status.expirationTimestamp.
    • “rotten” means a Bastion is older than the configured maxLifetime.

    The maxLifetime defaults to 24 hours and is an option in the BastionControllerConfiguration which is part of gardener-controller-managers ControllerManagerControllerConfiguration, see the example config file for details.

    The controller also deletes Bastions in case the referenced Shoot

    • does no longer exist
    • is marked for deletion (i.e., have a non-nil .metadata.deletionTimestamp)
    • was migrated to another seed (i.e., Shoot.spec.seedName is different than Bastion.spec.seedName).

    The deletion of Bastions triggers the gardenlet to perform the necessary cleanups in the Seed cluster, so some time can pass between deletion and the Bastion actually disappearing. Clients like gardenctl are advised to not re-use Bastions whose deletion timestamp has been set already.

    Refer to GEP-15 for more information on the lifecycle of Bastion resources.

    CertificateSigningRequest Controller

    After the gardenlet gets deployed on the Seed cluster it needs to establish itself as a trusted party to communicate with the Gardener API server. It runs through a bootstrap flow similar to the kubelet bootstrap process.

    On startup the gardenlet uses a kubeconfig with a bootstrap token which authenticates it as being part of the system:bootstrappers group. This kubeconfig is used to create a CertificateSigningRequest (CSR) against the Gardener API server.

    The controller in gardener-controller-manager checks whether the CertificateSigningRequest has the expected organisation, common name and usages which the gardenlet would request.

    It only auto-approves the CSR if the client making the request is allowed to “create” the certificatesigningrequests/seedclient subresource. Clients with the system:bootstrappers group are bound to the gardener.cloud:system:seed-bootstrapper ClusterRole, hence, they have such privileges. As the bootstrap kubeconfig for the gardenlet contains a bootstrap token which is authenticated as being part of the systems:bootstrappers group, its created CSR gets auto-approved.

    CloudProfile Controller

    CloudProfiles are essential when it comes to reconciling Shoots since they contain constraints (like valid machine types, Kubernetes versions, or machine images) and sometimes also some global configuration for the respective environment (typically via provider-specific configuration in .spec.providerConfig).

    Consequently, to ensure that CloudProfiles in-use are always present in the system until the last referring Shoot gets deleted, the controller adds a finalizer which is only released when there is no Shoot referencing the CloudProfile anymore.

    ControllerDeployment Controller

    Extensions are registered in garden cluster via ControllerRegistration and deployment of respective extensions are specified via ControllerDeployment. For more info refer here.

    This controller ensures that ControllerDeployment in-use always exists until the last ControllerRegistration referencing them gets deleted. The controller adds a finalizer which is only released when there is no ControllerRegistration referencing the ControllerDeployment anymore.

    ControllerRegistration Controller

    The ControllerRegistration controller makes sure that the required Gardener Extensions specified by the ControllerRegistration resources are present in the seed clusters. It also takes care of the creation and deletion of ControllerInstallation objects for a given seed cluster. The controller has three reconciliation loops.

    “Main” Reconciler

    This reconciliation loop watches the Seed objects and determines which ControllerRegistrations are required for them and reconciles the corresponding ControllerInstallation resources to reach the determined state. To begin with, it computes the kind/type combinations of extensions required for the seed. For this, the controller examines a live list of ControllerRegistrations, ControllerInstallations, BackupBuckets, BackupEntrys, Shoots, and Secrets from the garden cluster. For example, it examines the shoots running on the seed and deducts kind/type like Infrastructure/gcp. It also decides whether they should always be deployed based on the .spec.deployment.policy. For the configuration options, please see this section.

    Based on these required combinations, each of them are mapped to ControllerRegistration objects and then to their corresponding ControllerInstallation objects (if existing). The controller then creates or updates the required ControllerInstallation objects for the given seed. It also deletes every existing ControllerInstallation whose referenced ControllerRegistration is not part of the required list. For example, if the shoots in the seed are no longer using the DNS provider aws-route53, then the controller proceeds to delete the respective ControllerInstallation object.

    “ControllerRegistration” Reconciler

    This reconciliation loop watches the ControllerRegistration resource and adds finalizers to it when they are created. In case a deletion request comes in for the resource, i.e., if a .metadata.deletionTimestamp is set, it actively scans for a ControllerInstallation resource using this ControllerRegistration, and decides whether the deletion can be allowed. In case no related ControllerInstallation is present, it removes the finalizer and marks it for deletion.

    “Seed” Reconciler

    This loop also watches the Seed object and adds finalizers to it at creation. If a .metadata.deletionTimestamp is set for the seed then the controller checks for existing ControllerInstallation objects which reference this seed. If no such objects exist then it removes the finalizer and allows the deletion.

    Event Controller

    With the Gardener Event Controller you can prolong the lifespan of events related to Shoot clusters. This is an optional controller which will become active once you provide the below mentioned configuration.

    All events in K8s are deleted after a configurable time-to-live (controlled via a kube-apiserver argument called --event-ttl (defaulting to 1 hour)). The need to prolong the time-to-live for Shoot cluster events frequently arises when debugging customer issues on live systems. This controller leaves events involving Shoots untouched while deleting all other events after a configured time. In order to activate it, provide the following configuration:

    • concurrentSyncs: The amount of goroutines scheduled for reconciling events.
    • ttlNonShootEvents: When an event reaches this time-to-live it gets deleted unless it is a Shoot-related event (defaults to 1h, equivalent to the event-ttl default).

    ⚠️ In addition, you should also configure the --event-ttl for the kube-apiserver to define an upper-limit of how long Shoot-related events should be stored. The --event-ttl should be larger than the ttlNonShootEvents or this controller will have no effect.

    ExposureClass Controller

    ExposureClass abstracts the ability to expose a Shoot clusters control plane in certain network environments (e.g. corporate networks, DMZ, internet) on all Seeds or a subset of the Seeds. For more refer.

    Consequently, to ensure that ExposureClasss in-use are always present in the system until the last referring Shoot gets deleted, the controller adds a finalizer which is only released when there is no Shoot referencing the ExposureClass anymore.

    Quota Controller

    Quota object limits the resources consumed by shoot clusters either per provider secret or per project/namespace.

    Consequently, to ensure that Quotas in-use are always present in the system until the last SecretBinding that references them gets deleted, the controller adds a finalizer which is only released when there is no SecretBinding referencing the Quota anymore.

    Project Controller

    There are multiple controllers responsible for different aspects of Project objects. Please also refer to the Project documentation.

    “Main” Reconciler

    This reconciler manages a dedicated Namespace for each Project. The namespace name can either be specified explicitly in .spec.namespace (must be prefixed with garden-) or it will be determined by the controller. If .spec.namespace is set, it tries to create it. If it already exists, it tries to adopt it. This will only succeed if the Namespace was previously labeled with gardener.cloud/role=project and project.gardener.cloud/name=<project-name>. This is to prevent that end-users can adopt arbitrary namespaces and escalate their privileges, e.g. the kube-system namespace.

    After the namespace was created/adopted the controller creates several ClusterRoles and ClusterRoleBindings that allow the project members to access related resources based on their roles. These RBAC resources are prefixed with gardener.cloud:system:project{-member,-viewer}:<project-name>. Gardener administrators and extension developers can define their own roles, see this document for more information.

    In addition, operators can configure the Project controller to maintain a default ResourceQuota for project namespaces. Quotas can especially limit the creation of user facing resources, e.g. Shoots, SecretBindings, Secrets and thus protect the Garden cluster from massive resource exhaustion but also enable operators to align quotas with respective enterprise policies.

    ⚠️ Gardener itself is not exempted from configured quotas. For example, Gardener creates Secrets for every shoot cluster in the project namespace and at the same time increases the available quota count. Please mind this additional resource consumption.

    The controller configuration provides a template section controllers.project.quotas where such a ResourceQuota (see example below) can be deposited.

    controllers:
      project:
        quotas:
        - config:
            apiVersion: v1
            kind: ResourceQuota
            spec:
              hard:
                count/shoots.core.gardener.cloud: "100"
                count/secretbindings.core.gardener.cloud: "10"
                count/secrets: "800"
          projectSelector: {}
    

    The Project controller takes the specified config and creates a ResourceQuota with the name gardener in the project namespace. If a ResourceQuota resource with the name gardener already exists, the controller will only update fields in spec.hard which are unavailable at that time. This is done to configure a default Quota in all projects but to allow manual quota increases as the projects’ demands increase. spec.hard fields in the ResourceQuota object that are not present in the configuration are removed from the object. Labels and annotations on the ResourceQuota config get merged with the respective fields on existing ResourceQuotas. An optional projectSelector narrows down the amount of projects that are equipped with the given config. If multiple configs match for a project, then only the first match in the list is applied to the project namespace.

    The .status.phase of the Project resources is set to Ready or Failed by the reconciler to indicate whether the reconciliation loop was performed successfully. Also, it generates Events to provide further information about its operations.

    When a Project is marked for deletion, the controller ensures that there are no Shoots left in the project namespace. Once all Shoots are gone, the Namespace and Project is released.

    “Stale Projects” Reconciler

    As Gardener is a large-scale Kubernetes as a Service it is designed for being used by a large amount of end-users. Over time, it is likely to happen that some of the hundreds or thousands of Project resources are no longer actively used.

    Gardener offers the “stale projects” reconciler which will take care of identifying such stale projects, marking them with a “warning”, and eventually deleting them after a certain time period. This reconciler is enabled by default and works as following:

    1. Projects are considered as “stale”/not actively used when all of the following conditions apply: The namespace associated with the Project does not have any…
      1. Shoot resources.
      2. BackupEntry resources.
      3. Secret resources that are referenced by a SecretBinding that is in use by a Shoot (not necessarily in the same namespace).
      4. Quota resources that are referenced by a SecretBinding that is in use by a Shoot (not necessarily in the same namespace).
      5. The time period when the project was used for the last time (status.lastActivityTimestamp) is longer than the configured minimumLifetimeDays

    If a project is considered “stale” then its .status.staleSinceTimestamp will be set to the time when it was first detected to be stale. If it gets actively used again this timestamp will be removed. After some time the .status.staleAutoDeleteTimestamp will be set to a timestamp after which Gardener will auto-delete the Project resource if it still is not actively used.

    The component configuration of the gardener-controller-manager offers to configure the following options:

    • minimumLifetimeDays: Don’t consider newly created Projects as “stale” too early to give people/end-users some time to onboard and get familiar with the system. The “stale project” reconciler won’t set any timestamp for Projects younger than minimumLifetimeDays. When you change this value then projects marked as “stale” may be no longer marked as “stale” in case they are young enough, or vice versa.
    • staleGracePeriodDays: Don’t compute auto-delete timestamps for stale Projects that are unused for only less than staleGracePeriodDays. This is to not unnecessarily make people/end-users nervous “just because” they haven’t actively used their Project for a given amount of time. When you change this value then already assigned auto-delete timestamps may be removed again if the new grace period is not yet exceeded.
    • staleExpirationTimeDays: Expiration time after which stale Projects are finally auto-deleted (after .status.staleSinceTimestamp). If this value is changed and an auto-delete timestamp got already assigned to the projects then the new value will only take effect if it’s increased. Hence, decreasing the staleExpirationTimeDays will not decrease already assigned auto-delete timestamps.

    Gardener administrators/operators can exclude specific Projects from the stale check by annotating the related Namespace resource with project.gardener.cloud/skip-stale-check=true.

    “Activity” Reconciler

    Since the other two reconcilers are unable to actively monitor the relevant objects that are used in a Project (Shoot, Secret, etc.), there could be a situation where the user creates and deletes objects in a short period of time. In that case the Stale Project Reconciler could not see that there was any activity on that project and it will still mark it as a Stale, even though it is actively used.

    The Project Activity Reconciler is implemented to take care of such cases. An event handler will notify the reconciler for any acitivity and then it will update the status.lastActivityTimestamp. This update will also trigger the Stale Project Reconciler.

    SecretBinding Controller

    SecretBindings reference Secrets and Quotas and are themselves referenced by Shoots. The controller adds finalizers to the referenced objects to ensure they don’t get deleted while still being referenced. Similarly, to ensure that SecretBindings in-use are always present in the system until the last referring Shoot gets deleted, the controller adds a finalizer which is only released when there is no Shoot referencing the SecretBinding anymore.

    Referenced Secrets will also be labeled with provider.shoot.gardener.cloud/<type>=true where <type> is the value of the .provider.type of the SecretBinding. Also, all referenced Secrets as well as Quotas will be labeled with reference.gardener.cloud/secretbinding=true to allow easily filtering for objects referenced by SecretBindings.

    Seed Controller

    The Seed controller in the gardener-controller-manager reconciles Seed objects with the help of the following reconcilers.

    “Main” Reconciler

    This reconciliation loop takes care about seed related operations in the Garden cluster. When a new Seed object is created the reconciler creates a new Namespace in the garden cluster seed-<seed-name>. Namespaces dedicated to single seed clusters allow us to segregate access permissions i.e., a gardenlet must not have permissions to access objects in all Namespaces in the Garden cluster. There are objects in a Garden environment which are created once by the operator e.g., default domain secret, alerting credentials, and required for operations happening in the gardenlet. Therefore, we not only need a seed specific Namespace but also a copy of these “shared” objects.

    The “main” reconciler takes care about this replication:

    KindNamespaceLabel Selector
    Secretgardengardener.cloud/role

    “Backup Buckets Check” Reconciler

    Every time a BackupBucket object is created or updated, the referenced Seed object is enqueued for reconciliation. It’s the reconciler’s task to check the status subresource of all existing BackupBuckets that reference this Seed. If at least one BackupBucket has .status.lastError != nil, the BackupBucketsReady condition on the Seed will be set to False, and consequently the Seed is considered as NotReady. If the SeedBackupBucketsCheckControllerConfiguration (which is part of gardener-controller-managers component configuration) contains a conditionThreshold for the BackupBucketsReady, the condition will instead first be set to Progressing and eventually to False once the conditionThreshold expires, see the example config file for details. Once the BackupBucket is healthy again, the seed will be re-queued and the condition will turn true.

    “Extensions Check” Reconciler

    This reconciler reconciles Seed objects and checks whether all ControllerInstallations referencing them are in a healthy state. Concretely, all three conditions Valid, Installed, and Healthy must have status True and the Progressing condition must have status False. Based on this check, it maintains the ExtensionsReady condition in the respective Seed’s .status.conditions list.

    “Lifecycle” Reconciler

    The “Lifecycle” reconciler processes Seed objects which are enqueued every 10 seconds in order to check if the responsible gardenlet is still responding and operable. Therefore, it checks renewals via Lease objects of the seed in the garden cluster which are renewed regularly by the gardenlet.

    In case a Lease is not renewed for the configured amount in config.controllers.seed.monitorPeriod.duration:

    1. The reconciler assumes that the gardenlet stopped operating and updates the GardenletReady condition to Unknown.
    2. Additionally, conditions and constraints of all Shoot resources scheduled on the affected seed are set to Unknown as well because a striking gardenlet won’t be able to maintain these conditions any more.
    3. If the gardenlet’s client certificate has expired (identified based on the .status.clientCertificateExpirationTimestamp field in the Seed resource) and if it is managed by a ManagedSeed then this will be triggered for a reconciliation. This will trigger the bootstrapping process again and allows gardenlets to obtain a fresh client certificate.

    Shoot Controller

    “Conditions” Reconciler

    In case the reconciled Shoot is registered via a ManagedSeed as a seed cluster, this reconciler merges the conditions in the respective Seed’s .status.conditions into the .status.conditions of the Shoot. This is to provide a holistic view on the status of the registered seed cluster by just looking at the Shoot resource.

    “Hibernation” Reconciler

    This reconciler is responsible for hibernating or awakening shoot clusters based on the schedules defined in their .spec.hibernation.schedules. It ignores failed Shoots and those marked for deletion.

    “Maintenance” Reconciler

    This reconciler is responsible for maintaining shoot clusters based on the time window defined in their .spec.maintenance.timeWindow. It might auto-update the Kubernetes version or the operating system versions specified in the worker pools (.spec.provider.workers). It could also add some operation or task annotations, read more here.

    “Quota” Reconciler

    This reconciler might auto-delete shoot clusters in case their referenced SecretBinding is itself referencing a Quota with .spec.clusterLifetimeDays != nil. If the shoot cluster is older than the configured lifetime then it gets deleted. It maintains the expiration time of the Shoot in the value of the shoot.gardener.cloud/expiration-timestamp annotation. This annotation might be overridden, however only by at most twice the value of the .spec.clusterLifetimeDays.

    “Reference” Reconciler

    Shoot objects may specify references to other objects in the Garden cluster which are required for certain features. For example, users can configure various DNS providers via .spec.dns.providers and usually need to refer to a corresponding Secret with valid DNS provider credentials inside. Such objects need a special protection against deletion requests as long as they are still being referenced by one or multiple shoots.

    Therefore, this reconciler checks Shoots for referenced objects and adds the finalizer gardener.cloud/reference-protection to their .metadata.finalizers list. The reconciled Shoot also gets this finalizer to enable a proper garbage collection in case the gardener-controller-manager is offline at the moment of an incoming deletion request. When an object is not actively referenced anymore because the Shoot specification has changed or all related shoots were deleted (are in deletion), the controller will remove the added finalizer again so that the object can safely be deleted or garbage collected.

    This reconciler inspects the following references:

    • DNS provider secrets (.spec.dns.provider)
    • Audit policy configmaps (.spec.kubernetes.kubeAPIServer.auditConfig.auditPolicy.configMapRef)

    Further checks might be added in the future.

    “Retry” Reconciler

    This reconciler is responsible for retrying certain failed Shoots. Currently, the reconciler retries only failed Shoots with error code ERR_INFRA_RATE_LIMITS_EXCEEDED, see this document for more details.

    “Status Label” Reconciler

    This reconciler is responsible for maintaining the shoot.gardener.cloud/status label on Shoots, see this document for more details.

    1.2.8 - Etcd

    etcd - Key-Value Store for Kubernetes

    etcd is a strongly consistent key-value store and the most prevalent choice for the Kubernetes persistence layer. All API cluster objects like Pods, Deployments, Secrets, etc. are stored in etcd which makes it an essential part of a Kubernetes control plane.

    Shoot cluster persistence

    Each shoot cluster gets its very own persistence for the control plane. It runs in the shoot namespace on the respective seed cluster. Concretely, there are two etcd instances per shoot cluster which the Kube-Apiserver is configured to use in the following way:

    • etcd-main

    A store that contains all “cluster critical” or “long-term” objects. These object kinds are typically considered for a backup to prevent any data loss.

    • etcd-events

    A store that contains all Event objects (events.k8s.io) of a cluster. Events have usually a short retention period, occur frequently but are not essential for a disaster recovery.

    The setup above prevents both, the critical etcd-main is not flooded by Kubernetes Events as well as backup space is not occupied by non-critical data. This segmentation saves time and resources.

    etcd Operator

    Configuring, maintaining and health-checking etcd is outsourced to a dedicated operator called ETCD Druid. When Gardenlet reconciles a Shoot resource, it creates or updates an Etcd resources in the seed cluster, containing necessary information (backup information, defragmentation schedule, resources, etc.) etcd-druid needs to manage the lifecycle of the desired etcd instance (today main or events). Likewise, when the shoot is deleted, Gardenlet deletes the Etcd resource and ETCD Druid takes care about cleaning up all related objects, e.g. the backing StatefulSet.

    Autoscaling

    Gardenlet maintains HVPA objects for etcd StatefulSets if the corresponding feature gate is enabled. This enables a vertical scaling for etcd. Downscaling is handled more pessimistic to prevent many subsequent etcd restarts. Thus, for production and infrastructure clusters downscaling is deactivated and for all other clusters lower advertised requests/limits are only applied during a shoot’s maintenance time window.

    Backup

    If Seeds specify backups for etcd (example), then Gardener and the respective provider extensions are responsible for creating a bucket on the cloud provider’s side (modelled through BackupBucket resource). The bucket stores backups of shoots scheduled on that seed. Furthermore, Gardener creates a BackupEntry which subdivides the bucket and thus makes it possible to store backups of multiple shoot clusters.

    The etcd-main instance itself is configured to run with a special backup-restore sidecar. It takes care about regularly backing up etcd data and restoring it in case of data loss. More information can be found on the component’s GitHub page https://github.com/gardener/etcd-backup-restore.

    How long backups are stored in the bucket after a shoot has been deleted, depends on the configured retention period in the Seed resource. Please see this example configuration for more information.

    Housekeeping

    etcd maintenance tasks must be performed from time to time in order to re-gain database storage and to ensure the system’s reliability. The backup-restore sidecar takes care about this job as well. Gardener chooses a random time within the shoot’s maintenance time to schedule these tasks.

    1.2.9 - Gardenlet

    Gardenlet

    Gardener is implemented using the operator pattern: It uses custom controllers that act on our own custom resources, and apply Kubernetes principles to manage clusters instead of containers. Following this analogy, you can recognize components of the Gardener architecture as well-known Kubernetes components, for example, shoot clusters can be compared with pods, and seed clusters can be seen as worker nodes.

    The following Gardener components play a similar role as the corresponding components in the Kubernetes architecture:

    Gardener ComponentKubernetes Component
    gardener-apiserverkube-apiserver
    gardener-controller-managerkube-controller-manager
    gardener-schedulerkube-scheduler
    gardenletkubelet

    Similar to how the kube-scheduler of Kubernetes finds an appropriate node for newly created pods, the gardener-scheduler of Gardener finds an appropriate seed cluster to host the control plane for newly ordered clusters. By providing multiple seed clusters for a region or provider, and distributing the workload, Gardener also reduces the blast radius of potential issues.

    Kubernetes runs a primary “agent” on every node, the kubelet, which is responsible for managing pods and containers on its particular node. Decentralizing the responsibility to the kubelet has the advantage that the overall system is scalable. Gardener achieves the same for cluster management by using a gardenlet as primary “agent” on every seed cluster, and is only responsible for shoot clusters located in its particular seed cluster:

    Counterparts in the Gardener Architecture and the Kubernetes Architecture

    The gardener-controller-manager has control loops to manage resources of the Gardener API. However, instead of letting the gardener-controller-manager talk directly to seed clusters or shoot clusters, the responsibility isn’t only delegated to the gardenlet, but also managed using a reversed control flow: It’s up to the gardenlet to contact the Gardener API server, for example, to share a status for its managed seed clusters.

    Reversing the control flow allows placing seed clusters or shoot clusters behind firewalls without the necessity of direct access via VPN tunnels anymore.

    Reversed Control Flow Using a Gardenlet

    TLS Bootstrapping

    Kubernetes doesn’t manage worker nodes itself, and it’s also not responsible for the lifecycle of the kubelet running on the workers. Similarly, Gardener doesn’t manage seed clusters itself, so Gardener is also not responsible for the lifecycle of the gardenlet running on the seeds. As a consequence, both the gardenlet and the kubelet need to prepare a trusted connection to the Gardener API server and the Kubernetes API server correspondingly.

    To prepare a trusted connection between the gardenlet and the Gardener API server, the gardenlet initializes a bootstrapping process after you deployed it into your seed clusters:

    1. The gardenlet starts up with a bootstrap kubeconfig having a bootstrap token that allows to create CertificateSigningRequest (CSR) resources.

    2. After the CSR is signed, the gardenlet downloads the created client certificate, creates a new kubeconfig with it, and stores it inside a Secret in the seed cluster.

    3. The gardenlet deletes the bootstrap kubeconfig secret, and starts up with its new kubeconfig.

    4. The gardenlet starts normal operation.

    The gardener-controller-manager runs a control loop that automatically signs CSRs created by gardenlets.

    The gardenlet bootstrapping process is based on the kubelet bootstrapping process. More information: Kubelet’s TLS bootstrapping.

    If you don’t want to run this bootstrap process you can create a kubeconfig pointing to the garden cluster for the gardenlet yourself, and use field gardenClientConnection.kubeconfig in the gardenlet configuration to share it with the gardenlet.

    Gardenlet Certificate Rotation

    The certificate used to authenticate the gardenlet against the API server has a certain validity based on the configuration of the garden cluster (--cluster-signing-duration flag of the kube-controller-manager (default 1y)).

    If your garden cluster is of at least Kubernetes v1.22 then you can also configure the validity for the client certificate by specifying .gardenClientConnection.kubeconfigValidity.validity in the gardenlet’s component configuration. Note that changing this value will only take effect when the kubeconfig is rotated again (it is not picked up immediately). The minimum validity is 10m (that’s what is enforced by the CertificateSigningRequest API in Kubernetes which is used by gardenlet).

    By default, after about 70-90% of the validity expired, the gardenlet tries to automatically replace the current certificate with a new one (certificate rotation).

    You can change these boundaries by specifying .gardenClientConnection.kubeconfigValidity.autoRotationJitterPercentage{Min,Max} in the gardenlet’s component configuration.

    To use certificate rotation, you need to specify the secret to store the kubeconfig with the rotated certificate in field .gardenClientConnection.kubeconfigSecret of the gardenlet component configuration.

    Rotate certificates using bootstrap kubeconfig

    If the gardenlet created the certificate during the initial TLS Bootstrapping using the Bootstrap kubeconfig, certificates can be rotated automatically. The same control loop in the gardener-controller-manager that signs the CSRs during the initial TLS Bootstrapping also automatically signs the CSR during a certificate rotation.

    ℹ️ You can trigger an immediate renewal by annotating the Secret in the seed cluster stated in the .gardenClientConnection.kubeconfigSecret field with gardener.cloud/operation=renew and restarting the gardenlet. After it booted up again, gardenlet will issue a new certificate independent of the remaining validity of the existing one.

    Rotate Certificate Using Custom kubeconfig

    When trying to rotate a custom certificate that wasn’t created by gardenlet as part of the TLS Bootstrap, the x509 certificate’s Subject field needs to conform to the following:

    • the Common Name (CN) is prefixed with gardener.cloud:system:seed:
    • the Organization (O) equals gardener.cloud:system:seeds

    Otherwise, the gardener-controller-manager doesn’t automatically sign the CSR. In this case, an external component or user needs to approve the CSR manually, for example, using command kubectl certificate approve seed-csr-<...>). If that doesn’t happen within 15 minutes, the gardenlet repeats the process and creates another CSR.

    Configuring the Seed to work with

    The Gardenlet works with a single seed, which must be configured in the GardenletConfiguration under .seedConfig. This must be a copy of the Seed resource, for example (see example/20-componentconfig-gardenlet.yaml for a more complete example):

    apiVersion: gardenlet.config.gardener.cloud/v1alpha1
    kind: GardenletConfiguration
    seedConfig:
      metadata:
        name: my-seed
      spec:
        provider:
          type: aws
        # ...
        secretRef:
          name: my-seed-secret
          namespace: garden
    

    When using make start-gardenlet, the corresponding script will automatically fetch the seed cluster’s kubeconfig based on the seedConfig.spec.secretRef and set the environment accordingly.

    On startup, gardenlet registers a Seed resource using the given template in seedConfig if it’s not present already.

    Component Configuration

    In the component configuration for the gardenlet, it’s possible to define:

    • settings for the Kubernetes clients interacting with the various clusters
    • settings for the control loops inside the gardenlet
    • settings for leader election and log levels, feature gates, and seed selection or seed configuration.

    More information: Example Gardenlet Component Configuration.

    Heartbeats

    Similar to how Kubernetes uses Lease objects for node heart beats (see KEP), the gardenlet is using Lease objects for heart beats of the seed cluster. Every two seconds, the gardenlet checks that the seed cluster’s /healthz endpoint returns HTTP status code 200. If that is the case, the gardenlet renews the lease in the Garden cluster in the gardener-system-seed-lease namespace and updates the GardenletReady condition in the status.conditions field of the Seed resource(s).

    Similarly to the node-lifecycle-controller inside the kube-controller-manager, the gardener-controller-manager features a seed-lifecycle-controller that sets the GardenletReady condition to Unknown in case the gardenlet fails to renew the lease. As a consequence, the gardener-scheduler doesn’t consider this seed cluster for newly created shoot clusters anymore.

    /healthz Endpoint

    The gardenlet includes an HTTPS server that serves a /healthz endpoint. It’s used as a liveness probe in the Deployment of the gardenlet. If the gardenlet fails to renew its lease then the endpoint returns 500 Internal Server Error, otherwise it returns 200 OK.

    Please note that the /healthz only indicates whether the gardenlet could successfully probe the Seed’s API server and renew the lease with the Garden cluster. It does not show that the Gardener extension API server (with the Gardener resource groups) is available. However, the Gardenlet is designed to withstand such connection outages and retries until the connection is reestablished.

    Control Loops

    The gardenlet consists out of several controllers which are now described in more detail.

    ⚠️ This section is not necessarily complete and might be under construction.

    BackupEntry Controller

    The BackupEntry controller reconciles those core.gardener.cloud/v1beta1.BackupEntry resources whose .spec.seedName value is equal to the name of a Seed the respective gardenlet is responsible for. Those resources are created by the Shoot controller (only if backup is enabled for the respective Seed) and there is exactly one BackupEntry per Shoot.

    The controller creates an extensions.gardener.cloud/v1alpha1.BackupEntry resource (non-namespaced) in the seed cluster and waits until the responsible extension controller reconciled it (see this for more details). The status is populated in the .status.lastOperation field.

    The core.gardener.cloud/v1beta1.BackupEntry resource has an owner reference pointing to the corresponding Shoot. Hence, if the Shoot is deleted, also the BackupEntry resource gets deleted. In this case, the controller deletes the extensions.gardener.cloud/v1alpha1.BackupEntry resource in the seed cluster and waits until the responsible extension controller has deleted it. Afterwards, the finalizer of the core.gardener.cloud/v1beta1.BackupEntry resource is released so that it finally disappears from the system.

    Keep Backup for Deleted Shoots

    In some scenarios it might be beneficial to not immediately delete the BackupEntrys (and with them, the etcd backup) for deleted Shoots.

    In this case you can configure the .controllers.backupEntry.deletionGracePeriodHours field in the component configuration of the gardenlet. For example, if you set it to 48, then the BackupEntrys for deleted Shoots will only be deleted 48 hours after the Shoot was deleted.

    Additionally, you can limit the shoot purposes for which this applies by setting .controllers.backupEntry.deletionGracePeriodShootPurposes[]. For example, if you set it to [production] then only the BackupEntrys for Shoots with .spec.purpose=production will be deleted after the configured grace period. All others will be deleted immediately after the Shoot deletion.

    Managed Seeds

    Gardener users can use shoot clusters as seed clusters, so-called “managed seeds” (aka “shooted seeds”), by creating ManagedSeed resources. By default, the gardenlet that manages this shoot cluster then automatically creates a clone of itself with the same version and the same configuration that it currently has. Then it deploys the gardenlet clone into the managed seed cluster.

    If you want to prevent the automatic gardenlet deployment, specify the seedTemplate section in the ManagedSeed resource, and don’t specify the gardenlet section. In this case, you have to deploy the gardenlet on your own into the seed cluster.

    More information: Register Shoot as Seed

    Migrating from Previous Gardener Versions

    If your Gardener version doesn’t support gardenlets yet, no special migration is required, but the following prerequisites must be met:

    • Your Gardener version is at least 0.31 before upgrading to v1.
    • You have to make sure that your garden cluster is exposed in a way that it’s reachable from all your seed clusters.

    With previous Gardener versions, you had deployed the Gardener Helm chart (incorporating the API server, controller-manager, and scheduler). With v1, this stays the same, but you now have to deploy the gardenlet Helm chart as well into all of your seeds (if they aren’t managed, as mentioned earlier).

    More information: Deploy a Gardenlet for all instructions.

    Gardener Architecture

    Issue #356: Implement Gardener Scheduler

    PR #2309: Add /healthz endpoint for Gardenlet

    1.2.10 - Network Policies

    Network Policies in Gardener

    As Seed clusters can host the Kubernetes control planes of many Shoot clusters, it is necessary to isolate the control planes from each other for security reasons. Besides deploying each control plane in its own namespace, Gardener creates network policies to also isolate the networks. Essentially, network policies make sure that pods can only talk to other pods over the network they are supposed to. As such, network policies are an important part of Gardener’s tenant isolation.

    Gardener deploys network policies into

    • each namespace hosting the Kubernetes control plane of the Shoot cluster.
    • the namespace dedicated to Gardener seed-wide global controllers. This namespace is often called garden and contains e.g. the Gardenlet.
    • the kube-system namespace in the Shoot.

    The aforementioned namespaces in the Seed contain a deny-all network policy that denies all ingress and egress traffic. This secure by default setting requires pods to allow network traffic. This is done by pods having labels matching to the selectors of the network policies deployed by Gardener.

    More details on the deployed network policies can be found in the development and usage sections.

    1.2.11 - Resource Manager

    Gardener Resource Manager

    Initially, the gardener-resource-manager was a project similar to the kube-addon-manager. It manages Kubernetes resources in a target cluster which means that it creates, updates, and deletes them. Also, it makes sure that manual modifications to these resources are reconciled back to the desired state.

    In the Gardener project we were using the kube-addon-manager since more than two years. While we have progressed with our extensibility story (moving cloud providers out-of-tree) we had decided that the kube-addon-manager is no longer suitable for this use-case. The problem with it is that it needs to have its managed resources on its file system. This requires storing the resources in ConfigMaps or Secrets and mounting them to the kube-addon-manager pod during deployment time. The gardener-resource-manager uses CustomResourceDefinitions which allows to dynamically add, change, and remove resources with immediate action and without the need to reconfigure the volume mounts/restarting the pod.

    Meanwhile, the gardener-resource-manager has evolved to a more generic component comprising several controllers and webhook handlers. It is deployed by gardenlet once per seed (in the garden namespace) and once per shoot (in the respective shoot namespaces in the seed).

    Controllers

    ManagedResource controller

    This controller watches custom objects called ManagedResources in the resources.gardener.cloud/v1alpha1 API group. These objects contain references to secrets which itself contain the resources to be managed. The reason why a Secret is used to store the resources is that they could contain confidential information like credentials.

    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: managedresource-example1
      namespace: default
    type: Opaque
    data:
      objects.yaml: YXBpVmVyc2lvbjogdjEKa2luZDogQ29uZmlnTWFwCm1ldGFkYXRhOgogIG5hbWU6IHRlc3QtMTIzNAogIG5hbWVzcGFjZTogZGVmYXVsdAotLS0KYXBpVmVyc2lvbjogdjEKa2luZDogQ29uZmlnTWFwCm1ldGFkYXRhOgogIG5hbWU6IHRlc3QtNTY3OAogIG5hbWVzcGFjZTogZGVmYXVsdAo=
        # apiVersion: v1
        # kind: ConfigMap
        # metadata:
        #   name: test-1234
        #   namespace: default
        # ---
        # apiVersion: v1
        # kind: ConfigMap
        # metadata:
        #   name: test-5678
        #   namespace: default
    ---
    apiVersion: resources.gardener.cloud/v1alpha1
    kind: ManagedResource
    metadata:
      name: example
      namespace: default
    spec:
      secretRefs:
      - name: managedresource-example1
    

    In the above example, the controller creates two ConfigMaps in the default namespace. When a user is manually modifying them they will be reconciled back to the desired state stored in the managedresource-example secret.

    It is also possible to inject labels into all the resources:

    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: managedresource-example2
      namespace: default
    type: Opaque
    data:
      other-objects.yaml: YXBpVmVyc2lvbjogYXBwcy92MSAjIGZvciB2ZXJzaW9ucyBiZWZvcmUgMS45LjAgdXNlIGFwcHMvdjFiZXRhMgpraW5kOiBEZXBsb3ltZW50Cm1ldGFkYXRhOgogIG5hbWU6IG5naW54LWRlcGxveW1lbnQKc3BlYzoKICBzZWxlY3RvcjoKICAgIG1hdGNoTGFiZWxzOgogICAgICBhcHA6IG5naW54CiAgcmVwbGljYXM6IDIgIyB0ZWxscyBkZXBsb3ltZW50IHRvIHJ1biAyIHBvZHMgbWF0Y2hpbmcgdGhlIHRlbXBsYXRlCiAgdGVtcGxhdGU6CiAgICBtZXRhZGF0YToKICAgICAgbGFiZWxzOgogICAgICAgIGFwcDogbmdpbngKICAgIHNwZWM6CiAgICAgIGNvbnRhaW5lcnM6CiAgICAgIC0gbmFtZTogbmdpbngKICAgICAgICBpbWFnZTogbmdpbng6MS43LjkKICAgICAgICBwb3J0czoKICAgICAgICAtIGNvbnRhaW5lclBvcnQ6IDgwCg==
        # apiVersion: apps/v1
        # kind: Deployment
        # metadata:
        #   name: nginx-deployment
        # spec:
        #   selector:
        #     matchLabels:
        #       app: nginx
        #   replicas: 2 # tells deployment to run 2 pods matching the template
        #   template:
        #     metadata:
        #       labels:
        #         app: nginx
        #     spec:
        #       containers:
        #       - name: nginx
        #         image: nginx:1.7.9
        #         ports:
        #         - containerPort: 80
    
    ---
    apiVersion: resources.gardener.cloud/v1alpha1
    kind: ManagedResource
    metadata:
      name: example
      namespace: default
    spec:
      secretRefs:
      - name: managedresource-example2
      injectLabels:
        foo: bar
    

    In this example the label foo=bar will be injected into the Deployment as well as into all created ReplicaSets and Pods.

    Preventing Reconciliations

    If a ManagedResource is annotated with resources.gardener.cloud/ignore=true then it will be skipped entirely by the controller (no reconciliations or deletions of managed resources at all). However, when the ManagedResource itself is deleted (for example when a shoot is deleted) then the annotation is not respected and all resources will be deleted as usual. This feature can be helpful to temporarily patch/change resources managed as part of such ManagedResource. Condition checks will be skipped for such ManagedResources.

    Modes

    The gardener-resource-manager can manage a resource in different modes. The supported modes are:

    • Ignore
      • The corresponding resource is removed from the ManagedResource status (.status.resources). No action is performed on the cluster - the resource is no longer “managed” (updated or deleted).
      • The primary use case is a migration of a resource from one ManagedResource to another one.

    The mode for a resource can be specified with the resources.gardener.cloud/mode annotation. The annotation should be specified in the encoded resource manifest in the Secret that is referenced by the ManagedResource.

    Skipping health check

    If a resource in the ManagedResource is annotated with resources.gardener.cloud/skip-health-check=true then the resource will be skipped during health checks by the health controller. The ManagedResource conditions will not reflect the health condition of this resource anymore. The ResourcesProgressing condition will also be set to False.

    Resource Class

    By default, gardener-resource-manager controller watches for ManagedResources in all namespaces. --namespace flag can be specified to gardener-resource-manager binary to restrict the watch to ManagedResources in a single namespace. A ManagedResource has an optional .spec.class field that allows to indicate that it belongs to given class of resources. --resource-class flag can be specified to gardener-resource-manager binary to restrict the watch to ManagedResources with the given .spec.class. A default class is assumed if no class is specified.

    Conditions

    A ManagedResource has a ManagedResourceStatus, which has an array of Conditions. Conditions currently include:

    ConditionDescription
    ResourcesAppliedTrue if all resources are applied to the target cluster
    ResourcesHealthyTrue if all resources are present and healthy
    ResourcesProgressingFalse if all resources have been fully rolled out

    ResourcesApplied may be False when:

    • the resource apiVersion is not known to the target cluster
    • the resource spec is invalid (for example the label value does not match the required regex for it)

    ResourcesHealthy may be False when:

    • the resource is not found
    • the resource is a Deployment and the Deployment does not have the minimum availability.

    ResourcesProgressing may be True when:

    • a Deployment, StatefulSet or DaemonSet has not been fully rolled out yet, i.e. not all replicas have been updated with the latest changes to spec.template.

    Each Kubernetes resources has different notion for being healthy. For example, a Deployment is considered healthy if the controller observed its current revision and if the number of updated replicas is equal to the number of replicas.

    The following status.conditions section describes a healthy ManagedResource:

    conditions:
    - lastTransitionTime: "2022-05-03T10:55:39Z"
      lastUpdateTime: "2022-05-03T10:55:39Z"
      message: All resources are healthy.
      reason: ResourcesHealthy
      status: "True"
      type: ResourcesHealthy
    - lastTransitionTime: "2022-05-03T10:55:36Z"
      lastUpdateTime: "2022-05-03T10:55:36Z"
      message: All resources have been fully rolled out.
      reason: ResourcesRolledOut
      status: "False"
      type: ResourcesProgressing
    - lastTransitionTime: "2022-05-03T10:55:18Z"
      lastUpdateTime: "2022-05-03T10:55:18Z"
      message: All resources are applied.
      reason: ApplySucceeded
      status: "True"
      type: ResourcesApplied
    

    Ignoring Updates

    In some cases it is not desirable to update or re-apply some of the cluster components (for example, if customization is required or needs to be applied by the end-user). For these resources, the annotation “resources.gardener.cloud/ignore” needs to be set to “true” or a truthy value (Truthy values are “1”, “t”, “T”, “true”, “TRUE”, “True”) in the corresponding managed resource secrets, this can be done from the components that create the managed resource secrets, for example Gardener extensions or Gardener. Once this is done, the resource will be initially created and later ignored during reconciliation.

    Preserving replicas or resources in Workload Resources

    The objects which are part of the ManagedResource can be annotated with

    • resources.gardener.cloud/preserve-replicas=true in case the .spec.replicas field of workload resources like Deployments, StatefulSets, etc. shall be preserved during updates.
    • resources.gardener.cloud/preserve-resources=true in case the .spec.containers[*].resources fields of all containers of workload resources like Deployments, StatefulSets, etc. shall be preserved during updates.

    This can be useful if there are non-standard horizontal/vertical auto-scaling mechanisms in place. Standard mechanisms like HorizontalPodAutoscaler or VerticalPodAutoscaler will be auto-recognized by gardener-resource-manager, i.e., in such cases the annotations are not needed.

    Origin

    All the objects managed by the resource manager get a dedicated annotation resources.gardener.cloud/origin describing the ManagedResource object that describes this object. The default format is <namespace>/<objectname>.

    In multi-cluster scenarios (the ManagedResource objects are maintained in a cluster different from the one the described objects are managed), it might be useful to include the cluster identity, as well.

    This can be enforced by setting the --cluster-id option. Here, several possibilities are supported:

    • given a direct value: use this as id for the source cluster
    • <cluster>: read the cluster identity from a cluster-identity config map in the kube-system namespace (attribute cluster-identity). This is automatically maintained in all clusters managed or involved in a gardener landscape.
    • <default>: try to read the cluster identity from the config map. If not found, no identity is used
    • empty string: no cluster identity is used (completely cluster local scenarios)

    By default cluster id is not used. If cluster id is specified the format is <cluster id>:<namespace>/<objectname>.

    In addition to the origin annotation, all objects managed by the resource manager get a dedicated label resources.gardener.cloud/managed-by. This label can be used to describe these objects with a selector. By default it is set to “gardener”, but this can be overwritten by setting the --managed-by-label option.

    Garbage Collector For Immutable ConfigMaps/Secrets

    In Kubernetes, workload resources (e.g., Pods) can mount ConfigMaps or Secrets or reference them via environment variables in containers. Typically, when the content of such ConfigMap/Secret gets changed then the respective workload is usually not dynamically reloading the configuration, i.e., a restart is required. The most commonly used approach is probably having so-called checksum annotations in the pod template which makes Kubernetes to recreate the pod if the checksum changes. However, it has the downside that old, still running versions of the workload might not be able to properly work with the already updated content in the ConfigMap/Secret, potentially causing application outages.

    In order to protect users from such outages (and to also improve the performance of the cluster), the Kubernetes community provides the “immutable ConfigMaps/Secrets feature”. Enabling immutability requires ConfigMaps/Secrets to have unique names. Having unique names requires the client to delete ConfigMaps/Secret`s no longer in use.

    In order to provide a similarly lightweight experience for clients (compared to the well-established checksum annotation approach), the Gardener Resource Manager features an optional garbage collector controller (disabled by default). The purpose of this controller is cleaning up such immutable ConfigMaps/Secrets if they are no longer in use.

    How does the garbage collector work?

    The following algorithm is implemented in the GC controller:

    1. List all ConfigMaps and Secrets labeled with resources.gardener.cloud/garbage-collectable-reference=true.
    2. List all Deployments, StatefulSets, DaemonSets, Jobs, CronJobs, Pods and for each of them
      1. iterate over the .metadata.annotations and for each of them
        1. If the annotation key follows the reference.resources.gardener.cloud/{configmap,secret}-<hash> scheme and the value equals <name> then consider it as “in-use”.
    3. Delete all ConfigMaps and Secrets not considered as “in-use”.

    Consequently, clients need to

    1. Create immutable ConfigMaps/Secrets with unique names (e.g., a checksum suffix based on the .data).

    2. Label such ConfigMaps/Secrets with resources.gardener.cloud/garbage-collectable-reference=true.

    3. Annotate their workload resources with reference.resources.gardener.cloud/{configmap,secret}-<hash>=<name> for all ConfigMaps/Secrets used by the containers of the respective Pods.

      ⚠️ Add such annotations to .metadata.annotations as well as to all templates of other resources (e.g., .spec.template.metadata.annotations in Deployments or .spec.jobTemplate.metadata.annotations and .spec.jobTemplate.spec.template.metadata.annotations for CronJobs. This ensures that the GC controller does not unintentionally consider ConfigMaps/Secrets as “not in use” just because there isn’t a Pod referencing them anymore (e.g., they could still be used by a Deployment scaled down to 0).

    ℹ️ For the last step, there is a helper function InjectAnnotations in the pkg/controller/garbagecollector/references which you can use for your convenience.

    Example:

    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: test-1234
      namespace: default
      labels:
        resources.gardener.cloud/garbage-collectable-reference: "true"
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: test-5678
      namespace: default
      labels:
        resources.gardener.cloud/garbage-collectable-reference: "true"
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: example
      namespace: default
      annotations:
        reference.resources.gardener.cloud/configmap-82a3537f: test-5678
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        terminationGracePeriodSeconds: 2
    

    The GC controller would delete the ConfigMap/test-1234 because it is considered as not “in-use”.

    ℹ️ If the GC controller is activated then the ManagedResource controller will no longer delete ConfigMaps/Secrets having the above label.

    How to activate the garbage collector?

    The GC controller can be activated by providing the --garbage-collector-sync-period flag with a value larger than 0 (e.g., 1h) to the Gardener Resource Manager.

    TokenInvalidator

    The Kubernetes community is slowly transitioning from static ServiceAccount token Secrets to ServiceAccount Token Volume Projection. Typically, when you create a ServiceAccount

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: default
    

    then the serviceaccount-token controller (part of kube-controller-manager) auto-generates a Secret with a static token:

    apiVersion: v1
    kind: Secret
    metadata:
       annotations:
          kubernetes.io/service-account.name: default
          kubernetes.io/service-account.uid: 86e98645-2e05-11e9-863a-b2d4d086dd5a)
       name: default-token-ntxs9
    type: kubernetes.io/service-account-token
    data:
       ca.crt: base64(cluster-ca-cert)
       namespace: base64(namespace)
       token: base64(static-jwt-token)
    

    Unfortunately, when using ServiceAccount Token Volume Projection in a Pod, this static token is actually not used at all:

    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx
    spec:
      serviceAccountName: default
      containers:
      - image: nginx
        name: nginx
        volumeMounts:
        - mountPath: /var/run/secrets/tokens
          name: token
      volumes:
      - name: token
        projected:
          sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 7200
    

    While the Pod is now using an expiring and auto-rotated token, the static token is still generated and valid.

    As of Kubernetes v1.22, there is neither a way of preventing kube-controller-manager to generate such static tokens, nor a way to proactively remove or invalidate them:

    Disabling the serviceaccount-token controller is an option, however, especially in the Gardener context it may either break end-users or it may not even be possible to control such settings. Also, even if a future Kubernetes version supports native configuration of above behaviour, Gardener still supports older versions which won’t get such features but need a solution as well.

    This is where the TokenInvalidator comes into play: Since it is not possible to prevent kube-controller-manager from generating static ServiceAccount Secrets, the TokenInvalidator is - as its name suggests - just invalidating these tokens. It considers all such Secrets belonging to ServiceAccounts with .automountServiceAccountToken=false. By default, all namespaces in the target cluster are watched, however, this can be configured by specifying the --target-namespace flag.

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: my-serviceaccount
    automountServiceAccountToken: false
    

    This will result in a static ServiceAccount token secret whose token value is invalid:

    apiVersion: v1
    kind: Secret
    metadata:
      annotations:
        kubernetes.io/service-account.name: my-serviceaccount
        kubernetes.io/service-account.uid: 86e98645-2e05-11e9-863a-b2d4d086dd5a
      name: my-serviceaccount-token-ntxs9
    type: kubernetes.io/service-account-token
    data:
      ca.crt: base64(cluster-ca-cert)
      namespace: base64(namespace)
      token: AAAA
    

    Any attempt to regenerate the token or creating a new such secret will again make the component invalidating it.

    You can opt-out of this behaviour for ServiceAccounts setting .automountServiceAccountToken=false by labeling them with token-invalidator.resources.gardener.cloud/skip=true.

    In order to enable the TokenInvalidator you have to set --token-invalidator-max-concurrent-workers to a value larger than 0.

    Below graphic shows an overview of the Token Invalidator for Service account secrets in the Shoot cluster. image

    TokenRequestor

    This controller provides the service to create and auto-renew tokens via the TokenRequest API.

    It provides a functionality similar to the kubelet’s Service Account Token Volume Projection. It was created to handle the special case of issuing tokens to pods that run in a different cluster than the API server they communicate with (hence, using the native token volume projection feature is not possible).

    The controller differentiates between source cluster and target cluster. The source cluster hosts the gardener-resource-manager pod. Secrets in this cluster are watched and modified by the controller. The target cluster can be configured to point to another cluster. The existence of ServiceAccounts are ensured and token requests are issued against the target. When the gardener-resource-manager is deployed next to the Shoot’s controlplane in the Seed the source cluster is the Seed while the target cluster points to the Shoot.

    Reconciliation Loop

    This controller reconciles secrets in all namespaces in the source cluster with the label: resources.gardener.cloud/purpose: token-requestor. See here for an example of the secret.

    The controller ensures a ServiceAccount exists in the target cluster as specified in the annotations of the Secret in the source cluster:

    serviceaccount.resources.gardener.cloud/name: <sa-name>
    serviceaccount.resources.gardener.cloud/namespace: <sa-namespace>
    

    The requested tokens will act with the privileges which are assigned to this ServiceAccount.

    The controller will then request a token via the TokenRequest API and populate it into the .data.token field to the Secret in the source cluster.

    Alternatively, the client can provide a raw kubeconfig (in YAML or JSON format) via the Secret’s .data.kubeconfig field. The controller will then populate the requested token in the kubeconfig for the user used in the .current-context. For example, if .data.kubeconfig is

    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: AAAA
        server: some-server-url
      name: shoot--foo--bar
    contexts:
    - context:
        cluster: shoot--foo--bar
        user: shoot--foo--bar-token
      name: shoot--foo--bar
    current-context: shoot--foo--bar
    kind: Config
    preferences: {}
    users:
    - name: shoot--foo--bar-token
      user:
        token: ""
    

    then the .users[0].user.token field of the kubeconfig will be updated accordingly.

    The controller also adds an annotation to the Secret to keep track when to renew the token before it expires. By default, the tokens are issued to expire after 12 hours. The expiration time can be set with the following annotation:

    serviceaccount.resources.gardener.cloud/token-expiration-duration: 6h
    

    It automatically renews once 80% of the lifetime is reached or after 24h.

    Optionally, the controller can also populate the token into a Secret in the target cluster. This can be requested by annotating the Secret in the source cluster with

    token-requestor.resources.gardener.cloud/target-secret-name: "foo"
    token-requestor.resources.gardener.cloud/target-secret-namespace: "bar"
    

    Overall, the TokenRequestor controller provides credentials with limited lifetime (JWT tokens) used by Shoot control plane components running in the Seed to talk to the Shoot API Server. Please see the graphic below:

    image

    Webhooks

    Auto-Mounting Projected ServiceAccount Tokens

    When this webhook is activated then it automatically injects projected ServiceAccount token volumes into Pods and all its containers if all of the following preconditions are fulfilled:

    1. The Pod is NOT labeled with projected-token-mount.resources.gardener.cloud/skip=true.
    2. The Pod’s .spec.serviceAccountName field is NOT empty and NOT set to default.
    3. The ServiceAccount specified in the Pod’s .spec.serviceAccountName sets .automountServiceAccountToken=false.
    4. The Pod’s .spec.volumes[] DO NOT already contain a volume with a name prefixed with kube-api-access-.

    The projected volume will look as follows:

    spec:
      volumes:
      - name: kube-api-access-gardener
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              expirationSeconds: 43200
              path: token
          - configMap:
              items:
              - key: ca.crt
                path: ca.crt
              name: kube-root-ca.crt
          - downwardAPI:
              items:
              - fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
                path: namespace
    

    The expirationSeconds are defaulted to 12h and can be overwritten with the --projected-token-mount-expiration-seconds flag, or with the projected-token-mount.resources.gardener.cloud/expiration-seconds annotation on a Pod resource.

    The volume will be mounted into all containers specified in the Pod to the path /var/run/secrets/kubernetes.io/serviceaccount. This is the default location where client libraries expect to find the tokens and mimics the upstream ServiceAccount admission plugin, see this document for more information.

    Overall, this webhook is used to inject projected service account tokens into pods running in the Shoot and the Seed cluster. Hence, it is served from the Seed GRM and each Shoot GRM. Please find an overview below for pods deployed in the Shoot cluster:

    image

    Pod Zone Affinity

    When this webhook is activated and namespaces are annotated with control-plane.shoot.gardener.cloud/enforce-zone then it automatically adds a pod affinity to all Pods created in these namespaces:

    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector: {}
            topologyKey: topology.kubernetes.io/zone
    

    In addition, if the annotation key control-plane.shoot.gardener.cloud/enforce-zone has a value <zone-value>, i.e. zone assigned, this information is added as part of a node affinity.

    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: topology.kubernetes.io/zone
                  operator: In
                  values:
                  - <zone-value>
    

    Those terms let pods within a namespace being scheduled to nodes residing in the very same zone which is either randomly picked, or to a very specific zone. In addition, the webhook removes any (anti-)affinities with topology.kubernetes.io/zone because they potentially contradict the above shown configuration. Gardener uses this webhook to schedule control-plane pods within a single zone on a multi-zonal seed (seed with worker nodes across zones). The goal is to reduce cross zonal network traffic within the seed with this approach.

    Pod Topology Spread Constraints

    When this webhook is enabled then it mimics the topologyKey feature for Topology Spread Constraints (TSC) on the label pod-template-hash. Concretely, when a pod is labelled with pod-template-hash the handler of this webhook extends any topology spread constraint in the pod:

    metadata:
      labels:
        pod-template-hash: 123abc
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            pod-template-hash: 123abc # added by webhook
    

    The procedure circumvents a known limitation with TSCs which leads to imbalanced deployments after rolling updates. Gardener enables this webhook to schedule pods of deployments across nodes and zones.

    Please note, the gardener-resource-manager itself as well as pods labelled with topology-spread-constraints.resources.gardener.cloud/skip are excluded from any mutations.

    1.2.12 - Scheduler

    Gardener Scheduler

    The Gardener Scheduler is in essence a controller that watches newly created shoots and assigns a seed cluster to them. Conceptually, the task of the Gardener Scheduler is very similar to the task of the Kubernetes Scheduler: finding a seed for a shoot instead of a node for a pod.

    Either the scheduling strategy or the shoot cluster purpose hereby determines how the scheduler is operating. The following sections explain the configuration and flow in greater detail.

    Why is the Gardener Scheduler needed?

    1. Decoupling

    Previously, an admission plugin in the Gardener API server conducted the scheduling decisions. This implies changes to the API server whenever adjustments of the scheduling are needed. Decoupling the API server and the scheduler comes with greater flexibility to develop these components independently from each other.

    2. Extensibility

    It should be possible to easily extend and tweak the scheduler in the future. Possibly, similar to the Kubernetes scheduler, hooks could be provided which influence the scheduling decisions. It should be also possible to completely replace the standard Gardener Scheduler with a custom implementation.

    Algorithm overview

    The following sequence describes the steps involved to determine a seed candidate:

    1. Determine usable seeds with “usable” defined as follows:
      • no .metadata.deletionTimestamp
      • .spec.settings.scheduling.visible is true
      • conditions Bootstrapped, GardenletReady, BackupBucketsReady (if available) are true
    2. Filter seeds:
      • matching .spec.seedSelector in CloudProfile used by the Shoot
      • matching .spec.seedSelector in Shoot
      • having no network intersection with the Shoot’s networks (due to the VPN connectivity between seeds and shoots their networks must be disjoint)
      • having .spec.settings.shootDNS.enabled=false (only if the shoot specifies a DNS domain or does not use the unmanaged DNS provider)
      • whose taints (.spec.taints) are tolerated by the Shoot (.spec.tolerations)
      • whose capacity for shoots would not be exceeded if the shoot is scheduled onto the seed, see Ensuring seeds capacity for shoots is not exceeded
      • which are labelled with seed.gardener.cloud/multi-zonal if feature gate HAControlPlanes is turned on and shoot requests a high available control plane.
    3. Apply active strategy e.g., Minimal Distance strategy
    4. Choose least utilized seed, i.e., the one with the least number of shoot control planes, will be the winner and written to the .spec.seedName field of the Shoot.

    Configuration

    The Gardener Scheduler configuration has to be supplied on startup. It is a mandatory and also the only available flag. Here is an example scheduler configuration.

    Most of the configuration options are the same as in the Gardener Controller Manager (leader election, client connection, …). However, the Gardener Scheduler on the other hand does not need a TLS configuration, because there are currently no webhooks configurable.

    Strategies

    The scheduling strategy is defined in the candidateDeterminationStrategy of the scheduler’s configuration and can have the possible values SameRegion and MinimalDistance. The SameRegion strategy is the default strategy.

    1. Same Region strategy

      The Gardener Scheduler reads the spec.provider.type and .spec.region fields from the Shoot resource. It tries to find a seed that has the identical .spec.provider.type and .spec.provider.region fields set. If it cannot find a suitable seed, it adds an event to the shoot stating, that it is unschedulable.

    2. Minimal Distance strategy

      The Gardener Scheduler tries to find a valid seed with minimal distance to the shoot’s intended region. The distance is calculated based on the Levenshtein distance of the region. Therefore the region name is split into a base name and an orientation. Possible orientations are north, south, east, west and central. The distance then is twice the Levenshtein distance of the region’s base name plus a correction value based on the orientation and the provider.

      If the orientations of shoot and seed candidate match, the correction value is 0, if they differ it is 2 and if either the seed’s or the shoot’s region does not have an orientation it is 1. If the provider differs the correction value is additionally incremented by 2.

      Because of this a matching region with a matching provider is always prefered.

    In order to put the scheduling decision into effect, the scheduler sends an update request for the Shoot resource to the API server. After validation, the Gardener Aggregated API server updates the shoot to have the spec.seedName field set. Subsequently, the Gardenlet picks up and starts to create the cluster on the specified seed.

    1. Special handling based on shoot cluster purpose

    Every shoot cluster can have a purpose that describes what the cluster is used for, and also influences how the cluster is setup (see this document for more information).

    In case the shoot has the testing purpose then the scheduler only reads the .spec.provider.type from the Shoot resource and tries to find a Seed that has the identical .spec.provider.type. The region does not matter, i.e., testing shoots may also be scheduled on a seed in a complete different region if it is better for balancing the whole Gardener system.

    shoots/binding subresource

    The shoots/binding subresource is used to bind a Shoot to a Seed. On creation of shoot clusters, the scheduler updates the binding automatically if an appropriate seed cluster is available. Only an operator with necessary RBAC can update this binding manually. This can be done by changing the .spec.seedName of the shoot. However, if a different seed is already assigned to the shoot, this will trigger a control-plane migration. For required steps, Please see Triggering the Migration.

    spec.seedName field in the Shoot specification

    Similar to the .spec.nodeName field in Pods, the Shoot specification has an optional .spec.seedName field. If this field is set on creation, the shoot will be scheduled to this seed. However, this field can only be set by users having RBAC for the shoots/binding subresource. If this field is not set, the scheduler will assign a suitable seed automatically and populate this field with the seed name.

    seedSelector field in the Shoot specification

    Similar to the .spec.nodeSelector field in Pods, the Shoot specification has an optional .spec.seedSelector field. It allows the user to provide a label selector that must match the labels of Seeds in order to be scheduled to one of them. The labels on Seeds are usually controlled by Gardener administrators/operators - end users cannot add arbitrary labels themselves. If provided, the Gardener Scheduler will only consider those seeds as “suitable” whose labels match those provided in the .spec.seedSelector of the Shoot.

    By default only seeds with the same provider than the shoot are selected. By adding a providerTypes field to the seedSelector a dedicated set of possible providers (* means all provider types) can be selected.

    Ensuring seeds capacity for shoots is not exceeded

    Seeds have a practical limit of how many shoots they can accommodate. Exceeding this limit is undesirable as the system performance will be noticeably impacted. Therefore, the scheduler ensures that a seed’s capacity for shoots is not exceeded by taking into account a maximum number of shoots that can be scheduled onto a seed.

    This mechanism works as follows:

    • The gardenlet is configured with certain resources and their total capacity (and, for certain resources, the amount reserved for Gardener), see /example/20-componentconfig-gardenlet.yaml. Currently, the only such resource is the maximum number of shoots that can be scheduled onto a seed.
    • The gardenlet seed controller updates the capacity and allocatable fields in Seed status with the capacity of each resource and how much of it is actually available to be consumed by shoots. The allocatable value of a resource is equal to capacity minus reserved.
    • When scheduling shoots, the scheduler filters out all candidate seeds whose allocatable capacity for shoots would be exceeded if the shoot is scheduled onto the seed.

    Failure to determine a suitable seed

    In case the scheduler fails to find a suitable seed, the operation is being retried with exponential backoff.

    Current Limitation / Future Plans

    • Azure has unfortunately a geographically non-hierarchical naming pattern and does not start with the continent. This is the reason why we will exchange the implementation of the MinimalDistance strategy with a more suitable one in the future.

    1.2.13 - Seed Admission Controller

    Gardener Seed Admission Controller

    The Gardener Seed admission controller is deployed by the Gardenlet as part of its seed bootstrapping phase and, consequently, running in every seed cluster. It’s main purpose is to serve webhooks (validating or mutating) in order to admit or deny certain requests to the seed’s API server.

    What is it doing concretely?

    Validating Webhooks

    Unconfirmed Deletion Prevention

    As part of Gardener’s extensibility concepts a lot of CustomResourceDefinitions are deployed to the seed clusters that serve as extension points for provider-specific controllers. For example, the Infrastructure CRD triggers the provider extension to prepare the IaaS infrastructure of the underlying cloud provider for a to-be-created shoot cluster. Consequently, these extension CRDs have a lot of power and control large portions of the end-user’s shoot cluster. Accidental or undesired deletions of those resource can cause tremendous and hard-to-recover-from outages and should be prevented.

    Together with the deployment of the Gardener seed admission controller a ValidatingWebhookConfiguration for CustomResourceDefinitions and most (custom) resources in the extensions.gardener.cloud/v1alpha1 API group is registered. It prevents DELETE requests for those CustomResourceDefinitions labeled with gardener.cloud/deletion-protected=true, and for all mentioned custom resources if they were not previously annotated with the confirmation.gardener.cloud/deletion=true. This prevents that undesired kubectl delete <...> requests are accepted.

    Mutating Webhooks

    The admission controller endpoint /webhooks/default-pod-scheduler-name/gardener-kube-scheduler mutates pods and adds gardener-kube-scheduler to .spec.scheduleName.

    When SeedKubeScheduler feature gate is enabled, all control plane components are mutated. The scheduler scores Nodes with most resource usage higher than the rest, resulting in greater resource utilization.

    1.3 - Deployment

    1.3.1 - Authentication Gardener Control Plane

    Authentication of Gardener control plane components against the Garden cluster

    Note: This document refers to Gardener’s API server, admission controller, controller manager and scheduler components. Any reference to the term Gardener control plane component can be replaced with any of the mentioned above.

    There are several authentication possibilities depending on whether or not the concept of Virtual Garden is used.

    Virtual Garden is not used, i.e., the runtime Garden cluster is also the target Garden cluster.

    Automounted Service Account Token

    The easiest way to deploy a Gardener control plane component will be to not provide kubeconfig at all. This way in-cluster configuration and an automounted service account token will be used. The drawback of this approach is that the automounted token will not be automatically rotated.

    Service Account Token Volume Projection

    Another solution will be to use Service Account Token Volume Projection combined with a kubeconfig referencing a token file (see example below).

    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority-data: <CA-DATA>
        server: https://default.kubernetes.svc.cluster.local
      name: garden
    contexts:
    - context:
        cluster: garden
        user: garden
      name: garden
    current-context: garden
    users:
    - name: garden
      user:
        tokenFile: /var/run/secrets/projected/serviceaccount/token
    

    This will allow for automatic rotation of the service account token by the kubelet. The configuration can be achieved by setting both .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true and .Values.global.<GardenerControlPlaneComponent>.kubeconfig in the respective chart’s values.yaml file.

    Virtual Garden is used, i.e., the runtime Garden cluster is different from the target Garden cluster.

    Service Account

    The easiest way to setup the authentication will be to create a service account and the respective roles will be bound to this service account in the target cluster. Then use the generated service account token and craft a kubeconfig which will be used by the workload in the runtime cluster. This approach does not provide a solution for the rotation of the service account token. However, this setup can be achieved by setting .Values.global.deployment.virtualGarden.enabled: true and following these steps:

    1. Deploy the application part of the charts in the target cluster.
    2. Get the service account token and craft the kubeconfig.
    3. Set the crafted kubeconfig and deploy the runtime part of the charts in the runtime cluster.

    Client Certificate

    Another solution will be to bind the roles in the target cluster to a User subject instead of a service account and use a client certificate for authentication. This approach does not provide a solution for the client certificate rotation. However, this setup can be achieved by setting both .Values.global.deployment.virtualGarden.enabled: true and .Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name, then following these steps:

    1. Generate a client certificate for the target cluster for the respective user.
    2. Deploy the application part of the charts in the target cluster.
    3. Craft a kubeconfig using the already generated client certificate.
    4. Set the crafted kubeconfig and deploy the runtime part of the charts in the runtime cluster.

    Projected Service Account Token

    This approach requires an already deployed and configured oidc-webhook-authenticator for the target cluster. Also the runtime cluster should be registered as a trusted identity provider in the target cluster. Then projected service accounts tokens from the runtime cluster can be used to authenticate against the target cluster. The needed steps are as follows:

    1. Deploy OWA and establish the needed trust.
    2. Set .Values.global.deployment.virtualGarden.enabled: true and .Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name. Note: username value will depend on the trust configuration, e.g., <prefix>:system:serviceaccount:<namespace>:<serviceaccount>
    3. Set .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true and .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.audience. Note: audience value will depend on the trust configuration, e.g., <cliend-id-from-trust-config>.
    4. Craft a kubeconfig (see example below).
    5. Deploy the application part of the charts in the target cluster.
    6. Deploy the runtime part of the charts in the runtime cluster.
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority-data: <CA-DATA>
        server: https://virtual-garden.api
      name: virtual-garden
    contexts:
    - context:
        cluster: virtual-garden
        user: virtual-garden
      name: virtual-garden
    current-context: virtual-garden
    users:
    - name: virtual-garden
      user:
        tokenFile: /var/run/secrets/projected/serviceaccount/token
    

    1.3.2 - Configuring Logging

    Configuring the Logging stack via Gardenlet configurations

    Enable the Logging

    In order to install the Gardener logging stack the logging.enabled configuration option has to be enabled in the Gardenlet configuration:

    logging:
      enabled: true
    

    From now on each Seed is going to have a logging stack which will collect logs from all pods and some systemd services. Logs related to Shoots with testing purpose are dropped in the fluent-bit output plugin. Shoots with a purpose different than testing have the same type of log aggregator (but different instance) as the Seed. The logs can be viewed in the Grafana in the garden namespace for the Seed components and in the respective shoot control plane namespaces.

    Enable logs from the Shoot’s node systemd services.

    The logs from the systemd services on each node can be retrieved by enabling the logging.shootNodeLogging option in the Gardenlet configuration:

    logging:
      enabled: true
      shootNodeLogging:
        shootPurposes:
        - "evaluation"
        - "deployment"
    

    Under the shootPurpose section just list all the shoot purposes for which the Shoot node logging feature will be enabled. Specifying the testing purpose has no effect because this purpose prevents the logging stack installation. Logs can be viewed in the operator Grafana! The dedicated labels are unit, syslog_identifier and nodename in the Explore menu.

    Configuring the log processor

    Under logging.fluentBit there is three optional sections.

    • input: This overwrite the input configuration of the fluent-bit log processor.
    • output: This overwrite the output configuration of the fluent-bit log processor.
    • service: This overwrite the service configuration of the fluent-bit log processor.
    logging:
      enabled: true
      fluentBit:
        output: |-
          [Output]
              ...      
        input: |-
          [Input]
              ...      
        service: |-
          [Service]
              ...      
    

    additional egress IPBlock for allow-fluentbit NetworkPolicy

    The optional setting under logging.fluentBit.networkPolicy.additionalEgressIPBlocks add additional egress IPBlock to allow-fluentbit NetworkPolicy to forward logs to a central system.

    logging:
      enabled: true
      fluentBit:
        additionalEgressIpBlock:
          - 123.123.123.123/32
    

    Configure central logging

    For central logging, the output configuration of the fluent-bit log processor can be overwritten (logging.fluentBit.output) and the Loki instances deployments in Garden and Shoot namespace can be enabled/disabled (logging.loki.enabled), by default Loki is enabled.

    logging:
      enabled: true
      fluentBit:
        output: |-
          [Output]
              ...      
      loki:
        enabled: false
    

    Configuring central Loki storage capacity

    By default, the central Loki has 100Gi of storage capacity. To overwrite the current central Loki storage capacity, the logging.loki.garden.storage setting in the gardenlet’s component configuration should be altered. If you need to increase it you can do so without losing the current data by specifying higher capacity. Doing so, the Loki’s PersistentVolume capacity will be increased instead of deleting the current PV. However, if you specify less capacity then the PersistentVolume will be deleted and with it the logs, too.

    logging:
      enabled: true
      fluentBit:
        output: |-
          [Output]
              ...      
      loki:
        garden:
          storage: "200Gi"
    

    1.3.3 - Deploy Gardenlet

    Deploying Gardenlets

    Gardenlets act as decentral “agents” to manage shoot clusters of a seed cluster.

    To support scaleability in an automated way, gardenlets are deployed automatically. However, you can still deploy gardenlets manually to be more flexible, for example, when shoot clusters that need to be managed by Gardener are behind a firewall. The gardenlet only requires network connectivity from the gardenlet to the Garden cluster (not the other way round), so it can be used to register Kubernetes clusters with no public endpoint.

    Procedure

    1. First, an initial gardenlet needs to be deployed:

    2. To add additional seed clusters, it is recommended to use regular shoot clusters. You can do this by creating a ManagedSeed resource with a gardenlet section as described in Register Shoot as Seed.

    1.3.4 - Deploy Gardenlet Automatically

    Automatic Deployment of Gardenlets

    The gardenlet can automatically deploy itself into shoot clusters, and register this cluster as a seed cluster. These clusters are called “managed seeds” (aka “shooted seeds”). This procedure is the preferred way to add additional seed clusters, because shoot clusters already come with production-grade qualities that are also demanded for seed clusters.

    Prerequisites

    The only prerequisite is to register an initial cluster as a seed cluster that has already a gardenlet deployed:

    • This gardenlet was either deployed as part of a Gardener installation using a setup tool (for example, gardener/garden-setup) or
    • the gardenlet was deployed manually

    The initial cluster can be the garden cluster itself.

    Self-Deployment of Gardenlets in Additional Managed Seed Clusters

    For a better scalability, you usually need more seed clusters that you can create as follows:

    1. Use the initial cluster as the seed cluster for other managed seed clusters. It hosts the control planes of the other seed clusters.
    2. The gardenlet deployed in the initial cluster deploys itself automatically into the managed seed clusters.

    The advantage of this approach is that there’s only one initial gardenlet installation required. Every other managed seed cluster has a gardenlet deployed automatically.

    Register Shoot as Seed

    garden-setup

    1.3.5 - Deploy Gardenlet Manually

    Deploy a Gardenlet Manually

    Manually deploying a gardenlet is required in the following cases:

    • The Kubernetes cluster to be registered as a seed cluster has no public endpoint, because it is behind a firewall. The gardenlet must then be deployed into the cluster itself.

    • The Kubernetes cluster to be registered as a seed cluster is managed externally (the Kubernetes cluster is not a shoot cluster, so Automatic Deployment of Gardenlets cannot be used).

    • The gardenlet runs outside of the Kubernetes cluster that should be registered as a seed cluster. (The gardenlet is not restricted to run in the seed cluster or to be deployed into a Kubernetes cluster at all).

    Once you’ve deployed a gardenlet manually, for example, behind a firewall, you can deploy new gardenlets automatically. The manually deployed gardenlet is then used as a template for the new gardenlets. More information: Automatic Deployment of Gardenlets.

    Prerequisites

    Kubernetes cluster that should be registered as a seed cluster

    • Verify that the cluster has a supported Kubernetes version.

    • Determine the nodes, pods, and services CIDR of the cluster. You need to configure this information in the Seed configuration. Gardener uses this information to check that the shoot cluster isn’t created with overlapping CIDR ranges.

    • Every Seed cluster needs an Ingress controller which distributes external requests to internal components like grafana and prometheus. Gardener supports two approaches to achieve this:

    a. Gardener managed Ingress controller and DNS records. For this configure the following lines in your Seed resource:

    spec:
      dns:
        provider:
          type: aws-route53
          secretRef:
            name: ingress-secret
            namespace: garden
      ingress:
        domain: ingress.my-seed.example.com
        controller:
          kind: nginx
          providerConfig:
            <some-optional-provider-specific-config-for-the-ingressController>
    

    ⚠ Please note that if you set .spec.ingress then .spec.dns.ingressDomain must be nil.

    b. Self-managed DNS record and Ingress controller:

    ⚠️ There should exist a DNS record *.ingress.<SEED-CLUSTER-DOMAIN> where <SEED-CLUSTER-DOMAIN> is the value of the .dns.ingressDomain field of a Seed cluster resource (or the respective Gardenlet configuration).

    This is how it could be done for the Nginx ingress controller

    Deploy nginx into the kube-system namespace in the Kubernetes cluster that should be registered as a Seed.

    Nginx will on most cloud providers create the service with type LoadBalancer with an external ip.

    NAME                        TYPE           CLUSTER-IP    EXTERNAL-IP
    nginx-ingress-controller    LoadBalancer   10.0.15.46    34.200.30.30
    

    Create a wildcard A record (e.g *.ingress.sweet-seed.. IN A 34.200.30.30) with your DNS provider and point it to the external ip of the ingress service. This ingress domain is later required to register the Seed cluster.

    Please configure the ingress domain in the Seed specification as follows:

    spec:
      dns:
        ingressDomain: ingress.sweet-seed.<my-domain>
    

    ⚠ Please note that if you set .spec.dns.ingressDomain then .spec.ingress must be nil.

    kubeconfig for the Seed Cluster

    The kubeconfig is required to deploy the gardenlet Helm chart to the seed cluster. The gardenlet requires certain privileges to be able to operate. These privileges are described in RBAC resources in the gardenlet Helm chart (see charts/gardener/gardenlet/charts/runtime/templates). The Helm chart contains a service account gardenlet that the gardenlet deployment uses by default to talk to the Seed API server.

    If the gardenlet isn’t deployed in the seed cluster, the gardenlet can be configured to use a kubeconfig, which also requires the above-mentioned privileges, from a mounted directory. The kubeconfig is specified in section seedClientConnection.kubeconfig of the Gardenlet configuration. This configuration option isn’t used in the following, as this procedure only describes the recommended setup option where the gardenlet is running in the seed cluster itself.

    Procedure Overview

    1. Prepare the garden cluster:

      1. Create a bootstrap token secret in the kube-system namespace of the garden cluster
      2. Create RBAC roles for the gardenlet to allow bootstrapping in the garden cluster
    2. Prepare the gardenlet Helm chart.

    3. Automatically register shoot cluster as a seed cluster.

    4. Deploy the gardenlet

    5. Check that the gardenlet is successfully deployed

    Create a bootstrap token secret in the kube-system namespace of the garden cluster

    The gardenlet needs to talk to the Gardener API server residing in the garden cluster.

    The gardenlet can be configured with an already existing garden cluster kubeconfig in one of the following ways:

    • Either by specifying gardenClientConnection.kubeconfig in the Gardenlet configuration or

    • by supplying the environment variable GARDEN_KUBECONFIG pointing to a mounted kubeconfig file).

    The preferred way however, is to use the gardenlets ability to request a signed certificate for the garden cluster by leveraging Kubernetes Certificate Signing Requests. The gardenlet performs a TLS bootstrapping process that is similar to the Kubelet TLS Bootstrapping. Make sure that the API server of the garden cluster has bootstrap token authentication enabled.

    The client credentials required for the gardenlets TLS bootstrapping process, need to be either token or certificate (OIDC isn’t supported) and have permissions to create a Certificate Signing Request (CSR). It’s recommended to use bootstrap tokens due to their desirable security properties (such as a limited token lifetime).

    Therefore, first create a bootstrap token secret for the garden cluster:

    apiVersion: v1
    kind: Secret
    metadata:
      # Name MUST be of form "bootstrap-token-<token id>"
      name: bootstrap-token-07401b
      namespace: kube-system
    
    # Type MUST be 'bootstrap.kubernetes.io/token'
    type: bootstrap.kubernetes.io/token
    stringData:
      # Human readable description. Optional.
      description: "Token to be used by the gardenlet for Seed `sweet-seed`."
    
      # Token ID and secret. Required.
      token-id: 07401b # 6 characters
      token-secret: f395accd246ae52d # 16 characters
    
      # Expiration. Optional.
      # expiration: 2017-03-10T03:22:11Z
    
      # Allowed usages.
      usage-bootstrap-authentication: "true"
      usage-bootstrap-signing: "true"
    

    When you later prepare the gardenlet Helm chart, a kubeconfig based on this token is shared with the gardenlet upon deployment.

    Create RBAC roles for the gardenlet to allow bootstrapping in the garden cluster

    This step is only required if the gardenlet you deploy is the first gardenlet in the Gardener installation. Additionally, when using the control plane chart, the following resources are already contained in the Helm chart, that is, if you use it you can skip these steps as the needed RBAC roles already exist.

    The gardenlet uses the configured bootstrap kubeconfig in gardenClientConnection.bootstrapKubeconfig to request a signed certificate for the user gardener.cloud:system:seed:<seed-name> in the group gardener.cloud:system:seeds.

    Create a ClusterRole and ClusterRoleBinding that grant full admin permissions to authenticated gardenlets.

    Create the following resources in the garden cluster:

    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: gardener.cloud:system:seeds
    rules:
      - apiGroups:
          - '*'
        resources:
          - '*'
        verbs:
          - '*'
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: gardener.cloud:system:seeds
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: gardener.cloud:system:seeds
    subjects:
      - kind: Group
        name: gardener.cloud:system:seeds
        apiGroup: rbac.authorization.k8s.io
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: gardener.cloud:system:seed-bootstrapper
    rules:
      - apiGroups:
          - certificates.k8s.io
        resources:
          - certificatesigningrequests
        verbs:
          - create
          - get
      - apiGroups:
          - certificates.k8s.io
        resources:
          - certificatesigningrequests/seedclient
        verbs:
          - create
    ---
    # A kubelet/gardenlet authenticating using bootstrap tokens is authenticated as a user in the group system:bootstrappers
    # Allows the Gardenlet to create a CSR
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: gardener.cloud:system:seed-bootstrapper
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: gardener.cloud:system:seed-bootstrapper
    subjects:
      - kind: Group
        name: system:bootstrappers
        apiGroup: rbac.authorization.k8s.io
    

    ℹ️ After bootstrapping, the gardenlet has full administrative access to the garden cluster. You might be interested to harden this and limit its permissions to only resources related to the seed cluster it is responsible for. Please take a look into this document.

    Prepare the gardenlet Helm chart

    This section only describes the minimal configuration, using the global configuration values of the gardenlet Helm chart. For an overview over all values, see the configuration values. We refer to the global configuration values as gardenlet configuration in the remaining procedure.

    1. Create a gardenlet configuration gardenlet-values.yaml based on this template.

    2. Create a bootstrap kubeconfig based on the bootstrap token created in the garden cluster.

      Replace the <bootstrap-token> with token-id.token-secret (from our previous example: 07401b.f395accd246ae52d) from the bootstrap token secret.

      apiVersion: v1
      kind: Config
      current-context: gardenlet-bootstrap@default
      clusters:
      - cluster:
          certificate-authority-data: <ca-of-garden-cluster>
          server: https://<endpoint-of-garden-cluster>
        name: default
      contexts:
      - context:
          cluster: default
          user: gardenlet-bootstrap
        name: gardenlet-bootstrap@default
      users:
      - name: gardenlet-bootstrap
        user:
          token: <bootstrap-token>
      
    3. In section gardenClientConnection.bootstrapKubeconfig of your gardenlet configuration, provide the bootstrap kubeconfig together with a name and namespace to the gardenlet Helm chart.

      gardenClientConnection:
        bootstrapKubeconfig:
          name: gardenlet-kubeconfig-bootstrap
          namespace: garden
          kubeconfig: |
                  <bootstrap-kubeconfig>  # will be base64 encoded by helm
      

      The bootstrap kubeconfig is stored in the specified secret.

    4. In section gardenClientConnection.kubeconfigSecret of your gardenlet configuration, define a name and a namespace where the gardenlet stores the real kubeconfig that it creates during the bootstrap process. If the secret doesn’t exist, the gardenlet creates it for you.

      gardenClientConnection:
        kubeconfigSecret:
          name: gardenlet-kubeconfig
          namespace: garden
      

    Updating the garden cluster CA

    The kubeconfig created by the gardenlet in step 4 will not be recreated as long as it exists, even if a new bootstrap kubeconfig is provided. To enable rotation of the garden cluster CA certificate, a new bundle can be provided via the gardenClientConnection.gardenClusterCACert field. If the provided bundle differs from the one currently in the gardenlet’s kubeconfig secret then it will be updated. To remove the CA completely (e.g. when switching to a publicly trusted endpoint) this field can be set to either none or null.

    Automatically register shoot cluster as a seed cluster

    A seed cluster can either be registered by manually creating the Seed resource or automatically by the gardenlet. This functionality is useful for managed seed clusters, as the gardenlet in the garden cluster deploys a copy of itself into the cluster with automatic registration of the Seed configured. However, it can also be used to have a streamlined seed cluster registration process when manually deploying the gardenlet.

    This procedure doesn’t describe all the possible configurations for the Seed resource. For more information, see:

    Adjust the gardenlet component configuration

    1. Supply the Seed resource in section seedConfig of your gardenlet configuration gardenlet-values.yaml.

    2. Add the seedConfig to your gardenlet configuration gardenlet-values.yaml. The field seedConfig.spec.provider.type specifies the infrastructure provider type (for example, aws) of the seed cluster. For all supported infrastructure providers, see Known Extension Implementations.

      ....
      seedConfig:
        metadata:
          name: sweet-seed
        spec:
          dns:
            ingressDomain: ingress.sweet-seed.<my-domain> # see prerequisites
          networks: # see prerequisites
            nodes: 10.240.0.0/16
            pods: 100.244.0.0/16
            services: 100.32.0.0/13
            shootDefaults: # optional: non-overlapping default CIDRs for shoot clusters of that Seed
              pods: 100.96.0.0/11
              services: 100.64.0.0/13
          provider:
            region: eu-west-1
            type: <provider>
      

    Optional: Enable backup and restore

    The seed cluster can be set up with backup and restore for the main etcds of shoot clusters.

    Gardener uses etcd-backup-restore that integrates with different storage providers to store the shoot cluster’s main etcd backups. Make sure to obtain client credentials that have sufficient permissions with the chosen storage provider.

    Create a secret in the garden cluster with client credentials for the storage provider. The format of the secret is cloud provider specific and can be found in the repository of the respective Gardener extension. For example, the secret for AWS S3 can be found in the AWS provider extension (30-etcd-backup-secret.yaml).

    apiVersion: v1
    kind: Secret
    metadata:
      name: sweet-seed-backup
      namespace: garden
    type: Opaque
    data:
      # client credentials format is provider specific
    

    Configure the Seed resource in section seedConfig of your gardenlet configuration to use backup and restore:

    ...
    seedConfig:
      metadata:
        name: sweet-seed
      spec:
        backup:
          provider: <provider>
          secretRef:
            name: sweet-seed-backup
            namespace: garden
    

    Deploy the gardenlet

    The gardenlet doesn’t have to run in the same Kubernetes cluster as the seed cluster it’s registering and reconciling, but it is in most cases advantageous to use in-cluster communication to talk to the Seed API server. Running a gardenlet outside of the cluster is mostly used for local development.

    The gardenlet-values.yaml looks something like this (with automatic Seed registration and backup for shoot clusters enabled):

    global:
      # Gardenlet configuration values
      gardenlet:
        enabled: true
        ...
        <default config>
        ...
        config:
          gardenClientConnection:
            ...
            bootstrapKubeconfig:
              name: gardenlet-bootstrap-kubeconfig
              namespace: garden
              kubeconfig: |
                apiVersion: v1
                clusters:
                - cluster:
                    certificate-authority-data: <dummy>
                    server: <my-garden-cluster-endpoint>
                  name: my-kubernetes-cluster
                ....            
    
            kubeconfigSecret:
              name: gardenlet-kubeconfig
              namespace: garden
          ...
          <default config>
          ...
          seedConfig:
            metadata:
              name: sweet-seed
            spec:
              dns:
                ingressDomain: ingress.sweet-seed.<my-domain>
              networks:
                nodes: 10.240.0.0/16
                pods: 100.244.0.0/16
                services: 100.32.0.0/13
                shootDefaults:
                  pods: 100.96.0.0/11
                  services: 100.64.0.0/13
              provider:
                region: eu-west-1
                type: <provider>
              backup:
                provider: <provider>
                secretRef:
                  name: sweet-seed-backup
                  namespace: garden
    

    Deploy the gardenlet Helm chart to the Kubernetes cluster.

    helm install gardenlet charts/gardener/gardenlet \
      --namespace garden \
      -f gardenlet-values.yaml \
      --wait
    

    This helm chart creates:

    • A service account gardenlet that the gardenlet can use to talk to the Seed API server.
    • RBAC roles for the service account (full admin rights at the moment).
    • The secret (garden/gardenlet-bootstrap-kubeconfig) containing the bootstrap kubeconfig.
    • The gardenlet deployment in the garden namespace.

    Check that the gardenlet is successfully deployed

    1. Check that the gardenlets certificate bootstrap was successful.

      Check if the secret gardenlet-kubeconfig in the namespace garden in the seed cluster is created and contains a kubeconfig with a valid certificate.

      1. Get the kubeconfig from the created secret.

        $ kubectl -n garden get secret gardenlet-kubeconfig -o json | jq -r .data.kubeconfig | base64 -d
        
      2. Test against the garden cluster and verify it’s working.

      3. Extract the client-certificate-data from the user gardenlet.

      4. View the certificate:

        $ openssl x509 -in ./gardenlet-cert -noout -text
        

        Check that the certificate is valid for a year (that is the lifetime of new certificates).

    2. Check that the bootstrap secret gardenlet-bootstrap-kubeconfig has been deleted from the seed cluster in namespace garden.

    3. Check that the seed cluster is registered and READY in the garden cluster.

      Check that the seed cluster sweet-seed exists and all conditions indicate that it’s available. If so, the Gardenlet is sending regular heartbeats and the seed bootstrapping was successful.

      Check that the conditions on the Seed resource look similar to the following:

      $ kubectl get seed sweet-seed -o json | jq .status.conditions
      [
        {
          "lastTransitionTime": "2020-07-17T09:17:29Z",
          "lastUpdateTime": "2020-07-17T09:17:29Z",
          "message": "Gardenlet is posting ready status.",
          "reason": "GardenletReady",
          "status": "True",
          "type": "GardenletReady"
        },
        {
          "lastTransitionTime": "2020-07-17T09:17:49Z",
          "lastUpdateTime": "2020-07-17T09:53:17Z",
          "message": "Seed cluster has been bootstrapped successfully.",
          "reason": "BootstrappingSucceeded",
          "status": "True",
          "type": "Bootstrapped"
        },
        {
          "lastTransitionTime": "2020-07-17T09:17:49Z",
          "lastUpdateTime": "2020-07-17T09:53:17Z",
          "message": "Backup Buckets are available.",
          "reason": "BackupBucketsAvailable",
          "status": "True",
          "type": "BackupBucketsReady"
        }
      ]
      

    Issue #1724: Harden Gardenlet RBAC privileges.

    Backup and Restore.

    1.3.6 - Feature Gates

    Feature Gates in Gardener

    This page contains an overview of the various feature gates an administrator can specify on different Gardener components.

    Overview

    Feature gates are a set of key=value pairs that describe Gardener features. You can turn these features on or off using the a component configuration file for a specific component.

    Each Gardener component lets you enable or disable a set of feature gates that are relevant to that component. For example this is the configuration of the gardenlet component.

    The following tables are a summary of the feature gates that you can set on different Gardener components.

    • The “Since” column contains the Gardener release when a feature is introduced or its release stage is changed.
    • The “Until” column, if not empty, contains the last Gardener release in which you can still use a feature gate.
    • If a feature is in the Alpha or Beta state, you can find the feature listed in the Alpha/Beta feature gate table.
    • If a feature is stable you can find all stages for that feature listed in the Graduated/Deprecated feature gate table.
    • The Graduated/Deprecated feature gate table also lists deprecated and withdrawn features.

    Feature gates for Alpha or Beta features

    FeatureDefaultStageSinceUntil
    HVPAfalseAlpha0.31
    HVPAForShootedSeedfalseAlpha0.32
    ManagedIstiofalseAlpha1.51.18
    ManagedIstiotrueBeta1.19
    ManagedIstio (deprecated)trueBeta1.48
    APIServerSNIfalseAlpha1.71.18
    APIServerSNItrueBeta1.19
    APIServerSNI (deprecated)trueBeta1.48
    SeedChangefalseAlpha1.121.52
    SeedChangetrueBeta1.53
    ReversedVPNfalseAlpha1.221.41
    ReversedVPNtrueBeta1.42
    CopyEtcdBackupsDuringControlPlaneMigrationfalseAlpha1.371.52
    CopyEtcdBackupsDuringControlPlaneMigrationtrueBeta1.53
    ForceRestorefalseAlpha1.39
    HAControlPlanesfalseAlpha1.49
    DefaultSeccompProfilefalseAlpha1.54
    CoreDNSQueryRewritingfalseAlpha1.55

    Feature gates for graduated or deprecated features

    FeatureDefaultStageSinceUntil
    NodeLocalDNSfalseAlpha1.7
    NodeLocalDNSRemoved1.26
    KonnectivityTunnelfalseAlpha1.6
    KonnectivityTunnelRemoved1.27
    MountHostCADirectoriesfalseAlpha1.111.25
    MountHostCADirectoriestrueBeta1.261.27
    MountHostCADirectoriestrueGA1.27
    MountHostCADirectoriesRemoved1.30
    DisallowKubeconfigRotationForShootInDeletionfalseAlpha1.281.31
    DisallowKubeconfigRotationForShootInDeletiontrueBeta1.321.35
    DisallowKubeconfigRotationForShootInDeletiontrueGA1.36
    DisallowKubeconfigRotationForShootInDeletionRemoved1.38
    LoggingfalseAlpha0.131.40
    LoggingRemoved1.41
    AdminKubeconfigRequestfalseAlpha1.241.38
    AdminKubeconfigRequesttrueBeta1.391.41
    AdminKubeconfigRequesttrueGA1.421.49
    AdminKubeconfigRequestRemoved1.50
    UseDNSRecordsfalseAlpha1.271.38
    UseDNSRecordstrueBeta1.391.43
    UseDNSRecordstrueGA1.441.49
    UseDNSRecordsRemoved1.50
    CachedRuntimeClientsfalseAlpha1.71.33
    CachedRuntimeClientstrueBeta1.341.44
    CachedRuntimeClientstrueGA1.451.49
    CachedRuntimeClientsRemoved1.50
    DenyInvalidExtensionResourcesfalseAlpha1.311.41
    DenyInvalidExtensionResourcestrueBeta1.421.44
    DenyInvalidExtensionResourcestrueGA1.451.49
    DenyInvalidExtensionResourcesRemoved1.50
    RotateSSHKeypairOnMaintenancefalseAlpha1.281.44
    RotateSSHKeypairOnMaintenancetrueBeta1.451.47
    RotateSSHKeypairOnMaintenance (deprecated)falseBeta1.481.50
    RotateSSHKeypairOnMaintenance (deprecated)Removed1.51
    ShootMaxTokenExpirationOverwritefalseAlpha1.431.44
    ShootMaxTokenExpirationOverwritetrueBeta1.451.47
    ShootMaxTokenExpirationOverwritetrueGA1.481.50
    ShootMaxTokenExpirationOverwriteRemoved1.51
    ShootMaxTokenExpirationValidationfalseAlpha1.431.45
    ShootMaxTokenExpirationValidationtrueBeta1.461.47
    ShootMaxTokenExpirationValidationtrueGA1.481.50
    ShootMaxTokenExpirationValidationRemoved1.51
    WorkerPoolKubernetesVersionfalseAlpha1.351.45
    WorkerPoolKubernetesVersiontrueBeta1.461.49
    WorkerPoolKubernetesVersiontrueGA1.501.51
    WorkerPoolKubernetesVersionRemoved1.52
    DisableDNSProviderManagementfalseAlpha1.411.49
    DisableDNSProviderManagementtrueBeta1.501.51
    DisableDNSProviderManagementtrueGA1.52
    SecretBindingProviderValidationfalseAlpha1.381.50
    SecretBindingProviderValidationtrueBeta1.511.52
    SecretBindingProviderValidationtrueGA1.531.54
    SecretBindingProviderValidationRemoved1.55
    SeedKubeSchedulerfalseAlpha1.151.54
    SeedKubeSchedulerfalseDeprecated1.55
    ShootCARotationfalseAlpha1.421.50
    ShootCARotationtrueBeta1.511.56
    ShootCARotationtrueGA1.57
    ShootSARotationfalseAlpha1.481.50
    ShootSARotationtrueBeta1.511.56
    ShootSARotationtrueGA1.57

    Using a feature

    A feature can be in Alpha, Beta or GA stage. An Alpha feature means:

    • Disabled by default.
    • Might be buggy. Enabling the feature may expose bugs.
    • Support for feature may be dropped at any time without notice.
    • The API may change in incompatible ways in a later software release without notice.
    • Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.

    A Beta feature means:

    • Enabled by default.
    • The feature is well tested. Enabling the feature is considered safe.
    • Support for the overall feature will not be dropped, though details may change.
    • The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
    • Recommended for only non-critical uses because of potential for incompatible changes in subsequent releases.

    Please do try Beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.

    A General Availability (GA) feature is also referred to as a stable feature. It means:

    • The feature is always enabled; you cannot disable it.
    • The corresponding feature gate is no longer needed.
    • Stable versions of features will appear in released software for many subsequent versions.

    List of Feature Gates

    FeatureRelevant ComponentsDescription
    HVPAgardenletEnables simultaneous horizontal and vertical scaling in Seed Clusters.
    HVPAForShootedSeedgardenletEnables simultaneous horizontal and vertical scaling in managed seed (aka “shooted seed”) clusters.
    ManagedIstio (deprecated)gardenletEnables a Gardener-tailored Istio in each Seed cluster. Disable this feature if Istio is already installed in the cluster. Istio is not automatically removed if this feature is disabled. See the detailed documentation for more information.
    APIServerSNI (deprecated)gardenletEnables only one LoadBalancer to be used for every Shoot cluster API server in a Seed. Enable this feature when ManagedIstio is enabled or Istio is manually deployed in Seed cluster. See GEP-8 for more details.
    SeedChangegardener-apiserverEnables updating the spec.seedName field during shoot validation from a non-empty value in order to trigger shoot control plane migration.
    SeedKubeSchedulergardenletAdds custom kube-scheduler in gardener-kube-scheduler namespace. It schedules pods with scheduler name gardener-kube-scheduler on Nodes with higher resource utilization.
    ReversedVPNgardenletReverses the connection setup of the vpn tunnel between the Seed and the Shoot cluster(s). It allows Seed and Shoot clusters to be in different networks with only direct access in one direction (Shoot -> Seed). In addition to that, it reduces the amount of load balancers required, i.e. no load balancers are required for the vpn tunnel anymore. It requires APIServerSNI and kubernetes version 1.18 or higher to work. Details can be found in GEP-14.
    CopyEtcdBackupsDuringControlPlaneMigrationgardenletEnables the copy of etcd backups from the object store of the source seed to the object store of the destination seed during control plane migration.
    SecretBindingProviderValidationgardener-apiserverEnables validations on Gardener API server that:
    - requires the provider type of a SecretBinding to be set (on SecretBinding creation)
    - requires the SecretBinding provider type to match the Shoot provider type (on Shoot creation)
    - enforces immutability on the provider type of a SecretBinding
    ForceRestoregardenletEnables forcing the shoot’s restoration to the destination seed during control plane migration if the preparation for migration in the source seed is not finished after a certain grace period and is considered unlikely to succeed (falling back to the control plane migration “bad case” scenario). If you enable this feature gate, make sure to also enable CopyEtcdBackupsDuringControlPlaneMigration.
    DisableDNSProviderManagementgardenletDisables management of dns.gardener.cloud/v1alpha1.DNSProvider resources. In this case, the shoot-dns-service extension will take this over if it is installed.
    ShootCARotationgardener-apiserver, gardenletEnables the feature to trigger automated CA rotation for shoot clusters.
    ShootSARotationgardener-apiserver, gardenletEnables the feature to trigger automated service account signing key rotation for shoot clusters.
    HAControlPlanesgardener-scheduler, gardenletHAControlPlanes allows shoot control planes to be run in high availability mode.
    WorkerPoolKubernetesVersiongardener-apiserverAllows to overwrite the Kubernetes version used for shoot clusters per worker pool (see this document)
    DefaultSeccompProfilegardenletEnables the defaulting of the seccomp profile for Gardener managed workload in the seed to RuntimeDefault.
    CoreDNSQueryRewritinggardenletEnables automatic DNS query rewriting in shoot cluster’s CoreDNS to shortcut name resolution of fully qualified in-cluster and out-of-cluster names, which follow a user-defined pattern. Details can be found in DNS Search Path Optimization.

    1.3.7 - Gardenlet API Access

    Scoped API Access for Gardenlets

    By default, gardenlets have administrative access in the garden cluster. They are able to execute any API request on any object independent of whether the object is related to the seed cluster the gardenlet is responsible for. As RBAC is not powerful enough for fine-grained checks and for the sake of security, Gardener provides two optional but recommended configurations for your environments that scope the API access for gardenlets.

    Similar to the Node authorization mode in Kubernetes, Gardener features a SeedAuthorizer plugin. It is a special-purpose authorization plugin that specifically authorizes API requests made by the gardenlets.

    Likewise, similar to the NodeRestriction admission plugin in Kubernetes, Gardener features a SeedRestriction plugin. It is a special-purpose admission plugin that specifically limits the Kubernetes objects gardenlets can modify.

    📚 You might be interested to look into the design proposal for scoped Kubelet API access from the Kubernetes community. It can be translated to Gardener and Gardenlets with their Seed and Shoot resources.

    Flow Diagram

    The following diagram shows how the two plugins are included in the request flow of a gardenlet. When they are not enabled then the kube-apiserver is internally authorizing the request via RBAC before forwarding the request directly to the gardener-apiserver, i.e., the gardener-admission-controller would not be consulted (this is not entirely correct because it also serves other admission webhook handlers, but for simplicity reasons this document focuses on the API access scope only).

    When enabling the plugins, there is one additional step for each before the gardener-apiserver responds to the request.

    Flow Diagram

    Please note that the example shows a request to an object (Shoot) residing in one of the API groups served by gardener-apiserver. However, the gardenlet is also interacting with objects in API groups served by the kube-apiserver (e.g., Secret,ConfigMap, etc.). In this case, the consultation of the SeedRestriction admission plugin is performed by the kube-apiserver itself before it forwards the request to the gardener-apiserver.

    Today, the following rules are implemented:

    ResourceVerbsPath(s)Description
    BackupBucketget, list, watch, create, update, patch, deleteBackupBucket -> SeedAllow get, list, watch requests for all BackupBuckets. Allow only create, update, patch, delete requests for BackupBuckets assigned to the gardenlet’s Seed.
    BackupEntryget, list, watch, create, update, patchBackupEntry -> SeedAllow get, list, watch requests for all BackupEntrys. Allow only create, update, patch requests for BackupEntrys assigned to the gardenlet’s Seed and referencing BackupBuckets assigned to the gardenlet’s Seed.
    Bastionget, list, watch, create, update, patchBastion -> SeedAllow get, list, watch requests for all Bastions. Allow only create, update, patch requests for Bastions assigned to the gardenlet’s Seed.
    CertificateSigningRequestget, createCertificateSigningRequest -> SeedAllow only get, create requests for CertificateSigningRequests related to the gardenlet’s Seed.
    CloudProfilegetCloudProfile -> Shoot -> SeedAllow only get requests for CloudProfiles referenced by Shoots that are assigned to the gardenlet’s Seed.
    ClusterRoleBindingcreate, get, update, patch, deleteClusterRoleBinding -> ManagedSeed -> Shoot -> SeedAllow create, get, update, patch requests for ManagedSeeds in the bootstrapping phase assigned to the gardenlet’s Seeds. Allow delete requests from gardenlets bootstrapped via ManagedSeeds.
    ConfigMapgetConfigMap -> Shoot -> SeedAllow only get requests for ConfigMaps referenced by Shoots that are assigned to the gardenlet’s Seed. Allows reading the kube-system/cluster-identity ConfigMap.
    ControllerRegistrationget, list, watchControllerRegistration -> ControllerInstallation -> SeedAllow get, list, watch requests for all ControllerRegistrations.
    ControllerDeploymentgetControllerDeployment -> ControllerInstallation -> SeedAllow get requests for ControllerDeploymentss referenced by ControllerInstallations assigned to the gardenlet’s Seed.
    ControllerInstallationget, list, watch, update, patchControllerInstallation -> SeedAllow get, list, watch requests for all ControllerInstallations. Allow only update, patch requests for ControllerInstallations assigned to the gardenlet’s Seed.
    Eventcreate, patchnoneAllow to create or patch all kinds of Events.
    ExposureClassgetExposureClass -> Shoot -> SeedAllow get requests for ExposureClasses referenced by Shoots that are assigned to the gardenlet’s Seed. Deny get requests to other ExposureClasses.
    Leasecreate, get, watch, updateLease -> SeedAllow create, get, update, and delete requests for Leases of the gardenlet’s Seed.
    ManagedSeedget, list, watch, update, patchManagedSeed -> Shoot -> SeedAllow get, list, watch requests for all ManagedSeeds. Allow only update, patch requests for ManagedSeeds referencing a Shoot assigned to the gardenlet’s Seed.
    NamespacegetNamespace -> Shoot -> SeedAllow get requests for Namespaces of Shoots that are assigned to the gardenlet’s Seed. Always allow get requests for the garden Namespace.
    ProjectgetProject -> Namespace -> Shoot -> SeedAllow get requests for Projects referenced by the Namespace of Shoots that are assigned to the gardenlet’s Seed.
    SecretBindinggetSecretBinding -> Shoot -> SeedAllow only get requests for SecretBindings referenced by Shoots that are assigned to the gardenlet’s Seed.
    Secretcreate, get, update, patch, delete(, list, watch)Secret -> Seed, Secret -> Shoot -> Seed, Secret -> SecretBinding -> Shoot -> Seed, BackupBucket -> SeedAllow get, list, watch requests for all Secrets in the seed-<name> namespace. Allow only create, get, update, patch, delete requests for the Secrets related to resources assigned to the gardenlet's Seed`s.
    Seedget, list, watch, create, update, patch, deleteSeedAllow get, list, watch requests for all Seeds. Allow only create, update, patch, delete requests for the gardenlet’s Seeds. [1]
    ServiceAccountcreate, get, update, patch, deleteServiceAccount -> ManagedSeed -> Shoot -> SeedAllow create, get, update, patch requests for ManagedSeeds in the bootstrapping phase assigned to the gardenlet’s Seeds. Allow delete requests from gardenlets bootstrapped via ManagedSeeds.
    Shootget, list, watch, update, patchShoot -> SeedAllow get, list, watch requests for all Shoots. Allow only update, patch requests for Shoots assigned to the gardenlet’s Seed.
    ShootStateget, create, update, patchShootState -> Shoot -> SeedAllow only get, create, update, patch requests for ShootStates belonging by Shoots that are assigned to the gardenlet’s Seed.

    [1] If you use ManagedSeed resources then the gardenlet reconciling them (“parent gardenlet”) may be allowed to submit certain requests for the Seed resources resulting out of such ManagedSeed reconciliations (even if the “parent gardenlet” is not responsible for them):

    • ℹ️ It is allowed to delete the Seed resources if the corresponding ManagedSeed objects already have a deletionTimestamp (this is secure as gardenlets themselves don’t have permissions for deleting ManagedSeeds).
    • ⚠ It is allowed to create or update Seed resources if the corresponding ManagedSeed objects use a seed template, i.e., .spec.seedTemplate != nil. In this case, there is at least one gardenlet in your system which is responsible for two or more Seeds. Please keep in mind that this use case is not recommended for production scenarios (you should only have one dedicated gardenlet per seed cluster), hence, the security improvements discussed in this document might be limited.

    SeedAuthorizer Authorization Webhook Enablement

    The SeedAuthorizer is implemented as Kubernetes authorization webhook and part of the gardener-admission-controller component running in the garden cluster.

    🎛 In order to activate it, you have to follow these steps:

    1. Set the following flags for the kube-apiserver of the garden cluster (i.e., the kube-apiserver whose API is extended by Gardener):

      • --authorization-mode=RBAC,Node,Webhook (please note that Webhook should appear after RBAC in the list [1]; Node might not be needed if you use a virtual garden cluster)
      • --authorization-webhook-config-file=<path-to-the-webhook-config-file>
      • --authorization-webhook-cache-authorized-ttl=0
      • --authorization-webhook-cache-unauthorized-ttl=0
    2. The webhook config file (stored at <path-to-the-webhook-config-file>) should look as follows:

      apiVersion: v1
      kind: Config
      clusters:
      - name: garden
        cluster:
          certificate-authority-data: base64(CA-CERT-OF-GARDENER-ADMISSION-CONTROLLER)
          server: https://gardener-admission-controller.garden/webhooks/auth/seed
      users:
      - name: kube-apiserver
        user: {}
      contexts:
      - name: auth-webhook
        context:
          cluster: garden
          user: kube-apiserver
      current-context: auth-webhook
      
    3. When deploying the Gardener controlplane Helm chart, set .global.rbac.seedAuthorizer.enabled=true. This will prevent that the RBAC resources granting global access for all gardenlets will be deployed.

    4. Delete the existing RBAC resources granting global access for all gardenlets by running:

      kubectl delete \
        clusterrole.rbac.authorization.k8s.io/gardener.cloud:system:seeds \
        clusterrolebinding.rbac.authorization.k8s.io/gardener.cloud:system:seeds \
        --ignore-not-found
      

    Please note that you should activate the SeedRestriction admission handler as well.

    [1] The reason for the fact that Webhook authorization plugin should appear after RBAC is that the kube-apiserver will be depending on the gardener-admission-controller (serving the webhook). However, the gardener-admission-controller can only start when gardener-apiserver runs, but gardener-apiserver itself can only start when kube-apiserver runs. If Webhook is before RBAC then gardener-apiserver might not be able to start, leading to a deadlock.

    Authorizer Decisions

    As mentioned earlier, it’s the authorizer’s job to evaluate API requests and return one of the following decisions:

    • DecisionAllow: The request is allowed, further configured authorizers won’t be consulted.
    • DecisionDeny: The request is denied, further configured authorizers won’t be consulted.
    • DecisionNoOpinion: A decision cannot be made, further configured authorizers will be consulted.

    For backwards compatibility, no requests are denied at the moment, so that they are still deferred to a subsequent authorizer like RBAC. Though, this might change in the future.

    First, the SeedAuthorizer extracts the Seed name from the API request. This requires a proper TLS certificate the gardenlet uses to contact the API server and is automatically given if TLS bootstrapping is used. Concretely, the authorizer checks the certificate for name gardener.cloud:system:seed:<seed-name> and group gardener.cloud:system:seeds. In cases where this information is missing e.g., when a custom Kubeconfig is used, the authorizer cannot make any decision. Thus, RBAC is still a considerable option to restrict the gardenlet’s access permission if the above explained preconditions are not given.

    With the Seed name at hand, the authorizer checks for an existing path from the resource that a request is being made for to the Seed belonging to the gardenlet. Take a look at the Implementation Details section for more information.

    Implementation Details

    Internally, the SeedAuthorizer uses a directed, acyclic graph data structure in order to efficiently respond to authorization requests for gardenlets:

    • A vertex in this graph represents a Kubernetes resource with its kind, namespace, and name (e.g., Shoot:garden-my-project/my-shoot).
    • An edge from vertex u to vertex v in this graph exists when
      • (1) v is referred by u and v is a Seed, or when
      • (2) u is referred by v, or when
      • (3) u is strictly associated with v.

    For example, a Shoot refers to a Seed, a CloudProfile, a SecretBinding, etc., so it has an outgoing edge to the Seed (1) and incoming edges from the CloudProfile and SecretBinding vertices (2). However, there might also be a ShootState or a BackupEntry resource strictly associated with this Shoot, hence, it has incoming edges from these vertices (3).

    Resource Dependency Graph

    In above picture the resources that are actively watched have are shaded. Gardener resources are green while Kubernetes resources are blue. It shows the dependencies between the resources and how the graph is built based on above rules.

    ℹ️ Above picture shows all resources that may be accessed by gardenlets, except for the Quota resource which is only included for completeness.

    Now, when a gardenlet wants to access certain resources then the SeedAuthorizer uses a Depth-First traversal starting from the vertex representing the resource in question, e.g., from a Project vertex. If there is a path from the Project vertex to the vertex representing the Seed the gardenlet is responsible for then it allows the request.

    Metrics

    The SeedAuthorizer registers the following metrics related to the mentioned graph implementation:

    MetricDescription
    gardener_admission_controller_seed_authorizer_graph_update_duration_secondsHistogram of duration of resource dependency graph updates in seed authorizer, i.e., how long does it take to update the graph’s vertices/edges when a resource is created, changed, or deleted.
    gardener_admission_controller_seed_authorizer_graph_path_check_duration_secondsHistogram of duration of checks whether a path exists in the resource dependency graph in seed authorizer.

    Debug Handler

    When the .server.enableDebugHandlers field in the gardener-admission-controller’s component configuration is set to true then it serves a handler that can be used for debugging the resource dependency graph under /debug/resource-dependency-graph.

    🚨 Only use this setting for development purposes as it enables unauthenticated users to view all data if they have access to the gardener-admission-controller component.

    The handler renders an HTML page displaying the current graph with a list of vertices and its associated incoming and outgoing edges to other vertices. Depending on the size of the Gardener landscape (and consequently, the size of the graph), it might not be possible to render it in its entirety. If there are more than 2000 vertices then the default filtering will selected for kind=Seed to prevent overloading the output.

    Example output:

    -------------------------------------------------------------------------------
    |
    | # Seed:my-seed
    |   <- (11)
    |     BackupBucket:73972fe2-3d7e-4f61-a406-b8f9e670e6b7
    |     BackupEntry:garden-my-project/shoot--dev--my-shoot--4656a460-1a69-4f00-9372-7452cbd38ee3
    |     ControllerInstallation:dns-external-mxt8m
    |     ControllerInstallation:extension-shoot-cert-service-4qw5j
    |     ControllerInstallation:networking-calico-bgrb2
    |     ControllerInstallation:os-gardenlinux-qvb5z
    |     ControllerInstallation:provider-gcp-w4mvf
    |     Secret:garden/backup
    |     Shoot:garden-my-project/my-shoot
    |
    -------------------------------------------------------------------------------
    |
    | # Shoot:garden-my-project/my-shoot
    |   <- (5)
    |     CloudProfile:gcp
    |     Namespace:garden-my-project
    |     Secret:garden-my-project/my-dns-secret
    |     SecretBinding:garden-my-project/my-credentials
    |     ShootState:garden-my-project/my-shoot
    |   -> (1)
    |     Seed:my-seed
    |
    -------------------------------------------------------------------------------
    |
    | # ShootState:garden-my-project/my-shoot
    |   -> (1)
    |     Shoot:garden-my-project/my-shoot
    |
    -------------------------------------------------------------------------------
    
    ... (etc., similarly for the other resources)
    

    There are anchor links to easily jump from one resource to another, and the page provides means for filtering the results based on the kind, namespace, and/or name.

    Pitfalls

    When there is a relevant update to an existing resource, i.e., when a reference to another resource is changed, then the corresponding vertex (along with all associated edges) is first deleted from the graph before it gets added again with the up-to-date edges. However, this does only work for vertices belonging to resources that are only created in exactly one “watch handler”. For example, the vertex for a SecretBinding can either be created in the SecretBinding handler itself or in the Shoot handler. In such cases, deleting the vertex before (re-)computing the edges might lead to race conditions and potentially renders the graph invalid. Consequently, instead of deleting the vertex, only the edges the respective handler is responsible for are deleted. If the vertex ends up with no remaining edges then it also gets deleted automatically. Afterwards, the vertex can either be added again or the updated edges can be created.

    SeedRestriction Admission Webhook Enablement

    The SeedRestriction is implemented as Kubernetes admission webhook and part of the gardener-admission-controller component running in the garden cluster.

    🎛 In order to activate it, you have to set .global.admission.seedRestriction.enabled=true when using the Gardener controlplane Helm chart. This will add an additional webhook in the existing ValidatingWebhookConfiguration of the gardener-admission-controller which contains the configuration for the SeedRestriction handler. Please note that it should only be activated when the SeedAuthorizer is active as well.

    Admission Decisions

    The admission’s purpose is to perform extended validation on requests which require the body of the object in question. Additionally, it handles CREATE requests of gardenlets (above discussed resource dependency graph cannot be used in such cases because there won’t be any vertex/edge for non-existing resources).

    Gardenlets are restricted to only create new resources which are somehow related to the seed clusters they are responsible for.

    1.3.8 - Getting Started Locally

    Deploying Gardener locally

    This document will walk you through deploying Gardener on your local machine. If you encounter difficulties, please open an issue so that we can make this process easier.

    Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as both garden and seed cluster (please refer to the architecture overview) for simplicity.

    Based on Skaffold, the container images for all required components will be built and deployed into the cluster (via their Helm charts).

    Architecture Diagram

    Prerequisites

    • Make sure that you have followed the Local Setup guide up until the Get the sources step.
    • Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least 8 CPUs and 8Gi memory; see here how to configure the resources for Docker for Mac).

      Please note that 8 CPU / 8Gi memory might not be enough for more than two Shoot clusters, i.e., you might need to increase these values if you want to run additional Shoots. If you plan on following the optional steps to create a second seed cluster, the required resources will be more - at least 10 CPUs and 18Gi memory. Additionally, please configure at least 120Gi of disk size for the Docker daemon. Tip: With docker system df and docker system prune -a you can cleanup unused data.

    Setting up the KinD cluster (garden and seed)

    make kind-up
    

    This command sets up a new KinD cluster named gardener-local and stores the kubeconfig in the ./example/gardener-local/kind/kubeconfig file.

    It might be helpful to copy this file to $HOME/.kube/config since you will need to target this KinD cluster multiple times. Alternatively, make sure to set your KUBECONFIG environment variable to ./example/gardener-local/kind/kubeconfig for all future steps via export KUBECONFIG=example/gardener-local/kind/kubeconfig.

    All following steps assume that you are using this kubeconfig.

    Additionally, this command also deploys a local container registry to the cluster as well as a few registry mirrors, that are set up as a pull-through cache for all upstream registries Gardener uses by default. This is done to speed up image pulls across local clusters. The local registry can be accessed as localhost:5001 for pushing and pulling. The storage directories of the registries are mounted to the host machine under dev/local-registry. With this, mirrored images don’t have to be pulled again after recreating the cluster.

    The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy support (the default kindnet CNI doesn’t provide NetworkPolicy support). Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.

    Setting up Gardener

    make gardener-up
    

    This will first build the images based (which might take a bit if you do it for the first time). Afterwards, the Gardener resources will be deployed into the cluster.

    Creating a Shoot cluster

    You can wait for the Seed to be ready by running

    kubectl wait --for=condition=gardenletready --for=condition=extensionsready --for=condition=bootstrapped seed local --timeout=5m
    

    Alternatively, you can run kubectl get seed local and wait for the STATUS to indicate readiness:

    NAME    STATUS   PROVIDER   REGION   AGE     VERSION       K8S VERSION
    local   Ready    local      local    4m42s   vX.Y.Z-dev    v1.21.1
    

    In order to create a first shoot cluster, just run

    kubectl apply -f example/provider-local/shoot.yaml
    

    You can wait for the Shoot to be ready by running

    kubectl wait --for=condition=apiserveravailable --for=condition=controlplanehealthy --for=condition=everynodeready --for=condition=systemcomponentshealthy shoot local -n garden-local --timeout=10m
    

    Alternatively, you can run kubectl -n garden-local get shoot local and wait for the LAST OPERATION to reach 100%:

    NAME    CLOUDPROFILE   PROVIDER   REGION   K8S VERSION   HIBERNATION   LAST OPERATION            STATUS    AGE
    local   local          local      local    1.21.0        Awake         Create Processing (43%)   healthy   94s
    

    (Optional): You could also execute a simple e2e test (creating and deleting a shoot) by running

    make test-e2e-local-simple KUBECONFIG="$PWD/example/gardener-local/kind/kubeconfig"
    

    Accessing the Shoot cluster

    ⚠️ Please note that in this setup shoot clusters are not accessible by default when you download the kubeconfig and try to communicate with them. The reason is that your host most probably cannot resolve the DNS names of the clusters since provider-local extension runs inside the KinD cluster (see this for more details). Hence, if you want to access the shoot cluster, you have to run the following command which will extend your /etc/hosts file with the required information to make the DNS names resolvable:

    cat <<EOF | sudo tee -a /etc/hosts
    
    # Manually created to access local Gardener shoot clusters.
    # TODO: Remove this again when the shoot cluster access is no longer required.
    127.0.0.1 api.local.local.external.local.gardener.cloud
    127.0.0.1 api.local.local.internal.local.gardener.cloud
    
    127.0.0.1 api.e2e-managedseed.garden.external.local.gardener.cloud
    127.0.0.1 api.e2e-managedseed.garden.internal.local.gardener.cloud
    127.0.0.1 api.e2e-hibernated.local.external.local.gardener.cloud
    127.0.0.1 api.e2e-hibernated.local.internal.local.gardener.cloud
    127.0.0.1 api.e2e-unpriv.local.external.local.gardener.cloud
    127.0.0.1 api.e2e-unpriv.local.internal.local.gardener.cloud
    127.0.0.1 api.e2e-wake-up.local.external.local.gardener.cloud
    127.0.0.1 api.e2e-wake-up.local.internal.local.gardener.cloud
    127.0.0.1 api.e2e-migrate.local.external.local.gardener.cloud
    127.0.0.1 api.e2e-migrate.local.internal.local.gardener.cloud
    127.0.0.1 api.e2e-rotate.local.external.local.gardener.cloud
    127.0.0.1 api.e2e-rotate.local.internal.local.gardener.cloud
    127.0.0.1 api.e2e-default.local.external.local.gardener.cloud
    127.0.0.1 api.e2e-default.local.internal.local.gardener.cloud
    EOF
    

    Now you can access it by running

    kubectl -n garden-local get secret local.kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > /tmp/kubeconfig-shoot-local.yaml
    kubectl --kubeconfig=/tmp/kubeconfig-shoot-local.yaml get nodes
    

    (Optional): Setting up a second seed cluster

    There are cases where you would want to create a second cluster seed in your local setup. For example, if you want to test the control plane migration feature. The following steps describe how to do that.

    make kind2-up
    

    This command sets up a new KinD cluster named gardener-local2 and stores its kubeconfig in the ./example/gardener-local/kind2/kubeconfig file.

    In order to deploy required resources in the KinD cluster that you just created, run:

    make gardenlet-kind2-up
    

    The following steps assume that your are using the kubeconfig that points to the gardener-local cluster (first KinD cluster): export KUBECONFIG=example/gardener-local/kind/kubeconfig.

    You can wait for the local2 Seed to be ready by running:

    kubectl wait --for=condition=gardenletready --for=condition=extensionsready --for=condition=bootstrapped seed local2 --timeout=5m
    

    Alternatively, you can run kubectl get seed local2 and wait for the STATUS to indicate readiness:

    NAME    STATUS   PROVIDER   REGION   AGE     VERSION       K8S VERSION
    local2  Ready    local      local    4m42s   vX.Y.Z-dev    v1.21.1
    

    If you want to perform control plane migration you can follow the steps outlined here to migrate the shoot cluster to the second seed you just created.

    Deleting the Shoot cluster

    ./hack/usage/delete shoot local garden-local
    

    (Optional): Tear down the second seed cluster

    make kind2-down
    

    Tear down the Gardener environment

    make kind-down
    

    Further reading

    This setup makes use of the local provider extension. You can read more about it in this document.

    1.3.9 - Image Vector

    Image Vector

    The Gardenlet is deploying several different container images into the seed and the shoot clusters. The image repositories and tags are defined in a central image vector file. Obviously, the image versions defined there must fit together with the deployment manifests (e.g., some command-line flags do only exist in certain versions).

    Example

    images:
    - name: pause-container
      sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
      repository: gcr.io/google_containers/pause-amd64
      tag: "3.0"
      version: 1.17.x
      architectures:
      - amd64
    - name: pause-container
      sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
      repository: gcr.io/google_containers/pause-amd64
      tag: "3.1"
      version: ">= 1.18"
      architectures:
      - amd64
    ...
    

    That means that the Gardenlet will use the pause-container in with tag 3.0 for all seed/shoot clusters with Kubernetes version 1.17.x, and tag 3.1 for all clusters with Kubernetes >= 1.18.

    Image Vector Architecture

    images:
    - name: pause-container
      sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
      repository: registry.k8s.io/pause
      tag: "3.0"
      architectures:
      - amd64
    - name: pause-container
      sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
      repository: registry.k8s.io/pause
      tag: "3.0"
      architectures:
      - arm64
    - name: pause-container
      sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
      repository: registry.k8s.io/pause
      tag: "3.0"
      architectures:
      - amd64
      - arm64
    ...
    

    Architectures is an optional field of image. It is a list of strings specifying CPU architecture of machines on which this image can be used. The valid options for architectures field are as follow:

    • amd64 : This sepecifies image can run only on machines having CPU architecture amd64.
    • arm64 : This sepecifies image can run only on machines having CPU architecture arm64.

    If image doesn’t specifies any architectures then by default it is considered to support both amd64 and arm64 architectures.

    Overwrite image vector

    In some environment it is not possible to use these “pre-defined” images that come with a Gardener release. A prominent example for that is Alicloud in China which does not allow access to Google’s GCR. In these cases you might want to overwrite certain images, e.g., point the pause-container to a different registry.

    ⚠️ If you specify an image that does not fit to the resource manifest then the seed/shoot reconciliation might fail.

    In order to overwrite the images you must provide a similar file to Gardenlet:

    images:
    - name: pause-container
      sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
      repository: my-custom-image-registry/pause-amd64
      tag: "3.0"
      version: 1.17.x
    - name: pause-container
      sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
      repository: my-custom-image-registry/pause-amd64
      tag: "3.1"
      version: ">= 1.18"
    ...
    

    During deployment of the gardenlet create a ConfigMap containing the above content and mount it as a volume into the gardenlet pod. Next, specify the environment variable IMAGEVECTOR_OVERWRITE whose value must be the path to the file you just mounted:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: gardenlet-images-overwrite
      namespace: garden
    data:
      images_overwrite.yaml: |
        images:
        - ...    
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: gardenlet
      namespace: garden
    spec:
      template:
        ...
        spec:
          containers:
          - name: gardenlet
            env:
            - name: IMAGEVECTOR_OVERWRITE
              value: /charts-overwrite/images_overwrite.yaml
            volumeMounts:
            - name: gardenlet-images-overwrite
              mountPath: /charts-overwrite
            ...
          volumes:
          - name: gardenlet-images-overwrite
            configMap:
              name: gardenlet-images-overwrite
      ...
    

    Image vectors for dependent components

    The gardenlet is deploying a lot of different components that might deploy other images themselves. These components might use an image vector as well. Operators might want to customize the image locations for these transitive images as well, hence, they might need to specify an image vector overwrite for the components directly deployed by Gardener.

    It is possible to specify the IMAGEVECTOR_OVERWRITE_COMPONENTS environment variable to the gardenlet that points to a file with the following content:

    components:
    - name: etcd-druid
      imageVectorOverwrite: |
        images:
        - name: etcd
          tag: v1.2.3
          repository: etcd/etcd    
    ...
    

    The gardenlet will, if supported by the directly deployed component (etcd-druid in this example), inject the given imageVectorOverwrite into the Deployment manifest. The respective component is responsible for using the overwritten images instead of its defaults.

    1.3.10 - Migration V0 To V1

    Migration from Gardener v0 to v1

    Please refer to the document for older Gardener versions.

    1.3.11 - Secret Binding Provider Controller

    SecretBinding Provider Controller

    This page describes the process on how to enable the SecretBinding provider controller.

    Overview

    With Gardener v1.38.0 the SecretBinding resource does now contain a new optional field .provider.type (details about the motivation can be found in https://github.com/gardener/gardener/issues/4888). To make the process of setting the new field automated and afterwards to enforce validation on the new field in backwards compatible manner, Gardener features the SecretBinding provider controller and a feature gate - SecretBindingProviderValidation.

    Process

    A Gardener landscape operator can follow the following steps:

    1. Enable the SecretBinding provider controller of Gardener Controller Manager.

      The SecretBinding provider controller is responsible to populate the .provider.type field of a SecretBinding based on its current usage by Shoot resources. For example if a Shoot crazy-botany with .provider.type=aws is using a SecretBinding my-secret-binding, then the SecretBinding provider controller will take care to set the .provider.type field of the SecretBinding to the same provider type (aws). To enable the SecretBinding provider controller, in the ControllerManagerConfiguration set the controller.secretBindingProvider.concurentSyncs field (e.g set it to 5). Although that it is not recommended, the API allows Shoots from different provider types to reference the same SecretBinding (assuming that backing Secret contains data for both of the provider types). To preserve the backwards compatibility for such SecretBindings, the provider controller will maintain the multiple provider types in the field (it will join them with separator , - for example aws,gcp).

    2. Disable the SecretBinding provider controller and enable SecretBindingProviderValidation feature gate of Gardener API server.

      The SecretBindingProviderValidation feature gate of Gardener API server enables set of validations for the SecretBinding provider field. It forbids creating a Shoot that has a different provider type from the referenced SecretBinding’s one. It also enforces immutability on the field. After making sure that SecretBinding provider controller is enabled and it populated the .provider.type field of a majority of the SecretBindings on a Gardener landscape (the SecretBindings that are unused will have their provider type unset), a Gardener landscape operator has to disable the SecretBinding provider controller and to enable the SecretBindingProviderValidation feature gate of Gardener API server. To disable the SecretBinding provider controller, in the ControllerManagerConfiguration set the controller.secretBindingProvider.concurentSyncs field to 0.

    Implementation History

    • Gardener v1.38: The SecretBinding resource has a new optional field .provider.type. The SecretBinding provider controller is disabled by default. The SecretBindingProviderValidation feature gate of Gardener API server is disabled by default.
    • Gardener v1.42: The SecretBinding provider controller is enabled by default.
    • Gardener v1.51: The SecretBindingProviderValidation feature gate of Gardener API server is enabled by default and the SecretBinding provider controller is disabled by default.
    • Gardener v1.53: The SecretBindingProviderValidation feature gate of Gardener API server is unconditionally enabled (can no longer be disabled).
    • Gardener v1.55: The SecretBindingProviderValidation feature gate of Gardener API server and the SecretBinding provider controller are removed.

    1.3.12 - Setup Gardener

    Deploying the Gardener into a Kubernetes cluster

    Similar to Kubernetes, Gardener consists out of control plane components (Gardener API server, Gardener controller manager, Gardener scheduler), and an agent component (Gardenlet). The control plane is deployed in the so-called garden cluster while the agent is installed into every seed cluster. Please note that it is possible to use the garden cluster as seed cluster by simply deploying the Gardenlet into it.

    We are providing Helm charts in order to manage the various resources of the components. Please always make sure that you use the Helm chart version that matches the Gardener version you want to deploy.

    Deploying the Gardener control plane (API server, admission controller, controller manager, scheduler)

    The configuration values depict the various options to configure the different components. Please consult this document for component specific configurations and this document for authentication related specifics.

    Also note that all resources and deployments need to be created in the garden namespace (not overrideable). If you enable the Gardener admission controller as part of you setup, please make sure the garden namespace is labelled with app: gardener. Otherwise, the backing service account for the admission controller Pod might not be created successfully. No action is necessary, if you deploy the garden namespace with the Gardener control plane Helm chart.

    After preparing your values in a separate controlplane-values.yaml file (values.yaml can be used as starting point), you can run the following command against your garden cluster:

    helm install charts/gardener/controlplane \
      --namespace garden \
      --name gardener-controlplane \
      -f controlplane-values.yaml \
      --wait
    

    Deploying Gardener extensions

    Gardener is an extensible system that does not contain the logic for provider-specific things like DNS management, cloud infrastructures, network plugins, operating system configs, and many more.

    You have to install extension controllers for these parts. Please consult the documentation regarding extensions to get more information.

    Deploying the Gardener agent (Gardenlet)

    Please refer to this document on how to deploy a Gardenlet.

    1.3.13 - Version Skew Policy

    Version Skew Policy

    This document describes the maximum version skew supported between various Gardener components.

    Supported Gardener Versions

    Gardener versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version, following Semantic Versioning terminology.

    The Gardener project maintains release branches for the most recent three minor releases.

    Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility. Patch releases are cut from those branches at a regular cadence, plus additional urgent releases when required.

    For more information, see this document.

    Supported Version Skew

    Technically, we follow the same policy as the Kubernetes project. However, given that our release cadence is much more frequent compared to Kubernetes (every 14d vs. every 120d), in many cases it is possible to skip a version. Still, to be on the safe side, it is highly recommended to follow the described policy.

    gardener-apiserver

    In multi-instance setups of Gardener, the newest and oldest gardener-apiserver instances must be within one minor version.

    Example:

    • newest gardener-apiserver is at 1.37
    • other gardener-apiserver instances are supported at 1.37 and v1.36

    gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet

    gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet must not be newer than the gardener-apiserver instances they communicate with. They are expected to match the gardener-apiserver minor version, but may be up to one minor version older (to allow live upgrades).

    Example:

    • gardener-apiserver is at v1.37
    • gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet are supported at 1.37 and v1.36

    Supported Component Upgrade Order

    The supported version skew between components has implications on the order in which components must be upgraded. This section describes the order in which components must be upgraded to transition an existing Gardener installation from version 1.37 to version 1.38.

    gardener-apiserver

    Pre-requisites:

    • In a single-instance setup, the existing gardener-apiserver instance is 1.37
    • In a multi-instance setup, all gardener-apiserver instances are at 1.37 or 1.38 (this ensures maximum skew of 1 minor version between the oldest and newest gardener-apiserver instance)
    • The gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet instances that communicate with this gardener-apiserver are at version 1.37 (this ensures they are not newer than the existing API server version and are within 1 minor version of the new API server version)

    Action:

    • Upgrade gardener-apiserver to 1.38

    gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet

    Pre-requisites:

    • The gardener-apiserver instances these components communicate with are at 1.38 (in multi-instance setups in which these components can communicate with any gardener-apiserver instance in the cluster, all gardener-apiserver instances must be upgraded before upgrading these components)

    Action:

    • Upgrade gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet to 1.38

    Supported Kubernetes Versions

    Please refer to this document.

    1.4 - Development

    1.4.1 - Changing the APIs

    Extending the API

    This document describes the steps that need to be performed when changing the API. It provides guidance for API changes to both (Gardener system in general or component configurations).

    Generally, as Gardener is a Kubernetes-native extension, it follows the same API conventions and guidelines like Kubernetes itself. This document as well as this document already provide a good overview and general explanation of the basic concepts behind it. We are following the same approaches.

    Gardener API

    The Gardener API is defined in pkg/apis/{core,extensions,settings} directories and is the main point of interaction with the system. It must be ensured that the API is always backwards-compatible. If fields shall be removed permanently from the API then a proper deprecation period must be adhered to so that end-users have enough time adapt their clients.

    Checklist when changing the API:

    1. Modify the field(s) in the respective Golang files of all external and the internal version.
      1. Make sure new fields are being added as “optional” fields, i.e., they are of pointer types, they have the // +optional comment, and they have the omitempty JSON tag.
      2. Make sure that the existing field numbers in the protobuf tags are not changed.
    2. If necessary then implement/adapt the conversion logic defined in the versioned APIs (e.g., pkg/apis/core/v1beta1/conversions*.go).
    3. If necessary then implement/adapt defaulting logic defined in the versioned APIs (e.g., pkg/apis/core/v1beta1/defaults*.go).
    4. Run the code generation: make generate
    5. If necessary then implement/adapt validation logic defined in the internal API (e.g., pkg/apis/core/validation/validation*.go).
    6. If necessary then adapt the exemplary YAML manifests of the Gardener resources defined in example/*.yaml.
    7. In most cases it makes sense to add/adapt the documentation for administrators/operators and/or end-users in the docs folder to provide information on purpose and usage of the added/changed fields.
    8. When opening the pull request then always add a release note so that end-users are becoming aware of the changes.

    Component configuration APIs

    Most Gardener components have a component configuration that follows similar principles to the Gardener API. Those component configurations are defined in pkg/{controllermanager,gardenlet,scheduler},pkg/apis/config. Hence, the above checklist also applies for changes to those APIs. However, since these APIs are only used internally and only during the deployment of Gardener the guidelines with respect to changes and backwards-compatibility are slightly relaxed. If necessary then it is allowed to remove fields without a proper deprecation period if the release note uses the breaking operator keywords.

    In addition to the above checklist:

    1. If necessary then adapt the Helm chart of Gardener defined in charts/gardener. Adapt the values.yaml file as well as the manifest templates.

    1.4.2 - Dependencies

    Dependency Management

    We are using go modules for depedency management. In order to add a new package dependency to the project, you can perform go get <PACKAGE>@<VERSION> or edit the go.mod file and append the package along with the version you want to use.

    Updating Dependencies

    The Makefile contains a rule called revendor which performs go mod tidy and go mod vendor. go mod tidy makes sure go.mod matches the source code in the module. It adds any missing modules necessary to build the current module’s packages and dependencies, and it removes unused modules that don’t provide any relevant packages. go mod vendor resets the main module’s vendor directory to include all packages needed to build and test all the main module’s packages. It does not include test code for vendored packages.

    make revendor
    

    The dependencies are installed into the vendor folder which should be added to the VCS.

    ⚠️ Make sure that you test the code after you have updated the dependencies!

    Exported Packages

    This repository contains several packages that could be considered “exported packages”, in a sense that they are supposed to be reused in other Go projects. For example:

    • Gardener’s API packages: pkg/apis
    • Library for building Gardener extensions: extensions
    • Gardener’s Test Framework: test/framework

    There are a few more folders in this repository (non-Go sources) that are reused across projects in the gardener organization:

    • GitHub templates: .github
    • Concourse / cc-utils related helpers: hack/.ci
    • Development, build and testing helpers: hack

    These packages feature a dummy doc.go file to allow other Go projects to pull them in as go mod dependencies.

    These packages are explicitly not supposed to be used in other projects (consider them as “non-exported”):

    • API validation packages: pkg/apis/*/*/validation
    • Operation package (main Gardener business logic regarding Seed and Shoot clusters): pkg/operation
    • Third party code: third_party

    Currently, we don’t have a mechanism yet for selectively syncing out these exported packages into dedicated repositories like kube’s staging mechanism (publishing-bot).

    Import Restrictions

    We want to make sure, that other projects can depend on this repository’s “exported” packages without pulling in the entire repository (including “non-exported” packages) or a high number of other unwanted dependencies. Hence, we have to be careful when adding new imports or references between our packages.

    ℹ️ General rule of thumb: the mentioned “exported” packages should be as self-contained as possible and depend on as few other packages in the repository and other projects as possible.

    In order to support that rule and automatically check compliance with that goal, we leverage import-boss. The tool checks all imports of the given packages (including transitive imports) against rules defined in .import-restrictions files in each directory. An import is allowed if it matches at least one allowed prefix and does not match any forbidden prefixes. Note: '' (the empty string) is a prefix of everything. For more details, see: https://github.com/kubernetes/code-generator/tree/master/cmd/import-boss

    import-boss is executed on every pull request and blocks the PR if it doesn’t comply with the defined import restrictions. You can also run it locally using make check.

    Import restrictions should be changed in the following situations:

    • We spot a new pattern of imports across our packages that was not restricted before but makes it more difficult for other projects to depend on our “exported” packages. In that case, the imports should be further restricted to disallow such problematic imports, and the code/package structure should be reworked to comply with the newly given restrictions.
    • We want to share code between packages, but existing import restrictions prevent us from doing so. In that case, please consider what additional dependencies it will pull in, when loosening existing restrictions. Also consider possible alternatives, like code restructurings or extracting shared code into dedicated packages for minimal impact on dependent projects.

    1.4.3 - Getting Started Locally

    Running Gardener locally

    This document will walk you through running Gardener on your local machine for development purposes. If you encounter difficulties, please open an issue so that we can make this process easier.

    Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as both garden and seed cluster (please refer to the architecture overview) for simplicity.

    The Gardener components, however, will be run as regular processes on your machine (hence, no container images are being built).

    Architecture Diagram

    Prerequisites

    • Make sure that you have followed the Local Setup guide up until the Get the sources step.

    • Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least 4 CPUs and 4Gi memory; see here how to configure the resources for Docker for Mac).

      Please note that 4 CPU / 4Gi memory might not be enough for more than one Shoot cluster, i.e., you might need to increase these values if you want to run additional Shoots. If you plan on following the optional steps to create a second seed cluster, the required resources will be more - at least 10 CPUs and 16Gi memory.

      Additionally, please configure at least 120Gi of disk size for the Docker daemon.

      Tip: With docker system df and docker system prune -a you can cleanup unused data.

    • Make sure that you increase the maximum number of open files on your host:

      • On Mac, run sudo launchctl limit maxfiles 65536 200000

      • On Linux, extend the /etc/security/limits.conf file with

        * hard nofile 97816
        * soft nofile 97816
        

        and reload the terminal.

    Setting up the KinD cluster (garden and seed)

    make kind-up KIND_ENV=local
    

    This command sets up a new KinD cluster named gardener-local and stores the kubeconfig in the ./example/gardener-local/kind/kubeconfig file.

    It might be helpful to copy this file to $HOME/.kube/config since you will need to target this KinD cluster multiple times. Alternatively, make sure to set your KUBECONFIG environment variable to ./example/gardener-local/kind/kubeconfig for all future steps via export KUBECONFIG=example/gardener-local/kind/kubeconfig.

    All following steps assume that you are using this kubeconfig.

    Additionally, this command also deploys a local container registry to the cluster as well as a few registry mirrors, that are set up as a pull-through cache for all upstream registries Gardener uses by default. This is done to speed up image pulls across local clusters. The local registry can be accessed as localhost:5001 for pushing and pulling. The storage directories of the registries are mounted to the host machine under dev/local-registry. With this, mirrored images don’t have to be pulled again after recreating the cluster.

    The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy support (the default kindnet CNI doesn’t provide NetworkPolicy support). Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.

    Setting up Gardener

    make dev-setup                                                                # preparing the environment (without webhooks for now)
    kubectl wait --for=condition=ready pod -l run=etcd -n garden --timeout 2m     # wait for etcd to be ready
    make start-apiserver                                                          # starting gardener-apiserver
    

    In a new terminal pane, run

    kubectl wait --for=condition=available apiservice v1beta1.core.gardener.cloud # wait for gardener-apiserver to be ready
    make start-admission-controller                                               # starting gardener-admission-controller
    

    In a new terminal pane, run

    make dev-setup DEV_SETUP_WITH_WEBHOOKS=true                                   # preparing the environment with webhooks
    make start-controller-manager                                                 # starting gardener-controller-manager
    

    (Optional): In a new terminal pane, run

    make start-scheduler                                                          # starting gardener-scheduler
    

    In a new terminal pane, run

    make register-local-env                                                       # registering the local environment (CloudProfile, Seed, etc.)
    make start-gardenlet SEED_NAME=local                                          # starting gardenlet
    

    In a new terminal pane, run

    make start-extension-provider-local                                           # starting gardener-extension-provider-local
    

    ℹ️ The provider-local is started with elevated privileges since it needs to manipulate your /etc/hosts file to enable you accessing the created shoot clusters from your local machine, see this for more details.

    Creating a Shoot cluster

    You can wait for the Seed to be ready by running

    kubectl wait --for=condition=gardenletready --for=condition=extensionsready --for=condition=bootstrapped seed local --timeout=5m
    

    Alternatively, you can run kubectl get seed local and wait for the STATUS to indicate readiness:

    NAME    STATUS   PROVIDER   REGION   AGE     VERSION       K8S VERSION
    local   Ready    local      local    4m42s   vX.Y.Z-dev    v1.21.1
    

    In order to create a first shoot cluster, just run

    kubectl apply -f example/provider-local/shoot.yaml
    

    You can wait for the Shoot to be ready by running

    kubectl wait --for=condition=apiserveravailable --for=condition=controlplanehealthy --for=condition=everynodeready --for=condition=systemcomponentshealthy shoot local -n garden-local --timeout=10m
    

    Alternatively, you can run kubectl -n garden-local get shoot local and wait for the LAST OPERATION to reach 100%:

    NAME    CLOUDPROFILE   PROVIDER   REGION   K8S VERSION   HIBERNATION   LAST OPERATION            STATUS    AGE
    local   local          local      local    1.21.0        Awake         Create Processing (43%)   healthy   94s
    

    (Optional): You could also execute a simple e2e test (creating and deleting a shoot) by running

    make test-e2e-local-simple KUBECONFIG="$PWD/example/gardener-local/kind/kubeconfig"
    

    When the shoot got successfully created you can access it as follows:

    kubectl -n garden-local get secret local.kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > /tmp/kubeconfig-shoot-local.yaml
    kubectl --kubeconfig=/tmp/kubeconfig-shoot-local.yaml get nodes
    

    (Optional): Setting up a second seed cluster

    There are cases where you would want to create a second seed cluster in your local setup. For example, if you want to test the control plane migration feature. The following steps describe how to do that.

    Add a new IP address on your loopback device which will be necessary for the new KinD cluster that you will create. On Mac, the default loopback device is lo0.

    sudo ip addr add 127.0.0.2 dev lo0                                     # adding 127.0.0.2 ip to the loopback interface
    

    Next, setup the second KinD cluster:

    make kind2-up KIND_ENV=local
    

    This command sets up a new KinD cluster named gardener-local2 and stores its kubeconfig in the ./example/gardener-local/kind2/kubeconfig file. You will need this file when starting the provider-local extension controller for the second seed cluster.

    make register-kind2-env                                           # registering the local2 seed
    make start-gardenlet SEED_NAME=local2                             # starting gardenlet for the local2 seed
    

    In a new terminal pane, run

    export KUBECONFIG=./example/gardener-local/kind2/kubeconfig       # setting KUBECONFIG to point to second kind cluster
    make start-extension-provider-local \
      WEBHOOK_SERVER_PORT=9444 \
      WEBHOOK_CERT_DIR=/tmp/gardener-extension-provider-local2 \
      SERVICE_HOST_IP=127.0.0.2 \
      METRICS_BIND_ADDRESS=:8082 \
      HEALTH_BIND_ADDRESS=:8083                                       # starting gardener-extension-provider-local
    

    If you want to perform a control plane migration you can follow the steps outlined here to migrate the shoot cluster to the second seed you just created.

    Deleting the Shoot cluster

    ./hack/usage/delete shoot local garden-local
    

    (Optional): Tear down the second seed cluster

    make tear-down-kind2-env
    make kind2-down
    

    Tear down the Gardener environment

    make tear-down-local-env
    make kind-down
    

    Further reading

    This setup makes use of the local provider extension. You can read more about it in this document.

    1.4.4 - Kubernetes Clients

    Kubernetes Clients in Gardener

    This document aims at providing a general developer guideline on different aspects of using Kubernetes clients in a large-scale distributed system and project like Gardener. The points included here are not meant to be consulted as absolute rules, but rather as general rules of thumb, that allow developers to get a better feeling about certain gotchas and caveats. It should be updated with lessons learned from maintaining the project and running Gardener in production.

    Prerequisites:

    Please familiarize yourself with the following basic Kubernetes API concepts first, if you’re new to Kubernetes. A good understanding of these basics will help you better comprehend the following document.

    Client Types: Client-Go, Generated, Controller-Runtime

    For historical reasons, you will find different kinds of Kubernetes clients in Gardener:

    Client-Go Clients

    client-go is the default/official client for talking to the Kubernetes API in Golang. It features so called “client sets” for all built-in Kubernetes API groups and versions (e.g. v1 (aka core/v1), apps/v1, etc.). client-go clients are generated from the built-in API types using client-gen and are composed of interfaces for every known API GroupVersionKind. A typical client-go usage looks like this:

    var (
      ctx        context.Context
      c          kubernetes.Interface // "k8s.io/client-go/kubernetes"
      deployment *appsv1.Deployment   // "k8s.io/api/apps/v1"
    )
    
    updatedDeployment, err := c.AppsV1().Deployments("default").Update(ctx, deployment, metav1.UpdateOptions{})
    

    Important characteristics of client-go clients:

    • clients are specific to a given API GroupVersionKind, i.e., clients are hard-coded to corresponding API-paths (don’t need to use the discovery API to map GVK to a REST endpoint path).
    • client’s don’t modify the passed in-memory object (e.g. deployment in the above example). Instead, they return a new in-memory object.
      This means, controllers have to continue working with the new in-memory object or overwrite the shared object to not lose any state updates.

    Generated Client Sets for Gardener APIs

    Gardener’s APIs extend the Kubernetes API by registering an extension API server (in the garden cluster) and CustomResourceDefinitions (on Seed clusters), meaning that the Kubernetes API will expose additional REST endpoints to manage Gardener resources in addition to the built-in API resources. In order to talk to these extended APIs in our controllers and components, client-gen is used to generate client-go-style clients to pkg/client/{core,extensions,seedmanagement,...}.

    Usage of these clients is equivalent to client-go clients, and the same characteristics apply. For example:

    var (
      ctx   context.Context
      c     gardencoreclientset.Interface // "github.com/gardener/gardener/pkg/client/core/clientset/versioned"
      shoot *gardencorev1beta1.Shoot      // "github.com/gardener/gardener/pkg/apis/core/v1beta1"
    )
    
    updatedShoot, err := c.CoreV1beta1().Shoots("garden-my-project").Update(ctx, shoot, metav1.UpdateOptions{})
    

    Controller-Runtime Clients

    controller-runtime is a Kubernetes community project (kubebuilder subproject) for building controllers and operators for custom resources. Therefore, it features a generic client, that follows a different approach and does not rely on generated client sets. Instead, the client can be used for managing any Kubernetes resources (built-in or custom) homogeneously. For example:

    var (
      ctx        context.Context
      c          client.Client            // "sigs.k8s.io/controller-runtime/pkg/client"
      deployment *appsv1.Deployment       // "k8s.io/api/apps/v1"
      shoot      *gardencorev1beta1.Shoot // "github.com/gardener/gardener/pkg/apis/core/v1beta1"
    )
    
    err := c.Update(ctx, deployment)
    // or
    err = c.Update(ctx, shoot)
    

    A brief introduction to controller-runtime and its basic constructs can be found here.

    Important characteristics of controller-runtime clients:

    • The client functions take a generic client.Object or client.ObjectList value. These interfaces are implemented by all Golang types, that represent Kubernetes API objects or lists respectively which can be interacted with via usual API requests. [1]
    • The client first consults a runtime.Scheme (configured during client creation) for recognizing the object’s GroupVersionKind (this happens on the client-side only).
      A runtime.Scheme is basically a registry for Golang API types, defaulting and conversion functions. Schemes are usually provided per GroupVersion (see this example for apps/v1) and can be combined to one single scheme for further usage (example). In controller-runtime clients, schemes are used only for mapping a typed API object to its GroupVersionKind.
    • It then consults a meta.RESTMapper (also configured during client creation) for mapping the GroupVersionKind to a RESTMapping, which contains the GroupVersionResource and Scope (namespaced or cluster-scoped). From these values, the client can unambiguously determine the REST endpoint path of the corresponding API resource. For instance: appsv1.DeploymentList is available at /apis/apps/v1/deployments or /apis/apps/v1/namespaces/<namespace>/deployments respectively.
      • There are different RESTMapper implementations, but generally they are talking to the API server’s discovery API for retrieving RESTMappings for all API resources known to the API server (either built-in, registered via API extension or CustomResourceDefinitions).
      • The default implementation of controller-runtime (which Gardener uses as well), is the dynamic RESTMapper. It caches discovery results (i.e. RESTMappings) in-memory and only re-discovers resources from the API server, when a client tries to use an unknown GroupVersionKind, i.e., when it encounters a No{Kind,Resource}MatchError.
    • The client writes back results from the API server into the passed in-memory object.
      • This means, that controllers don’t have to worry about copying back the results and should just continue to work on the given in-memory object.
      • This is a nice and flexible pattern and helper functions should try to follow it wherever applicable. Meaning, if possible accept an object param, pass it down to clients and keep working on the same in-memory object instead of creating a new one in your helper function.
      • The benefit is, that you don’t lose updates to the API object and always have the last-known state in memory. Therefore, you don’t have to read it again, e.g., for getting the current resourceVersion when working with optimistic locking, and thus minimize the chances for running into conflicts.
      • However, controllers must not use the same in-memory object concurrently in multiple goroutines. For example, decoding results from the API server in multiple goroutines into the same maps (e.g., labels, annotations) will cause panics because of “concurrent map writes”. Also, reading from an in-memory API object in one goroutine while decoding into it in another goroutine will yield non-atomic reads, meaning data might be corrupt and represent a non-valid/non-existing API object.
      • Therefore, if you need to use the same in-memory object in multiple goroutines concurrently (e.g., shared state), remember to leverage proper synchronization techniques like channels, mutexes, atomic.Value and/or copy the object prior to use. The average controller however, will not need to share in-memory API objects between goroutines, and it’s typically an indicator that the controller’s design should be improved.
    • The client decoder erases the object’s TypeMeta (apiVersion and kind fields) after retrieval from the API server, see kubernetes/kubernetes#80609, kubernetes-sigs/controller-runtime#1517. Unstructured and metadata-only requests objects are an exception to this because the contained TypeMeta is the only way to identify the object’s type. Because of this behavior, obj.GetObjectKind().GroupVersionKind() is likely to return an empty GroupVersionKind. I.e., you must not rely on TypeMeta being set or GetObjectKind() to return something usable.
      If you need to identify an object’s GroupVersionKind, use a scheme and its ObjectKinds function instead (or the helper function apiutil.GVKForObject). This is not specific to controller-runtime clients and applies to client-go clients as well.

    [1] Other lower level, config or internal API types (e.g., such as AdmissionReview) don’t implement client.Object. However, you also can’t interact with such objects via the Kubernetes API and thus also not via a client, so this can be disregarded at this point.

    Metadata-Only Clients

    Additionally, controller-runtime clients can be used to easily retrieve metadata-only objects or lists. This is useful for efficiently checking if at least one object of a given kind exists, or retrieving metadata of an object, if one is not interested in the rest (e.g., spec/status).
    The Accept header sent to the API server then contains application/json;as=PartialObjectMetadataList;g=meta.k8s.io;v=v1, which makes the API server only return metadata of the retrieved object(s). This saves network traffic and cpu/memory load on the API server and client side. If the client fully lists all objects of a given kind including their spec/status, the resulting list can be quite large and easily exceed the controllers available memory. That’s why it’s important to carefully check, if a full list is actually needed or if metadata-only list can be used instead.

    For example:

    var (
      ctx       context.Context
      c         client.Client                         // "sigs.k8s.io/controller-runtime/pkg/client"
      shootList = &metav1.PartialObjectMetadataList{} // "k8s.io/apimachinery/pkg/apis/meta/v1"
    )
    shootList.SetGroupVersionKind(gardencorev1beta1.SchemeGroupVersion.WithKind("ShootList"))
    
    if err := c.List(ctx, shootList, client.InNamespace("garden-my-project"), client.Limit(1)); err != nil {
      return err
    }
    
    if len(shootList.Items) > 0 {
      // project has at least one shoot
    } else {
      // project doesn't have any shoots
    }
    

    Gardener’s Client Collection, ClientMaps

    The Gardener codebase has a collection of clients (kubernetes.Interface), which can return all the above mentioned client types. Additionally, it contains helpers for rendering and applying helm charts (ChartRender, ChartApplier) and retrieving the API server’s version (Version).
    Client sets are managed by so called ClientMaps, which are a form of registry for all client set for a given type of cluster, i.e., Garden, Seed and Shoot. ClientMaps manage the whole lifecycle of clients: they take care of creating them if they don’t exist already, running their caches, refreshing their cached server version and invalidating them when they are no longer needed.

    var (
      ctx   context.Context
      cm    clientmap.ClientMap // "github.com/gardener/gardener/pkg/client/kubernetes/clientmap"
      shoot *gardencorev1beta1.Shoot
    )
    
    cs, err := cm.GetClient(ctx, keys.ForShoot(shoot)) // kubernetes.Interface
    if err != nil {
      return err
    }
    
    c := cs.Client() // client.Client
    

    The client collection mainly exist for historical reasons (there used to be a lot of code using the client-go style clients). However, Gardener is in the process of moving more towards controller-runtime and only using their clients, as they provide many benefits and are much easier to use. Also, gardener/gardener#4251 aims at refactoring our controller and admission components to native controller-runtime components.

    ⚠️ Please always prefer controller-runtime clients over other clients when writing new code or refactoring existing code.

    Cache Types: Informers, Listers, Controller-Runtime Caches

    Similar to the different types of client(set)s, there are also different kinds of Kubernetes client caches. However, all of them are based on the same concept: Informers. An Informer is a watch-based cache implementation, meaning it opens watch connections to the API server and continuously updates cached objects based on the received watch events (ADDED, MODIFIED, DELETED). Informers offer to add indices to the cache for efficient object lookup (e.g., by name or labels) and to add EventHandlers for the watch events. The latter is used by controllers to fill queues with objects that should be reconciled on watch events.

    Informers are used in and created via several higher-level constructs:

    SharedInformerFactories, Listers

    The generated clients (built-in as well as extended) feature a SharedInformerFactory for every API group, which can be used to create and retrieve Informers for all GroupVersionKinds. Similarly, it can be used to retrieve Listers, that allow getting and listing objects from the Informer’s cache. However, both of these constructs are only used for historical reasons, and we are in the process of migrating away from them in favor of cached controller-runtime clients (see gardener/gardener#2414, gardener/gardener#2822). Thus, they are described only briefly here.

    Important characteristics of Listers:

    • Objects read from Informers and Listers can always be slightly out-out-date (i.e., stale) because the client has to first observe changes to API objects via watch events (which can intermittently lag behind by a second or even more).
    • Thus, don’t make any decisions based on data read from Listers if the consequences of deciding wrongfully based on stale state might be catastrophic (e.g. leaking infrastructure resources). In such cases, read directly from the API server via a client instead.
    • Objects retrieved from Informers or Listers are pointers to the cached objects, so they must not be modified without copying them first, otherwise the objects in the cache are also modified.

    Controller-Runtime Caches

    controller-runtime features a cache implementation that can be used equivalently as their clients. In fact, it implements a subset of the client.Client interface containing the Get and List functions. Under the hood, a cache.Cache dynamically creates Informers (i.e., opens watches) for every object GroupVersionKind that is being retrieved from it.

    Note, that the underlying Informers of a controller-runtime cache (cache.Cache) and the ones of a SharedInformerFactory (client-go) are not related in any way. Both create Informers and watch objects on the API server individually. This means, that if you read the same object from different cache implementations, you may receive different versions of the object because the watch connections of the individual Informers are not synced.

    ⚠️ Because of this, controllers/reconcilers should get the object from the same cache in the reconcile loop, where the EventHandler was also added to set up the controller. For example, if a SharedInformerFactory is used for setting up the controller then read the object in the reconciler from the Lister instead of from a cached controller-runtime client.

    By default, the client.Client created by a controller-runtime Manager is a DelegatingClient. It delegates Get and List calls to a Cache and all other calls to a client, that talks directly to the API server. Exceptions are requests with *unstructured.Unstructured objects and object kinds that were configured to be excluded from the cache in the DelegatingClient.

    ℹ️ kubernetes.Interface.Client() returns a DelegatingClient that uses the cache returned from kubernetes.Interface.Cache() under the hood. This means, all Client() usages need to be ready for cached clients and should be able to cater with stale cache reads.

    Important characteristics of cached controller-runtime clients:

    • Like for Listers, objects read from a controller-runtime cache can always be slightly out of date. Hence, don’t base any important decisions on data read from the cache (see above).
    • In contrast to Listers, controller-runtime caches fill the passed in-memory object with the state of the object in the cache (i.e., they perform something like a “deep copy into”). This means that objects read from a controller-runtime cache can safely be modified without unintended side effects.
    • Reading from a controller-runtime cache or a cached controller-runtime client implicitly starts a watch for the given object kind under the hood. This has important consequences:
      • Reading a given object kind from the cache for the first time can take up to a few seconds depending on size and amount of objects as well as API server latency. This is because the cache has to do a full list operation and wait for an initial watch sync before returning results.
      • ⚠️ Controllers need appropriate RBAC permissions for the object kinds they retrieve via cached clients (i.e., list and watch).
      • ⚠️ By default, watches started by a controller-runtime cache are cluster-scoped, meaning it watches and caches objects across all namespaces. Thus, be careful which objects to read from the cache as it might significantly increase the controller’s memory footprint.
    • There is no interaction with the cache on writing calls (Create, Update, Patch and Delete), see below.

    Uncached objects, filtered caches, APIReaders:

    In order to allow more granular control over which object kinds should be cached and which calls should bypass the cache, controller-runtime offers a few mechanisms to further tweak the client/cache behavior:

    • When creating a DelegatingClient, certain object kinds can be configured to always be read directly from the API instead of from the cache. Note that this does not prevent starting a new Informer when retrieving them directly from the cache.
    • Watches can be restricted to a given (set of) namespace(s) by using cache.MultiNamespacedCacheBuilder or setting cache.Options.Namespace.
    • Watches can be filtered (e.g., by label) per object kind by configuring cache.Options.SelectorsByObject on creation of the cache.
    • Retrieving metadata-only objects or lists from a cache results in a metadata-only watch/cache for that object kind.
    • The APIReader can be used to always talk directly to the API server for a given Get or List call (use with care and only as a last resort!).

    To Cache or Not to Cache

    Although watch-based caches are an important factor for the immense scalability of Kubernetes, it definitely comes at a price (mainly in terms of memory consumption). Thus, developers need to be careful when introducing new API calls and caching new object kinds. Here are some general guidelines on choosing whether to read from a cache or not:

    • Always try to use the cache wherever possible and make your controller able to tolerate stale reads.
      • Leverage optimistic locking: use deterministic naming for objects you create (this is what the Deployment controller does [2]).
      • Leverage optimistic locking / concurrency control of the API server: send updates/patches with the last-known resourceVersion from the cache (see below). This will make the request fail, if there were concurrent updates to the object (conflict error), which indicates that we have operated on stale data and might have made wrong decisions. In this case, let the controller handle the error with exponential backoff. This will make the controller eventually consistent.
      • Track the actions you took, e.g., when creating objects with generateName (this is what the ReplicaSet controller does [3]). The actions can be tracked in memory and repeated if the expected watch events don’t occur after a given amount of time.
      • Always try to write controllers with the assumption that data will only be eventually correct and can be slightly out of date (even if read directly from the API server!).
      • If there is already some other code that needs a cache (e.g., a controller watch), reuse it instead of doing extra direct reads.
      • Don’t read an object again if you just sent a write request. Write requests (Create, Update, Patch and Delete) don’t interact with the cache. Hence, use the current state that the API server returned (filled into the passed in-memory object), which is basically a “free direct read”, instead of reading the object again from a cache, because this will probably set back the object to an older resourceVersion.
    • If you are concerned about the impact of the resulting cache, try to minimize that by using filtered or metadata-only watches.
    • If watching and caching an object type is not feasible, for example because there will be a lot of updates, and you are only interested in the object every ~5m, or because it will blow up the controllers memory footprint, fallback to a direct read. This can either be done by disabling caching the object type generally or doing a single request via an APIReader. In any case, please bear in mind that every direct API call results in a quorum read from etcd, which can be costly in a heavily-utilized cluster and impose significant scalability limits. Thus, always try to minimize the impact of direct calls by filtering results by namespace or labels, limiting the number of results and/or using metadata-only calls.

    [2] The Deployment controller uses the pattern <deployment-name>-<podtemplate-hash> for naming ReplicaSets. This means, the name of a ReplicaSet it tries to create/update/delete at any given time is deterministically calculated based on the Deployment object. By this, it is insusceptible to stale reads from its ReplicaSets cache.

    [3] In simple terms, the ReplicaSet controller tracks its CREATE pod actions as follows: when creating new Pods, it increases a counter of expected ADDED watch events for the corresponding ReplicaSet. As soon as such events arrive, it decreases the counter accordingly. It only creates new Pods for a given ReplicaSet, once all expected events occurred (counter is back to zero) or a timeout occurred. This way, it prevents creating more Pods than desired because of stale cache reads and makes the controller eventually consistent.

    Conflicts, Concurrency Control and Optimistic Locking

    Every Kubernetes API object contains the metadata.resourceVersion field, which identifies an object’s version in the backing data store, i.e., etcd. Every write to an object in etcd results in a newer resourceVersion. This field is mainly used for concurrency control on the API server in an optimistic locking fashion, but also for efficient resumption of interrupted watch connections.

    Optimistic locking in the Kubernetes API sense means that when a client wants to update an API object then it includes the object’s resourceVersion in the request to indicate the object’s version the modifications are based on. If the resourceVersion in etcd has not changed in the meantime, the update request is accepted by the API server and the updated object is written to etcd. If the resourceVersion sent by the client does not match the one of the object stored in etcd, there were concurrent modifications to the object. Consequently, the request is rejected with a conflict error (status code 409, API reason Conflict), for example:

    {
      "kind": "Status",
      "apiVersion": "v1",
      "metadata": {},
      "status": "Failure",
      "message": "Operation cannot be fulfilled on configmaps \"foo\": the object has been modified; please apply your changes to the latest version and try again",
      "reason": "Conflict",
      "details": {
        "name": "foo",
        "kind": "configmaps"
      },
      "code": 409
    }
    

    This concurrency control is an important mechanism in Kubernetes as there are typically multiple clients acting on API objects at the same time (humans, different controllers, etc.). If a client receives a conflict error, it should read the object’s latest version from the API server, make the modifications based on the newest changes and retry the update. The reasoning behind this is that a client might choose to make different decisions based on the concurrent changes made by other actors compared to the outdated version that it operated on.

    Important points about concurrency control and conflicts:

    • The resourceVersion field carries a string value and clients must not assume numeric values (the type and structure of versions depend on the backing data store). This means clients may compare resourceVersion values to detect whether objects were changed. But they must not compare resourceVersions to figure out which one is newer/older, i.e., no greater/less-than comparisons are allowed.
    • By default, update calls (e.g. via client-go and controller-runtime clients) use optimistic locking as the passed in-memory usually object contains the latest resourceVersion known to the controller which is then also sent to the API server.
    • API servers can also choose to accept update calls without optimistic locking (i.e., without a resourceVersion in the object’s metadata) for any given resource. However, sending update requests without optimistic locking is strongly discouraged as doing so overwrites the entire object discarding any concurrent changes made to it.
    • On the other side, patch requests can always be executed either with or without optimistic locking, by (not) including the resourceVersion in the patched object’s metadata. Sending patch requests without optimistic locking might be safe and even desirable as a patch typically updates only a specific section of the object. However, there are also situations where patching without optimistic locking is not safe (see below).

    Don’t Retry on Conflict

    Similar to how a human would typically handle a conflict error, there are helper functions implementing RetryOnConflict-semantics, i.e., try an update call, then re-read the object if a conflict occurs, apply the modification again and retry the update. However, controllers should generally not use RetryOnConflict-semantics. Instead, controllers should abort their current reconciliation run and let the queue handle the conflict error with exponential backoff. The reasoning behind this is, that a conflict error indicates that the controller has operated on stale data and might have made wrong decisions earlier on in the reconciliation. When using a helper function that implements RetryOnConflict-semantics, the controller doesn’t check which fields were changed and doesn’t revise its previous decisions accordingly. Instead, retrying on conflict basically just ignores any conflict error and blindly applies the modification.

    To properly solve the conflict situation, controllers should immediately return with the error from the update call. This will cause retries with exponential backoff so that the cache has a chance to observe the latest changes to the object. In a later run, the controller will then make correct decisions based on the newest version of the object, not run into conflict errors and will then be able to successfully reconcile the object. This way, the controller becomes eventually consistent.

    The other way to solve the situation is to modify objects without optimistic locking in order to avoid running into a conflict in the first place (only if this is safe). This can be a preferable solution for controllers with long-running reconciliations (which is actually an anti-pattern but quite unavoidable in some of Gardener’s controllers). Aborting the entire reconciliation run is rather undesirable in such cases as it will add a lot of unnecessary waiting time for end users and overhead in terms of compute and network usage.

    However, in any case retrying on conflict is probably not the right option to solve the situation (there are some correct use cases for it, though, they are very rare). Hence, don’t retry on conflict.

    To Lock or Not to Lock

    As explained before, conflicts are actually important and prevent clients from doing wrongful concurrent updates. This means, conflicts are not something we generally want to avoid or ignore. However, in many cases controllers are exclusive owners of the fields they want to update and thus it might be safe to run without optimistic locking.

    For example, the gardenlet is the exclusive owner of the spec section of the Extension resources it creates on behalf of a Shoot (e.g., the Infrastructure resource for creating VPC, etc.). Meaning, it knows the exact desired state and no other actor is supposed to update the Infrastructure’s spec fields. When the gardenlet now updates the Infrastructures spec section as part of the Shoot reconciliation, it can simply issue a PATCH request that only updates the spec and runs without optimistic locking. If another controller concurrently updated the object in the meantime (e.g., the status section), the resourceVersion got changed which would cause a conflict error if running with optimistic locking. However, concurrent status updates would not change the gardenlet’s mind on the desired spec of the Infrastructure resource as it is determined only by looking at the Shoot’s specification. If the spec section was changed concurrently, it’s still fine to overwrite it because the gardenlet should reconcile the spec back to its desired state.

    Generally speaking, if a controller is the exclusive owner of a given set of fields and they are independent of concurrent changes to other fields in that object, it can patch these fields without optimistic locking. This might ignore concurrent changes to other fields or blindly overwrite changes to the same fields, but this is fine if the mentioned conditions apply. Obviously, this applies only to patch requests that modify only a specific set of fields but not to update requests that replace the entire object.

    In such cases, it’s even desirable to run without optimistic locking as it will be more performant and save retries. If certain requests are made with high frequency and have a good chance of causing conflicts, retries because of optimistic locking can cause a lot of additional network traffic in a large-scale Gardener installation.

    Updates, Patches, Server-side Apply

    There are different ways of modifying Kubernetes API objects. The following snippet demonstrates how to do a given modification with the most frequently used options using a controller-runtime client:

    var (
      ctx   context.Context
      c     client.Client
      shoot *gardencorev1beta1.Shoot
    )
    
    // update
    shoot.Spec.Kubernetes.Version = "1.22"
    err := c.Update(ctx, shoot)
    
    // json merge patch
    patch := client.MergeFrom(shoot.DeepCopy())
    shoot.Spec.Kubernetes.Version = "1.22"
    err = c.Patch(ctx, shoot, patch)
    
    // strategic merge patch
    patch = client.StrategicMergeFrom(shoot.DeepCopy())
    shoot.Spec.Kubernetes.Version = "1.22"
    err = c.Patch(ctx, shoot, patch)
    

    Important characteristics of the shown request types:

    • Update requests always send the entire object to the API server and update all fields accordingly. By default, optimistic locking is used (resourceVersion is included).
    • Both patch types run without optimistic locking by default. However, it can be enabled explicitly if needed:
      // json merge patch + optimistic locking
      patch := client.MergeFromWithOptions(shoot.DeepCopy(), client.MergeFromWithOptimisticLock{})
      // ...
      
      // strategic merge patch + optimistic locking
      patch = client.StrategicMergeFrom(shoot.DeepCopy(), client.MergeFromWithOptimisticLock{})
      // ...
      
    • Patch requests only contain the changes made to the in-memory object between the copy passed to client.*MergeFrom and the object passed to Client.Patch(). The diff is calculated on the client-side based on the in-memory objects only. This means, if in the meantime some fields were changed on the API server to a different value than the one on the client-side, the fields will not be changed back as long as they are not changed on the client-side as well (there will be no diff in memory).
    • Thus, if you want to ensure a given state using patch requests, always read the object first before patching it, as there will be no diff otherwise, meaning the patch will be empty. Also see gardener/gardener#4057 and comments in gardener/gardener#4027.
    • Also, always send updates and patch requests even if your controller hasn’t made any changes to the current state on the API server. I.e., don’t make any optimization for preventing empty patches or no-op updates. There might be mutating webhooks in the system that will modify the object and that rely on update/patch requests being sent (even if they are no-op). Gardener’s extension concept makes heavy use of mutating webhooks, so it’s important to keep this in mind.
    • JSON merge patches always replace lists as a whole and don’t merge them. Keep this in mind when operating on lists with merge patch requests. If the controller is the exclusive owner of the entire list, it’s safe to run without optimistic locking. Though, if you want to prevent overwriting concurrent changes to the list or its items made by other actors (e.g., additions/removals to the metadata.finalizers list), enable optimistic locking.
    • Strategic merge patches are able to make more granular modifications to lists and their elements without replacing the entire list. It uses Golang struct tags of the API types to determine which and how lists should be merged. See this document or the strategic merge patch documentation for more in-depth explanations and comparison with JSON merge patches. With this, controllers might be able to issue patch requests for individual list items without optimistic locking, even if they are not exclusive owners of the entire list. Remember to check the patchStrategy and patchMergeKey struct tags of the fields you want to modify before blindly adding patch requests without optimistic locking.
    • Strategic merge patches are only supported by built-in Kubernetes resources and custom resources served by Extension API servers. Strategic merge patches are not supported by custom resources defined by CustomResourceDefinitions (see this comparison). In that case, fallback to JSON merge patches.
    • Server-side Apply is yet another mechanism to modify API objects, which is supported by all API resources (in newer Kubernetes versions). However, it has a few problems and more caveats preventing us from using it in Gardener at the time of writing. See gardener/gardener#4122 for more details.

    Generally speaking, patches are often the better option compared to update requests because they can save network traffic, encoding/decoding effort and avoid conflicts under the presented conditions. If choosing a patch type, consider which type is supported by the resource you’re modifying and what will happen in case of a conflict. Consider whether your modification is safe to run without optimistic locking. However, there is no simple rule of thumb on which patch type to choose.

    On Helper Functions

    Here is a note on some helper functions, that should be avoided and why:

    controllerutil.CreateOrUpdate does a basic get, mutate and create or update call chain, which is often used in controllers. We should avoid using this helper function in Gardener, because it is likely to cause conflicts for cached clients and doesn’t send no-op requests if nothing was changed, which can cause problems because of the heavy use of webhooks in Gardener extensions (see above). That’s why usage of this function was completely replaced in gardener/gardener#4227 and similar PRs.

    controllerutil.CreateOrPatch is similar to CreateOrUpdate but does a patch request instead of an update request. It has the same drawback as CreateOrUpdate regarding no-op updates. Also, controllers can’t use optimistic locking or strategic merge patches when using CreateOrPatch. Another reason for avoiding use of this function is, that it also implicitly patches the status section if it was changed, which is confusing for others reading the code. To accomplish this, the func does some back and forth conversion, comparison and checks, which are unnecessary in most of our cases and simply wasted CPU cycles and complexity we want to avoid.

    There were some Try{Update,UpdateStatus,Patch,PatchStatus} helper functions in Gardener that were already removed by gardener/gardener#4378 but are still used in some extension code at the time of writing. The reason for eliminating these functions is that they implement RetryOnConflict-semantics. Meaning, they first get the object, mutate it, then try to update and retry if a conflict error occurs. As explained above, retrying on conflict is a controller anti-pattern and should be avoided in almost every situation. The other problem with these functions is that they read the object first from the API server (always do a direct call), although in most cases we already have a recent version of the object at hand. So, using this function generally does unnecessary API calls and therefore causes unwanted compute and network load.

    For the reasons explained above, there are similar helper functions that accomplish similar things but address the mentioned drawbacks: controllerutils.{GetAndCreateOrMergePatch,GetAndCreateOrStrategicMergePatch}. These can be safely used as replacements for the aforementioned helper funcs. If they are not fitting for your use case, for example because you need to use optimistic locking, just do the appropriate calls in the controller directly.

    Further Resources

    These resources are only partially related to the topics covered in this doc, but might still be interesting for developer seeking a deeper understanding of Kubernetes API machinery, architecture and foundational concepts.

    1.4.5 - Local Setup

    Overview

    Conceptually, all Gardener components are designed to run as a Pod inside a Kubernetes cluster. The Gardener API server extends the Kubernetes API via the user-aggregated API server concepts. However, if you want to develop it, you may want to work locally with the Gardener without building a Docker image and deploying it to a cluster each and every time. That means that the Gardener runs outside a Kubernetes cluster which requires providing a Kubeconfig in your local filesystem and point the Gardener to it when starting it (see below).

    Further details can be found in

    1. Principles of Kubernetes, and its components
    2. Kubernetes Development Guide
    3. Architecture of Gardener

    This guide is split into three main parts:

    Limitations of the local development setup

    You can run Gardener (API server, controller manager, scheduler, gardenlet) against any local Kubernetes cluster, however, your seed and shoot clusters must be deployed to a cloud provider. Currently, it is not possible to run Gardener entirely isolated from any cloud provider. This means that to be able create Shoot clusters you need to register an external Seed cluster (e.g., one created in AWS).

    Preparing the Setup

    [macOS only] Installing homebrew

    The copy-paste instructions in this guide are designed for macOS and use the package manager Homebrew.

    On macOS run

    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
    

    Installing git

    We use git as VCS which you need to install. On macOS run

    brew install git
    

    For other OS, please check the Git installation documentation.

    Installing Go

    Install the latest version of Go. On macOS run

    brew install go
    

    For other OS, please check Go installation documentation.

    Installing kubectl

    Install kubectl. Please make sure that the version of kubectl is at least v1.11.x. On macOS run

    brew install kubernetes-cli
    

    For other OS, please check the kubectl installation documentation.

    Installing helm

    You also need the Helm CLI. On macOS run

    brew install helm
    

    For other OS please check the Helm installation documentation.

    Installing openvpn

    We use OpenVPN to establish network connectivity from the control plane running in the Seed cluster to the Shoot’s worker nodes running in private networks. To harden the security we need to generate another secret to encrypt the network traffic (details). Please install the openvpn binary. On macOS run

    brew install openvpn
    export PATH=$(brew --prefix openvpn)/sbin:$PATH
    

    For other OS, please check the OpenVPN downloads page.

    Installing Docker

    You need to have docker installed and running. On macOS run

    brew install --cask docker
    

    For other OS please check the docker installation documentation.

    Installing iproute2

    iproute2 provides a collection of utilities for network administration and configuration. On macOS run

    brew install iproute2mac
    

    Installing jq

    brew install jq
    

    Installing GNU Parallel

    GNU Parallel is a shell tool for executing jobs in parallel, used by the code generation scripts (make generate). On macOS run

    brew install parallel
    

    [macOS only] Install GNU core utilities

    When running on macOS, install the GNU core utilities and friends:

    brew install coreutils gnu-sed gnu-tar grep
    

    This will create symbolic links for the GNU utilities with g prefix in /usr/local/bin, e.g., gsed or gbase64. To allow using them without the g prefix please put /usr/local/opt/coreutils/libexec/gnubin etc. at the beginning of your PATH environment variable, e.g., export PATH=/usr/local/opt/coreutils/libexec/gnubin:$PATH (brew will print out instructions for each installed formula).

    export PATH=/usr/local/opt/coreutils/libexec/gnubin:$PATH
    export PATH=/usr/local/opt/gnu-sed/libexec/gnubin:$PATH
    export PATH=/usr/local/opt/gnu-tar/libexec/gnubin:$PATH
    export PATH=/usr/local/opt/grep/libexec/gnubin:$PATH
    

    [Windows only] WSL2

    Apart from Linux distributions and macOS, the local gardener setup can also run on the Windows Subsystem for Linux 2.

    While WSL1, plain docker for windows and various Linux distributions and local Kubernetes environments may be supported, this setup was verified with:

    The Gardener repository and all the above-mentioned tools (git, golang, kubectl, …) should be installed in your WSL2 distro, according to the distribution-specific Linux installation instructions.

    Start Gardener locally

    Get the sources

    Clone the repository from GitHub into your $GOPATH.

    mkdir -p $GOPATH/src/github.com/gardener
    cd $GOPATH/src/github.com/gardener
    git clone git@github.com:gardener/gardener.git
    cd gardener
    

    Note: Gardener is using Go modules and cloning the repository into $GOPATH is not a hard requirement. However it is still recommended to clone into $GOPATH because k8s.io/code-generator does not work yet outside of $GOPATH - kubernetes/kubernetes#86753.

    Start the Gardener

    ℹ️ In the following guide, you have to define the configuration (CloudProfiles, SecretBindings, Seeds, etc.) manually for the infrastructure environment you want to develop against. Additionally, you have to register the respective Gardener extensions manually. If you are rather looking for a quick start guide to develop entirely locally on your machine (no real cloud provider or infrastructure involved) then you should rather follow this guide.

    Start a local kubernetes cluster

    For the development of Gardener you need a Kubernetes API server on which you can register Gardener’s own Extension API Server as APIService. This cluster doesn’t need any worker nodes to run pods, though, therefore, you can use the “nodeless Garden cluster setup” residing in hack/local-garden. This will start all minimally required components of a Kubernetes cluster (etcd, kube-apiserver, kube-controller-manager) and an etcd Instance for the gardener-apiserver as Docker containers. This is the easiest way to get your Gardener development setup up and running.

    Using the nodeless cluster setup

    Use the provided Makefile rules to start your local Garden:

    make local-garden-up
    [...]
    Starting gardener-dev kube-etcd cluster..!
    Starting gardener-dev kube-apiserver..!
    Starting gardener-dev kube-controller-manager..!
    Starting gardener-dev gardener-etcd cluster..!
    namespace/garden created
    clusterrole.rbac.authorization.k8s.io/gardener.cloud:admin created
    clusterrolebinding.rbac.authorization.k8s.io/front-proxy-client created
    [...]
    

    ℹ️ [Optional] If you want to develop the SeedAuthorization feature then you have to run make ACTIVATE_SEEDAUTHORIZER=true local-garden-up. However, please note that this forces you to start the gardener-admission-controller via make start-admission-controller.

    To tear down the local Garden cluster and remove the Docker containers, simply run:

    make local-garden-down
    
    Alternative: Using a local kubernetes cluster

    Instead of starting a kubernetes API server and etcd as docker containers, you can also opt for running a local kubernetes cluster, provided by e.g. minikube, kind or docker desktop.

    Note: Gardener requires self-contained kubeconfig files because of a security issue. You can configure your minikube to create self-contained kubeconfig files via:

    minikube config set embed-certs true
    

    or when starting the local cluster

    minikube start --embed-certs
    
    Alternative: Using a remote kubernetes cluster

    For some testing scenarios, you may want to use a remote cluster instead of a local one as your Garden cluster. To do this, you can use the “remote Garden cluster setup” residing in hack/remote-garden. This will start an etcd instance for the gardener-apiserver as a Docker container, and open tunnels for accessing local gardener components from the remote cluster.

    To avoid mistakes, the remote cluster must have a garden namespace labeled with gardener.cloud/purpose=remote-garden. You must create the garden namespace and label it manually before running make remote-garden-up as described below.

    Use the provided Makefile rules to bootstrap your remote Garden:

    export KUBECONFIG=<path to kubeconfig>
    make remote-garden-up
    [...]
    # Start gardener etcd used to store gardener resources (e.g., seeds, shoots)
    Starting gardener-dev-remote gardener-etcd cluster!
    [...]
    # Open tunnels for accessing local gardener components from the remote cluster
    [...]
    

    To close the tunnels and remove the locally-running Docker containers, run:

    make remote-garden-down
    

    Note: The minimum K8S version of the remote cluster that can be used as Garden cluster is 1.19.x.

    ℹ️ [Optional] If you want to use the remote Garden cluster setup with the SeedAuthorization feature you have to adapt the kube-apiserver process of your remote Garden cluster. To do this, perform the following steps after running make remote-garden-up:

    • Create an authorization webhook configuration file using the IP of the garden/quic-server pod running in your remote Garden cluster and port 10444 that tunnels to your locally running gardener-admission-controller process.

      apiVersion: v1
      kind: Config
      current-context: seedauthorizer
      clusters:
      - name: gardener-admission-controller
        cluster:
          insecure-skip-tls-verify: true
          server: https://<quic-server-pod-ip>:10444/webhooks/auth/seed
      users:
      - name: kube-apiserver
        user: {}
      contexts:
      - name: seedauthorizer
        context:
          cluster: gardener-admission-controller
          user: kube-apiserver
      
    • Change or add the following command line parameters to your kube-apiserver process:

      • --authorization-mode=<...>,Webhook
      • --authorization-webhook-config-file=<path to config file>
      • --authorization-webhook-cache-authorized-ttl=0
      • --authorization-webhook-cache-unauthorized-ttl=0
    • Delete the cluster role and rolebinding gardener.cloud:system:seeds from your remote Garden cluster.

    If your remote Garden cluster is a Gardener shoot, and you can access the seed on which this shoot is scheduled, you can automate the above steps by running the enable-seed-authorizer script and passing the kubeconfig of the seed cluster and the shoot namespace as parameters:

    hack/local-development/remote-garden/enable-seed-authorizer <seed kubeconfig> <namespace>
    

    Note: The configuration changes introduced by this script result in a working SeedAuthorization feature only on shoots for which the ReversedVPN feature is not enabled. If the corresponding feature gate is enabled in gardenlet, add the annotation alpha.featuregates.shoot.gardener.cloud/reversed-vpn: 'false' to the remote Garden shoot to disable it for that particular shoot.

    To prevent Gardener from reconciling the shoot and overwriting your changes, add the annotation shoot.gardener.cloud/ignore: 'true' to the remote Garden shoot. Note that this annotation takes effect only if it is enabled via the constollers.shoot.respectSyncPeriodOverwrite: true option in the gardenlet configuration.

    To disable the seed authorizer again, run the same script with -d as a third parameter:

    hack/local-development/remote-garden/enable-seed-authorizer <seed kubeconfig> <namespace> -d
    

    If the seed authorizer is enabled, you also have to start the gardener-admission-controller via make start-admission-controller.

    ⚠️ In the remote garden setup all Gardener components run with administrative permissions, i.e., there is no fine-grained access control via RBAC (as opposed to productive installations of Gardener).

    Prepare the Gardener

    Now, that you have started your local cluster, we can go ahead and register the Gardener API Server. Just point your KUBECONFIG environment variable to the cluster you created in the previous step and run:

    make dev-setup
    [...]
    namespace/garden created
    namespace/garden-dev created
    deployment.apps/etcd created
    service/etcd created
    service/gardener-apiserver created
    service/gardener-admission-controller created
    endpoints/gardener-apiserver created
    endpoints/gardener-admission-controller created
    apiservice.apiregistration.k8s.io/v1alpha1.core.gardener.cloud created
    apiservice.apiregistration.k8s.io/v1beta1.core.gardener.cloud created
    apiservice.apiregistration.k8s.io/v1alpha1.seedmanagement.gardener.cloud created
    apiservice.apiregistration.k8s.io/v1alpha1.settings.gardener.cloud created
    

    ℹ️ [Optional] If you want to enable logging, in the Gardenlet configuration add:

    logging:
      enabled: true
    

    The Gardener exposes the API servers of Shoot clusters via Kubernetes services of type LoadBalancer. In order to establish stable endpoints (robust against changes of the load balancer address), it creates DNS records pointing to these load balancer addresses. They are used internally and by all cluster components to communicate. You need to have control over a domain (or subdomain) for which these records will be created. Please provide an internal domain secret (see this for an example) which contains credentials with the proper privileges. Further information can be found here.

    kubectl apply -f example/10-secret-internal-domain-unmanaged.yaml
    secret/internal-domain-unmanaged created
    

    Run the Gardener

    Next, run the Gardener API Server, the Gardener Controller Manager (optionally), the Gardener Scheduler (optionally), and the Gardenlet in different terminal windows/panes using rules in the Makefile.

    make start-apiserver
    [...]
    I0306 15:23:51.044421   74536 plugins.go:84] Registered admission plugin "ResourceReferenceManager"
    I0306 15:23:51.044523   74536 plugins.go:84] Registered admission plugin "DeletionConfirmation"
    [...]
    I0306 15:23:51.626836   74536 secure_serving.go:116] Serving securely on [::]:8443
    [...]
    

    (Optional) Now you are ready to launch the Gardener Controller Manager.

    make start-controller-manager
    time="2019-03-06T15:24:17+02:00" level=info msg="Starting Gardener controller manager..."
    time="2019-03-06T15:24:17+02:00" level=info msg="Feature Gates: "
    time="2019-03-06T15:24:17+02:00" level=info msg="Starting HTTP server on 0.0.0.0:2718"
    time="2019-03-06T15:24:17+02:00" level=info msg="Acquired leadership, starting controllers."
    time="2019-03-06T15:24:18+02:00" level=info msg="Starting HTTPS server on 0.0.0.0:2719"
    time="2019-03-06T15:24:18+02:00" level=info msg="Found internal domain secret internal-domain-unmanaged for domain nip.io."
    time="2019-03-06T15:24:18+02:00" level=info msg="Successfully bootstrapped the Garden cluster."
    time="2019-03-06T15:24:18+02:00" level=info msg="Gardener controller manager (version 1.0.0-dev) initialized."
    time="2019-03-06T15:24:18+02:00" level=info msg="ControllerRegistration controller initialized."
    time="2019-03-06T15:24:18+02:00" level=info msg="SecretBinding controller initialized."
    time="2019-03-06T15:24:18+02:00" level=info msg="Project controller initialized."
    time="2019-03-06T15:24:18+02:00" level=info msg="Quota controller initialized."
    time="2019-03-06T15:24:18+02:00" level=info msg="CloudProfile controller initialized."
    [...]
    

    (Optional) Now you are ready to launch the Gardener Scheduler.

    make start-scheduler
    time="2019-05-02T16:31:50+02:00" level=info msg="Starting Gardener scheduler ..."
    time="2019-05-02T16:31:50+02:00" level=info msg="Starting HTTP server on 0.0.0.0:10251"
    time="2019-05-02T16:31:50+02:00" level=info msg="Acquired leadership, starting scheduler."
    time="2019-05-02T16:31:50+02:00" level=info msg="Gardener scheduler initialized (with Strategy: SameRegion)"
    time="2019-05-02T16:31:50+02:00" level=info msg="Scheduler controller initialized."
    [...]
    

    The Gardener should now be ready to operate on Shoot resources. You can use

    kubectl get shoots
    No resources found.
    

    to operate against your local running Gardener API Server.

    Note: It may take several seconds until the Gardener API server has been started and is available. No resources found is the expected result of our initial development setup.

    Create a Shoot

    The steps below describe the general process of creating a Shoot. Have in mind that the steps do not provide full example manifests. The reader needs to check the provider documentation and adapt the manifests accordingly.

    1. Copy the example manifests

    The next steps require modifications of the example manifests. These modifications are part of local setup and should not be git push-ed. To do not interfere with git, let’s copy the example manifests to dev/ which is ignored by git.

    cp example/*.yaml dev/
    

    2. Create a Project

    Every Shoot is associated with a Project. Check the corresponding example manifests dev/00-namespace-garden-dev.yaml and dev/05-project-dev.yaml. Adapt them and create them.

    kubectl apply -f dev/00-namespace-garden-dev.yaml
    kubectl apply -f dev/05-project-dev.yaml
    

    Make sure that the Project is successfully reconciled:

    $ kubectl get project dev
    NAME   NAMESPACE    STATUS   OWNER                  CREATOR            AGE
    dev    garden-dev   Ready    john.doe@example.com   kubernetes-admin   6s
    

    3. Create a CloudProfile

    The CloudProfile resource is provider specific and describes the underlying cloud provider (available machine types, regions, machine images, etc.). Check the corresponding example manifest dev/30-cloudprofile.yaml. Check also the documentation and example manifests of the provider extension. Adapt dev/30-cloudprofile.yaml and apply it.

    kubectl apply -f dev/30-cloudprofile.yaml
    

    4. Install necessary Gardener Extensions

    The Known Extension Implementations section contains a list of available extension implementations. You need to create a ControllerRegistration and ControllerDeployment for

    • at least one infrastructure provider
    • a dns provider (if the DNS for the Seed is not disabled)
    • at least one operating system extension
    • at least one network plugin extension

    As a convention, the example ControllerRegistration manifest (containing also the necessary ControllerDeployment) for an extension is located under example/controller-registration.yaml in the corresponding repository (for example for AWS the ControllerRegistration can be found here). An example creation for provider-aws (make sure to replace <version> with the newest released version tag):

    kubectl apply -f https://raw.githubusercontent.com/gardener/gardener-extension-provider-aws/<version>/example/controller-registration.yaml
    

    Instead of updating extensions manually you can use Gardener Extensions Manager to install and update extension controllers. This is especially useful if you want to keep and maintain your development setup for a longer time. Also, please refer to this document for further information about how extensions are registered in case you want to use other versions than the latest releases.

    5. Register a Seed

    Shoot controlplanes run in seed clusters, so we need to create our first Seed now.

    Check the corresponding example manifest dev/40-secret-seed.yaml and dev/50-seed.yaml. Update dev/40-secret-seed.yaml with base64 encoded kubeconfig of the cluster that will be used as Seed (the scope of the permissions should be identical to the kubeconfig that the Gardenlet creates during bootstrapping - for now, cluster-admin privileges are recommended).

    kubectl apply -f dev/40-secret-seed.yaml
    

    Adapt dev/50-seed.yaml - adjust .spec.secretRef to refer the newly created Secret, adjust .spec.provider with the Seed cluster provider and revise the other fields.

    kubectl apply -f dev/50-seed.yaml
    

    6. Start Gardenlet

    Once the Seed is created, start the Gardenlet to reconcile it. The make start-gardenlet command will automatically configure the local Gardenlet process to use the Seed and its kubeconfig. If you have multiple Seeds, you have to specify which to use by setting the SEED_NAME environment variable like in make start-gardenlet SEED_NAME=my-first-seed.

    make start-gardenlet
    time="2019-11-06T15:24:17+02:00" level=info msg="Starting Gardenlet..."
    time="2019-11-06T15:24:17+02:00" level=info msg="Feature Gates: HVPA=true, Logging=true"
    time="2019-11-06T15:24:17+02:00" level=info msg="Acquired leadership, starting controllers."
    time="2019-11-06T15:24:18+02:00" level=info msg="Found internal domain secret internal-domain-unmanaged for domain nip.io."
    time="2019-11-06T15:24:18+02:00" level=info msg="Gardenlet (version 1.0.0-dev) initialized."
    time="2019-11-06T15:24:18+02:00" level=info msg="ControllerInstallation controller initialized."
    time="2019-11-06T15:24:18+02:00" level=info msg="Shoot controller initialized."
    time="2019-11-06T15:24:18+02:00" level=info msg="Seed controller initialized."
    [...]
    

    The Gardenlet will now reconcile the Seed. Check the progess from time to time until it’s Ready:

    kubectl get seed
    NAME       STATUS    PROVIDER    REGION      AGE    VERSION       K8S VERSION
    seed-aws   Ready     aws         eu-west-1   4m     v1.11.0-dev   v1.18.12
    

    7. Create a Shoot

    A Shoot requires a SecretBinding. The SecretBinding refers to a Secret that contains the cloud provider credentials. The Secret data keys are provider specific and you need to check the documentation of the provider to find out which data keys are expected (for example for AWS the related documentation can be found here). Adapt dev/70-secret-provider.yaml and dev/80-secretbinding.yaml and apply them.

    kubectl apply -f dev/70-secret-provider.yaml
    kubectl apply -f dev/80-secretbinding.yaml
    

    After the SecretBinding creation, you are ready to proceed with the Shoot creation. You need to check the documentation of the provider to find out the expected configuration (for example for AWS the related documentation and example Shoot manifest can be found here). Adapt dev/90-shoot.yaml and apply it.

    To make sure that a specific Seed cluster will be chosen or to skip the scheduling (the sheduling requires Gardener Scheduler to be running), specify the .spec.seedName field (see here).

    kubectl apply -f dev/90-shoot.yaml
    

    Watch the progress of the operation and make sure that the Shoot will be successfully created.

    watch kubectl get shoot --all-namespaces
    

    1.4.6 - Log Parsers

    How to create log parser for container into fluent-bit

    If our log message is parsed correctly, it has to be showed in Grafana like this:

      {"log":"OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io","pid":"1","severity":"INFO","source":"controller.go:107"}
    

    Otherwise it will looks like this:

    {
      "log":"{
      \"level\":\"info\",\"ts\":\"2020-06-01T11:23:26.679Z\",\"logger\":\"gardener-resource-manager.health-reconciler\",\"msg\":\"Finished ManagedResource health checks\",\"object\":\"garden/provider-aws-dsm9r\"
      }\n"
      }
    }
    

    Lets make a custom parser now

    • First of all we need to know how does the log for the specific container look like (for example lets take a log from the alertmanager : level=info ts=2019-01-28T12:33:49.362015626Z caller=main.go:175 build_context="(go=go1.11.2, user=root@4ecc17c53d26, date=20181109-15:40:48))

    • We can see that this log contains 4 subfields(severity=info, timestamp=2019-01-28T12:33:49.362015626Z, source=main.go:175 and the actual message). So we have to write a regex which matches this log in 4 groups(We can use https://regex101.com/ like helping tool). So for this purpose our regex looks like this:

    ^level=(?<severity>\w+)\s+ts=(?<time>\d{4}-\d{2}-\d{2}[Tt].*[zZ])\s+caller=(?<source>[^\s]*+)\s+(?<log>.*)
    
    %Y-%m-%dT%H:%M:%S.%L
    
    • It’s a time to apply our new regex into fluent-bit configuration. Go to fluent-bit-configmap.yaml and create new filter using the following template:
    [FILTER]
            Name                parser
            Match               kubernetes.<< pod-name >>*<< container-name >>*
            Key_Name            log
            Parser              << parser-name >>
            Reserve_Data        True
    
    EXAMPLE
    [FILTER]
            Name                parser
            Match               kubernetes.alertmanager*alertmanager*
            Key_Name            log
            Parser              alermanagerParser
            Reserve_Data        True
    
    • Now lets check if there is already exists parser with such a regex and time format that we need. if not, let`s create one:
    [PARSER]
            Name        << parser-name >>
            Format      regex
            Regex       << regex >>
            Time_Key    time
            Time_Format << time-format >>
    
    EXAMPLE
    [PARSER]
            Name        alermanagerParser
            Format      regex
            Regex       ^level=(?<severity>\w+)\s+ts=(?<time>\d{4}-\d{2}-\d{2}[Tt].*[zZ])\s+caller=(?<source>[^\s]*+)\s+(?<log>.*)
            Time_Key    time
            Time_Format %Y-%m-%dT%H:%M:%S.%L
    
    Follow your development setup to validate that parsers are working correctly.
    

    1.4.7 - Logging

    Logging in Gardener Components

    This document aims at providing a general developer guideline on different aspects of logging practices and conventions used in the Gardener codebase. It contains mostly Gardener-specific points and references other existing and commonly accepted logging guidelines for general advice. Developers and reviewers should consult this guide when writing, refactoring and reviewing Gardener code. If parts are unclear or new learnings arise, this guide should be adapted accordingly.

    Logging Libraries / Implementations

    Historically, Gardener components have been using logrus. There is a global logrus logger (logger.Logger) that is initialized by components on startup and used across the codebase. In most places, it is used as a printf-style logger and only in some instances we make use of logrus’ structured logging functionality.

    In the process of migrating our components to native controller-runtime components (see gardener/gardener#4251), we also want to make use of controller-runtime’s built-in mechanisms for streamlined logging. controller-runtime uses logr, a simple structured logging interface, for library-internal logging and logging in controllers.

    logr itself is only an interface and doesn’t provide an implementation out of the box. Instead, it needs to be backed by a logging implementation like zapr. Code that uses the logr interface is thereby not tied to a specific logging implementation and makes the implementation easily exchangeable. controller-runtime already provides a set of helpers for constructing zapr loggers, i.e., logr loggers backed by zap, which is a popular logging library in the go community. Hence, we are migrating our component logging from logrus to logr (backed by zap) as part of gardener/gardener#4251.

    ⚠️ logger.Logger (logrus logger) is deprecated in Gardener and shall not be used in new code – use logr loggers when writing new code! (also see Migration from logrus to logr)

    ℹ️ Don’t use zap loggers directly, always use the logr interface in order to avoid tight coupling to a specific logging implementation.

    gardener-apiserver differs from the other components as it is based on the apiserver library and therefore uses klog – just like kube-apiserver. As gardener-apiserver writes (almost) no logs in our coding (outside the apiserver library), there is currently no plan for switching the logging implementation. Hence, the following sections focus on logging in the controller and admission components only.

    logcheck Tool

    To ensure a smooth migration to logr and make logging in Gardener components more consistent, the logcheck tool was added. It enforces (parts of) this guideline and detects programmer-level errors early on in order to prevent bugs. Please check out the tool’s documentation for a detailed description.

    Structured Logging

    Similar to efforts in the Kubernetes project, we want to migrate our component logs to structured logging. As motivated above, we will use the logr interface instead of klog though.

    You can read more about the motivation behind structured logging in logr’s background and FAQ (also see this blog post by Dave Cheney). Also, make sure to check out controller-runtime’s logging guideline with specifics for projects using the library. The following sections will focus on the most important takeaways from those guidelines and give general instructions on how to apply them to Gardener and its controller-runtime components. Note: some parts in this guideline differ slightly from controller-runtime’s document.

    TL;DR of Structured Logging

    ❌ stop using printf-style logging:

    var logger *logrus.Logger
    logger.Infof("Scaling deployment %s/%s to %d replicas", deployment.Namespace, deployment.Name, replicaCount)
    

    ✅ instead, write static log messages and enrich them with additional structured information in form of key-value pairs:

    var logger logr.Logger
    logger.Info("Scaling deployment", "deployment", client.ObjectKeyFromObject(deployment), "replicas", replicaCount)
    

    Log Configuration

    Gardener components can be configured to either log in json (default) or text format: json format is supposed to be used in production, while text format might be nicer for development.

    # json
    {"level":"info","ts":"2021-12-16T08:32:21.059+0100","msg":"Hello botanist","garden":"eden"}
    
    # text
    2021-12-16T08:32:21.059+0100    INFO    Hello botanist  {"garden": "eden"}
    

    Components can be set to one of the following log levels (with increasing verbosity): error, info (default), debug.

    ℹ️ Note: some Gardener components don’t feature a configurable log level and format yet. In this case, they log at info in json format. We might add configuration options via command line flags that can be used in all components in the future though (see gardener/gardener#5191).

    Log Levels

    logr uses V-levels (numbered log levels), higher V-level means higher verbosity. V-levels are relative (in contrast to klog’s absolute V-levels), i.e., V(1) creates a logger, that is one level more verbose than its parent logger.

    In Gardener components, the mentioned log levels in the component config (error, info, debug) map to the zap levels with the same names (see here). Hence, our loggers follow the same mapping from numerical logr levels to named zap levels like described in zapr, i.e.:

    • component config specifies debug ➡️ both V(0) and V(1) are enabled
    • component config specifies info ➡️ V(0) is enabled, V(1) will not be shown
    • component config specifies error ➡️ neither V(0) nor V(1) will be shown
    • Error() logs will always be shown

    This mapping applies to the components’ root loggers (the ones that are not “derived” from any other logger; constructed on component startup). If you derive a new logger with e.g. V(1), the mapping will shift by one. For example, V(0) will then log at zap’s debug level.

    There is no warning level (see Dave Cheney’s post). If there is an error condition (e.g., unexpected error received from a called function), the error should either be handled or logged at error if it is neither handled nor returned. If you have an error value at hand that doesn’t represent an actual error condition, but you still want to log it as an informational message, log it at info level with key err.

    We might consider to make use of a broader range of log levels in the future when introducing more logs and common command line flags for our components (comparable to --v of Kubernetes components). For now, we stick to the mentioned two log levels like controller-runtime: info (V(0)) and debug (V(1)).

    Logging in Controllers

    Named Loggers

    Controllers should use named loggers that include their name, e.g.:

    controllerLogger := rootLogger.WithName("controller").WithName("shoot")
    controllerLogger.Info("Deploying kube-apiserver")
    

    results in

    2021-12-16T09:27:56.550+0100    INFO    controller.shoot    Deploying kube-apiserver
    

    Logger names are hierarchical. You can make use of it, where controllers are composed of multiple “subcontrollers”, e.g., controller.shoot.hibernation or controller.shoot.maintenance.

    Using the global logger logf.Log directly is discouraged and should be rather exceptional because it makes correlating logs with code harder. Preferably, all parts of the code should use some named logger.

    Reconciler Loggers

    In your Reconcile function, retrieve a logger from the given context.Context. It inherits from the controller’s logger (i.e., is already named) and is preconfigured with name and namespace values for the reconciliation request:

    func (r *reconciler) Reconcile(ctx context.Context, request reconcile.Request) (reconcile.Result, error) {
      log := logf.FromContext(ctx)
      log.Info("Reconciling Shoot")
      // ...
      return reconcile.Result{}, nil
    }
    

    results in

    2021-12-16T09:35:59.099+0100    INFO    controller.shoot    Reconciling Shoot        {"name": "sunflower", "namespace": "garden-greenhouse"}
    

    The logger is injected by controller-runtime’s Controller implementation and our controllerutils.CreateWorker alike (if a logger is passed using controllerutils.WithLogger). The logger returned by logf.FromContext is never nil. If the context doesn’t carry a logger, it falls back to the global logger (logf.Log), which might discard logs if not configured, but is also never nil.

    ⚠️ Make sure that you don’t overwrite the name or namespace value keys for such loggers, otherwise you will lose information about the reconciled object.

    The controller implementation (controller-runtime / CreateWorker) itself takes care of logging the error returned by reconcilers. Hence, don’t log an error that you are returning. Generally, functions should not return an error, if they already logged it, because that means the error is already handled and not an error anymore. See Dave Cheney’s post for more on this.

    Messages

    • Log messages should be static. Don’t put variable content in there, i.e., no fmt.Sprintf or string concatenation (+). Use key-value pairs instead.
    • Log messages should be capitalized. Note: this contrasts with error messages, that should not be capitalized. However, both should not end with a punctuation mark.

    Keys and Values

    • Use WithValues instead of repeatedly adding key-value pairs for multiple log statements. WithValues creates a new logger from the parent, that carries the given key-value pairs. E.g., use it when acting on one object in multiple steps and logging something for each step:

      log := parentLog.WithValues("infrastructure", client.ObjectKeyFromObject(infrastrucutre))
      // ...
      log.Info("Creating Infrastructure")
      // ...
      log.Info("Waiting for Infrastructure to be reconciled")
      // ...
      

      Note: WithValues bypasses controller-runtime’s special zap encoder that nicely encodes ObjectKey/NamespacedName and runtime.Object values, see kubernetes-sigs/controller-runtime#1290. Thus, the end result might look different depending on the value and its Stringer implementation.

    • Use lowerCamelCase for keys. Don’t put spaces in keys, as it will make log processing with simple tools like jq harder.

    • Keys should be constant, human-readable, consistent across the codebase and naturally match parts of the log message, see logr guideline.

    • When logging object keys (name and namespace), use the object’s type as the log key and a client.ObjectKey/types.NamespacedName value as value, e.g.:

      var deployment *appsv1.Deployment
      log.Info("Creating Deployment", "deployment", client.ObjectKeyFromObject(deployment))
      

      which results in

      {"level":"info","ts":"2021-12-16T08:32:21.059+0100","msg":"Creating Deployment","deployment":{"name": "bar", "namespace": "foo"}}
      

      Earlier, we often used kutil.ObjectName() for logging object keys, which encodes them into a flat string like foo/bar. However, this flat string cannot be processed so easily by logging stacks (or jq) like a structured log. Hence, the use of kutil.ObjectName() for logging object keys is discouraged. Existing usages should be refactored to use client.ObjectKeyFromObject() instead.

    • There are cases where you don’t have the full object key or the object itself at hand, e.g., if an object references another object (in the same namespace) by name (think secretRef or similar). In such a cases, either construct the full object key including the implied namespace or log the object name under a key ending in Name, e.g.:

      var (
        // object to reconcile
        shoot *gardencorev1beta1.Shoot
        // retrieved via logf.FromContext, preconfigured by controller with namespace and name of reconciliation request
        log logr.Logger
      )
      
      // option a: full object key, manually constructed
      log.Info("Shoot uses SecretBinding", "secretBinding", client.ObjectKey{Namespace: shoot.Namespace, Name: shoot.Spec.SecretBindingName})
      // option b: only name under respective *Name log key
      log.Info("Shoot uses SecretBinding", "secretBindingName", shoot.Spec.SecretBindingName)
      

      Both options result in well-structured logs, that are easy to interpret and process:

      {"level":"info","ts":"2022-01-18T18:00:56.672+0100","msg":"Shoot uses SecretBinding","name":"my-shoot","namespace":"garden-project","secretBinding":{"namespace":"garden-project","name":"aws"}}
      {"level":"info","ts":"2022-01-18T18:00:56.673+0100","msg":"Shoot uses SecretBinding","name":"my-shoot","namespace":"garden-project","secretBindingName":"aws"}
      
    • When handling generic client.Object values (e.g. in helper funcs), use object as key.

    • When adding timestamps to key-value pairs, use time.Time values. By this, they will be encoded in the same format as the log entry’s timestamp.
      Don’t use metav1.Time values, as they will be encoded in a different format by their Stringer implementation. Pass <someTimestamp>.Time to loggers in case you have a metav1.Time value at hand.

    • Same applies to durations. Use time.Duration values instead of *metav1.Duration. Durations can be handled specially by zap just like timestamps.

    • Event recorders not only create Event objects but also log them. However, both Gardener’s manually instantiated event recorders and the ones that controller-runtime provides log to debug level and use generic formats, that are not very easy to interpret or process (no structured logs). Hence, don’t use event recorders as replacements for well-structured logs. If a controller records an event for a completed action or important information, it should probably log it as well, e.g.:

      log.Info("Creating ManagedSeed", "replica", r.GetObjectKey())
      a.recorder.Eventf(managedSeedSet, corev1.EventTypeNormal, EventCreatingManagedSeed, "Creating ManagedSeed %s", r.GetFullName())
      

    Logging in Test Code

    • If the tested production code requires a logger, you can pass logr.Discard() or logf.NullLogger{} in your test, which simply discards all logs.

    • logf.Log is safe to use in tests and will not cause a nil pointer deref, even if it’s not initialized via logf.SetLogger. It is initially set to a NullLogger by default, which means all logs are discarded, unless logf.SetLogger is called in the first 30 seconds of execution.

    • Pass zap.WriteTo(GinkgoWriter) in tests where you want to see the logs on test failure but not on success, for example:

      logf.SetLogger(logger.MustNewZapLogger(logger.DebugLevel, logger.FormatJSON, zap.WriteTo(GinkgoWriter)))
      log := logf.Log.WithName("test")
      

    Migration from logrus to logr

    These points might be helpful when refactoring existing code during the migration period:

    • For migrating an existing controller to logr:
      • Create a named logger (example).
      • Pass controllerutils.WithLogger to CreateWorker (example). This allows logf.FromContext to be used in reconcilers.
      • Use logf.FromContext in Reconcile to retrieve the logr logger and use it from there on (example).
      • Make sure to follow the other guidelines mentioned above as well (see Logging in Controllers).
    • Libraries might expect a different logging implementation than the component which uses it. E.g., a controller that already uses logr might want to use the flow package which still uses logrus. In such cases:
      • You can consider refactoring the library along with the component itself, if feasible.
      • It is acceptable for the migration period to use a logger derived from the respective global logger (logger.Logger or logf.Log) and pass it to the library. However, please add a TODO for cleaning it up later on, once the migration is completed. E.g.:
        // TODO: switch to logr once flow package is migrated
        err := shootFlow.Run(flow.Opts{
          Logger: logger.Logger.WithFields(logrus.Fields{"logger": "controller." + ControllerName, "name": shoot.Name, "namespace": shoot.Namespace})
        })
        

    1.4.8 - Monitoring Stack

    Extending the Monitoring Stack

    This document provides instructions to extend the Shoot cluster monitoring stack by integrating new scrape targets, alerts and dashboards.

    Please ensure that you have understood the basic principles of Prometheus and its ecosystem before you continue.

    ‼️ The purpose of the monitoring stack is to observe the behaviour of the control plane and the system components deployed by Gardener onto the worker nodes. Monitoring of custom workloads running in the cluster is out of scope.

    Overview

    Monitoring Architecture

    Each Shoot cluster comes with its own monitoring stack. The following components are deployed into the seed and shoot:

    In each Seed cluster there is a Prometheus in the garden namespace responsible for collecting metrics from the Seed kubelets and cAdvisors. These metrics are provided to each Shoot Prometheus via federation.

    The alerts for all Shoot clusters hosted on a Seed are routed to a central Alertmanger running in the garden namespace of the Seed. The purpose of this central alertmanager is to forward all important alerts to the operators of the Gardener setup.

    The Alertmanager in the Shoot namespace on the Seed is only responsible for forwarding alerts from its Shoot cluster to a cluster owner/cluster alert receiver via email. The Alertmanager is optional and the conditions for a deployment are already described here.

    Adding New Monitoring Targets

    After exploring the metrics which your component provides or adding new metrics, you should be aware which metrics are required to write the needed alerts and dashboards.

    Prometheus prefers a pull based metrics collection approach and therefore the targets to observe need to be defined upfront. The targets are defined in charts/seed-monitoring/charts/prometheus/templates/config.yaml. New scrape jobs can be added in the section scrape_configs. Detailed information how to configure scrape jobs and how to use the kubernetes service discovery are available in the Prometheus documentation.

    The job_name of a scrape job should be the name of the component e.g. kube-apiserver or vpn. The collection interval should be the default of 30s. You do not need to specify this in the configuration.

    Please do not ingest all metrics which are provided by a component. Rather collect only those metrics which are needed to define the alerts and dashboards (i.e. whitelist). This can be achieved by adding the following metric_relabel_configs statement to your scrape jobs (replace exampleComponent with component name).

        - job_name: example-component
          ...
          metric_relabel_configs:
    {{ include "prometheus.keep-metrics.metric-relabel-config" .Values.allowedMetrics.exampleComponent | indent 6 }}
    

    The whitelist for the metrics of your job can be maintained in charts/seed-monitoring/charts/prometheus/values.yaml in section allowedMetrics.exampleComponent (replace exampleComponent with component name). Check the following example:

    allowedMetrics:
      ...
      exampleComponent:
      * metrics_name_1
      * metrics_name_2
      ...
    

    Adding Alerts

    The alert definitons are located in charts/seed-monitoring/charts/prometheus/rules. There are two approaches for adding new alerts.

    1. Adding additional alerts for a component which already has a set of alerts. In this case you have to extend the existing rule file for the component.
    2. Adding alerts for a new component. In this case a new rule file with name scheme example-component.rules.yaml needs to be added.
    3. Add the new alert to alertInhibitionGraph.dot, add any required inhibition flows and render the new graph. To render the graph run:
    dot -Tpng ./content/alertInhibitionGraph.dot -o ./content/alertInhibitionGraph.png
    
    1. Create a test for the new alert. See Alert Tests.

    Example alert:

    groups:
    * name: example.rules
      rules:
      * alert: ExampleAlert
        expr: absent(up{job="exampleJob"} == 1)
        for: 20m
        labels:
          service: example
          severity: critical # How severe is the alert? (blocker|critical|info|warning)
          type: shoot # For which topology is the alert relevant? (seed|shoot)
          visibility: all # Who should receive the alerts? (all|operator|owner)
        annotations:
          description: A longer description of the example alert that should also explain the impact of the alert.
          summary: Short summary of an example alert.
    

    If the deployment of component is optional then the alert definitions needs to be added to charts/seed-monitoring/charts/prometheus/optional-rules instead. Furthermore the alerts for component need to be activatable in charts/seed-monitoring/charts/prometheus/values.yaml via rules.optional.example-component.enabled. The default should be true.

    Basic instruction how to define alert rules can be found in the Prometheus documentation.

    Routing tree

    The Alertmanager is grouping incoming alerts based on labels into buckets. Each bucket has its own configuration like alert receivers, initial delaying duration or resending frequency etc. You can find more information about Alertmanager routing in the Prometheus/Alertmanager documentation. The routing trees for the Alertmanagers deployed by Gardener are depicted below.

    Central Seed Alertmanager

    ∟ main route (all alerts for all shoots on the seed will enter)
      ∟ group by project and shoot name
        ∟ group by visibility "all" and "operator"
          ∟ group by severity "blocker", "critical", and "info" → route to Garden operators
          ∟ group by severity "warning" (dropped)
        ∟ group by visibility "owner" (dropped)
    

    Shoot Alertmanager

    ∟ main route (only alerts for one Shoot will enter)
      ∟ group by visibility "all" and "owner"
        ∟ group by severity "blocker", "critical", and "info" → route to cluster alert receiver
        ∟ group by severity "warning" (dropped, will change soon → route to cluster alert receiver)
      ∟ group by visibility "operator" (dropped)
    

    Alert Inhibition

    All alerts related to components running on the Shoot workers are inhibited in case of an issue with the vpn connection, because those components can’t be scraped anymore and Prometheus will fire alerts in consequence. The components running on the workers are probably healthy and the alerts are presumably false positives. The inhibition flow is shown in the figure below. If you add a new alert make sure to add it to the diagram.

    alertDiagram

    Alert Attributes

    Each alert rule definition has to contain the following annotations:

    • summary: A short description of the issue.
    • description: A detailed explanation of the issue with hints to the possible root causes and the impact assessment of the issue.

    In addtion each alert must contain the following labels:

    • type
      • shoot: Components running on the Shoot worker nodes in the kube-system namespace.
      • seed: Components running on the Seed in the Shoot namespace as part of/next to the control plane.
    • service
      • Name of the component (in lowercase) e.g. kube-apiserver, alertmanager or vpn.
    • severity
      • blocker: All issues which make the cluster entirely unusable e.g. KubeAPIServerDown or KubeSchedulerDown
      • critical: All issues which affect single functionalities/components but not affect the cluster in its core functionality e.g. VPNDown or KubeletDown.
      • info: All issues that do not affect the cluster or its core functionality, but if this component is down we cannot determine if a blocker alert is firing. (i.e. A component with an info level severity is a dependency for a component with a blocker severity)
      • warning: No current existing issue, rather a hint for situations which could lead to real issue in the close future e.g. HighLatencyApiServerToWorkers or ApiServerResponseSlow.

    Alert Tests

    To test the Prometheus alerts:

    make test-prometheus
    

    If you want to add alert tests:

    1. Create a new file in rules-tests in the form <alert-group-name>.rules.test.yaml or if the alerts are for an existing component with existing tests, simply add the tests to the appropriate files.

    2. Make sure that newly added tests succeed. See above.

    Adding Grafana Dashboards

    The dashboard definition files are located in charts/seed-monitoring/charts/grafana/dashboards. Every dashboard needs its own file.

    If you are adding a new component dashboard please also update the overview dashboard by adding a chart for its current up/down status and with a drill down option to the component dashboard.

    Dashboard Structure

    The dashboards should be structured in the following way. The assignment of the component dashboards to the categories should be handled via dashboard tags.

    • Kubernetes control plane components (Tag: control-plane)
      • All components which are part of the Kubernetes control plane e. g. Kube API Server, Kube Controller Manager, Kube Scheduler and Cloud Controller Manager
      • ETCD + Backup/Restore
      • Kubernetes Addon Manager
    • Node/Machine components (Tag: node/machine)
      • All metrics which are related to the behaviour/control of the Kubernetes nodes and kubelets
      • Machine-Controller-Manager + Cluster Autoscaler
    • Networking components (Tag: network)
      • CoreDNS, KubeProxy, Calico, VPN, Nginx Ingress
    • Addon components (Tag: addon)
      • Cert Broker
    • Monitoring components (Tag: monitoring)
    • Logging components (Tag: logging)

    Mandatory Charts for Component Dashboards

    For each new component, its corresponding dashboard should contain the following charts in the first row, before adding custom charts for the component in the subsequent rows.

    1. Pod up/down status up{job="example-component"}
    2. Pod/containers cpu utilization
    3. Pod/containers memorty consumption
    4. Pod/containers network i/o

    These information is provided by the cAdvisor metrics. These metrics are already integrated. Please check the other dashboards for detailed information on how to query.

    Chart Requirements

    Each chart needs to contain:

    • a meaningful name
    • a detailed description (for non trivial charts)
    • appropriate x/y axis descriptions
    • appropriate scaling levels for the x/y axis
    • proper units for the x/y axis
    Dashboard Parameters

    The following parameters should be added to all dashboards to ensure a homogeneous experience across all dashboards.

    Dashboards have to …

    • contain a title which refers to the component name(s)
    • contain a timezone statement which should be the browser time
    • contain tags which express where the component is running (seed or shoot) and to which category the component belong (see dashboard structure)
    • contain a version statement with a value of 1
    • be immutable

    Example dashboard configuration

    {
      "title": "example-component",
      "timezone": "utc",
      "tags": [
        "seed",
        "control-plane"
      ],
      "version": 1,
      "editable": "false"
    }
    

    Furthermore all dashboards should contain the following time options:

    {
      "time": {
        "from": "now-1h",
        "to": "now"
      },
      "timepicker": {
        "refresh_intervals": [
          "30s",
          "1m",
          "5m"
        ],
        "time_options": [
          "5m",
          "15m",
          "1h",
          "6h",
          "12h",
          "24h",
          "2d",
          "10d"
        ]
      }
    }
    

    1.4.9 - New Cloud Provider

    Adding Cloud Providers

    This document provides an overview of how to integrate a new cloud provider into Gardener. Each component that requires integration has a detailed description of how to integrate it and the steps required.

    Cloud Components

    Gardener is composed of 2 or more Kubernetes clusters:

    • Shoot: These are the end-user clusters, the regular Kubernetes clusters you have seen. They provide places for your workloads to run.
    • Seed: This is the “management” cluster. It manages the control planes of shoots by running them as native Kubernetes workloads.

    These two clusters can run in the same cloud provider, but they do not need to. For example, you could run your Seed in AWS, while having one shoot in Azure, two in Google, two in Alicloud, and three in Equinix Metal.

    The Seed cluster deploys and manages the Shoot clusters. Importantly, for this discussion, the etcd data store backing each Shoot runs as workloads inside the Seed. Thus, to use the above example, the clusters in Azure, Google, Alicloud and Equinix Metal will have their worker nodes and master nodes running in those clouds, but the etcd clusters backing them will run as separate deployments in the Seed Kubernetes cluster on AWS.

    This distinction becomes important when preparing the integration to a new cloud provider.

    Gardener Cloud Integration

    Gardener and its related components integrate with cloud providers at the following key lifecycle elements:

    • Create/destroy/get/list machines for the Shoot
    • Create/destroy/get/list infrastructure components for the Shoot, e.g. VPCs, subnets, routes, etc.
    • Backup/restore etcd for the Seed via writing files to and reading them from object storage

    Thus, the integrations you need for your cloud provider depend on whether you want to deploy Shoot clusters to the provider, Seed or both.

    • Shoot Only: machine lifecycle management, infrastructure.
    • Seed: etcd backup/restore

    Gardener API

    In addition to the requirements to integrate with the cloud provider, you also need to enable the core Gardener app to receive, validate and process requests to use that cloud provider.

    • Expose the cloud provider to the consumers of the Gardener API, so it can be told to use that cloud provider as an option
    • Validate that API as requests come in
    • Write cloud provider specific implementation (called “provider extension”)

    Cloud Provider API Requirements

    In order for a cloud provider to integrate with Gardener, the provider must have an API to perform machine lifecycle events, specifically:

    • Create a machine
    • Destroy a machine
    • Get information about a machine and its state
    • List machines

    In addition, if the Seed is to run on the given provider, it also must have an API to save files to block storage and retrieve them, for etcd backup/restore.

    The current integration with cloud providers is to add their API calls to Gardener and the Machine Controller Manager. As both Gardener and the Machine Controller Manager are written in go, the cloud provider should have a go SDK. However, if it has an API that is wrappable in go, e.g. a REST API, then you can use that to integrate.

    The Gardener team is working on bringing cloud provider integrations out-of-tree, making them pluggable, which should simplify the process and make it possible to use other SDKs.

    Summary

    To add a new cloud provider, you need some or all of the following. Each repository contains instructions on how to extend it to a new cloud provider.

    TypePurposeLocationDocumentation
    Seed or ShootMachine Lifecyclemachine-controller-managerMCM new cloud provider
    Seed onlyetcd backup/restoreetcd-backup-restoreIn process
    AllExtension implementationgardenerExtension controller

    1.4.10 - New Kubernetes Version

    Adding Support For A New Kubernetes Version

    This document describes the steps needed to perform in order to confidently add support for a new Kubernetes minor version.

    ⚠️ Typically, once a minor Kubernetes version vX.Y is supported by Gardener then all patch versions vX.Y.Z are also automatically supported without any required action. This is because patch versions do not introduce any new feature or API changes, so there is nothing that needs to be adapted in gardener/gardener code.

    The Kubernetes community release a new minor version roughly every 4 months. Please refer to the official documentation about their release cycles for any additional information.

    Shortly before a new release, an “umbrella” issue should be opened which is used to collect the required adaptations and to track the work items. For example, #5102 can be used as a template for the issue description.
    As you can see, the task of supporting a new Kubernetes version also includes the provider extensions maintained in the gardener GitHub organization and is not restricted to gardener/gardener only.

    Generally, the work items can be split into two groups: The first group contains Kubernetes release-independent tasks, the second group contains tasks specific to the changes in the given Kubernetes release.

    ℹ️ Upgrading the k8s.io/* and sigs.k8s.io/controller-runtime Golang dependencies is typically tracked and worked on separately (see e.g. #4772 or #5282).

    Deriving Release-Specific Tasks

    Most new minor Kubernetes releases incorporate API changes, deprecations or new features. The community announces them via their change logs. In order to derive the release-specific tasks, the respective change log for the new version vX.Y has to be read and understood (for example, this document for v1.24).

    As already mentioned, typical changes to watch out for are:

    • API version promotions or deprecations
    • Feature gate promotions or deprecations
    • CLI flag changes for Kubernetes components
    • New default values in resources
    • New available fields in resources
    • New features potentially relevant for the Gardener system
    • Changes of labels or annotations Gardener relies on

    Obviously, this requires a certain experience and understanding of the Gardener project so that all “relevant changes” can be identified. While reading the change log, add the tasks (along with the respective PR in kubernetes/kubernetes to the umbrella issue).

    ℹ️ Some of the changes might be specific to certain cloud providers. Pay attention to those as well and add related tasks to the issue.

    List Of Release-Independent Tasks

    The following paragraphs describe recurring tasks that need to be performed for each new release.

    Releasing A New hyperkube Image

    The gardener/hyperkube repository is used to release container images consisting of the kubectl and kubelet binaries.

    Run the .ci/check-and-release script to automatically build the image (make sure Docker is running!), push the images to the GCR (make sure gcloud is configured properly!) and publish the release on GitHub (make sure git is configured properly!).

    Adapting Gardener

    • Allow instantiation of a Kubernetes client for the new minor version and update the README.md:
      • See this example commit.
    • Maintain the Kubernetes feature gates used for validation of Shoot resources:
      • The feature gates are maintained in this file.
      • To maintain this list for new Kubernetes versions, run hack/compare-k8s-feature-gates.sh <old-version> <new-version> (e.g. hack/compare-k8s-feature-gates.sh v1.22 v1.23).
      • It will present 3 lists of feature gates: those added and those removed in <new-version> compared to <old-version> and feature gates that got locked to default in <new-version>.
      • Add all added feature gates to the map with <new-version> as AddedInVersion and no RemovedInVersion.
      • For any removed feature gates, add <new-version> as RemovedInVersion to the already existing feature gate in the map.
      • For feature gates locked to default, add <new-version> as LockedToDefaultInVersion to the already existing feature gate in the map.
      • See this example commit.
    • Maintain the Kubernetes kube-apiserver admission plugins used for validation of Shoot resources:
      • The admission plugins are maintained in this file.
      • To maintain this list for new Kubernetes versions, run hack/compare-k8s-admission-plugins.sh <old-version> <new-version> (e.g. hack/compare-k8s-admission-plugins.sh 1.24 1.25).
      • It will present 2 lists of admission plugins: those added and those removed in <new-version> compared to <old-version>.
      • Add all added admission plugins to the admissionPluginsVersionRanges map with <new-version> as AddedInVersion and no RemovedInVersion.
      • For any removed admission plugins, add <new-version> as RemovedInVersion to the already existing admission plugin in the map.
      • Flag any admission plugins that are required (plugins that must not be disabled in the Shoot spec) by setting the Required boolean variable to true for the admission plugin in the map.
      • Flag any admission plugins that are forbidden by setting the Forbidden boolean variable to true for the admission plugin in the map.
    • Maintain the ServiceAccount names for the controllers part of kube-controller-manager:
      • The names are maintained in this file.
      • To maintain this list for new Kubernetes versions, run hack/compare-k8s-controllers.sh <old-version> <new-version> (e.g. hack/compare-k8s-controllers.sh 1.22 1.23).
      • It will present 2 lists of controllers: those added and those removed in <new-version> compared to <old-version>.
      • Double check whether such ServiceAccount indeed appears in the kube-system namespace when creating a cluster with <new-version>. Note that it sometimes might be hidden behind a default-off feature gate. You can create a local cluster with the new version using the local provider.
      • If it appears, add all added controllers to the list based on the Kubernetes version (example).
      • For any removed controllers, add them only to the Kubernetes version if it is low enough.
    • Bump the used Kubernetes version for local Shoot and local e2e test.
      • See this example commit.

    Filing The Pull Request

    Work on all the tasks you have collected and validate them using the local provider. Execute the e2e tests and if everything looks good, then go ahead and file the PR (example PR). Generally, it is great if you add the PRs also to the umbrella issue so that they can be tracked more easily.

    Adapting Provider Extensions

    After the PR in gardener/gardener for the support of the new version has been merged, you can go ahead and work on the provider extensions.

    Actually, you can already start even if the PR is not yet merged and use the branch of your fork.

    • Revendor the github.com/gardener/gardener dependency in the extension and update the README.md.
    • Work on release-specific tasks related to this provider.

    Maintaining The cloud-controller-manager Images

    Some of the cloud providers are not yet using upstream cloud-controller-manager images. Instead, we build and maintain them ourselves:

    Until we switch to upstream images, you need to revendor the Kubernetes dependencies and release a new image. The required steps are as follows:

    • Checkout the legacy-cloud-provider branch of the respective repository
    • Bump the versions in the Dockerfile (example commit).
    • Update the VERSION to vX.Y.Z-dev where Z is the latest available Kubernetes patch version for the vX.Y minor version.
    • Update the k8s.io/* dependencies in the go.mod file to vX.Y.Z and run go mod vendor and go mod tidy (example commit).
    • Checkout a new release-vX.Y branch and release it (example)

    As you are already on it, it is great if you also bump the k8s.io/* dependencies for the last three minor releases as well. In this case, you need to checkout the release-vX.{Y-{1,2,3}} branches and only perform the last three steps (example branch, example commit).

    Now you need to update the new releases in the charts/images.yaml of the respective provider extension so that they are used (see this example commit for reference).

    Filing The Pull Request

    Again, work on all the tasks you have collected. This time, you cannot use the local provider for validation but should create real clusters on the various infrastructures. Typically, the following validations should be performed:

    • Create new clusters with versions < vX.Y
    • Create new clusters with version = vX.Y
    • Upgrade old clusters from version vX.{Y-1} to version vX.Y
    • Delete clusters with versions < vX.Y
    • Delete clusters with version = vX.Y

    If everything looks good, then go ahead and file the PR (example PR). Generally, it is again great if you add the PRs also to the umbrella issue so that they can be tracked more easily.

    1.4.11 - Priority Classes

    PriorityClasses in Gardener Clusters

    Gardener makes use of PriorityClasses to improve overall robustness of the system. In order to benefit from the full potential of PriorityClasses, gardenlet manages a set of well-known PriorityClasses with fine-granular priority values.

    All components of the system should use these well-known PriorityClasses instead of creating and using separate ones with arbitrary values, which would compromise the overall goal of using PriorityClasses in the first place. Gardenlet manages the well-known PriorityClasses listed in this document, so that third parties (e.g., Gardener extensions) can rely on them to be present when deploying components to Seed and Shoot clusters.

    The listed well-known PriorityClasses follow this rough concept:

    • Values are close to the maximum that can be declared by the user. This is important to ensure that Shoot system components have higher priority than the workload deployed by end-users.
    • Values have a bit of headroom in between to ensure flexibility when the need for intermediate priority values arises.
    • Values of PriorityClasses created on Seed clusters are lower than the ones on Shoots to ensure that Shoot system components have higher priority than Seed components, if the Seed is backed by a Shoot (ManagedSeed), e.g. coredns should have higher priority than gardenlet.
    • Names simply include the last digits of the value to minimize confusion caused by many (similar) names like critical, importance-high, etc.

    PriorityClasses for Shoot System Components

    NamePriorityAssociated Components (Examples)
    system-node-critical (created by Kubernetes)2000001000calico-node, kube-proxy, apiserver-proxy, csi-driver, egress-filter-applier
    system-cluster-critical (created by Kubernetes)2000000000calico-typha, calico-kube-controllers, coredns, vpn-shoot
    gardener-shoot-system-900999999900node-problem-detector
    gardener-shoot-system-800999999800calico-typha-horizontal-autoscaler, calico-typha-vertical-autoscaler
    gardener-shoot-system-700999999700blackbox-exporter, node-exporter
    gardener-shoot-system-600999999600addons-nginx-ingress-controller, addons-nginx-ingress-k8s-backend, kubernetes-dashboard, kubernetes-metrics-scraper

    PriorityClasses for Seed System Components

    NamePriorityAssociated Components (Examples)
    gardener-system-critical999998950gardenlet, gardener-resource-manager, istio-ingressgateway, istiod
    gardener-system-900999998900Extensions, gardener-seed-admission-controller, reversed-vpn-auth-server
    gardener-system-800999998800dependency-watchdog-endpoint, dependency-watchdog-probe, etcd-druid, (auditlog-)mutator, vpa-admission-controller
    gardener-system-700999998700auditlog-seed-controller, hvpa-controller, vpa-recommender, vpa-updater
    gardener-system-600999998600aggregate-alertmanager, alertmanager, fluent-bit, grafana, kube-state-metrics, nginx-ingress-controller, nginx-k8s-backend, prometheus, loki, seed-prometheus, vpa-exporter
    gardener-reserve-excess-capacity-5reserve-excess-capacity (ref)

    PriorityClasses for Shoot Control Plane Components

    NamePriorityAssociated Components (Examples)
    gardener-system-500999998500etcd-events, etcd-main, kube-apiserver
    gardener-system-400999998400gardener-resource-manager
    gardener-system-300999998300cloud-controller-manager, cluster-autoscaler, csi-driver-controller, kube-controller-manager, kube-scheduler, machine-controller-manager, terraformer, vpn-seed-server`
    gardener-system-200999998200csi-snapshot-controller, csi-snapshot-validation, cert-controller-manager, shoot-dns-service, vpa-admission-controller, vpa-recommender, vpa-updater
    gardener-system-100999998100alertmanager, grafana-operators, grafana-users, kube-state-metrics, prometheus, loki, event-logger

    There is also a legacy PriorityClass called gardener-shoot-controlplane with value 100. This PriorityClass is deprecated and will be removed in a future release. Make sure to migrate all your components to the above listed fine-granular PriorityClasses.

    1.4.12 - Process

    Releases, Features, Hotfixes

    This document describes how to contribute features or hotfixes, and how new Gardener releases are usually scheduled, validated, etc.

    Releases

    The @gardener-maintainers are trying to provide a new release roughly every other week (depending on their capacity and the stability/robustness of the master branch).

    Hotfixes are usually maintained for the latest three minor releases, though, there are no fixed release dates.

    Release Responsible Plan

    VersionWeek NoBegin Validation PhaseDue DateRelease Responsible
    v1.56Week 37-38September 12, 2022September 25, 2022@shafeeqes
    v1.57Week 39-40September 26, 2022October 9, 2022@ary1992
    v1.58Week 41-42October 10, 2022October 23, 2022@plkokanov
    v1.59Week 43-44October 24, 2022November 6, 2022@rfranzke
    v1.60Week 45-46November 7, 2022November 20, 2022@acumino
    v1.61Week 47-48November 21, 2022December 4, 2022@ialidzhikov
    v1.62Week 49-50December 5, 2022December 18, 2022@oliver-goetz
    v1.63Week 01-04January 2, 2023January 29, 2023@shafeeqes

    Apart from the release of the next version, the release responsible is also taking care of potential hotfix releases of the last three minor versions. The release responsible is the main contact person for coordinating new feature PRs for the next minor versions or cherry-pick PRs for the last three minor versions.

    Click to expand the archived release responsible associations!
    VersionWeek NoBegin Validation PhaseDue DateRelease Responsible
    v1.17Week 07-08February 15, 2021February 28, 2021@rfranzke
    v1.18Week 09-10March 1, 2021March 14, 2021@danielfoehrKn
    v1.19Week 11-12March 15, 2021March 28, 2021@timebertt
    v1.20Week 13-14March 29, 2021April 11, 2021@vpnachev
    v1.21Week 15-16April 12, 2021April 25, 2021@timuthy
    v1.22Week 17-18April 26, 2021May 9, 2021@BeckerMax
    v1.23Week 19-20May 10, 2021May 23, 2021@ialidzhikov
    v1.24Week 21-22May 24, 2021June 5, 2021@stoyanr
    v1.25Week 23-24June 7, 2021June 20, 2021@rfranzke
    v1.26Week 25-26June 21, 2021July 4, 2021@danielfoehrKn
    v1.27Week 27-28July 5, 2021July 18, 2021@timebertt
    v1.28Week 29-30July 19, 2021August 1, 2021@ialidzhikov
    v1.29Week 31-32August 2, 2021August 15, 2021@timuthy
    v1.30Week 33-34August 16, 2021August 29, 2021@BeckerMax
    v1.31Week 35-36August 30, 2021September 12, 2021@stoyanr
    v1.32Week 37-38September 13, 2021September 26, 2021@vpnachev
    v1.33Week 39-40September 27, 2021October 10, 2021@voelzmo
    v1.34Week 41-42October 11, 2021October 24, 2021@plkokanov
    v1.35Week 43-44October 25, 2021November 7, 2021@kris94
    v1.36Week 45-46November 8, 2021November 21, 2021@timebertt
    v1.37Week 47-48November 22, 2021December 5, 2021@danielfoehrKn
    v1.38Week 49-50December 6, 2021December 19, 2021@rfranzke
    v1.39Week 01-04January 3, 2022January 30, 2022@ialidzhikov, @timuthy
    v1.40Week 05-06January 31, 2022February 13, 2022@BeckerMax
    v1.41Week 07-08February 14, 2022February 27, 2022@plkokanov
    v1.42Week 09-10February 28, 2022March 13, 2022@kris94
    v1.43Week 11-12March 14, 2022March 27, 2022@rfranzke
    v1.44Week 13-14March 28, 2022April 10, 2022@timebertt
    v1.45Week 15-16April 11, 2022April 24, 2022@acumino
    v1.46Week 17-18April 25, 2022May 8, 2022@ialidzhikov
    v1.47Week 19-20May 9, 2022May 22, 2022@shafeeqes
    v1.48Week 21-22May 23, 2022June 5, 2022@ary1992
    v1.49Week 23-24June 6, 2022June 19, 2022@plkokanov
    v1.50Week 25-26June 20, 2022July 3, 2022@rfranzke
    v1.51Week 27-28July 4, 2022July 17, 2022@timebertt
    v1.52Week 29-30July 18, 2022July 31, 2022@acumino
    v1.53Week 31-32August 1, 2022August 14, 2022@kris94
    v1.54Week 33-34August 15, 2022August 28, 2022@ialidzhikov
    v1.55Week 35-36August 29, 2022September 11, 2022@oliver-goetz

    Release Validation

    The release phase for a new minor version lasts two weeks. Typically, the first week is used for the validation of the release. This phase includes the following steps:

    1. master (or latest release-* branch) is deployed to a development landscape that already hosts some existing seed and shoot clusters.
    2. An extended test suite is triggered by the “release responsible” which
      1. executes the Gardener integration tests for different Kubernetes versions, infrastructures, and Shoot settings.
      2. executes the Kubernetes conformance tests.
      3. executes further tests like Kubernetes/OS patch/minor version upgrades.
    3. Additionally, every four hours (or on demand) more tests (e.g., including the Kubernetes e2e test suite) are executed for different infrastructures.
    4. The “release responsible” is verifying new features or other notable changes (derived of the draft release notes) in this development system.

    Usually, the new release is triggered in the beginning of the second week if all tests are green, all checks were successful, and if all of the planned verifications were performed by the release responsible.

    Contributing new Features or Fixes

    Please refer to the Gardener contributor guide. Besides a lot of a general information, it also provides a checklist for newly created pull requests that may help you to prepare your changes for an efficient review process. If you are contributing a fix or major improvement, please take care to open cherry-pick PRs to all affected and still supported versions once the change is approved and merged in the master branch.

    ⚠️ Please ensure that your modifications pass the verification checks (linting, formatting, static code checks, tests, etc.) by executing

    make verify
    

    before filing your pull request.

    The guide applies for both changes to the master and to any release-* branch. All changes must be submitted via a pull request and be reviewed and approved by at least one code owner.

    Cherry Picks

    This section explains how to initiate cherry picks on release branches within the gardener/gardener repository.

    Prerequisites

    Before you initiate a cherry pick, make sure that the following prerequisites are accomplished.

    • A pull request merged against the master branch.
    • The release branch exists (check in the branches section)
    • Have the gardener/gardener repository cloned as follows:
      • the origin remote should point to your fork (alternatively this can be overwritten by passing FORK_REMOTE=<fork-remote>)
      • the upstream remote should point to the Gardener github org (alternatively this can be overwritten by passing UPSTREAM_REMOTE=<upstream-remote>)
    • Have hub installed, which is most easily installed via go get github.com/github/hub assuming you have a standard golang development environment.
    • A github token which has permissions to create a PR in an upstream branch.

    Initiate a Cherry Pick

    • Run the cherry pick script

      This example applies a master branch PR #3632 to the remote branch upstream/release-v3.14:

      GITHUB_USER=<your-user> hack/cherry-pick-pull.sh upstream/release-v3.14 3632
      
      • Be aware the cherry pick script assumes you have a git remote called upstream that points at the Gardener github org.

      • You will need to run the cherry pick script separately for each patch release you want to cherry pick to. Cherry picks should be applied to all active release branches where the fix is applicable.

      • When asked for your github password, provide the created github token rather than your actual github password. Refer https://github.com/github/hub/issues/2655#issuecomment-735836048

    1.4.13 - Secrets Management

    Secrets Management for Seed and Shoot Cluster

    The gardenlet needs to create quite some amount of credentials (certificates, private keys, passwords, etc.) for seed and shoot clusters in order to ensure secure deployments. Such credentials typically should be renewed automatically when their validity expires, rotated regularly, and they potentially need to be persisted such that they don’t get lost in case of a control plane migration or a lost seed cluster.

    SecretsManager Introduction

    These requirements can be covered by using the SecretsManager package maintained in pkg/utils/secrets/manager. It is built on top of the ConfigInterface and DataInterface interfaces part of pkg/utils/secrets and provides the following functions:

    • Generate(context.Context, secrets.ConfigInterface, ...GenerateOption) (*corev1.Secret, error)

      This method either retrieves the current secret for the given configuration or it (re)generates it in case the configuration changed, the signing CA changed (for certificate secrets), or when proactive rotation was triggered. If the configuration describes a certificate authority secret then this method automatically generates a bundle secret containing the current and potentially the old certificate.
      Available GenerateOptions:

      • SignedByCA(string, ...SignedByCAOption): This is only valid for certificate secrets and automatically retrieves the correct certificate authority in order to sign the provided server or client certificate.
        • There are two SignedByCAOptions:
          • UseCurrentCA. This option will sign server certificates with the new/current CA in case of a CA rotation. For more information, please refer to the “Certificate Signing” section below.
          • UseOldCA. This option will sign client certificates with the old CA in case of a CA rotation. For more information, please refer to the “Certificate Signing” section below.
      • Persist(): This marks the secret such that it gets persisted in the ShootState resource in the garden cluster. Consequently, it should only be used for secrets related to a shoot cluster.
      • Rotate(rotationStrategy): This specifies the strategy in case this secret is to be rotated or regenerated (either InPlace which immediately forgets about the old secret, or KeepOld which keeps the old secret in the system).
      • IgnoreOldSecrets(): This specifies that old secrets should not be considered and loaded (contrary to the default behavior). It should be used when old secrets are no longer important and can be “forgotten” (e.g. in “phase 2” (t2) of the CA certificate rotation). Such old secrets will be deleted on Cleanup().
      • IgnoreOldSecretsAfter(time.Duration): This specifies that old secrets should not be considered and loaded once a given duration after rotation has passed. It can be used to clean up old secrets after automatic rotation (e.g. the Seed cluster CA is automatically rotated when its validity will soon end and the old CA will be cleaned up 24 hours after triggering the rotation).
      • Validity(time.Duration): This specifies how long the secret should be valid. For certificate secret configurations, the manager will automatically deduce this information from the generated certificate.
    • Get(string, ...GetOption) (*corev1.Secret, bool)

      This method retrieves the current secret for the given name. In case the secret in question is a certificate authority secret then it retrieves the bundle secret by default. It is important that this method only knows about secrets for which there were prior Generate calls.
      Available GetOptions:

      • Bundle (default): This retrieves the bundle secret.
      • Current: This retrieves the current secret.
      • Old: This retrieves the old secret.
    • Cleanup(context.Context) error

      This method deletes secrets which are no longer required. No longer required secrets are those still existing in the system which weren’t detected by prior Generate calls. Consequently, only call Cleanup after you have executed Generate calls for all desired secrets.

    Some exemplary usages would look as follows:

    secret, err := k.secretsManager.Generate(
        ctx,
        &secrets.CertificateSecretConfig{
            Name:                        "my-server-secret",
            CommonName:                  "server-abc",
            DNSNames:                    []string{"first-name", "second-name"},
            CertType:                    secrets.ServerCert,
            SkipPublishingCACertificate: true,
        },
        secretsmanager.SignedByCA("my-ca"),
        secretsmanager.Persist(),
        secretsmanager.Rotate(secretsmanager.InPlace),
    )
    if err != nil {
        return err
    }
    

    As explained above, the caller does not need to care about the renewal, rotation or the persistence of this secret - all of these concerns are handled by the secrets manager. Automatic renewal of secrets happens when their validity approaches 80% or less than 10d are left until expiration.

    In case a CA certificate is needed by some component then it can be retrieved as follows:

    caSecret, found := k.secretsManager.Get("my-ca")
    if !found {
        return fmt.Errorf("secret my-ca not found")
    }
    

    As explained above, this returns the bundle secret for the CA my-ca which might potentially contain both the current and the old CA (in case of rotation/regeneration).

    Certificate Signing

    By default, client certificates are always signed by the current CA while server certificate are signed by the old CA (if it exists). This is to ensure a smooth exchange of certificate during a CA rotation (typically has two phases, ref GEP-18):

    • Client certificates:
      • In phase 1, clients get new certificates as soon as possible to ensure that all clients have been adapted before phase 2.
      • In phase 2, the respective server drops accepting certificates signed by the old CA.
    • Server certificates:
      • In phase 1, servers still use their old/existing certificates to allow clients to update their CA bundle used for verification of the servers’ certificates.
      • In phase 2, the old CA is dropped, hence servers need to get a certificate signed by the new/current CA. At this point in time, clients have already adapted their CA bundles.

    Always Sign Server Certificates With Current CA

    In case you control all clients and update them at the same time as the server, it is possible to make the secrets manager generate even server certificates with the new/current CA. This can help to prevent certificate mismatches when the CA bundle is already exchanged while the server still serves with a certificate signed by a CA no longer part of the bundle.

    Let’s consider the two following examples:

    1. gardenlet deploys a webhook server (gardener-resource-manager) and a corresponding MutatingWebhookConfiguration at the same time. In this case, the server certificate should be generated with the new/current CA to avoid above mentioned certificate mismatches during a CA rotation.
    2. gardenlet deploys a server (etcd) in one step, and a client (kube-apiserver) in a subsequent step. In this case, the default behaviour should apply (server certificate should be signed by old/existing CA).

    Always Sign Client Certificate With Old CA

    In the unusual case where the client is deployed before the server, it might be useful to always use the old CA for signing the client’s certificate. This can help to prevent certificate mismatches when the client already gets a new certificate while the server still only accepts certificates signed by the old CA.

    Let’s consider the following example:

    1. gardenlet deploys the kube-apiserver before the kubelet. However, the kube-apiserver has a client certificate signed by the ca-kubelet in order to communicate with it (e.g., when retrieving logs or forwarding ports). In this case, the client certificate should be generated with the old CA to avoid above mentioned certificate mismatches during a CA rotation.

    Reusing the SecretsManager in Other Components

    While the SecretsManager is primarily used by gardenlet, it can be reused by other components (e.g. extensions) as well for managing secrets that are specific to the component or extension. For example, provider extensions might use their own SecretsManager instance for managing the serving certificate of cloud-controller-manager.

    External components that want to reuse the SecretsManager should consider the following aspects:

    • On initialization of a SecretsManager, pass an identity specific to the component, controller and purpose. For example, gardenlet’s shoot controller uses gardenlet as the SecretsManager’s identity, the Worker controller in provider-foo should use provider-foo-worker and the ControlPlane controller should use provider-foo-controlplane-exposure for ControlPlane objects of purpose exposure. The given identity is added as a value for the manager-identity label on managed Secrets. This label is used by the Cleanup function to select only those Secrets that are actually managed by the particular SecretManager instance. This is done to prevent removing still needed Secrets that are managed by other instances.
    • Generate dedicated CAs for signing certificates instead of depending on CAs managed by gardenlet.
    • Names of Secrets managed by external SecretsManager instances must not conflict with Secret names from other instances (e.g. gardenlet).
    • For CAs that should be rotated in lock-step with the Shoot CAs managed by gardenlet, components need to pass information about the last rotation initiation time and the current rotation phase to the SecretsManager upon initialization. The relevant information can be retrieved from the Cluster resource under .spec.shoot.status.credentials.rotation.certificateAuthorities.
    • Independent of the specific identity, secrets marked with the Persist option are automatically saved in the ShootState resource by gardenlet and are also restored by gardenlet on Control Plane Migration to the new Seed.

    Implementation Details

    The source of truth for the secrets manager is the list of Secrets in the Kubernetes cluster it acts upon (typically, the seed cluster). The persisted secrets in the ShootState are only used if and only if the shoot is in the Restore phase - in this case all secrets are just synced to the seed cluster so that they can be picked up by the secrets manager.

    In order to prevent kubelets from unneeded watches (thus, causing some significant traffic against the kube-apiserver), the Secrets are marked as immutable. Consequently, they have a unique, deterministic name which is computed as follows:

    • For CA secrets, the name is just exactly the name specified in the configuration (e.g., ca). This is for backwards-compatibility and will be dropped in a future release once all components depending on the static name have been adapted.
    • For all other secrets, the name specified in the configuration is used as prefix followed by an 8-digit hash. This hash is computed out of the checksum of the secret configuration and the checksum of the certificate of the signing CA (only for certificate configurations).

    In all cases, the name of the secrets is suffixed with a 5-digit hash computed out of the time when the rotation for this secret was last started.

    1.4.14 - Seed Network Policies

    Network Policies in the Seed Cluster

    This document describes the Kubernetes network policies deployed by Gardener into the Seed cluster. For network policies deployed into the Shoot kube-system namespace, please see the usage section.

    Network policies deployed by Gardener have names and annotations describing their purpose, so this document does only highlight a subset of the policies in detail.

    Network policies in the Shoot namespace in the Seed

    The network policies in the Shoot namespace in the Seed can roughly be grouped into policies required for the control plane components and for logging & monitoring.

    The network policy deny-all plays a special role. This policy denies all ingress and egress traffic from each pod in the Shoot namespace. So per default, a pod running in the control plane cannot talk to any other pod in the whole Seed cluster. This means the pod needs to have labels matching to appropriate network policies allowing it to talk to exactly the components required to execute its desired functionality. This has also implications for Gardener extensions that need to deploy additional components into the Shoot's control plane.

    Network Policies for Control Plane Components

    This section highlights a selection of network policies that exist in the Shoot namespace in the Seed cluster. In general, the control plane components serve different purposes and thus need access to different pods and network ranges.

    In contrast to other network policies, the policy allow-to-shoot-networks is tailored to the individual Shoot cluster, because it is based on the network configuration in the Shoot manifest. It allows pods with the label networking.gardener.cloud/to-shoot-networks=allowed to access pods in the Shoot pod, service and node CIDR range. This is used by the Shoot API Server and the prometheus pods to communicate over VPN/proxy with pods in the Shoot cluster. This network policy is only useful if reversed vpn is disabled as otherwise the vpn-seed-server pod in the control plane is the only pod with layer 3 routing to the shoot network.

    The policy allow-to-blocked-cidrs allows pods with the label networking.gardener.cloud/to-blocked-cidrs=allowed to access IPs that are explicitly blocked for all control planes in a Seed cluster (configurable via spec.networks.blockCIDRS). This is used for instance to block the cloud provider’s metadata service.

    Another network policy to be highlighted is allow-to-seed-apiserver. Some components need access to the Seed API Server. This can be allowed by labeling the pod with networking.gardener.cloud/to-seed-apiserver=allowed. This policy allows exactly the IPs of the kube-apiserver of the Seed. While all other policies have a static set of permissions (do not change during the lifecycle of the Shoot), the policy allow-to-seed-apiserver is reconciled to reflect the endpoints in the default namespace. This is required because endpoint IPs are not necessarily stable (think of scaling the Seed API Server pods or hibernating the Seed cluster (acting as a managed seed) in a local development environment).

    Furthermore, the following network policies exist in the Shoot namespace. These policies are the same for every Shoot control plane.

    NAME                              POD-SELECTOR      
    # Pods that need to access the Shoot API server. Used by all Kubernetes control plane components.
    allow-to-shoot-apiserver          networking.gardener.cloud/to-shoot-apiserver=allowed
    
    # allows access to kube-dns/core-dns pods for DNS queries                       
    allow-to-dns                      networking.gardener.cloud/to-dns=allowed
    
    # allows access to private IP address ranges 
    allow-to-private-networks         networking.gardener.cloud/to-private-networks=allowed
    
    # allows access to all but private IP address ranges 
    allow-to-public-networks          networking.gardener.cloud/to-public-networks=allowed
    
    # allows Ingress to etcd pods from the Shoot's Kubernetes API Server
    allow-etcd                        app=etcd-statefulset,garden.sapcloud.io/role=controlplane
    
    # used by the Shoot API server to allows ingress from pods labeled
    # with'networking.gardener.cloud/to-shoot-apiserver=allowed', from Prometheus, and allows Egress to etcd pods
    allow-kube-apiserver              app=kubernetes,gardener.cloud/role=controlplane,role=apiserver
    

    Network policies for Logging & Monitoring

    Gardener currently introduces a logging stack based on Loki. So this section is subject to change. Please checkout the Community Meeting for more information.

    These are the logging and monitoring related network policies:

    NAME                              POD-SELECTOR                                                             
    allow-from-prometheus             networking.gardener.cloud/from-prometheus=allowed
    allow-grafana                     component=grafana,gardener.cloud/role=monitoring
    allow-prometheus                  app=prometheus,gardener.cloud/role=monitoring,role=monitoring
    allow-to-aggregate-prometheus     networking.gardener.cloud/to-aggregate-prometheus=allowed
    allow-to-loki                     networking.gardener.cloud/to-loki=allowed
    

    Let’s take for instance a look at the network policy from-prometheus. As part of the shoot reconciliation flow, Gardener deploys a shoot-specific Prometheus into the shoot namespace. Each pod that should be scraped for metrics must be labeled with networking.gardener.cloud/from-prometheus=allowed to allow incoming network requests by the prometheus pod. Most components of the Shoot cluster’s control plane expose metrics and are therefore labeled appropriately.

    Implications for Gardener Extensions

    Gardener extensions sometimes need to deploy additional components into the Shoot namespace in the Seed hosting the control plane. For example the Gardener extension provider-aws deploys the MachineControllerManager into the Shoot namespace, that is ultimately responsible to create the VMs with the cloud provider AWS.

    Every Shoot namespace in the Seed contains the network policy deny-all. This requires a pod deployed by a Gardener extension to have labels from network policies, that exist in the Shoot namespace, that allow the required network ranges.

    Additionally, extensions could also deploy their own network policies. This is used e.g by the Gardener extension provider-aws to serve Admission Webhooks for the Shoot API server that need to be reachable from within the Shoot namespace.

    The pod can use an arbitrary combination of network policies.

    Network policies in the garden namespace

    The network policies in the garden namespace are, with a few exceptions (e.g Kubernetes control plane specific policies), the same as in the Shoot namespaces. For your reference, these are all the deployed network policies.

    NAME                              POD-SELECTOR  
    allow-fluentbit                   app=fluent-bit,gardener.cloud/role=logging,role=logging              
    allow-from-aggregate-prometheus   networking.gardener.cloud/from-aggregate-prometheus=allowed              
    allow-to-aggregate-prometheus     networking.gardener.cloud/to-aggregate-prometheus=allowed                
    allow-to-all-shoot-apiservers     networking.gardener.cloud/to-all-shoot-apiservers=allowed                
    allow-to-blocked-cidrs            networking.gardener.cloud/to-blocked-cidrs=allowed                       
    allow-to-dns                      networking.gardener.cloud/to-dns=allowed                                 
    allow-to-loki                     networking.gardener.cloud/to-loki=allowed                       
    allow-to-private-networks         networking.gardener.cloud/to-private-networks=allowed                    
    allow-to-public-networks          networking.gardener.cloud/to-public-networks=allowed                     
    allow-to-seed-apiserver           networking.gardener.cloud/to-seed-apiserver=allowed                      
    deny-all                          networking.gardener.cloud/to-all=disallowed                              
    

    This section describes the network policies that are unique to the garden namespace.

    The network policy allow-to-all-shoot-apiservers allows pods to access every Shoot API server in the Seed. This is for instance used by the dependency watchdog to regularly check the health of all the Shoot API servers.

    Gardener deploys a central Prometheus instance in the garden namespace that fetches metrics and data from all seed cluster nodes and all seed cluster pods. The network policies allow-to-aggregate-prometheus and allow-from-aggregate-prometheus allow traffic from and to this prometheus instance.

    Worth mentioning is, that the network policy allow-to-shoot-networks does not exist in the garden namespace. This is to forbid Gardener system components to talk to workload deployed in the Shoot VPC.

    1.4.15 - Testing

    Testing Strategy and Developer Guideline

    This document walks you through

    • what kind of tests we have in Gardener
    • how to run each of them
    • what purpose each kind of test serves
    • how to best write tests that are correct, stable, fast and maintainable
    • how to debug tests that are not working as expected

    The document is aimed towards developers that want to contribute code and need to write tests, as well as maintainers and reviewers that review test code. It serves as a common guide that we commit to follow in our project to ensure consistency in our tests, good coverage for high confidence and good maintainability.

    The guidelines are not meant to be absolute rules. Always apply common sense and adapt the guideline if it doesn’t make much sense for some cases. If in doubt, don’t hesitate to ask questions during PR review (as an author but also as a reviewer). Add new learnings as soon as we make them!

    Generally speaking, tests are a strict requirement for contributing new code. If you touch code that is currently untested, you need to add tests for the new cases that you introduce as a minimum. Ideally though, you would add the missing test cases for the current code as well (boy scout rule – “always leave the campground cleaner than you found it”).

    Writing Tests (Relevant for All Kinds)

    • we follow BDD (behavior-driven development) testing principles and use Ginkgo along with Gomega
      • make sure to check out their extensive guides for more information and how to best leverage all of their features
    • use By to structure test cases with multiple steps, so that steps are easy to follow in the logs: example test
    • call defer GinkgoRecover() if making assertions in goroutines: doc, example test
    • use DeferCleanup instead of cleaning up manually (or use custom coding from the test framework): example test, example test
      • DeferCleanup makes sure to run the cleanup code in the right point in time, e.g., a DeferCleanup added in BeforeEach is executed with AfterEach
    • test failures should point to an exact location, so that failures in CI aren’t too difficult to debug/fix
      • use ExpectWithOffset for making assertions in helper funcs like expectSomethingWasCreated: example test
      • make sure to add additional descriptions to Gomega matchers if necessary (e.g. in a loop): example test
    • introduce helper functions for assertions to make test more readable where applicable: example test
    • introduce custom matchers to make tests more readable where applicable: example matcher
    • don’t rely on accurate timing of time.Sleep and friends
    • use the same client schemes that are also used by production code to avoid subtle bugs/regressions: example PR, production schemes, usage in test
    • make sure, your test is actually asserting the right thing and it doesn’t pass if the exact bug is introduced that you want to prevent
      • use specific error matchers instead of asserting any error has happened, make sure that the corresponding branch in the code is tested, e.g., prefer
        Expect(err).To(MatchError("foo"))
        
        over
        Expect(err).To(HaveOccurred())
        
      • if you’re unsure about your test’s behavior, attaching the debugger can sometimes be helpful to make sure your test is correct
    • about overwriting global variables
      • this is a common pattern (or hack?) in go for faking calls to external functions
      • however, this can lead to races, when the global variable is used from a goroutine (e.g., the function is called)
      • alternatively, set fields on structs (passed via parameter or set directly): this is not racy, as struct values are typically (and should be) only used for a single test case
      • alternative to dealing with function variables and fields:
        • add an interface, which your code depends on
        • write a fake and a real implementation (similar to clock.Clock.Sleep)
        • the real implementation calls the actual function (clock.RealClock.Sleep calls time.Sleep)
        • the fake implementation does whatever you want it to do for your test (clock.FakeClock.Sleep waits until the test code advanced the time)
    • use constants in test code with care
      • typically, you should not use constants from the same package as the tested code, instead use literals
      • if the constant value is changed, tests using the constant will still pass, although the “specification” is not fulfilled anymore
      • there are cases where it’s fine to use constants, but keep this caveat in mind when doing so
    • creating sample data for tests can be a high effort
      • if valuable, add a package for generating common sample data, e.g. Shoot/Cluster objects
    • make use of the testdata directory for storing arbitrary sample data needed by tests (helm charts, YAML manifests, etc.), example PR

    Unit Tests

    Running Unit Tests

    Run all unit tests:

    make test
    

    Run all unit tests with test coverage:

    make test-cov
    open test.coverage.html
    make test-cov-clean
    

    Run unit tests of specific packages:

    # run with same settings like in CI (race dector, timeout, ...)
    ./hack/test.sh ./pkg/resourcemanager/controller/... ./pkg/utils/secrets/...
    
    # freestyle
    go test ./pkg/resourcemanager/controller/... ./pkg/utils/secrets/...
    ginkgo run ./pkg/resourcemanager/controller/... ./pkg/utils/secrets/...
    

    Debugging Unit Tests

    Use ginkgo to focus on (a set of) test specs via code or via CLI flags. Remember to unfocus specs before contributing code, otherwise your PR tests will fail.

    $ ginkgo run --focus "should delete the unused resources" ./pkg/resourcemanager/controller/garbagecollector
    ...
    Will run 1 of 3 specs
    SS•
    
    Ran 1 of 3 Specs in 0.003 seconds
    SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 2 Skipped
    PASS
    

    Use ginkgo to run tests until they fail:

    $ ginkgo run --until-it-fails ./pkg/resourcemanager/controller/garbagecollector
    ...
    Ran 3 of 3 Specs in 0.004 seconds
    SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 0 Skipped
    PASS
    
    All tests passed...
    Will keep running them until they fail.
    This was attempt #58
    No, seriously... you can probably stop now.
    

    Use the stress tool for deflaking tests that fail sporadically in CI, e.g., due resource contention (CPU throttling):

    # get the stress tool
    go install golang.org/x/tools/cmd/stress@latest
    
    # build a test binary
    ginkgo build ./pkg/resourcemanager/controller/garbagecollector
    # alternatively
    go test -c ./pkg/resourcemanager/controller/garbagecollector
    
    # run the test in parallel and report any failures
    $ stress -p 16 ./pkg/resourcemanager/controller/garbagecollector/garbagecollector.test -ginkgo.focus "should delete the unused resources"
    5s: 1077 runs so far, 0 failures
    10s: 2160 runs so far, 0 failures
    

    stress will output a path to a file containing the full failure message, when a test run fails.

    Purpose of Unit Tests

    • unit tests prove correctness of a single unit according to the specification of its interface
      • think: is the unit that I introduced doing what it is supposed to do for all cases?
    • unit tests protect against regressions caused by adding new functionality to or refactoring of a single unit
      • think: is the unit that was introduced earlier (by someone else) and that I changed still doing what it was supposed to do for all cases?
    • example units: functions (conversion, defaulting, validation, helpers), structs (helpers, basic building blocks like the Secrets Manager), predicates, event handlers
    • for these purposes, unit tests need to cover all important cases of input for a single unit and cover edge cases / negative paths as well (e.g., errors)
      • because of the possible high dimensionality of test input, unit tests need to be fast to execute: individual test cases should not take more than a few seconds, test suites not more than 2 minutes
      • fuzzing can be used as a technique in addition to usual test cases for covering edge cases
    • test coverage can be used as a tool during test development for covering all cases of a unit
    • however, test coverage data can be a false safety net
      • full line coverage doesn’t mean you have covered all cases of valid input
      • we don’t have strict requirements for test coverage, as it doesn’t necessarily yield the desired outcome
    • unit tests should not test too large components, e.g. entire controller Reconcile functions
      • if a function/component does many steps, it’s probably better to split it up into multiple functions/components that can be unit tested individually
      • there might be special cases for very small Reconcile functions
      • if there are a lot of edge cases, extract dedicated functions that cover them and use unit tests to test them
      • usual-sized controllers should rather be tested in integration tests
      • individual parts (e.g. helper functions) should still be tested in unit test for covering all cases, though
    • unit tests are especially easy to run with a debugger and can help in understanding concrete behavior of components

    Writing Unit Tests

    • for the sake of execution speed, fake expensive calls/operations, e.g. secret generation: example test
    • generally, prefer fakes over mocks, e.g., use controller-runtime fake client over mock clients
      • mocks decrease maintainability because they expect the tested component to follow a certain way to reach the desired goal (e.g., call specific functions with particular arguments), example consequence
      • generally, fakes should be used in “result-oriented” test code (e.g., that a certain object was labelled, but the test doesn’t care if it was via patch or update as both a valid ways to reach the desired goal)
      • although rare, there are valid use cases for mocks, e.g. if the following aspects are important for correctness:
        • asserting that an exact function is called
        • asserting that functions are called in a specific order
        • asserting that exact parameters/values/… are passed
        • asserting that a certain function was not called
        • many of these can also be verified with fakes, although mocks might be simpler
      • only use mocks if the tested code directly calls the mock; never if the tested code only calls the mock indirectly (e.g., through a helper package/function)
      • keep in mind the maintenance implications of using mocks:
        • can you make a valid non-behavioral change in the code without breaking the test or dependent tests?
      • it’s valid to mix fakes and mocks in the same test or between test cases
    • generally, use the go test package, i.e., declare package <production_package>_test
      • helps in avoiding cyclic dependencies between production, test and helper packages
      • also forces you to distinguish between the public (exported) API surface of your code and internal state that might not be of interest to tests
      • it might be valid to use the same package as the tested code if you want to test unexported functions
      • helpers can also be exported if no one is supposed to import the containing package (e.g. controller package)

    Integration Tests (envtests)

    Integration tests in Gardener use the sigs.k8s.io/controller-runtime/pkg/envtest package. It sets up a temporary control plane (etcd + kube-apiserver) and runs the test against it.

    Package github.com/gardener/gardener/pkg/envtest augments controller-runtime’s envtest package by starting and registering gardener-apiserver. This is used to test controllers that act on resources in the Gardener APIs (aggregated APIs).

    Historically, test machinery tests have also been called “integration tests”. However, test machinery does not perform integration testing but rather executes a form of end-to-end tests against a real landscape. Hence, we tried to sharpen the terminology that we use to distinguish between “real” integration tests and test machinery tests but you might still find “integration tests” referring to test machinery tests in old issues or outdated documents.

    Running Integration Tests

    The test-integration make rule prepares the environment automatically by downloading the respective binaries (if not yet present) and sets the necessary environment variables.

    make test-integration
    

    If you want to run a specific set of integration tests, you can also execute them using ./hack/test-integration.sh directly instead of using the test-integration rule. For example:

    ./hack/test-integration.sh ./test/integration/resourcemanager/tokenrequestor
    

    The script takes care of preparing the environment for you. If you want to execute the test suites directly via go test or ginkgo, you have to point the KUBEBUILDER_ASSETS environment variable to the path that contains the etcd and kube-apiserver binaries. Alternatively, you can install the binaries to /usr/local/kubebuilder/bin.

    Debugging Integration Tests

    You can configure envtest to use an existing cluster instead of starting a temporary control plane for your test. This can be helpful for debugging integration tests, because you can easily inspect what is going on in your test cluster with kubectl.

    Run an envtest suite (not using gardener-apiserver) against an existing cluster:

    make kind-up
    export KUBECONFIG=$PWD/example/gardener-local/kind/kubeconfig
    export USE_EXISTING_CLUSTER=true
    
    # run test with verbose output
    ./hack/test-integration.sh -v ./test/integration/resourcemanager/health -ginkgo.v
    
    # watch test objects
    k get managedresource -A -w
    

    Run a gardenerenvtest suite (using gardener-apiserver) against an existing gardener setup:

    make kind-up
    export KUBECONFIG=$PWD/example/gardener-local/kind/kubeconfig
    make dev-setup
    # you might need to disable some admission plugins in hack/local-development/start-apiserver
    # via --disable-admission-plugins depending on the test suite
    make start-apiserver
    export USE_EXISTING_GARDENER=true
    
    # run test with verbose output
    ./hack/test-integration.sh -v ./test/integration/controllermanager/bastion -ginkgo.v
    
    # watch test objects
    k get bastion -A -w
    

    Similar to debugging unit tests, the stress tool can help hunting flakes in integration tests. Though, you might need to run less tests in parallel though (specified via -p) and have a bit more patience. Generally, reproducing flakes in integration tests is easier when stress-testing against an existing cluster instead of starting temporary individual control planes per test run.

    Stress-test an envtest suite (not using gardener-apiserver):

    # build a test binary
    ginkgo build ./test/integration/resourcemanager/health
    
    # prepare a cluster to run the test against
    make kind-up
    export KUBECONFIG=$PWD/example/gardener-local/kind/kubeconfig
    export USE_EXISTING_CLUSTER=true
    
    # use same timeout settings like in CI
    source ./hack/test-integration.env
    
    # run the test in parallel and report any failures
    $ stress -ignore "unable to grab random port" -p 16 ./test/integration/resourcemanager/health/health.test
    ...
    

    Stress-test a gardenerenvtest suite (using gardener-apiserver):

    # build a test binary
    ginkgo build ./test/integration/controllermanager/bastion
    
    # prepare a cluster including gardener-apiserver to run the test against
    make kind-up
    export KUBECONFIG=$PWD/example/gardener-local/kind/kubeconfig
    make dev-setup
    # you might need to disable some admission plugins in hack/local-development/start-apiserver
    # via --disable-admission-plugins depending on the test suite
    # especially the ResourceQuota plugin can quickly lead to test failures when stress-testing
    make start-apiserver
    export USE_EXISTING_GARDENER=true
    
    # use same timeout settings like in CI
    source ./hack/test-integration.env
    
    # run the test in parallel and report any failures
    $ stress -ignore "unable to grab random port" -p 16 ./test/integration/controllermanager/bastion/bastion.test
    ...
    

    Purpose of Integration Tests

    • integration tests prove that multiple units are correctly integrated into a fully-functional component of the system
    • example components with multiple units:
      • a controller with its reconciler, watches, predicates, event handlers, queues, etc.
      • a webhook with its server, handler, decoder and webhook configuration
    • integration tests set up a full component (including used libraries) and run it against a test environment close to the actual setup
      • e.g., start controllers against a real Kubernetes control plane to catch bugs that can only happen when talking to a real API server
      • integration tests are generally more expensive to run (e.g., in terms of execution time)
    • integration tests should not cover each and every detailed case
      • rather cover a good portion of the “usual” cases that components will face during normal operation (positive and negative test cases)
      • but don’t cover all failure cases or all cases of predicates -> they should be covered in unit tests already
      • generally, not supposed to “generate test coverage” but to provide confidence that components work well
    • as integration tests typically test only one component (or a cohesive set of components) isolated from others, they cannot catch bugs that occur when multiple controllers interact (could be discovered by e2e tests, though)
    • rule of thumb: a new integration tests should be added for each new controller (an integration test doesn’t replace unit tests though)

    Writing Integration Tests

    • make sure to have a clean test environment on both test suite and test case level:
      • set up dedicated test environments (envtest instances) per test suite
      • use dedicated namespaces per test suite
        • use GenerateName with a test-specific prefix: example test
        • restrict the controller-runtime manager to the test namespace by setting manager.Options.Namespace: example test
        • alternatively, use a test-specific prefix with a random suffix determined upfront: example test
          • this can be used to restrict webhooks to a dedicated test namespace: example test
        • this allows running a test in parallel against the same existing cluster for deflaking and stress testing: example PR
      • if the controller works on cluster-scoped resources
        • label the resources with a label specific to the test run, e.g. the test namespace’s name: example test
        • restrict the manager’s cache for these objects with a corresponding label selector: example test
        • alternatively, use a default label selector for all objects in the manager’s cache: example test
        • this allows running a test in parallel against the same existing cluster for deflaking and stress testing, even if it works with cluster-scoped resources that are visible to all parallel test runs: example PR
      • use dedicated test resources for each test case
        • use GenerateName: example test
        • alternatively, use a checksum of CurrentSpecReport().LeafNodeLocation.String(): example test
        • logging the created object names is generally a good idea to support debugging failing or flaky tests: example test
        • always delete all resources after the test case (e.g., via DeferCleanup) that were created for the test case
        • this avoids conflicts between test cases and cascading failures which distract from the actual root failures
      • don’t tolerate already existing resources (~dirty test environment), code smell: ignoring already exist errors
    • don’t use a cached client in test code (e.g., the one from a controller-runtime manager), always construct a dedicated test client (uncached): example test
    • use asynchronous assertions: Eventually and Consistently
      • never Expect anything to happen synchronously (immediately)
      • don’t use retry or wait until functions -> use Eventually, Consistently instead: example test
      • this allows to override the interval/timeout values from outside instead of hard-coding this in the test (see hack/test-integration.sh): example PR
      • beware of the default Eventually / Consistently timeouts / poll intervals: docs
      • don’t set custom (high) timeouts and intervals in test code: example PR
        • instead, shorten sync period of controllers, overwrite intervals of the tested code, or use fake clocks: example test
      • pass g Gomega to Eventually/Consistently and use g.Expect in it: docs, example test, example PR
      • don’t forget to call {Eventually,Consistently}.Should(), otherwise the assertions always silently succeeds without errors: onsi/gomega#561
    • when using Gardener’s envtest (envtest.GardenerTestEnvironment):
      • disable gardener-apiserver’s admission plugins that are not relevant to the integration test itself by passing --disable-admission-plugins: example test
      • this makes setup / teardown code simpler and ensures to only test code relevant to the tested component itself (but not the entire set of admission plugins)
      • e.g., you can disable the ShootValidator plugin to create Shoots that reference non-existing SecretBindings or disable the DeletionConfirmation plugin to delete Gardener resources without adding a deletion confirmation first.
    • use a custom rate limiter for controllers in integration tests: example test
      • this can be used for limiting exponential backoff to shorten wait times
      • otherwise, if using the default rate limiter, exponential backoff might exceed the timeout of Eventually calls and cause flakes

    End-to-end (e2e) Tests (using provider-local)

    We run a suite of e2e tests on every pull request and periodically on the master branch. It uses a KinD cluster and skaffold to boostrap a full installation of Gardener based on the current revision, including provider-local. This allows us to run e2e tests in an isolated test environment and fully locally without any infrastructure interaction. The tests perform a set of operations on Shoot clusters, e.g. creating, deleting, hibernating and waking up.

    These tests are executed in our prow instance at prow.gardener.cloud, see job definition and job history.

    Running e2e Tests

    You can also run these tests on your development machine, using the following commands:

    make kind-up
    export KUBECONFIG=$PWD/example/gardener-local/kind/kubeconfig
    make gardener-up
    make test-e2e-local  # alternatively: make test-e2e-local-simple
    

    If you want to run a specific set of e2e test cases, you can also execute them using ./hack/test-e2e-local.sh directly in combination with ginkgo label filters. For example:

    ./hack/test-e2e-local.sh --label-filter "Shoot && credentials-rotation"
    

    If you want to use an existing shoot instead of creating a new one for the test case and deleting it afterwards, you can specify the existing shoot via the following flags. This can be useful to speed of the development of e2e tests.

    ./hack/test-e2e-local.sh --label-filter "Shoot && credentials-rotation" -- --project-namespace=garden-local --existing-shoot-name=local
    

    Also see: developing Gardener locally and deploying Gardener locally.

    Debugging e2e Tests

    When debugging e2e test failures in CI, logs of the cluster components can be very helpful. Our e2e test jobs export logs of all containers running in the kind cluster to prow’s artifacts storage. You can find them by clicking the Artifacts link in the top bar in prow’s job view and navigating to artifacts. This directory will contain all cluster component logs grouped by node.

    Pull all artifacts using gsutil for searching and filtering the logs locally (use the path displayed in the artifacts view):

    gsutil cp -r gs://gardener-prow/pr-logs/pull/gardener_gardener/6136/pull-gardener-e2e-kind/1542030416616099840/artifacts/gardener-local-control-plane /tmp
    

    Purpose of e2e Tests

    • e2e tests provide a high level of confidence that our code runs as expected by users when deployed to production
    • they are supposed to catch bugs resulting from interaction between multiple components
    • test cases should be as close as possible to real usage by endusers
      • should test “from the perspective of the user” (or operator)
      • example: I create a Shoot and expect to be able to connect to it via the provided kubeconfig
      • accordingly, don’t assert details of the system
        • e.g., the user also wouldn’t expect that there is a kube-apiserver deployment in the seed, they rather expect that they can talk to it no matter how it is deployed
        • only assert details of the system if the tested feature is not fully visible to the end-user and there is no other way of ensuring that the feature works reliably
        • e.g., the Shoot CA rotation is not fully visible to the user but is assertable by looking at the secrets in the Seed.
    • pro: can be executed by developers and users without any real infrastructure (provider-local)
    • con: they currently cannot be executed with real infrastructure (e.g., provider-aws), we will work on this as part of #6016
    • keep in mind that the tested scenario is still artificial in a sense of using default configuration, only a few objects, only a few config/settings combinations are covered
      • we will never be able to cover the full “test matrix” and this should not be our goal
      • bugs will still be released and will still happen in production; we can’t avoid it
      • instead, we should add test cases for preventing bugs in features or settings that were frequently regressed: example PR
    • usually e2e tests cover the “straight-forward cases”
      • however, negative test cases can also be included, especially if they are important from the user’s perspective

    Writing e2e Tests

    • always wrap API calls and similar things in Eventually blocks: example test
      • at this point, we are pretty much working with a distributed system and failures can happen anytime
      • wrapping calls in Eventually makes tests more stable and more realistic (usually, you wouldn’t call the system broken if a single API call fails because of a short connectivity issue)
    • most of the points from writing integration tests are relevant for e2e tests as well (especially the points about asynchronous assertions)
    • in contrast to integration tests, in e2e tests, it might make sense to specify higher timeouts for Eventually calls, e.g., when waiting for a Shoot to be reconciled
      • generally, try to use the default settings for Eventually specified via the environment variables
      • only set higher timeouts if waiting for long-running reconciliations to be finished

    Test Machinery Tests

    Please see Test Machinery Tests.

    Purpose of Test Machinery Tests

    • test machinery tests have to be executed against full-blown Gardener installations
    • they can provide a very high level of confidence that an installation is functional in its current state, this includes: all Gardener components, Extensions, the used Cloud Infrastructure, all relevant settings/configuration
    • this brings the following benefits:
      • they test more realistic scenarios than e2e tests (real configuration, real infrastructure, etc.)
      • tests run “where the users are”
    • however, this also brings significant drawbacks:
      • tests are difficult to develop and maintain
      • tests require a full Gardener installation and cannot be executed in CI (on PR-level or against master)
      • tests require real infrastructure (think cloud provider credentials, cost)
      • using TestDefinitions under .test-defs requires a full test machinery installation
      • accordingly, tests are heavyweight and expensive to run
      • testing against real infrastructure can cause flakes sometimes (e.g., in outage situations)
      • failures are hard to debug, because clusters are deleted after the test (for obvious cost reasons)
      • bugs can only be caught, once it’s “too late”, i.e., when code is merged and deployed
    • today, test machinery tests cover a bigger “test matrix” (e.g., Shoot creation across infrastructures, kubernetes versions, machine image versions, etc.)
    • test machinery also runs Kubernetes conformance tests
    • however, because of the listed drawbacks, we should rather focus on augmenting our e2e tests, as we can run them locally and in CI in order to catch bugs before they get merged
    • it’s still a good idea to add test machinery tests if a feature needs to be tested that is depending on some installation-specific configuration

    Writing Test Machinery Tests

    • generally speaking, most points from writing integration tests and writing e2e tests apply here as well
    • however, test machinery tests contain a lot of technical debt and existing code doesn’t follow these best practices
    • as test machinery tests are out of our general focus, we don’t intend on reworking the tests soon or providing more guidance on how to write new ones

    Manual Tests

    • manual tests can be useful when the cost of trying to automatically test certain functionality are too high
    • useful for PR verification, if a reviewer wants to verify that all cases are properly tested by automated tests
    • currently, it’s the simplest option for testing upgrade scenarios
      • e.g. migration coding is probably best tested manually, as it’s a high effort to write an automated test for little benefit
    • obviously, the need for manual tests should be kept at a bare minimum
      • instead, we should add e2e tests wherever sensible/valuable
      • we want to implement some form of general upgrade tests as part of #6016

    1.4.16 - Testmachinery Tests

    Test Machinery Tests

    In order to automatically qualify Gardener releases, we execute a set of end-to-end tests using Test Machinery. This requires a full Gardener installation including infrastructure extensions as well as a setup of Test Machinery itself. These tests operate on Shoot clusters across different Cloud Providers, using different supported Kubernetes versions and various configuration options (huge test matrix).

    This manual gives an overview about test machinery tests in Gardener.

    Structure

    Gardener test machinery tests are split into two test suites that can be found under test/testmachinery/suites:

    • The Gardener Test Suite contains all tests that only require a running gardener instance.
    • The Shoot Test Suite contains all tests that require a predefined running shoot cluster.

    The corresponding tests of a test suite are defined in the import statement of the suite definition see shoot/run_suite_test.go and their source code can be found under test/testmachinery

    The test directory is structured as follows:

    test
    ├── e2e           # end-to-end tests (using provider-local)
    │  └── shoot
    ├── framework     # helper code shared across integration, e2e and testmachinery tests
    ├── integration   # integration tests (envtests)
    │  ├── controllermanager
    │  ├── envtest
    │  ├── resourcemanager
    │  ├── scheduler
    │  ├── seedadmissioncontroller
    │  ├── shootmaintenance
    │  └── ...
    └── testmachinery # test machinery tests
       ├── gardener   # actual test cases imported by suites/gardener
       │  └── security
       ├── shoots     # actual test cases imported by suites/shoot
       │  ├── applications
       │  ├── care
       │  ├── logging
       │  ├── operatingsystem
       │  ├── operations
       │  └── vpntunnel
       ├── suites     # suites that run agains a running garden or shoot cluster
       │  ├── gardener
       │  └── shoot
       └── system     # suites that are used for building a full test flow
          ├── complete_reconcile
          ├── managed_seed_creation
          ├── managed_seed_deletion
          ├── shoot_cp_migration
          ├── shoot_creation
          ├── shoot_deletion
          ├── shoot_hibernation
          ├── shoot_hibernation_wakeup
          └── shoot_update
    

    A suite can be executed by running the suite definition with ginkgo’s focus and skip flags to control the execution of specific labeled test. See example below:

    go test -timeout=0 -mod=vendor ./test/testmachinery/suites/shoot \
          --v -ginkgo.v -ginkgo.progress -ginkgo.no-color \
          --report-file=/tmp/report.json \                     # write elasticsearch formatted output to a file
          --disable-dump=false \                               # disables dumping of teh current state if a test fails
          -kubecfg=/path/to/gardener/kubeconfig \
          -shoot-name=<shoot-name> \                           # Name of the shoot to test
          -project-namespace=<gardener project namespace> \    # Name of the gardener project the test shoot resides
          -ginkgo.focus="\[RELEASE\]" \                        # Run all tests that are tagged as release
          -ginkgo.skip="\[SERIAL\]|\[DISRUPTIVE\]"             # Exclude all tests that are tagged SERIAL or DISRUPTIVE
    

    Add a new test

    To add a new test the framework requires the following steps (step 1. and 2. can be skipped if the test is added to an existing package):

    1. Create a new test file e.g. test/testmachinery/shoot/security/my-sec-test.go
    2. Import the test into the appropriate test suite (gardener or shoot): import _ "github.com/gardener/gardener/test/testmachinery/shoot/security"
    3. Define your test with the testframework. The framework will automatically add its initialization, cleanup and dump functions.
    var _ = ginkgo.Describe("my suite", func(){
      f := framework.NewShootFramework(nil)
    
      f.Beta().CIt("my first test", func(ctx context.Context) {
        f.ShootClient.Get(xx)
        // testing ...
      })
    })
    

    The newly created test can be tested by focusing the test with the default ginkgo focus f.Beta().FCIt("my first test", func(ctx context.Context) and run the shoot test suite with:

    go test -timeout=0 -mod=vendor ./test/testmachinery/suites/shoot \
          --v -ginkgo.v -ginkgo.progress -ginkgo.no-color \
          --report-file=/tmp/report.json \                     # write elasticsearch formatted output to a file
          --disable-dump=false \                               # disables dumping of the current state if a test fails
          -kubecfg=/path/to/gardener/kubeconfig \
          -shoot-name=<shoot-name> \                           # Name of the shoot to test
          -project-namespace=<gardener project namespace> \
          -fenced=<true|false>                                 # Tested shoot is running in a fenced environment and cannot be reached by gardener
    

    or for the gardener suite with:

    go test -timeout=0 -mod=vendor ./test/testmachinery/suites/gardener \
          --v -ginkgo.v -ginkgo.progress -ginkgo.no-color \
          --report-file=/tmp/report.json \                     # write elasticsearch formatted output to a file
          --disable-dump=false \                               # disables dumping of the current state if a test fails
          -kubecfg=/path/to/gardener/kubeconfig \
          -project-namespace=<gardener project namespace>
    

    ⚠️ Make sure that you do not commit any focused specs as this feature is only intended for local development! Ginkgo will fail the test suite if there are any focused specs.

    Alternatively, a test can be triggered by specifying a ginkgo focus regex with the name of the test e.g.

    go test -timeout=0 -mod=vendor ./test/testmachinery/suites/gardener \
          --v -ginkgo.v -ginkgo.progress -ginkgo.no-color \
          --report-file=/tmp/report.json \                     # write elasticsearch formatted output to a file
          -kubecfg=/path/to/gardener/kubeconfig \
          -project-namespace=<gardener project namespace> \
          -ginkgo.focus="my first test"                        # regex to match test cases
    

    Test Labels

    Every test should be labeled by using the predefined labels available with every framework to have consistent labeling across all test machinery tests.

    The labels are applied to every new It()/CIt() definition by:

    f := framework.NewCommonFramework()
    f.Default().Serial().It("my test") => "[DEFAULT] [SERIAL] my test"
    
    f := framework.NewShootFramework()
    f.Default().Serial().It("my test") => "[DEFAULT] [SERIAL] [SHOOT] my test"
    
    f := framework.NewGardenerFramework()
    f.Default().Serial().It("my test") => "[DEFAULT] [GARDENER] [SERIAL] my test"
    

    Labels:

    • Beta: Newly created tests with no experience on stableness should be first labeled as beta tests. They should be watched (and probably improved) until stable enough to be promoted to Default.
    • Default: Tests that were Beta before and proved to be stable are promoted to Default eventually. Default tests run more often, produce alerts and are considered during the release decision although they don’t necessarily block a release.
    • Release: Test are release relevant. A failing Release test blocks the release pipeline. Therefore these tests need to be stable. Only tests proven to be stable will eventually be promoted to Release.

    Behavior Labels:

    • Serial: The test should always be executed in serial with no other tests running as it may impact other tests.
    • Destructive: The test is destructive. Which means that is runs with no other tests and may break gardener or the shoot. Only create such tests if really necessary as the execution will be expensive (neither gardener nor the shoot can be reused in this case for other tests).

    Framework

    The framework directory contains all the necessary functions / utilities for running test machinery tests. For example, there are methods for creation/deletion of shoots, waiting for shoot deletion/creation, downloading/installing/deploying helm charts, logging, etc.

    The framework itself consists of 3 different framework that expect different prerequisites and offer context specific functionality.

    • CommonFramework: The common framework is the base framework that handles logging and setup of commonly needed resources like helm. It also contains common functions for interacting with kubernetes clusters like Waiting for resources to be ready or Exec into a running pod.
    • GardenerFramework contains all functions of the common framework and expects a running gardener instance with the provided gardener kubeconfig and a project namespace. It also contains functions to interact with gardener like Waiting for a shoot to be reconciled or Patch a shoot or Get a seed.
    • ShootFramework: contains all functions of the common and the gardener framework. It expects a running shoot cluster defined by the shoot’s name and namespace(project namespace). This framework contains functions to directly interact with the specific shoot.

    The whole framework also includes commonly used checks, ginkgo wrapper, etc. as well as commonly used tests. Theses common application tests (like the guestbook test) can be used within multiple tests to have a default application (with ingress, deployment, stateful backend) to test external factors.

    Config

    Every framework commandline flag can also be defined by a configuration file (the value of the configuration file is only used if flag is not specified by commandline). The test suite searches for a configuration file (yaml is preferred) if the command line flag --config=/path/to/config/file is provided. A framework can be defined in the configuration file by just using the flag name as root key e.g.

    verbose: debug
    kubecfg: /kubeconfig/path
    project-namespace: garden-it
    

    Report

    The framework automatically writes the default ginkgo default report to stdout and a specifically structured elastichsearch bulk report file to a specified location. The elastichsearch bulk report will write one json document per testcase and injects metadata of the whole testsuite. An example document for one test case would look like the following document:

    {
        "suite": {
            "name": "Shoot Test Suite",
            "phase": "Succeeded",
            "tests": 3,
            "failures": 1,
            "errors": 0,
            "time": 87.427
        },
        "name": "Shoot application testing  [DEFAULT] [RELEASE] [SHOOT] should download shoot kubeconfig successfully",
        "shortName": "should download shoot kubeconfig successfully",
        "labels": [
            "DEFAULT",
            "RELEASE",
            "SHOOT"
        ],
        "phase": "Succeeded",
        "time": 0.724512057
    }
    

    Resources

    The resources directory contains all the templates, helm config files (e.g., repositories.yaml, charts, and cache index which are downloaded upon the start of the test), shoot configs, etc.

    resources
    ├── charts
    ├── repository
    │   └── repositories.yaml
    └── templates
        ├── guestbook-app.yaml.tpl
        └── logger-app.yaml.tpl
    

    There are two special directories that are dynamically filled with the correct test files:

    • charts: the charts will be downloaded and saved in this directory
    • repository contains the repository.yaml file that the target helm repos will be read from and the cache where the stable-index.yaml file will be created

    System Tests

    This directory contains the system tests that have a special meaning for the testmachinery with their own Test Definition. Currently these system tests consists of:

    • Shoot creation
    • Shoot deletion
    • Shoot Kubernetes update
    • Gardener Full reconcile check

    Shoot Creation test

    Create Shoot test is meant to test shoot creation.

    Example Run

    go test -mod=vendor -timeout=0 ./test/testmachinery/system/shoot_creation \
      --v -ginkgo.v -ginkgo.progress \
      -kubecfg=$HOME/.kube/config \
      -shoot-name=$SHOOT_NAME \
      -cloud-profile=$CLOUDPROFILE \
      -seed=$SEED \
      -secret-binding=$SECRET_BINDING \
      -provider-type=$PROVIDER_TYPE \
      -region=$REGION \
      -k8s-version=$K8S_VERSION \
      -project-namespace=$PROJECT_NAMESPACE \
      -annotations=$SHOOT_ANNOTATIONS \
      -infrastructure-provider-config-filepath=$INFRASTRUCTURE_PROVIDER_CONFIG_FILEPATH \
      -controlplane-provider-config-filepath=$CONTROLPLANE_PROVIDER_CONFIG_FILEPATH \
      -workers-config-filepath=$$WORKERS_CONFIG_FILEPATH \
      -worker-zone=$ZONE \
      -networking-pods=$NETWORKING_PODS \
      -networking-services=$NETWORKING_SERVICES \
      -networking-nodes=$NETWORKING_NODES \
      -start-hibernated=$START_HIBERNATED
    

    Shoot Deletion test

    Delete Shoot test is meant to test the deletion of a shoot.

    Example Run

    go test -mod=vendor -timeout=0 -ginkgo.v -ginkgo.progress \
      ./test/testmachinery/system/shoot_deletion \
      -kubecfg=$HOME/.kube/config \
      -shoot-name=$SHOOT_NAME \
      -project-namespace=$PROJECT_NAMESPACE
    

    Shoot Update test

    The Update Shoot test is meant to test the kubernetes version update of a existing shoot. If no specific version is provided the next patch version is automatically selected. If there is no available newer version this test is a noop.

    Example Run

    go test -mod=vendor -timeout=0 ./test/testmachinery/system/shoot_update \
      --v -ginkgo.v -ginkgo.progress \
      -kubecfg=$HOME/.kube/config \
      -shoot-name=$SHOOT_NAME \
      -project-namespace=$PROJECT_NAMESPACE \
      -version=$K8S_VERSION
    

    Gardener Full Reconcile test

    The Gardener Full Reconcile test is meant to test if all shoots of a gardener instance are successfully reconciled.

    Example Run

    go test -mod=vendor -timeout=0 ./test/testmachinery/system/complete_reconcile \
      --v -ginkgo.v -ginkgo.progress \
      -kubecfg=$HOME/.kube/config \
      -project-namespace=$PROJECT_NAMESPACE \
      -gardenerVersion=$GARDENER_VERSION # needed to validate the last acted gardener version of a shoot
    

    1.5 - Extensions

    1.5.1 - Admission

    Extension Admission

    The extensions are expected to validate their respective resources for their extension specific configurations, when the resources are newly created or updated. For example, provider extensions would validate spec.provider.infrastructureConfig and spec.provider.controlPlaneConfig in the Shoot resource and spec.providerConfig in the CloudProfile resource, networking extensions would validate spec.networking.providerConfig in the Shoot resource. As best practice, the validation should be performed only if there is a change in the spec of the resource. Please find an exemplary implementation here.

    When a resource is newly created or updated, Gardener adds an extension label for all the extension types referenced in the spec of the resource. This label is of the form <extension-type>.extensions.gardener.cloud/<extension-name> : "true". For example, an extension label for provider extension type aws, looks like provider.extensions.gardener.cloud/aws : "true". The extensions should add object selectors in their admission webhooks for these labels, to filter out the objects they are responsible for. At present, these labels are added to BackupEntrys, BackupBuckets, CloudProfiles, Seeds, SecretBindings and Shoots. Please see this for the full list of extension labels.

    1.5.2 - BackupBucket

    Contract: BackupBucket Resource

    The Gardener project features a sub-project called etcd-backup-restore to take periodic backups of etcd backing Shoot clusters. It demands the bucket (or its equivalent in different object store providers) to be created and configured externally with appropriate credentials. The BackupBucket resource takes this responsibility in Gardener.

    Before introducing the BackupBucket extension resource, Gardener was using Terraform in order to create and manage these provider-specific resources (e.g., see AWS Backup). Now, Gardener commissions an external, provider-specific controller to take over this task. You can also refer to backupInfra proposal documentation to get idea about how the transition was done and understand the resource in broader scope.

    What is the Scope of Bucket?

    A bucket will be provisioned per Seed. So, a backup of every Shoot created on that Seed will be stored under a different shoot specific prefix under the bucket. For the backup of the Shoot rescheduled on different Seed, it will continue to use the same bucket.

    What is the Lifespan of BackupBucket?

    The bucket associated with BackupBucket will be created at the creation of the Seed. And as per current implementation, it will also be deleted on deletion of the Seed, if there isn’t any BackupEntry resource associated with it.

    In the future, we plan to introduce a schedule for BackupBucket - the deletion logic for the BackupBucket resource, which will reschedule it on different available Seeds on deletion or failure of a health check for the currently associated seed. In that case, the BackupBucket will be deleted only if there isn’t any schedulable Seed available and there isn’t any associated BackupEntry resource.

    What Needs to be Implemented to Support a New Infrastructure Provider?

    As part of the seed flow, Gardener will create a special CRD in the seed cluster that needs to be reconciled by an extension controller, for example:

    ---
    apiVersion: extensions.gardener.cloud/v1alpha1
    kind: BackupBucket
    metadata:
      name: foo
    spec:
      type: azure
      providerConfig:
        <some-optional-provider-specific-backupbucket-configuration>
      region: eu-west-1
      secretRef:
        name: backupprovider
        namespace: shoot--foo--bar
    

    The .spec.secretRef contains a reference to the provider secret pointing to the account that shall be used to create the needed resources. This provider secret will be configured by the Gardener operator in the Seed resource and propagated over there by the seed controller.

    After your controller has created the required bucket, if required, it generates the secret to access the objects in the bucket and put a reference to it in status. This secret is supposed to be used by Gardener or eventually a BackupEntry resource and etcd-backup-restore component to backup the etcd.

    In order to support a new infrastructure provider, you need to write a controller that watches all BackupBuckets with .spec.type=<my-provider-name>. You can take a look at the below referenced example implementation for the Azure provider.

    References and Additional Resources

    1.5.3 - BackupEntry

    Contract: BackupEntry Resource

    The Gardener project features a sub-project called etcd-backup-restore to take periodic backups of etcd backing Shoot clusters. It demands the bucket (or its equivalent in different object store providers) access credentials to be created and configured externally with appropriate credentials. The BackupEntry resource takes this responsibility in Gardener to provide this information by creating a secret specific to the component.

    That being said, the core motivation for introducing this resource was to support retention of backups post deletion of Shoot. The etcd-backup-restore components take responsibility of garbage collecting old backups out of the defined period. Once a shoot is deleted, we need to persist the backups for few days. Hence, Gardener uses the BackupEntry resource for this housekeeping work post deletion of a Shoot. The BackupEntry resource is responsible for shoot specific prefix under referred bucket.

    Before introducing the BackupEntry extension resource, Gardener was using Terraform in order to create and manage these provider-specific resources (e.g., see AWS Backup). Now, Gardener commissions an external, provider-specific controller to take over this task. You can also refer to backupInfra proposal documentation to get idea about how the transition was done and understand the resource in broader scope.

    What is the Lifespan of BackupEntry?

    The bucket associated with BackupEntry will be created by using a BackupBucket resource. The BackupEntry resource will be created as a part of the Shoot creation. But resources might continue to exist post deletion of a Shoot (see Gardenlet for more details).

    What Needs to be Implemented to Support a New Infrastructure Provider?

    As part of the shoot flow, Gardener will create a special CRD in the seed cluster that needs to be reconciled by an extension controller, for example:

    ---
    apiVersion: extensions.gardener.cloud/v1alpha1
    kind: BackupEntry
    metadata:
      name: shoot--foo--bar
    spec:
      type: azure
      providerConfig:
        <some-optional-provider-specific-backup-bucket-configuration>
      backupBucketProviderStatus:
        <some-optional-provider-specific-backup-bucket-status>
      region: eu-west-1
      bucketName: foo
      secretRef:
        name: backupprovider
        namespace: shoot--foo--bar
    

    The .spec.secretRef contains a reference to the provider secret pointing to the account that shall be used to create the needed resources. This provider secret will be propagated from the BackupBucket resource by the shoot controller.

    Your controller is supposed to create the etcd-backup secret in the control plane namespace of a shoot. This secret is supposed to be used by Gardener or eventually by the etcd-backup-restore component to backup the etcd. The controller implementation should clean up the objects created under the shoot specific prefix in the bucket equivalent to the name of the BackupEntry resource.

    In order to support a new infrastructure provider, you need to write a controller that watches all the BackupBuckets with .spec.type=<my-provider-name>. You can take a look at the below referenced example implementation for the Azure provider.

    References and Additional Resources

    1.5.4 - Bastion

    Contract: Bastion resource

    The Gardener project allows users to connect to Shoot worker nodes via SSH. As nodes are usually firewalled and not directly accessible from the public internet, GEP-15 introduced the concept of “Bastions”. A bastion is a dedicated server that only serves to allow SSH ingress to the worker nodes.

    Bastion resources contain the user’s public SSH key and IP address, in order to provision the server accordingly: The public key is put onto the Bastion and SSH ingress is only authorized for the given IP address (in fact, it’s not a single IP address, but a set of IP ranges, however for most purposes a single IP is be used).

    What is the lifespan of Bastion?

    Once a Bastion has been created in the garden, it will be replicated to the appropriate seed cluster, where a controller then reconciles a server and firewall rules etc. on the cloud provider used by the target Shoot. When the Bastion is ready (i.e. has a public IP), that IP is stored in the Bastion’s status and from there is picked up by the garden cluster and gardenctl eventually.

    To make multiple SSH sessions possible, the existence of the Bastion is not directly tied to the execution of gardenctl: users can exit out of gardenctl and use ssh manually to connect to the bastion and worker nodes.

    However, Bastions have an expiry date, after which they will be garbage collected.

    What needs to be implemented to support a new infrastructure provider?

    As part of the shoot flow Gardener will create a special CRD in the seed cluster that needs to be reconciled by an extension controller, for example:

    ---
    apiVersion: extensions.gardener.cloud/v1alpha1
    kind: Bastion
    metadata:
      name: mybastion
      namespace: shoot--foo--bar
    spec:
      type: aws
      # userData is base64-encoded cloud provider user data; this contains the
      # user's SSH key
      userData: IyEvYmluL2Jhc2ggL....Nlcgo=
      ingress:
        - ipBlock:
            cidr: 192.88.99.0/32 # this is most likely the user's IP address
    

    Your controller is supposed to create a new instance at the given cloud provider, firewall it to only allow SSH (TCP port 22) from the given IP blocks, and then to configure the firewall for the worker nodes to allow SSH from the bastion instance. When a Bastion is deleted, all these changes need to be reverted.

    Implementation details

    ConfigValidator interface

    For bastion controllers, the generic Reconciler also delegates to a ConfigValidator interface that contains a single Validate method. This method is called by the generic Reconciler at the beginning of every reconciliation, and can be implemented by the extension to validate the .spec.providerConfig part of the Bastion resource with the respective cloud provider, typically the existence and validity of cloud provider resources such as VPCs, Images.

    The Validate method returns a list of errors. If this list is non-empty, the generic Reconciler will fail with an error. This error will have the error code ERR_CONFIGURATION_PROBLEM, unless there is at least one error in the list that has its ErrorType field set to field.ErrorTypeInternal.

    References and additional resources

    1.5.5 - CA Rotation

    CA Rotation in Extensions

    GEP-18 proposes adding support for automated rotation of Shoot cluster certificate authorities (CAs). This document outlines all requirements that Gardener extensions need to fulfill in order to support the CA rotation feature.

    Requirements for Shoot Cluster CA Rotation

    • Extensions must not rely on static CA Secret names managed by gardenlet, because their names are changing during CA rotation.
    • Extensions cannot issue or use client certificates for authenticating against shoot API servers. Instead, they should use short-lived auto-rotated ServiceAccount tokens via gardener-resource-manager’s TokenRequestor. Also see Conventions and TokenRequestor documents.
    • Extensions need to generate dedicated CAs for signing server certificates (e.g. cloud-controller-manager). There should be one CA per controller and purpose in order to bind the lifecycle to the reconciliation cycle of the respective object for which it is created.
    • CAs managed by extensions should be rotated in lock-step with the shoot cluster CA. When the user triggers a rotation, gardenlet writes phase and initiation time to Shoot.status.credentials.rotation.certificateAuthorities.{phase,lastInitiationTime}. See GEP-18 for a detailed description on what needs to happen in each phase. Extensions can retrieve this information from Cluster.shoot.status.

    Utilities for Secrets Management

    In order to fulfill the requirements listed above, extension controllers can reuse the SecretsManager that gardenlet uses to manage all shoot cluster CAs, certificates, and other secrets as well. It implements the core logic for managing secrets that need to be rotated, auto-renewed etc.

    Additionally, there are utilities for reusing SecretsManager in extension controllers. They already implement above requirements based on the Cluster resource and allow focusing on the extension controllers’ business logic.

    For example, a simple SecretsManager usage in an extension controller could look like this:

    const (
      // identity for SecretsManager instance in ControlPlane controller
      identity = "provider-foo-controlplane"
      // secret config name of the dedicated CA
      caControlPlaneName = "ca-provider-foo-controlplane"
    )
    
    func Reconcile() {
      var (
        cluster *extensionscontroller.Cluster
        client  client.Client
    
        // define wanted secrets with options
        secretConfigs = []extensionssecretsmanager.SecretConfigWithOptions{
          {
            // dedicated CA for ControlPlane controller
            Config: &secretutils.CertificateSecretConfig{
              Name:       caControlPlaneName,
              CommonName: "ca-provider-foo-controlplane",
              CertType:   secretutils.CACert,
            },
            // persist CA so that it gets restored on control plane migration
            Options: []secretsmanager.GenerateOption{secretsmanager.Persist()},
          },
          {
            // server cert for control plane component
            Config: &secretutils.CertificateSecretConfig{
              Name:       "cloud-controller-manager",
              CommonName: "cloud-controller-manager",
              DNSNames:   kutil.DNSNamesForService("cloud-controller-manager", namespace),
              CertType:   secretutils.ServerCert,
            },
            // sign with our dedicated CA
            Options: []secretsmanager.GenerateOption{secretsmanager.SignedByCA(caControlPlaneName)},
          },
        }
      )
    
      // initialize SecretsManager based on Cluster object
      sm, err := extensionssecretsmanager.SecretsManagerForCluster(ctx, logger.WithName("secretsmanager"), clock.RealClock{}, client, cluster, identity, secretConfigs)
      
      // generate all wanted secrets (first CAs, then the rest)
      secrets, err := extensionssecretsmanager.GenerateAllSecrets(ctx, sm, secretConfigs)
    
      // cleanup any secrets that are not needed any more (e.g. after rotation)
      err = sm.Cleanup(ctx)
    }
    

    Please pay attention to the following points:

    • There should be one SecretsManager identity per controller (and purpose if applicable) in order to prevent conflicts between different instances. E.g., there should be different identities for Infrastructrue, Worker controller etc. and the ControlPlane controller should use dedicated SecretsManager identities per purpose (e.g. provider-foo-controlplane and provider-foo-controlplane-exposure).
    • All other points in Reusing the SecretsManager in Other Components

    1.5.6 - Cluster

    Cluster resource

    As part of the extensibility epic a lot of responsibility that was previously taken over by Gardener directly has now been shifted to extension controllers running in the seed clusters. These extensions often serve a well-defined purpose, e.g. the management of DNS records, infrastructure, etc. We have introduced a couple of extension CRDs in the seeds whose specification is written by Gardener, and which are acted up by the extensions.

    However, the extensions sometimes require more information that is not directly part of the specification. One example of that is the GCP infrastructure controller which needs to know the shoot’s pod and service network. Another example is the Azure infrastructure controller which requires some information out of the CloudProfile resource. The problem is that Gardener does not know which extension requires which information so that it can write it into their specific CRDs.

    In order to deal with this problem we have introduced the Cluster extension resource. This CRD is written into the seeds, however, it does not contain a status, so it is not expected that something acts upon it. Instead, you can treat it like a ConfigMap which contains data that might be interesting for you. In the context of Gardener, seeds and shoots, and extensibility the Cluster resource contains the CloudProfile, Seed, and Shoot manifest. Extension controllers can take whatever information they want out of it that might help completing their individual tasks.

    ---
    
    apiVersion: extensions.gardener.cloud/v1alpha1
    kind: Cluster
    metadata:
      name: shoot--foo--bar
    spec:
      cloudProfile:
        apiVersion: core.gardener.cloud/v1beta1
        kind: CloudProfile
        ...
      seed:
        apiVersion: core.gardener.cloud/v1beta1
        kind: Seed
        ...
      shoot:
        apiVersion: core.gardener.cloud/v1beta1
        kind: Shoot
        ...
    

    The resource is written by Gardener before it starts the reconciliation flow of the shoot.

    ⚠️ All Gardener components use the core.gardener.cloud/v1beta1 version, i.e., the Cluster resource will contain the objects in this version.

    Important information that should be taken into account

    There are some fields in the Shoot specification that might be interesting to take into account.

    • .spec.hibernation.enabled={true,false}: Extension controllers might want to behave differently if the shoot is hibernated or not (probably they might want to scale down their control plane components, for example).
    • .status.lastOperation.state=Failed: If Gardener sets the shoot’s last operation state to Failed it means that Gardener won’t automatically retry to finish the reconciliation/deletion flow because an error occurred that could not be resolved within the last 24h (default). In this case end-users are expected to manually re-trigger the reconciliation flow in case they want Gardener to try again. Extension controllers are expected to follow the same principle. This means they have to read the shoot state out of the Cluster resource.

    Extension resources not associated with a shoot

    In some cases, Gardener may create extension resources that are not associated with a shoot, but are needed to support some functionality internal to Gardener. Such resources will be created in the garden namespace of a seed cluster.

    For example, if the managed ingress controller is active on the seed, Gardener will create a DNSRecord resource(s) in the garden namespace of the seed cluster for the ingress DNS record.

    Extension controllers that may be expected to reconcile extension resources in the garden namespace should make sure that they can tolerate the absence of a cluster resource. This means that they should not attempt to read the cluster resource in such cases, or if they do they should ignore the “not found” error.

    References and additional resources

    1.5.7 - ContainerRuntime

    Gardener Container Runtime Extension

    At the lowest layers of a Kubernetes node is the software that, among other things, starts and stops containers. It is called “Container Runtime”. The most widely known container runtime is Docker, but it is not alone in this space. In fact, the container runtime space has been rapidly evolving.

    Kubernetes supports different container runtimes using Container Runtime Interface (CRI) – a plugin interface which enables kubelet to use a wide variety of container runtimes.

    Gardener supports creation of Worker machines using CRI, more information can be found here: CRI Support.

    Motivation

    Prior to the Container Runtime Extensibility concept, Gardener used Docker as the only container runtime to use in shoot worker machines. Because of the wide variety of different container runtimes offers multiple important features (for example enhanced security concepts) it is important to enable end users to use other container runtimes as well.

    The ContainerRuntime Extension Resource

    Here is what a typical ContainerRuntime resource would look-like:

    ---
    apiVersion: extensions.gardener.cloud/v1alpha1
    kind: ContainerRuntime
    metadata:
      name: my-container-runtime
    spec:
      binaryPath: /var/bin/containerruntimes
      type: gvisor
      workerPool:
        name: worker-ubuntu
        selector:
          matchLabels:
            worker.gardener.cloud/pool: worker-ubuntu
    

    Gardener deploys one ContainerRuntime resource per worker pool per CRI. To exemplify this, consider a Shoot having two worker pools (worker-one, worker-two) using containerd as the CRI as well as gvisor and kata as enabled container runtimes. Gardener would deploy four ContainerRuntime resources. For worker-one: one ContainerRuntime for type gvisor and one for type kata. The same resource are being deployed for worker-two.

    Supporting a new Container Runtime Provider

    To add support for another container runtime (e.g., gvisor, kata-containers, etc.) a container runtime extension controller needs to be implemented. It should support Gardener’s supported CRI plugins.

    The container runtime extension should install the necessary resources into the shoot cluster (e.g., RuntimeClasses), and it should copy the runtime binaries to the relevant worker machines in path: spec.binaryPath. Gardener labels the shoot nodes according to the CRI configured: worker.gardener.cloud/cri-name=<value> (e.g worker.gardener.cloud/cri-name=containerd) and multiple labels for each of the container runtimes configured for the shoot Worker machine: containerruntime.worker.gardener.cloud/<container-runtime-type-value>=true (e.g containerruntime.worker.gardener.cloud/gvisor=true). The way to install the binaries is by creating a daemon set which copies the binaries from an image in a docker registry to the relevant labeled Worker’s nodes (avoid downloading binaries from internet to also cater with isolated environments).

    For additional reference, please have a look at the runtime-gvsior provider extension, which provides more information on how to configure the necessary charts as well as the actuators required to reconcile container runtime inside the Shoot cluster to the desired state.

    1.5.8 - ControllerRegistration

    Registering Extension Controllers

    Extensions are registered in the garden cluster via ControllerRegistration resources. Deployment for respective extensions are specified via ControllerDeployment resources. Gardener evaluates the registrations and deployments and creates ControllerInstallation resources which describe the request “please install this controller X to this seed Y”.

    Similar to how CloudProfile or Seed resources get into the system, the Gardener administrator must deploy the ControllerRegistration and ControllerDeployment resources (this does not happen automatically in any way - the administrator decides which extensions shall be enabled).

    The specification mainly describes which of Gardener’s extension CRDs are managed, for example:

    apiVersion: core.gardener.cloud/v1beta1
    kind: ControllerDeployment
    metadata:
      name: os-gardenlinux
    type: helm
    providerConfig:
      chart: H4sIFAAAAAAA/yk... # <base64-gzip-chart>
      values:
        foo: bar
    ---
    apiVersion: core.gardener.cloud/v1beta1
    kind: ControllerRegistration
    metadata:
      name: os-gardenlinux
    spec:
      deployment:
        deploymentRefs:
        - name: os-gardenlinux
      resources:
      - kind: OperatingSystemConfig
        type: gardenlinux
        primary: true
    

    This information tells Gardener that there is an extension controller that can handle OperatingSystemConfig resources of type gardenlinux. A reference to the shown ControllerDeployment specifies how the deployment of the extension controller is accomplished.

    Also, it specifies that this controller is the primary one responsible for the lifecycle of the OperatingSystemConfig resource. Setting primary to false would allow to register additional, secondary controllers that may also watch/react on the OperatingSystemConfig/coreos resources, however, only the primary controller may change/update the main status of the extension object (that are used to “communicate” with the Gardenlet). Particularly, only the primary controller may set .status.lastOperation, .status.lastError, .status.observedGeneration, and .status.state. Secondary controllers may contribute to the .status.conditions[] if they like, of course.

    Secondary controllers might be helpful in scenarios where additional tasks need to be completed which are not part of the reconciliation logic of the primary controller but separated out into a dedicated extension.

    ⚠️ There must be exactly one primary controller for every registered kind/type combination. Also, please note that the primary field cannot be changed after creation of the ControllerRegistration.

    Deploying Extension Controllers

    Submitting above ControllerDeployment and ControllerRegistration will create a ControllerInstallation resource:

    apiVersion: core.gardener.cloud/v1beta1
    kind: ControllerInstallation
    metadata:
      name: os-gardenlinux
    spec:
      deploymentRef:
        name: os-gardenlinux
      registrationRef:
        name: os-gardenlinux
      seedRef:
        name: aws-eu1
    

    This resource expresses that Gardener requires the os-gardenlinux extension controller to run on the aws-eu1 seed cluster.

    The Gardener Controller Manager does automatically determine which extension is required on which seed cluster and will only create ControllerInstallation objects for those. Also, it will automatically delete ControllerInstallations referencing extension controllers that are no longer required on a seed (e.g., because all shoots on it have been deleted). There are additional configuration options, please see this section.

    How do extension controllers get deployed to seeds?

    After Gardener has written the ControllerInstallation resource some component must satisfy this request and start deploying the extension controller to the seed. Depending on the complexity of the controller’s lifecycle management, configuration, etc. there are two possible scenarios:

    Scenario 1: Deployed by Gardener

    In many cases the extension controllers are easy to deploy and configure. It is sufficient to simply create a Helm chart (standardized way of packaging software in the Kubernetes context) and deploy it together with some static configuration values. Gardener supports this scenario and allows to provide arbitrary deployment information in the ControllerDeployment resource’s .providerConfig section: