This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Gardener

The core component providing the extension API server of your Kubernetes cluster

Documentation Index

Overview

Concepts

Usage

API Reference

Proposals

Development

Extensions

Deployment

Operations

Monitoring

1.1 - Authentication

Packages:

authentication.gardener.cloud/v1alpha1

Package v1alpha1 is a version of the API. “authentication.gardener.cloud/v1alpha1” API is already used for CRD registration and must not be served by the API server.

Resource Types:

    AdminKubeconfigRequest

    AdminKubeconfigRequest can be used to request a kubeconfig with admin credentials for a Shoot cluster.

    FieldDescription
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    AdminKubeconfigRequestSpec

    Spec is the specification of the AdminKubeconfigRequest.



    expirationSeconds
    int64
    (Optional)

    ExpirationSeconds is the requested validity duration of the credential. The credential issuer may return a credential with a different validity duration so a client needs to check the ‘expirationTimestamp’ field in a response. Defaults to 1 hour.

    status
    AdminKubeconfigRequestStatus

    Status is the status of the AdminKubeconfigRequest.

    AdminKubeconfigRequestSpec

    (Appears on: AdminKubeconfigRequest)

    AdminKubeconfigRequestSpec contains the expiration time of the kubeconfig.

    FieldDescription
    expirationSeconds
    int64
    (Optional)

    ExpirationSeconds is the requested validity duration of the credential. The credential issuer may return a credential with a different validity duration so a client needs to check the ‘expirationTimestamp’ field in a response. Defaults to 1 hour.

    AdminKubeconfigRequestStatus

    (Appears on: AdminKubeconfigRequest)

    AdminKubeconfigRequestStatus is the status of the AdminKubeconfigRequest containing the kubeconfig and expiration of the credential.

    FieldDescription
    kubeconfig
    []byte

    Kubeconfig contains the kubeconfig with cluster-admin privileges for the shoot cluster.

    expirationTimestamp
    Kubernetes meta/v1.Time

    ExpirationTimestamp is the expiration timestamp of the returned credential.

    ViewerKubeconfigRequest

    ViewerKubeconfigRequest can be used to request a kubeconfig with viewer credentials (excluding Secrets) for a Shoot cluster.

    FieldDescription
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ViewerKubeconfigRequestSpec

    Spec is the specification of the ViewerKubeconfigRequest.



    expirationSeconds
    int64
    (Optional)

    ExpirationSeconds is the requested validity duration of the credential. The credential issuer may return a credential with a different validity duration so a client needs to check the ‘expirationTimestamp’ field in a response. Defaults to 1 hour.

    status
    ViewerKubeconfigRequestStatus

    Status is the status of the ViewerKubeconfigRequest.

    ViewerKubeconfigRequestSpec

    (Appears on: ViewerKubeconfigRequest)

    ViewerKubeconfigRequestSpec contains the expiration time of the kubeconfig.

    FieldDescription
    expirationSeconds
    int64
    (Optional)

    ExpirationSeconds is the requested validity duration of the credential. The credential issuer may return a credential with a different validity duration so a client needs to check the ‘expirationTimestamp’ field in a response. Defaults to 1 hour.

    ViewerKubeconfigRequestStatus

    (Appears on: ViewerKubeconfigRequest)

    ViewerKubeconfigRequestStatus is the status of the ViewerKubeconfigRequest containing the kubeconfig and expiration of the credential.

    FieldDescription
    kubeconfig
    []byte

    Kubeconfig contains the kubeconfig with viewer privileges (excluding Secrets) for the shoot cluster.

    expirationTimestamp
    Kubernetes meta/v1.Time

    ExpirationTimestamp is the expiration timestamp of the returned credential.


    Generated with gen-crd-api-reference-docs

    1.2 - Core

    Packages:

    core.gardener.cloud/v1beta1

    Package v1beta1 is a version of the API.

    Resource Types:

    BackupBucket

    BackupBucket holds details about backup bucket

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    BackupBucket
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    BackupBucketSpec

    Specification of the Backup Bucket.



    provider
    BackupBucketProvider

    Provider holds the details of cloud provider of the object store. This field is immutable.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig is the configuration passed to BackupBucket resource.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the credentials to access object store.

    seedName
    string
    (Optional)

    SeedName holds the name of the seed allocated to BackupBucket for running controller. This field is immutable.

    status
    BackupBucketStatus

    Most recently observed status of the Backup Bucket.

    BackupEntry

    BackupEntry holds details about shoot backup.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    BackupEntry
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    BackupEntrySpec
    (Optional)

    Spec contains the specification of the Backup Entry.



    bucketName
    string

    BucketName is the name of backup bucket for this Backup Entry.

    seedName
    string
    (Optional)

    SeedName holds the name of the seed to which this BackupEntry is scheduled

    status
    BackupEntryStatus
    (Optional)

    Status contains the most recently observed status of the Backup Entry.

    CloudProfile

    CloudProfile represents certain properties about a provider environment.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    CloudProfile
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    CloudProfileSpec
    (Optional)

    Spec defines the provider environment properties.



    caBundle
    string
    (Optional)

    CABundle is a certificate bundle which will be installed onto every host machine of shoot cluster targeting this profile.

    kubernetes
    KubernetesSettings

    Kubernetes contains constraints regarding allowed values of the ‘kubernetes’ block in the Shoot specification.

    machineImages
    []MachineImage

    MachineImages contains constraints regarding allowed values for machine images in the Shoot specification.

    machineTypes
    []MachineType

    MachineTypes contains constraints regarding allowed values for machine types in the ‘workers’ block in the Shoot specification.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig contains provider-specific configuration for the profile.

    regions
    []Region

    Regions contains constraints regarding allowed values for regions and zones.

    seedSelector
    SeedSelector
    (Optional)

    SeedSelector contains an optional list of labels on Seed resources that marks those seeds whose shoots may use this provider profile. An empty list means that all seeds of the same provider type are supported. This is useful for environments that are of the same type (like openstack) but may have different “instances”/landscapes. Optionally a list of possible providers can be added to enable cross-provider scheduling. By default, the provider type of the seed must match the shoot’s provider.

    type
    string

    Type is the name of the provider.

    volumeTypes
    []VolumeType
    (Optional)

    VolumeTypes contains constraints regarding allowed values for volume types in the ‘workers’ block in the Shoot specification.

    ControllerDeployment

    ControllerDeployment contains information about how this controller is deployed.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    ControllerDeployment
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    type
    string

    Type is the deployment type.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension

    ProviderConfig contains type-specific configuration. It contains assets that deploy the controller.

    ControllerInstallation

    ControllerInstallation represents an installation request for an external controller.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    ControllerInstallation
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ControllerInstallationSpec

    Spec contains the specification of this installation. If the object’s deletion timestamp is set, this field is immutable.



    registrationRef
    Kubernetes core/v1.ObjectReference

    RegistrationRef is used to reference a ControllerRegistration resource. The name field of the RegistrationRef is immutable.

    seedRef
    Kubernetes core/v1.ObjectReference

    SeedRef is used to reference a Seed resource. The name field of the SeedRef is immutable.

    deploymentRef
    Kubernetes core/v1.ObjectReference
    (Optional)

    DeploymentRef is used to reference a ControllerDeployment resource.

    status
    ControllerInstallationStatus

    Status contains the status of this installation.

    ControllerRegistration

    ControllerRegistration represents a registration of an external controller.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    ControllerRegistration
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ControllerRegistrationSpec

    Spec contains the specification of this registration. If the object’s deletion timestamp is set, this field is immutable.



    resources
    []ControllerResource
    (Optional)

    Resources is a list of combinations of kinds (DNSProvider, Infrastructure, Generic, …) and their actual types (aws-route53, gcp, auditlog, …).

    deployment
    ControllerRegistrationDeployment
    (Optional)

    Deployment contains information for how this controller is deployed.

    ExposureClass

    ExposureClass represents a control plane endpoint exposure strategy.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    ExposureClass
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    handler
    string

    Handler is the name of the handler which applies the control plane endpoint exposure strategy. This field is immutable.

    scheduling
    ExposureClassScheduling
    (Optional)

    Scheduling holds information how to select applicable Seed’s for ExposureClass usage. This field is immutable.

    InternalSecret

    InternalSecret holds secret data of a certain type. The total bytes of the values in the Data field must be less than MaxSecretSize bytes.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    InternalSecret
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object’s metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    immutable
    bool
    (Optional)

    Immutable, if set to true, ensures that data stored in the Secret cannot be updated (only object metadata can be modified). If not set to true, the field can be modified at any time. Defaulted to nil.

    data
    map[string][]byte
    (Optional)

    Data contains the secret data. Each key must consist of alphanumeric characters, ‘-’, ‘_’ or ‘.’. The serialized form of the secret data is a base64 encoded string, representing the arbitrary (possibly non-string) data value here. Described in https://tools.ietf.org/html/rfc4648#section-4

    stringData
    map[string]string
    (Optional)

    stringData allows specifying non-binary secret data in string form. It is provided as a write-only input field for convenience. All keys and values are merged into the data field on write, overwriting any existing values. The stringData field is never output when reading from the API.

    type
    Kubernetes core/v1.SecretType
    (Optional)

    Used to facilitate programmatic handling of secret data. More info: https://kubernetes.io/docs/concepts/configuration/secret/#secret-types

    NamespacedCloudProfile

    NamespacedCloudProfile represents certain properties about a provider environment.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    NamespacedCloudProfile
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    NamespacedCloudProfileSpec

    Spec defines the provider environment properties.



    caBundle
    string
    (Optional)

    CABundle is a certificate bundle which will be installed onto every host machine of shoot cluster targeting this profile.

    kubernetes
    KubernetesSettings
    (Optional)

    Kubernetes contains constraints regarding allowed values of the ‘kubernetes’ block in the Shoot specification.

    machineImages
    []MachineImage
    (Optional)

    MachineImages contains constraints regarding allowed values for machine images in the Shoot specification.

    machineTypes
    []MachineType
    (Optional)

    MachineTypes contains constraints regarding allowed values for machine types in the ‘workers’ block in the Shoot specification.

    regions
    []Region
    (Optional)

    Regions contains constraints regarding allowed values for regions and zones.

    volumeTypes
    []VolumeType
    (Optional)

    VolumeTypes contains constraints regarding allowed values for volume types in the ‘workers’ block in the Shoot specification.

    parent
    CloudProfileReference

    Parent contains a reference to a CloudProfile it inherits from.

    status
    NamespacedCloudProfileStatus

    Most recently observed status of the NamespacedCloudProfile.

    Project

    Project holds certain properties about a Gardener project.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    Project
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ProjectSpec
    (Optional)

    Spec defines the project properties.



    createdBy
    Kubernetes rbac/v1.Subject
    (Optional)

    CreatedBy is a subject representing a user name, an email address, or any other identifier of a user who created the project. This field is immutable.

    description
    string
    (Optional)

    Description is a human-readable description of what the project is used for.

    owner
    Kubernetes rbac/v1.Subject
    (Optional)

    Owner is a subject representing a user name, an email address, or any other identifier of a user owning the project. IMPORTANT: Be aware that this field will be removed in the v1 version of this API in favor of the owner role. The only way to change the owner will be by moving the owner role. In this API version the only way to change the owner is to use this field. TODO: Remove this field in favor of the owner role in v1.

    purpose
    string
    (Optional)

    Purpose is a human-readable explanation of the project’s purpose.

    members
    []ProjectMember
    (Optional)

    Members is a list of subjects representing a user name, an email address, or any other identifier of a user, group, or service account that has a certain role.

    namespace
    string
    (Optional)

    Namespace is the name of the namespace that has been created for the Project object. A nil value means that Gardener will determine the name of the namespace. This field is immutable.

    tolerations
    ProjectTolerations
    (Optional)

    Tolerations contains the tolerations for taints on seed clusters.

    dualApprovalForDeletion
    []DualApprovalForDeletion
    (Optional)

    DualApprovalForDeletion contains configuration for the dual approval concept for resource deletion.

    status
    ProjectStatus
    (Optional)

    Most recently observed status of the Project.

    Quota

    Quota represents a quota on resources consumed by shoot clusters either per project or per provider secret.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    Quota
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    QuotaSpec
    (Optional)

    Spec defines the Quota constraints.



    clusterLifetimeDays
    int32
    (Optional)

    ClusterLifetimeDays is the lifetime of a Shoot cluster in days before it will be terminated automatically.

    metrics
    Kubernetes core/v1.ResourceList

    Metrics is a list of resources which will be put under constraints.

    scope
    Kubernetes core/v1.ObjectReference

    Scope is the scope of the Quota object, either ‘project’ or ‘secret’. This field is immutable.

    SecretBinding

    SecretBinding represents a binding to a secret in the same or another namespace.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    SecretBinding
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret object in the same or another namespace. This field is immutable.

    quotas
    []Kubernetes core/v1.ObjectReference
    (Optional)

    Quotas is a list of references to Quota objects in the same or another namespace. This field is immutable.

    provider
    SecretBindingProvider
    (Optional)

    Provider defines the provider type of the SecretBinding. This field is immutable.

    Seed

    Seed represents an installation request for an external controller.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    Seed
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    SeedSpec

    Spec contains the specification of this installation.



    backup
    SeedBackup
    (Optional)

    Backup holds the object store configuration for the backups of shoot (currently only etcd). If it is not specified, then there won’t be any backups taken for shoots associated with this seed. If backup field is present in seed, then backups of the etcd from shoot control plane will be stored under the configured object store.

    dns
    SeedDNS

    DNS contains DNS-relevant information about this seed cluster.

    networks
    SeedNetworks

    Networks defines the pod, service and worker network of the Seed cluster.

    provider
    SeedProvider

    Provider defines the provider type and region for this Seed cluster.

    taints
    []SeedTaint
    (Optional)

    Taints describes taints on the seed.

    volume
    SeedVolume
    (Optional)

    Volume contains settings for persistentvolumes created in the seed cluster.

    settings
    SeedSettings
    (Optional)

    Settings contains certain settings for this seed cluster.

    ingress
    Ingress
    (Optional)

    Ingress configures Ingress specific settings of the Seed cluster. This field is immutable.

    status
    SeedStatus

    Status contains the status of this installation.

    Shoot

    Shoot represents a Shoot cluster created and managed by Gardener.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    Shoot
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ShootSpec
    (Optional)

    Specification of the Shoot cluster. If the object’s deletion timestamp is set, this field is immutable.



    addons
    Addons
    (Optional)

    Addons contains information about enabled/disabled addons and their configuration.

    cloudProfileName
    string

    CloudProfileName is a name of a CloudProfile object. This field is immutable.

    dns
    DNS
    (Optional)

    DNS contains information about the DNS settings of the Shoot.

    extensions
    []Extension
    (Optional)

    Extensions contain type and provider information for Shoot extensions.

    hibernation
    Hibernation
    (Optional)

    Hibernation contains information whether the Shoot is suspended or not.

    kubernetes
    Kubernetes

    Kubernetes contains the version and configuration settings of the control plane components.

    networking
    Networking
    (Optional)

    Networking contains information about cluster networking such as CNI Plugin type, CIDRs, …etc.

    maintenance
    Maintenance
    (Optional)

    Maintenance contains information about the time window for maintenance operations and which operations should be performed.

    monitoring
    Monitoring
    (Optional)

    Monitoring contains information about custom monitoring configurations for the shoot.

    provider
    Provider

    Provider contains all provider-specific and provider-relevant information.

    purpose
    ShootPurpose
    (Optional)

    Purpose is the purpose class for this cluster.

    region
    string

    Region is a name of a region. This field is immutable.

    secretBindingName
    string
    (Optional)

    SecretBindingName is the name of the a SecretBinding that has a reference to the provider secret. The credentials inside the provider secret will be used to create the shoot in the respective account. This field is immutable.

    seedName
    string
    (Optional)

    SeedName is the name of the seed cluster that runs the control plane of the Shoot.

    seedSelector
    SeedSelector
    (Optional)

    SeedSelector is an optional selector which must match a seed’s labels for the shoot to be scheduled on that seed.

    resources
    []NamedResourceReference
    (Optional)

    Resources holds a list of named resource references that can be referred to in extension configs by their names.

    tolerations
    []Toleration
    (Optional)

    Tolerations contains the tolerations for taints on seed clusters.

    exposureClassName
    string
    (Optional)

    ExposureClassName is the optional name of an exposure class to apply a control plane endpoint exposure strategy. This field is immutable.

    systemComponents
    SystemComponents
    (Optional)

    SystemComponents contains the settings of system components in the control or data plane of the Shoot cluster.

    controlPlane
    ControlPlane
    (Optional)

    ControlPlane contains general settings for the control plane of the shoot.

    schedulerName
    string
    (Optional)

    SchedulerName is the name of the responsible scheduler which schedules the shoot. If not specified, the default scheduler takes over. This field is immutable.

    cloudProfile
    CloudProfileReference
    (Optional)

    CloudProfile contains a reference to a CloudProfile or a NamespacedCloudProfile.

    status
    ShootStatus
    (Optional)

    Most recently observed status of the Shoot cluster.

    ShootState

    ShootState contains a snapshot of the Shoot’s state required to migrate the Shoot’s control plane to a new Seed.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1beta1
    kind
    string
    ShootState
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ShootStateSpec
    (Optional)

    Specification of the ShootState.



    gardener
    []GardenerResourceData
    (Optional)

    Gardener holds the data required to generate resources deployed by the gardenlet

    extensions
    []ExtensionResourceState
    (Optional)

    Extensions holds the state of custom resources reconciled by extension controllers in the seed

    resources
    []ResourceData
    (Optional)

    Resources holds the data of resources referred to by extension controller states

    APIServerLogging

    (Appears on: KubeAPIServerConfig)

    APIServerLogging contains configuration for the logs level and http access logs

    FieldDescription
    verbosity
    int32
    (Optional)

    Verbosity is the kube-apiserver log verbosity level Defaults to 2.

    httpAccessVerbosity
    int32
    (Optional)

    HTTPAccessVerbosity is the kube-apiserver access logs level

    APIServerRequests

    (Appears on: KubeAPIServerConfig)

    APIServerRequests contains configuration for request-specific settings for the kube-apiserver.

    FieldDescription
    maxNonMutatingInflight
    int32
    (Optional)

    MaxNonMutatingInflight is the maximum number of non-mutating requests in flight at a given time. When the server exceeds this, it rejects requests.

    maxMutatingInflight
    int32
    (Optional)

    MaxMutatingInflight is the maximum number of mutating requests in flight at a given time. When the server exceeds this, it rejects requests.

    Addon

    (Appears on: KubernetesDashboard, NginxIngress)

    Addon allows enabling or disabling a specific addon and is used to derive from.

    FieldDescription
    enabled
    bool

    Enabled indicates whether the addon is enabled or not.

    Addons

    (Appears on: ShootSpec)

    Addons is a collection of configuration for specific addons which are managed by the Gardener.

    FieldDescription
    kubernetesDashboard
    KubernetesDashboard
    (Optional)

    KubernetesDashboard holds configuration settings for the kubernetes dashboard addon.

    nginxIngress
    NginxIngress
    (Optional)

    NginxIngress holds configuration settings for the nginx-ingress addon.

    AdmissionPlugin

    (Appears on: KubeAPIServerConfig)

    AdmissionPlugin contains information about a specific admission plugin and its corresponding configuration.

    FieldDescription
    name
    string

    Name is the name of the plugin.

    config
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    Config is the configuration of the plugin.

    disabled
    bool
    (Optional)

    Disabled specifies whether this plugin should be disabled.

    kubeconfigSecretName
    string
    (Optional)

    KubeconfigSecretName specifies the name of a secret containing the kubeconfig for this admission plugin.

    Alerting

    (Appears on: Monitoring)

    Alerting contains information about how alerting will be done (i.e. who will receive alerts and how).

    FieldDescription
    emailReceivers
    []string
    (Optional)

    MonitoringEmailReceivers is a list of recipients for alerts

    AuditConfig

    (Appears on: KubeAPIServerConfig)

    AuditConfig contains settings for audit of the api server

    FieldDescription
    auditPolicy
    AuditPolicy
    (Optional)

    AuditPolicy contains configuration settings for audit policy of the kube-apiserver.

    AuditPolicy

    (Appears on: AuditConfig)

    AuditPolicy contains audit policy for kube-apiserver

    FieldDescription
    configMapRef
    Kubernetes core/v1.ObjectReference
    (Optional)

    ConfigMapRef is a reference to a ConfigMap object in the same namespace, which contains the audit policy for the kube-apiserver.

    AvailabilityZone

    (Appears on: Region)

    AvailabilityZone is an availability zone.

    FieldDescription
    name
    string

    Name is an availability zone name.

    unavailableMachineTypes
    []string
    (Optional)

    UnavailableMachineTypes is a list of machine type names that are not availability in this zone.

    unavailableVolumeTypes
    []string
    (Optional)

    UnavailableVolumeTypes is a list of volume type names that are not availability in this zone.

    BackupBucketProvider

    (Appears on: BackupBucketSpec)

    BackupBucketProvider holds the details of cloud provider of the object store.

    FieldDescription
    type
    string

    Type is the type of provider.

    region
    string

    Region is the region of the bucket.

    BackupBucketSpec

    (Appears on: BackupBucket)

    BackupBucketSpec is the specification of a Backup Bucket.

    FieldDescription
    provider
    BackupBucketProvider

    Provider holds the details of cloud provider of the object store. This field is immutable.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig is the configuration passed to BackupBucket resource.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the credentials to access object store.

    seedName
    string
    (Optional)

    SeedName holds the name of the seed allocated to BackupBucket for running controller. This field is immutable.

    BackupBucketStatus

    (Appears on: BackupBucket)

    BackupBucketStatus holds the most recently observed status of the Backup Bucket.

    FieldDescription
    providerStatus
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderStatus is the configuration passed to BackupBucket resource.

    lastOperation
    LastOperation
    (Optional)

    LastOperation holds information about the last operation on the BackupBucket.

    lastError
    LastError
    (Optional)

    LastError holds information about the last occurred error during an operation.

    observedGeneration
    int64
    (Optional)

    ObservedGeneration is the most recent generation observed for this BackupBucket. It corresponds to the BackupBucket’s generation, which is updated on mutation by the API Server.

    generatedSecretRef
    Kubernetes core/v1.SecretReference
    (Optional)

    GeneratedSecretRef is reference to the secret generated by backup bucket, which will have object store specific credentials.

    BackupEntrySpec

    (Appears on: BackupEntry)

    BackupEntrySpec is the specification of a Backup Entry.

    FieldDescription
    bucketName
    string

    BucketName is the name of backup bucket for this Backup Entry.

    seedName
    string
    (Optional)

    SeedName holds the name of the seed to which this BackupEntry is scheduled

    BackupEntryStatus

    (Appears on: BackupEntry)

    BackupEntryStatus holds the most recently observed status of the Backup Entry.

    FieldDescription
    lastOperation
    LastOperation
    (Optional)

    LastOperation holds information about the last operation on the BackupEntry.

    lastError
    LastError
    (Optional)

    LastError holds information about the last occurred error during an operation.

    observedGeneration
    int64
    (Optional)

    ObservedGeneration is the most recent generation observed for this BackupEntry. It corresponds to the BackupEntry’s generation, which is updated on mutation by the API Server.

    seedName
    string
    (Optional)

    SeedName is the name of the seed to which this BackupEntry is currently scheduled. This field is populated at the beginning of a create/reconcile operation. It is used when moving the BackupEntry between seeds.

    migrationStartTime
    Kubernetes meta/v1.Time
    (Optional)

    MigrationStartTime is the time when a migration to a different seed was initiated.

    CARotation

    (Appears on: ShootCredentialsRotation)

    CARotation contains information about the certificate authority credential rotation.

    FieldDescription
    phase
    CredentialsRotationPhase

    Phase describes the phase of the certificate authority credential rotation.

    lastCompletionTime
    Kubernetes meta/v1.Time
    (Optional)

    LastCompletionTime is the most recent time when the certificate authority credential rotation was successfully completed.

    lastInitiationTime
    Kubernetes meta/v1.Time
    (Optional)

    LastInitiationTime is the most recent time when the certificate authority credential rotation was initiated.

    lastInitiationFinishedTime
    Kubernetes meta/v1.Time
    (Optional)

    LastInitiationFinishedTime is the recent time when the certificate authority credential rotation initiation was completed.

    lastCompletionTriggeredTime
    Kubernetes meta/v1.Time
    (Optional)

    LastCompletionTriggeredTime is the recent time when the certificate authority credential rotation completion was triggered.

    CRI

    (Appears on: MachineImageVersion, Worker)

    CRI contains information about the Container Runtimes.

    FieldDescription
    name
    CRIName

    The name of the CRI library. Supported values are containerd.

    containerRuntimes
    []ContainerRuntime
    (Optional)

    ContainerRuntimes is the list of the required container runtimes supported for a worker pool.

    CRIName (string alias)

    (Appears on: CRI)

    CRIName is a type alias for the CRI name string.

    CloudProfileReference

    (Appears on: NamespacedCloudProfileSpec, ShootSpec)

    CloudProfileReference holds the information about the parent of the NamespacedCloudProfile.

    FieldDescription
    kind
    string

    Kind contains a CloudProfile kind.

    name
    string

    Name contains the name of the referenced CloudProfile.

    CloudProfileSpec

    (Appears on: CloudProfile, NamespacedCloudProfileStatus)

    CloudProfileSpec is the specification of a CloudProfile. It must contain exactly one of its defined keys.

    FieldDescription
    caBundle
    string
    (Optional)

    CABundle is a certificate bundle which will be installed onto every host machine of shoot cluster targeting this profile.

    kubernetes
    KubernetesSettings

    Kubernetes contains constraints regarding allowed values of the ‘kubernetes’ block in the Shoot specification.

    machineImages
    []MachineImage

    MachineImages contains constraints regarding allowed values for machine images in the Shoot specification.

    machineTypes
    []MachineType

    MachineTypes contains constraints regarding allowed values for machine types in the ‘workers’ block in the Shoot specification.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig contains provider-specific configuration for the profile.

    regions
    []Region

    Regions contains constraints regarding allowed values for regions and zones.

    seedSelector
    SeedSelector
    (Optional)

    SeedSelector contains an optional list of labels on Seed resources that marks those seeds whose shoots may use this provider profile. An empty list means that all seeds of the same provider type are supported. This is useful for environments that are of the same type (like openstack) but may have different “instances”/landscapes. Optionally a list of possible providers can be added to enable cross-provider scheduling. By default, the provider type of the seed must match the shoot’s provider.

    type
    string

    Type is the name of the provider.

    volumeTypes
    []VolumeType
    (Optional)

    VolumeTypes contains constraints regarding allowed values for volume types in the ‘workers’ block in the Shoot specification.

    ClusterAutoscaler

    (Appears on: Kubernetes)

    ClusterAutoscaler contains the configuration flags for the Kubernetes cluster autoscaler.

    FieldDescription
    scaleDownDelayAfterAdd
    Kubernetes meta/v1.Duration
    (Optional)

    ScaleDownDelayAfterAdd defines how long after scale up that scale down evaluation resumes (default: 1 hour).

    scaleDownDelayAfterDelete
    Kubernetes meta/v1.Duration
    (Optional)

    ScaleDownDelayAfterDelete how long after node deletion that scale down evaluation resumes, defaults to scanInterval (default: 0 secs).

    scaleDownDelayAfterFailure
    Kubernetes meta/v1.Duration
    (Optional)

    ScaleDownDelayAfterFailure how long after scale down failure that scale down evaluation resumes (default: 3 mins).

    scaleDownUnneededTime
    Kubernetes meta/v1.Duration
    (Optional)

    ScaleDownUnneededTime defines how long a node should be unneeded before it is eligible for scale down (default: 30 mins).

    scaleDownUtilizationThreshold
    float64
    (Optional)

    ScaleDownUtilizationThreshold defines the threshold in fraction (0.0 - 1.0) under which a node is being removed (default: 0.5).

    scanInterval
    Kubernetes meta/v1.Duration
    (Optional)

    ScanInterval how often cluster is reevaluated for scale up or down (default: 10 secs).

    expander
    ExpanderMode
    (Optional)

    Expander defines the algorithm to use during scale up (default: least-waste). See: https://github.com/gardener/autoscaler/blob/machine-controller-manager-provider/cluster-autoscaler/FAQ.md#what-are-expanders.

    maxNodeProvisionTime
    Kubernetes meta/v1.Duration
    (Optional)

    MaxNodeProvisionTime defines how long CA waits for node to be provisioned (default: 20 mins).

    maxGracefulTerminationSeconds
    int32
    (Optional)

    MaxGracefulTerminationSeconds is the number of seconds CA waits for pod termination when trying to scale down a node (default: 600).

    ignoreTaints
    []string
    (Optional)

    IgnoreTaints specifies a list of taint keys to ignore in node templates when considering to scale a node group.

    newPodScaleUpDelay
    Kubernetes meta/v1.Duration
    (Optional)

    NewPodScaleUpDelay specifies how long CA should ignore newly created pods before they have to be considered for scale-up (default: 0s).

    maxEmptyBulkDelete
    int32
    (Optional)

    MaxEmptyBulkDelete specifies the maximum number of empty nodes that can be deleted at the same time (default: 10).

    ignoreDaemonsetsUtilization
    bool
    (Optional)

    IgnoreDaemonsetsUtilization allows CA to ignore DaemonSet pods when calculating resource utilization for scaling down (default: false).

    verbosity
    int32
    (Optional)

    Verbosity allows CA to modify its log level (default: 2).

    ClusterAutoscalerOptions

    (Appears on: Worker)

    ClusterAutoscalerOptions contains the cluster autoscaler configurations for a worker pool.

    FieldDescription
    scaleDownUtilizationThreshold
    float64
    (Optional)

    ScaleDownUtilizationThreshold defines the threshold in fraction (0.0 - 1.0) under which a node is being removed.

    scaleDownGpuUtilizationThreshold
    float64
    (Optional)

    ScaleDownGpuUtilizationThreshold defines the threshold in fraction (0.0 - 1.0) of gpu resources under which a node is being removed.

    scaleDownUnneededTime
    Kubernetes meta/v1.Duration
    (Optional)

    ScaleDownUnneededTime defines how long a node should be unneeded before it is eligible for scale down.

    scaleDownUnreadyTime
    Kubernetes meta/v1.Duration
    (Optional)

    ScaleDownUnreadyTime defines how long an unready node should be unneeded before it is eligible for scale down.

    maxNodeProvisionTime
    Kubernetes meta/v1.Duration
    (Optional)

    MaxNodeProvisionTime defines how long CA waits for node to be provisioned.

    Condition

    (Appears on: ControllerInstallationStatus, SeedStatus, ShootStatus)

    Condition holds the information about the state of a resource.

    FieldDescription
    type
    ConditionType

    Type of the condition.

    status
    ConditionStatus

    Status of the condition, one of True, False, Unknown.

    lastTransitionTime
    Kubernetes meta/v1.Time

    Last time the condition transitioned from one status to another.

    lastUpdateTime
    Kubernetes meta/v1.Time

    Last time the condition was updated.

    reason
    string

    The reason for the condition’s last transition.

    message
    string

    A human readable message indicating details about the transition.

    codes
    []ErrorCode
    (Optional)

    Well-defined error codes in case the condition reports a problem.

    ConditionStatus (string alias)

    (Appears on: Condition)

    ConditionStatus is the status of a condition.

    ConditionType (string alias)

    (Appears on: Condition)

    ConditionType is a string alias.

    ContainerRuntime

    (Appears on: CRI)

    ContainerRuntime contains information about worker’s available container runtime

    FieldDescription
    type
    string

    Type is the type of the Container Runtime.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig is the configuration passed to container runtime resource.

    ControlPlane

    (Appears on: ShootSpec)

    ControlPlane holds information about the general settings for the control plane of a shoot.

    FieldDescription
    highAvailability
    HighAvailability
    (Optional)

    HighAvailability holds the configuration settings for high availability of the control plane of a shoot.

    ControllerDeploymentPolicy (string alias)

    (Appears on: ControllerRegistrationDeployment)

    ControllerDeploymentPolicy is a string alias.

    ControllerInstallationSpec

    (Appears on: ControllerInstallation)

    ControllerInstallationSpec is the specification of a ControllerInstallation.

    FieldDescription
    registrationRef
    Kubernetes core/v1.ObjectReference

    RegistrationRef is used to reference a ControllerRegistration resource. The name field of the RegistrationRef is immutable.

    seedRef
    Kubernetes core/v1.ObjectReference

    SeedRef is used to reference a Seed resource. The name field of the SeedRef is immutable.

    deploymentRef
    Kubernetes core/v1.ObjectReference
    (Optional)

    DeploymentRef is used to reference a ControllerDeployment resource.

    ControllerInstallationStatus

    (Appears on: ControllerInstallation)

    ControllerInstallationStatus is the status of a ControllerInstallation.

    FieldDescription
    conditions
    []Condition
    (Optional)

    Conditions represents the latest available observations of a ControllerInstallations’s current state.

    providerStatus
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderStatus contains type-specific status.

    ControllerRegistrationDeployment

    (Appears on: ControllerRegistrationSpec)

    ControllerRegistrationDeployment contains information for how this controller is deployed.

    FieldDescription
    policy
    ControllerDeploymentPolicy
    (Optional)

    Policy controls how the controller is deployed. It defaults to ‘OnDemand’.

    seedSelector
    Kubernetes meta/v1.LabelSelector
    (Optional)

    SeedSelector contains an optional label selector for seeds. Only if the labels match then this controller will be considered for a deployment. An empty list means that all seeds are selected.

    deploymentRefs
    []DeploymentRef
    (Optional)

    DeploymentRefs holds references to ControllerDeployments. Only one element is supported currently.

    ControllerRegistrationSpec

    (Appears on: ControllerRegistration)

    ControllerRegistrationSpec is the specification of a ControllerRegistration.

    FieldDescription
    resources
    []ControllerResource
    (Optional)

    Resources is a list of combinations of kinds (DNSProvider, Infrastructure, Generic, …) and their actual types (aws-route53, gcp, auditlog, …).

    deployment
    ControllerRegistrationDeployment
    (Optional)

    Deployment contains information for how this controller is deployed.

    ControllerResource

    (Appears on: ControllerRegistrationSpec)

    ControllerResource is a combination of a kind (DNSProvider, Infrastructure, Generic, …) and the actual type for this kind (aws-route53, gcp, auditlog, …).

    FieldDescription
    kind
    string

    Kind is the resource kind, for example “OperatingSystemConfig”.

    type
    string

    Type is the resource type, for example “coreos” or “ubuntu”.

    globallyEnabled
    bool
    (Optional)

    GloballyEnabled determines if this ControllerResource is required by all Shoot clusters. This field is defaulted to false when kind is “Extension”.

    reconcileTimeout
    Kubernetes meta/v1.Duration
    (Optional)

    ReconcileTimeout defines how long Gardener should wait for the resource reconciliation. This field is defaulted to 3m0s when kind is “Extension”.

    primary
    bool
    (Optional)

    Primary determines if the controller backed by this ControllerRegistration is responsible for the extension resource’s lifecycle. This field defaults to true. There must be exactly one primary controller for this kind/type combination. This field is immutable.

    lifecycle
    ControllerResourceLifecycle
    (Optional)

    Lifecycle defines a strategy that determines when different operations on a ControllerResource should be performed. This field is defaulted in the following way when kind is “Extension”. Reconcile: “AfterKubeAPIServer” Delete: “BeforeKubeAPIServer” Migrate: “BeforeKubeAPIServer”

    workerlessSupported
    bool
    (Optional)

    WorkerlessSupported specifies whether this ControllerResource supports Workerless Shoot clusters. This field is only relevant when kind is “Extension”.

    ControllerResourceLifecycle

    (Appears on: ControllerResource)

    ControllerResourceLifecycle defines the lifecycle of a controller resource.

    FieldDescription
    reconcile
    ControllerResourceLifecycleStrategy
    (Optional)

    Reconcile defines the strategy during reconciliation.

    delete
    ControllerResourceLifecycleStrategy
    (Optional)

    Delete defines the strategy during deletion.

    migrate
    ControllerResourceLifecycleStrategy
    (Optional)

    Migrate defines the strategy during migration.

    ControllerResourceLifecycleStrategy (string alias)

    (Appears on: ControllerResourceLifecycle)

    ControllerResourceLifecycleStrategy is a string alias.

    CoreDNS

    (Appears on: SystemComponents)

    CoreDNS contains the settings of the Core DNS components running in the data plane of the Shoot cluster.

    FieldDescription
    autoscaling
    CoreDNSAutoscaling
    (Optional)

    Autoscaling contains the settings related to autoscaling of the Core DNS components running in the data plane of the Shoot cluster.

    rewriting
    CoreDNSRewriting
    (Optional)

    Rewriting contains the setting related to rewriting of requests, which are obviously incorrect due to the unnecessary application of the search path.

    CoreDNSAutoscaling

    (Appears on: CoreDNS)

    CoreDNSAutoscaling contains the settings related to autoscaling of the Core DNS components running in the data plane of the Shoot cluster.

    FieldDescription
    mode
    CoreDNSAutoscalingMode

    The mode of the autoscaling to be used for the Core DNS components running in the data plane of the Shoot cluster. Supported values are horizontal and cluster-proportional.

    CoreDNSAutoscalingMode (string alias)

    (Appears on: CoreDNSAutoscaling)

    CoreDNSAutoscalingMode is a type alias for the Core DNS autoscaling mode string.

    CoreDNSRewriting

    (Appears on: CoreDNS)

    CoreDNSRewriting contains the setting related to rewriting requests, which are obviously incorrect due to the unnecessary application of the search path.

    FieldDescription
    commonSuffixes
    []string
    (Optional)

    CommonSuffixes are expected to be the suffix of a fully qualified domain name. Each suffix should contain at least one or two dots (‘.’) to prevent accidental clashes.

    CredentialsRotationPhase (string alias)

    (Appears on: CARotation, ETCDEncryptionKeyRotation, ServiceAccountKeyRotation)

    CredentialsRotationPhase is a string alias.

    DNS

    (Appears on: ShootSpec)

    DNS holds information about the provider, the hosted zone id and the domain.

    FieldDescription
    domain
    string
    (Optional)

    Domain is the external available domain of the Shoot cluster. This domain will be written into the kubeconfig that is handed out to end-users. This field is immutable.

    providers
    []DNSProvider
    (Optional)

    Providers is a list of DNS providers that shall be enabled for this shoot cluster. Only relevant if not a default domain is used. Deprecated: Configuring multiple DNS providers is deprecated and will be forbidden in a future release. Please use the DNS extension provider config (e.g. shoot-dns-service) for additional providers.

    DNSIncludeExclude

    (Appears on: DNSProvider)

    DNSIncludeExclude contains information about which domains shall be included/excluded.

    FieldDescription
    include
    []string
    (Optional)

    Include is a list of domains that shall be included.

    exclude
    []string
    (Optional)

    Exclude is a list of domains that shall be excluded.

    DNSProvider

    (Appears on: DNS)

    DNSProvider contains information about a DNS provider.

    FieldDescription
    domains
    DNSIncludeExclude
    (Optional)

    Domains contains information about which domains shall be included/excluded for this provider. Deprecated: This field is deprecated and will be removed in a future release. Please use the DNS extension provider config (e.g. shoot-dns-service) for additional configuration.

    primary
    bool
    (Optional)

    Primary indicates that this DNSProvider is used for shoot related domains. Deprecated: This field is deprecated and will be removed in a future release. Please use the DNS extension provider config (e.g. shoot-dns-service) for additional and non-primary providers.

    secretName
    string
    (Optional)

    SecretName is a name of a secret containing credentials for the stated domain and the provider. When not specified, the Gardener will use the cloud provider credentials referenced by the Shoot and try to find respective credentials there (primary provider only). Specifying this field may override this behavior, i.e. forcing the Gardener to only look into the given secret.

    type
    string
    (Optional)

    Type is the DNS provider type.

    zones
    DNSIncludeExclude
    (Optional)

    Zones contains information about which hosted zones shall be included/excluded for this provider. Deprecated: This field is deprecated and will be removed in a future release. Please use the DNS extension provider config (e.g. shoot-dns-service) for additional configuration.

    DataVolume

    (Appears on: Worker)

    DataVolume contains information about a data volume.

    FieldDescription
    name
    string

    Name of the volume to make it referencable.

    type
    string
    (Optional)

    Type is the type of the volume.

    size
    string

    VolumeSize is the size of the volume.

    encrypted
    bool
    (Optional)

    Encrypted determines if the volume should be encrypted.

    DeploymentRef

    (Appears on: ControllerRegistrationDeployment)

    DeploymentRef contains information about ControllerDeployment references.

    FieldDescription
    name
    string

    Name is the name of the ControllerDeployment that is being referred to.

    DualApprovalForDeletion

    (Appears on: ProjectSpec)

    DualApprovalForDeletion contains configuration for the dual approval concept for resource deletion.

    FieldDescription
    resource
    string

    Resource is the name of the resource this applies to.

    selector
    Kubernetes meta/v1.LabelSelector

    Selector is the label selector for the resources.

    includeServiceAccounts
    bool
    (Optional)

    IncludeServiceAccounts specifies whether the concept also applies when deletion is triggered by ServiceAccounts. Defaults to true.

    ETCDEncryptionKeyRotation

    (Appears on: ShootCredentialsRotation)

    ETCDEncryptionKeyRotation contains information about the ETCD encryption key credential rotation.

    FieldDescription
    phase
    CredentialsRotationPhase

    Phase describes the phase of the ETCD encryption key credential rotation.

    lastCompletionTime
    Kubernetes meta/v1.Time
    (Optional)

    LastCompletionTime is the most recent time when the ETCD encryption key credential rotation was successfully completed.

    lastInitiationTime
    Kubernetes meta/v1.Time
    (Optional)

    LastInitiationTime is the most recent time when the ETCD encryption key credential rotation was initiated.

    lastInitiationFinishedTime
    Kubernetes meta/v1.Time
    (Optional)

    LastInitiationFinishedTime is the recent time when the certificate authority credential rotation initiation was completed.

    lastCompletionTriggeredTime
    Kubernetes meta/v1.Time
    (Optional)

    LastCompletionTriggeredTime is the recent time when the certificate authority credential rotation completion was triggered.

    EncryptionConfig

    (Appears on: KubeAPIServerConfig)

    EncryptionConfig contains customizable encryption configuration of the API server.

    FieldDescription
    resources
    []string

    Resources contains the list of resources that shall be encrypted in addition to secrets. Each item is a Kubernetes resource name in plural (resource or resource.group) that should be encrypted. Note that configuring a custom resource is only supported for versions >= 1.26. Wildcards are not supported for now. See https://github.com/gardener/gardener/blob/master/docs/usage/etcd_encryption_config.md for more details.

    ErrorCode (string alias)

    (Appears on: Condition, LastError)

    ErrorCode is a string alias.

    ExpanderMode (string alias)

    (Appears on: ClusterAutoscaler)

    ExpanderMode is type used for Expander values

    ExpirableVersion

    (Appears on: KubernetesSettings, MachineImageVersion)

    ExpirableVersion contains a version and an expiration date.

    FieldDescription
    version
    string

    Version is the version identifier.

    expirationDate
    Kubernetes meta/v1.Time
    (Optional)

    ExpirationDate defines the time at which this version expires.

    classification
    VersionClassification
    (Optional)

    Classification defines the state of a version (preview, supported, deprecated)

    ExposureClassScheduling

    (Appears on: ExposureClass)

    ExposureClassScheduling holds information to select applicable Seed’s for ExposureClass usage.

    FieldDescription
    seedSelector
    SeedSelector
    (Optional)

    SeedSelector is an optional label selector for Seed’s which are suitable to use the ExposureClass.

    tolerations
    []Toleration
    (Optional)

    Tolerations contains the tolerations for taints on Seed clusters.

    Extension

    (Appears on: ShootSpec)

    Extension contains type and provider information for Shoot extensions.

    FieldDescription
    type
    string

    Type is the type of the extension resource.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig is the configuration passed to extension resource.

    disabled
    bool
    (Optional)

    Disabled allows to disable extensions that were marked as ‘globally enabled’ by Gardener administrators.

    ExtensionResourceState

    (Appears on: ShootStateSpec)

    ExtensionResourceState contains the kind of the extension custom resource and its last observed state in the Shoot’s namespace on the Seed cluster.

    FieldDescription
    kind
    string

    Kind (type) of the extension custom resource

    name
    string
    (Optional)

    Name of the extension custom resource

    purpose
    string
    (Optional)

    Purpose of the extension custom resource

    state
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    State of the extension resource

    resources
    []NamedResourceReference
    (Optional)

    Resources holds a list of named resource references that can be referred to in the state by their names.

    FailureTolerance

    (Appears on: HighAvailability)

    FailureTolerance describes information about failure tolerance level of a highly available resource.

    FieldDescription
    type
    FailureToleranceType

    Type specifies the type of failure that the highly available resource can tolerate

    FailureToleranceType (string alias)

    (Appears on: FailureTolerance)

    FailureToleranceType specifies the type of failure that a highly available shoot control plane that can tolerate.

    Gardener

    (Appears on: SeedStatus, ShootStatus)

    Gardener holds the information about the Gardener version that operated a resource.

    FieldDescription
    id
    string

    ID is the container id of the Gardener which last acted on a resource.

    name
    string

    Name is the hostname (pod name) of the Gardener which last acted on a resource.

    version
    string

    Version is the version of the Gardener which last acted on a resource.

    GardenerResourceData

    (Appears on: ShootStateSpec)

    GardenerResourceData holds the data which is used to generate resources, deployed in the Shoot’s control plane.

    FieldDescription
    name
    string

    Name of the object required to generate resources

    type
    string

    Type of the object

    data
    k8s.io/apimachinery/pkg/runtime.RawExtension

    Data contains the payload required to generate resources

    labels
    map[string]string
    (Optional)

    Labels are labels of the object

    HelmControllerDeployment

    HelmControllerDeployment configures how an extension controller is deployed using helm. This is the legacy structure that used to be defined in gardenlet’s ControllerInstallation controller for ControllerDeployment’s with type=helm. While this is not a proper API type, we need to define the structure in the API package so that we can convert it to the internal API version in the new representation.

    FieldDescription
    chart
    []byte

    Chart is a Helm chart tarball.

    values
    Kubernetes apiextensions/v1.JSON

    Values is a map of values for the given chart.

    Hibernation

    (Appears on: ShootSpec)

    Hibernation contains information whether the Shoot is suspended or not.

    FieldDescription
    enabled
    bool
    (Optional)

    Enabled specifies whether the Shoot needs to be hibernated or not. If it is true, the Shoot’s desired state is to be hibernated. If it is false or nil, the Shoot’s desired state is to be awakened.

    schedules
    []HibernationSchedule
    (Optional)

    Schedules determine the hibernation schedules.

    HibernationSchedule

    (Appears on: Hibernation)

    HibernationSchedule determines the hibernation schedule of a Shoot. A Shoot will be regularly hibernated at each start time and will be woken up at each end time. Start or End can be omitted, though at least one of each has to be specified.

    FieldDescription
    start
    string
    (Optional)

    Start is a Cron spec at which time a Shoot will be hibernated.

    end
    string
    (Optional)

    End is a Cron spec at which time a Shoot will be woken up.

    location
    string
    (Optional)

    Location is the time location in which both start and shall be evaluated.

    HighAvailability

    (Appears on: ControlPlane)

    HighAvailability specifies the configuration settings for high availability for a resource. Typical usages could be to configure HA for shoot control plane or for seed system components.

    FieldDescription
    failureTolerance
    FailureTolerance

    FailureTolerance holds information about failure tolerance level of a highly available resource.

    HorizontalPodAutoscalerConfig

    (Appears on: KubeControllerManagerConfig)

    HorizontalPodAutoscalerConfig contains horizontal pod autoscaler configuration settings for the kube-controller-manager. Note: Descriptions were taken from the Kubernetes documentation.

    FieldDescription
    cpuInitializationPeriod
    Kubernetes meta/v1.Duration
    (Optional)

    The period after which a ready pod transition is considered to be the first.

    downscaleStabilization
    Kubernetes meta/v1.Duration
    (Optional)

    The configurable window at which the controller will choose the highest recommendation for autoscaling.

    initialReadinessDelay
    Kubernetes meta/v1.Duration
    (Optional)

    The configurable period at which the horizontal pod autoscaler considers a Pod “not yet ready” given that it’s unready and it has transitioned to unready during that time.

    syncPeriod
    Kubernetes meta/v1.Duration
    (Optional)

    The period for syncing the number of pods in horizontal pod autoscaler.

    tolerance
    float64
    (Optional)

    The minimum change (from 1.0) in the desired-to-actual metrics ratio for the horizontal pod autoscaler to consider scaling.

    IPFamily (string alias)

    (Appears on: Networking, SeedNetworks)

    IPFamily is a type for specifying an IP protocol version to use in Gardener clusters.

    Ingress

    (Appears on: SeedSpec)

    Ingress configures the Ingress specific settings of the cluster

    FieldDescription
    domain
    string

    Domain specifies the IngressDomain of the cluster pointing to the ingress controller endpoint. It will be used to construct ingress URLs for system applications running in Shoot/Garden clusters. Once set this field is immutable.

    controller
    IngressController

    Controller configures a Gardener managed Ingress Controller listening on the ingressDomain

    IngressController

    (Appears on: Ingress)

    IngressController enables a Gardener managed Ingress Controller listening on the ingressDomain

    FieldDescription
    kind
    string

    Kind defines which kind of IngressController to use. At the moment only nginx is supported

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig specifies infrastructure specific configuration for the ingressController

    KubeAPIServerConfig

    (Appears on: Kubernetes)

    KubeAPIServerConfig contains configuration settings for the kube-apiserver.

    FieldDescription
    KubernetesConfig
    KubernetesConfig

    (Members of KubernetesConfig are embedded into this type.)

    admissionPlugins
    []AdmissionPlugin
    (Optional)

    AdmissionPlugins contains the list of user-defined admission plugins (additional to those managed by Gardener), and, if desired, the corresponding configuration.

    apiAudiences
    []string
    (Optional)

    APIAudiences are the identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. Defaults to [“kubernetes”].

    auditConfig
    AuditConfig
    (Optional)

    AuditConfig contains configuration settings for the audit of the kube-apiserver.

    oidcConfig
    OIDCConfig
    (Optional)

    OIDCConfig contains configuration settings for the OIDC provider.

    runtimeConfig
    map[string]bool
    (Optional)

    RuntimeConfig contains information about enabled or disabled APIs.

    serviceAccountConfig
    ServiceAccountConfig
    (Optional)

    ServiceAccountConfig contains configuration settings for the service account handling of the kube-apiserver.

    watchCacheSizes
    WatchCacheSizes
    (Optional)

    WatchCacheSizes contains configuration of the API server’s watch cache sizes. Configuring these flags might be useful for large-scale Shoot clusters with a lot of parallel update requests and a lot of watching controllers (e.g. large ManagedSeed clusters). When the API server’s watch cache’s capacity is too small to cope with the amount of update requests and watchers for a particular resource, it might happen that controller watches are permanently stopped with too old resource version errors. Starting from kubernetes v1.19, the API server’s watch cache size is adapted dynamically and setting the watch cache size flags will have no effect, except when setting it to 0 (which disables the watch cache).

    requests
    APIServerRequests
    (Optional)

    Requests contains configuration for request-specific settings for the kube-apiserver.

    enableAnonymousAuthentication
    bool
    (Optional)

    EnableAnonymousAuthentication defines whether anonymous requests to the secure port of the API server should be allowed (flag --anonymous-auth). See: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

    eventTTL
    Kubernetes meta/v1.Duration
    (Optional)

    EventTTL controls the amount of time to retain events. Defaults to 1h.

    logging
    APIServerLogging
    (Optional)

    Logging contains configuration for the log level and HTTP access logs.

    defaultNotReadyTolerationSeconds
    int64
    (Optional)

    DefaultNotReadyTolerationSeconds indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration (flag --default-not-ready-toleration-seconds). The field has effect only when the DefaultTolerationSeconds admission plugin is enabled. Defaults to 300.

    defaultUnreachableTolerationSeconds
    int64
    (Optional)

    DefaultUnreachableTolerationSeconds indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration (flag --default-unreachable-toleration-seconds). The field has effect only when the DefaultTolerationSeconds admission plugin is enabled. Defaults to 300.

    encryptionConfig
    EncryptionConfig
    (Optional)

    EncryptionConfig contains customizable encryption configuration of the Kube API server.

    KubeControllerManagerConfig

    (Appears on: Kubernetes)

    KubeControllerManagerConfig contains configuration settings for the kube-controller-manager.

    FieldDescription
    KubernetesConfig
    KubernetesConfig

    (Members of KubernetesConfig are embedded into this type.)

    horizontalPodAutoscaler
    HorizontalPodAutoscalerConfig
    (Optional)

    HorizontalPodAutoscalerConfig contains horizontal pod autoscaler configuration settings for the kube-controller-manager.

    nodeCIDRMaskSize
    int32
    (Optional)

    NodeCIDRMaskSize defines the mask size for node cidr in cluster (default is 24). This field is immutable.

    podEvictionTimeout
    Kubernetes meta/v1.Duration
    (Optional)

    PodEvictionTimeout defines the grace period for deleting pods on failed nodes. Defaults to 2m.

    Deprecated: The corresponding kube-controller-manager flag --pod-eviction-timeout is deprecated in favor of the kube-apiserver flags --default-not-ready-toleration-seconds and --default-unreachable-toleration-seconds. The --pod-eviction-timeout flag does not have effect when the taint besed eviction is enabled. The taint based eviction is beta (enabled by default) since Kubernetes 1.13 and GA since Kubernetes 1.18. Hence, instead of setting this field, set the spec.kubernetes.kubeAPIServer.defaultNotReadyTolerationSeconds and spec.kubernetes.kubeAPIServer.defaultUnreachableTolerationSeconds.

    nodeMonitorGracePeriod
    Kubernetes meta/v1.Duration
    (Optional)

    NodeMonitorGracePeriod defines the grace period before an unresponsive node is marked unhealthy.

    KubeProxyConfig

    (Appears on: Kubernetes)

    KubeProxyConfig contains configuration settings for the kube-proxy.

    FieldDescription
    KubernetesConfig
    KubernetesConfig

    (Members of KubernetesConfig are embedded into this type.)

    mode
    ProxyMode
    (Optional)

    Mode specifies which proxy mode to use. defaults to IPTables.

    enabled
    bool
    (Optional)

    Enabled indicates whether kube-proxy should be deployed or not. Depending on the networking extensions switching kube-proxy off might be rejected. Consulting the respective documentation of the used networking extension is recommended before using this field. defaults to true if not specified.

    KubeSchedulerConfig

    (Appears on: Kubernetes)

    KubeSchedulerConfig contains configuration settings for the kube-scheduler.

    FieldDescription
    KubernetesConfig
    KubernetesConfig

    (Members of KubernetesConfig are embedded into this type.)

    kubeMaxPDVols
    string
    (Optional)

    KubeMaxPDVols allows to configure the KUBE_MAX_PD_VOLS environment variable for the kube-scheduler. Please find more information here: https://kubernetes.io/docs/concepts/storage/storage-limits/#custom-limits Note that using this field is considered alpha-/experimental-level and is on your own risk. You should be aware of all the side-effects and consequences when changing it.

    profile
    SchedulingProfile
    (Optional)

    Profile configures the scheduling profile for the cluster. If not specified, the used profile is “balanced” (provides the default kube-scheduler behavior).

    KubeletConfig

    (Appears on: Kubernetes, WorkerKubernetes)

    KubeletConfig contains configuration settings for the kubelet.

    FieldDescription
    KubernetesConfig
    KubernetesConfig

    (Members of KubernetesConfig are embedded into this type.)

    cpuCFSQuota
    bool
    (Optional)

    CPUCFSQuota allows you to disable/enable CPU throttling for Pods.

    cpuManagerPolicy
    string
    (Optional)

    CPUManagerPolicy allows to set alternative CPU management policies (default: none).

    evictionHard
    KubeletConfigEviction
    (Optional)

    EvictionHard describes a set of eviction thresholds (e.g. memory.available<1Gi) that if met would trigger a Pod eviction. Default: memory.available: “100Mi/1Gi/5%” nodefs.available: “5%” nodefs.inodesFree: “5%” imagefs.available: “5%” imagefs.inodesFree: “5%”

    evictionMaxPodGracePeriod
    int32
    (Optional)

    EvictionMaxPodGracePeriod describes the maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. Default: 90

    evictionMinimumReclaim
    KubeletConfigEvictionMinimumReclaim
    (Optional)

    EvictionMinimumReclaim configures the amount of resources below the configured eviction threshold that the kubelet attempts to reclaim whenever the kubelet observes resource pressure. Default: 0 for each resource

    evictionPressureTransitionPeriod
    Kubernetes meta/v1.Duration
    (Optional)

    EvictionPressureTransitionPeriod is the duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. Default: 4m0s

    evictionSoft
    KubeletConfigEviction
    (Optional)

    EvictionSoft describes a set of eviction thresholds (e.g. memory.available<1.5Gi) that if met over a corresponding grace period would trigger a Pod eviction. Default: memory.available: “200Mi/1.5Gi/10%” nodefs.available: “10%” nodefs.inodesFree: “10%” imagefs.available: “10%” imagefs.inodesFree: “10%”

    evictionSoftGracePeriod
    KubeletConfigEvictionSoftGracePeriod
    (Optional)

    EvictionSoftGracePeriod describes a set of eviction grace periods (e.g. memory.available=1m30s) that correspond to how long a soft eviction threshold must hold before triggering a Pod eviction. Default: memory.available: 1m30s nodefs.available: 1m30s nodefs.inodesFree: 1m30s imagefs.available: 1m30s imagefs.inodesFree: 1m30s

    maxPods
    int32
    (Optional)

    MaxPods is the maximum number of Pods that are allowed by the Kubelet. Default: 110

    podPidsLimit
    int64
    (Optional)

    PodPIDsLimit is the maximum number of process IDs per pod allowed by the kubelet.

    failSwapOn
    bool
    (Optional)

    FailSwapOn makes the Kubelet fail to start if swap is enabled on the node. (default true).

    kubeReserved
    KubeletConfigReserved
    (Optional)

    KubeReserved is the configuration for resources reserved for kubernetes node components (mainly kubelet and container runtime). When updating these values, be aware that cgroup resizes may not succeed on active worker nodes. Look for the NodeAllocatableEnforced event to determine if the configuration was applied. Default: cpu=80m,memory=1Gi,pid=20k

    systemReserved
    KubeletConfigReserved
    (Optional)

    SystemReserved is the configuration for resources reserved for system processes not managed by kubernetes (e.g. journald). When updating these values, be aware that cgroup resizes may not succeed on active worker nodes. Look for the NodeAllocatableEnforced event to determine if the configuration was applied.

    imageGCHighThresholdPercent
    int32
    (Optional)

    ImageGCHighThresholdPercent describes the percent of the disk usage which triggers image garbage collection. Default: 50

    imageGCLowThresholdPercent
    int32
    (Optional)

    ImageGCLowThresholdPercent describes the percent of the disk to which garbage collection attempts to free. Default: 40

    serializeImagePulls
    bool
    (Optional)

    SerializeImagePulls describes whether the images are pulled one at a time. Default: true

    registryPullQPS
    int32
    (Optional)

    RegistryPullQPS is the limit of registry pulls per second. The value must not be a negative number. Setting it to 0 means no limit. Default: 5

    registryBurst
    int32
    (Optional)

    RegistryBurst is the maximum size of bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registryPullQPS. The value must not be a negative number. Only used if registryPullQPS is greater than 0. Default: 10

    seccompDefault
    bool
    (Optional)

    SeccompDefault enables the use of RuntimeDefault as the default seccomp profile for all workloads. This requires the corresponding SeccompDefault feature gate to be enabled as well. This field is only available for Kubernetes v1.25 or later.

    containerLogMaxSize
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    A quantity defines the maximum size of the container log file before it is rotated. For example: “5Mi” or “256Ki”. Default: 100Mi

    containerLogMaxFiles
    int32
    (Optional)

    Maximum number of container log files that can be present for a container.

    protectKernelDefaults
    bool
    (Optional)

    ProtectKernelDefaults ensures that the kernel tunables are equal to the kubelet defaults. Defaults to true for Kubernetes v1.26 or later.

    streamingConnectionIdleTimeout
    Kubernetes meta/v1.Duration
    (Optional)

    StreamingConnectionIdleTimeout is the maximum time a streaming connection can be idle before the connection is automatically closed. This field cannot be set lower than “30s” or greater than “4h”. Default: “4h” for Kubernetes < v1.26. “5m” for Kubernetes >= v1.26.

    memorySwap
    MemorySwapConfiguration
    (Optional)

    MemorySwap configures swap memory available to container workloads.

    KubeletConfigEviction

    (Appears on: KubeletConfig)

    KubeletConfigEviction contains kubelet eviction thresholds supporting either a resource.Quantity or a percentage based value.

    FieldDescription
    memoryAvailable
    string
    (Optional)

    MemoryAvailable is the threshold for the free memory on the host server.

    imageFSAvailable
    string
    (Optional)

    ImageFSAvailable is the threshold for the free disk space in the imagefs filesystem (docker images and container writable layers).

    imageFSInodesFree
    string
    (Optional)

    ImageFSInodesFree is the threshold for the available inodes in the imagefs filesystem.

    nodeFSAvailable
    string
    (Optional)

    NodeFSAvailable is the threshold for the free disk space in the nodefs filesystem (docker volumes, logs, etc).

    nodeFSInodesFree
    string
    (Optional)

    NodeFSInodesFree is the threshold for the available inodes in the nodefs filesystem.

    KubeletConfigEvictionMinimumReclaim

    (Appears on: KubeletConfig)

    KubeletConfigEvictionMinimumReclaim contains configuration for the kubelet eviction minimum reclaim.

    FieldDescription
    memoryAvailable
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    MemoryAvailable is the threshold for the memory reclaim on the host server.

    imageFSAvailable
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    ImageFSAvailable is the threshold for the disk space reclaim in the imagefs filesystem (docker images and container writable layers).

    imageFSInodesFree
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    ImageFSInodesFree is the threshold for the inodes reclaim in the imagefs filesystem.

    nodeFSAvailable
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    NodeFSAvailable is the threshold for the disk space reclaim in the nodefs filesystem (docker volumes, logs, etc).

    nodeFSInodesFree
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    NodeFSInodesFree is the threshold for the inodes reclaim in the nodefs filesystem.

    KubeletConfigEvictionSoftGracePeriod

    (Appears on: KubeletConfig)

    KubeletConfigEvictionSoftGracePeriod contains grace periods for kubelet eviction thresholds.

    FieldDescription
    memoryAvailable
    Kubernetes meta/v1.Duration
    (Optional)

    MemoryAvailable is the grace period for the MemoryAvailable eviction threshold.

    imageFSAvailable
    Kubernetes meta/v1.Duration
    (Optional)

    ImageFSAvailable is the grace period for the ImageFSAvailable eviction threshold.

    imageFSInodesFree
    Kubernetes meta/v1.Duration
    (Optional)

    ImageFSInodesFree is the grace period for the ImageFSInodesFree eviction threshold.

    nodeFSAvailable
    Kubernetes meta/v1.Duration
    (Optional)

    NodeFSAvailable is the grace period for the NodeFSAvailable eviction threshold.

    nodeFSInodesFree
    Kubernetes meta/v1.Duration
    (Optional)

    NodeFSInodesFree is the grace period for the NodeFSInodesFree eviction threshold.

    KubeletConfigReserved

    (Appears on: KubeletConfig)

    KubeletConfigReserved contains reserved resources for daemons

    FieldDescription
    cpu
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    CPU is the reserved cpu.

    memory
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    Memory is the reserved memory.

    ephemeralStorage
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    EphemeralStorage is the reserved ephemeral-storage.

    pid
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    PID is the reserved process-ids.

    Kubernetes

    (Appears on: ShootSpec)

    Kubernetes contains the version and configuration variables for the Shoot control plane.

    FieldDescription
    clusterAutoscaler
    ClusterAutoscaler
    (Optional)

    ClusterAutoscaler contains the configuration flags for the Kubernetes cluster autoscaler.

    kubeAPIServer
    KubeAPIServerConfig
    (Optional)

    KubeAPIServer contains configuration settings for the kube-apiserver.

    kubeControllerManager
    KubeControllerManagerConfig
    (Optional)

    KubeControllerManager contains configuration settings for the kube-controller-manager.

    kubeScheduler
    KubeSchedulerConfig
    (Optional)

    KubeScheduler contains configuration settings for the kube-scheduler.

    kubeProxy
    KubeProxyConfig
    (Optional)

    KubeProxy contains configuration settings for the kube-proxy.

    kubelet
    KubeletConfig
    (Optional)

    Kubelet contains configuration settings for the kubelet.

    version
    string
    (Optional)

    Version is the semantic Kubernetes version to use for the Shoot cluster. Defaults to the highest supported minor and patch version given in the referenced cloud profile. The version can be omitted completely or partially specified, e.g. <major>.<minor>.

    verticalPodAutoscaler
    VerticalPodAutoscaler
    (Optional)

    VerticalPodAutoscaler contains the configuration flags for the Kubernetes vertical pod autoscaler.

    enableStaticTokenKubeconfig
    bool
    (Optional)

    EnableStaticTokenKubeconfig indicates whether static token kubeconfig secret will be created for the Shoot cluster. Defaults to true for Shoots with Kubernetes versions < 1.26. Defaults to false for Shoots with Kubernetes versions >= 1.26. Starting Kubernetes 1.27 the field will be locked to false.

    KubernetesConfig

    (Appears on: KubeAPIServerConfig, KubeControllerManagerConfig, KubeProxyConfig, KubeSchedulerConfig, KubeletConfig)

    KubernetesConfig contains common configuration fields for the control plane components.

    FieldDescription
    featureGates
    map[string]bool
    (Optional)

    FeatureGates contains information about enabled feature gates.

    KubernetesDashboard

    (Appears on: Addons)

    KubernetesDashboard describes configuration values for the kubernetes-dashboard addon.

    FieldDescription
    Addon
    Addon

    (Members of Addon are embedded into this type.)

    authenticationMode
    string
    (Optional)

    AuthenticationMode defines the authentication mode for the kubernetes-dashboard.

    KubernetesSettings

    (Appears on: CloudProfileSpec, NamespacedCloudProfileSpec)

    KubernetesSettings contains constraints regarding allowed values of the ‘kubernetes’ block in the Shoot specification.

    FieldDescription
    versions
    []ExpirableVersion
    (Optional)

    Versions is the list of allowed Kubernetes versions with optional expiration dates for Shoot clusters.

    LastError

    (Appears on: BackupBucketStatus, BackupEntryStatus, ShootStatus)

    LastError indicates the last occurred error for an operation on a resource.

    FieldDescription
    description
    string

    A human readable message indicating details about the last error.

    taskID
    string
    (Optional)

    ID of the task which caused this last error

    codes
    []ErrorCode
    (Optional)

    Well-defined error codes of the last error(s).

    lastUpdateTime
    Kubernetes meta/v1.Time
    (Optional)

    Last time the error was reported

    LastMaintenance

    (Appears on: ShootStatus)

    LastMaintenance holds information about a maintenance operation on the Shoot.

    FieldDescription
    description
    string

    A human-readable message containing details about the operations performed in the last maintenance.

    triggeredTime
    Kubernetes meta/v1.Time

    TriggeredTime is the time when maintenance was triggered.

    state
    LastOperationState

    Status of the last maintenance operation, one of Processing, Succeeded, Error.

    failureReason
    string
    (Optional)

    FailureReason holds the information about the last maintenance operation failure reason.

    LastOperation

    (Appears on: BackupBucketStatus, BackupEntryStatus, SeedStatus, ShootStatus)

    LastOperation indicates the type and the state of the last operation, along with a description message and a progress indicator.

    FieldDescription
    description
    string

    A human readable message indicating details about the last operation.

    lastUpdateTime
    Kubernetes meta/v1.Time

    Last time the operation state transitioned from one to another.

    progress
    int32

    The progress in percentage (0-100) of the last operation.

    state
    LastOperationState

    Status of the last operation, one of Aborted, Processing, Succeeded, Error, Failed.

    type
    LastOperationType

    Type of the last operation, one of Create, Reconcile, Delete, Migrate, Restore.

    LastOperationState (string alias)

    (Appears on: LastMaintenance, LastOperation)

    LastOperationState is a string alias.

    LastOperationType (string alias)

    (Appears on: LastOperation)

    LastOperationType is a string alias.

    Machine

    (Appears on: Worker)

    Machine contains information about the machine type and image.

    FieldDescription
    type
    string

    Type is the machine type of the worker group.

    image
    ShootMachineImage
    (Optional)

    Image holds information about the machine image to use for all nodes of this pool. It will default to the latest version of the first image stated in the referenced CloudProfile if no value has been provided.

    architecture
    string
    (Optional)

    Architecture is CPU architecture of machines in this worker pool.

    MachineControllerManagerSettings

    (Appears on: Worker)

    MachineControllerManagerSettings contains configurations for different worker-pools. Eg. MachineDrainTimeout, MachineHealthTimeout.

    FieldDescription
    machineDrainTimeout
    Kubernetes meta/v1.Duration
    (Optional)

    MachineDrainTimeout is the period after which machine is forcefully deleted.

    machineHealthTimeout
    Kubernetes meta/v1.Duration
    (Optional)

    MachineHealthTimeout is the period after which machine is declared failed.

    machineCreationTimeout
    Kubernetes meta/v1.Duration
    (Optional)

    MachineCreationTimeout is the period after which creation of the machine is declared failed.

    maxEvictRetries
    int32
    (Optional)

    MaxEvictRetries are the number of eviction retries on a pod after which drain is declared failed, and forceful deletion is triggered.

    nodeConditions
    []string
    (Optional)

    NodeConditions are the set of conditions if set to true for the period of MachineHealthTimeout, machine will be declared failed.

    MachineImage

    (Appears on: CloudProfileSpec, NamespacedCloudProfileSpec)

    MachineImage defines the name and multiple versions of the machine image in any environment.

    FieldDescription
    name
    string

    Name is the name of the image.

    versions
    []MachineImageVersion

    Versions contains versions, expiration dates and container runtimes of the machine image

    updateStrategy
    MachineImageUpdateStrategy
    (Optional)

    UpdateStrategy is the update strategy to use for the machine image. Possible values are: - patch: update to the latest patch version of the current minor version. - minor: update to the latest minor and patch version. - major: always update to the overall latest version (default).

    MachineImageUpdateStrategy (string alias)

    (Appears on: MachineImage)

    MachineImageUpdateStrategy is the update strategy to use for a machine image

    MachineImageVersion

    (Appears on: MachineImage)

    MachineImageVersion is an expirable version with list of supported container runtimes and interfaces

    FieldDescription
    ExpirableVersion
    ExpirableVersion

    (Members of ExpirableVersion are embedded into this type.)

    cri
    []CRI
    (Optional)

    CRI list of supported container runtime and interfaces supported by this version

    architectures
    []string
    (Optional)

    Architectures is the list of CPU architectures of the machine image in this version.

    kubeletVersionConstraint
    string
    (Optional)

    KubeletVersionConstraint is a constraint describing the supported kubelet versions by the machine image in this version. If the field is not specified, it is assumed that the machine image in this version supports all kubelet versions. Examples: - ‘>= 1.26’ - supports only kubelet versions greater than or equal to 1.26 - ‘< 1.26’ - supports only kubelet versions less than 1.26

    MachineType

    (Appears on: CloudProfileSpec, NamespacedCloudProfileSpec)

    MachineType contains certain properties of a machine type.

    FieldDescription
    cpu
    k8s.io/apimachinery/pkg/api/resource.Quantity

    CPU is the number of CPUs for this machine type.

    gpu
    k8s.io/apimachinery/pkg/api/resource.Quantity

    GPU is the number of GPUs for this machine type.

    memory
    k8s.io/apimachinery/pkg/api/resource.Quantity

    Memory is the amount of memory for this machine type.

    name
    string

    Name is the name of the machine type.

    storage
    MachineTypeStorage
    (Optional)

    Storage is the amount of storage associated with the root volume of this machine type.

    usable
    bool
    (Optional)

    Usable defines if the machine type can be used for shoot clusters.

    architecture
    string
    (Optional)

    Architecture is the CPU architecture of this machine type.

    MachineTypeStorage

    (Appears on: MachineType)

    MachineTypeStorage is the amount of storage associated with the root volume of this machine type.

    FieldDescription
    class
    string

    Class is the class of the storage type.

    size
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    StorageSize is the storage size.

    type
    string

    Type is the type of the storage.

    minSize
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    MinSize is the minimal supported storage size. This overrides any other common minimum size configuration from spec.volumeTypes[*].minSize.

    Maintenance

    (Appears on: ShootSpec)

    Maintenance contains information about the time window for maintenance operations and which operations should be performed.

    FieldDescription
    autoUpdate
    MaintenanceAutoUpdate
    (Optional)

    AutoUpdate contains information about which constraints should be automatically updated.

    timeWindow
    MaintenanceTimeWindow
    (Optional)

    TimeWindow contains information about the time window for maintenance operations.

    confineSpecUpdateRollout
    bool
    (Optional)

    ConfineSpecUpdateRollout prevents that changes/updates to the shoot specification will be rolled out immediately. Instead, they are rolled out during the shoot’s maintenance time window. There is one exception that will trigger an immediate roll out which is changes to the Spec.Hibernation.Enabled field.

    MaintenanceAutoUpdate

    (Appears on: Maintenance)

    MaintenanceAutoUpdate contains information about which constraints should be automatically updated.

    FieldDescription
    kubernetesVersion
    bool

    KubernetesVersion indicates whether the patch Kubernetes version may be automatically updated (default: true).

    machineImageVersion
    bool
    (Optional)

    MachineImageVersion indicates whether the machine image version may be automatically updated (default: true).

    MaintenanceTimeWindow

    (Appears on: Maintenance)

    MaintenanceTimeWindow contains information about the time window for maintenance operations.

    FieldDescription
    begin
    string

    Begin is the beginning of the time window in the format HHMMSS+ZONE, e.g. “220000+0100”. If not present, a random value will be computed.

    end
    string

    End is the end of the time window in the format HHMMSS+ZONE, e.g. “220000+0100”. If not present, the value will be computed based on the “Begin” value.

    MemorySwapConfiguration

    (Appears on: KubeletConfig)

    MemorySwapConfiguration contains kubelet swap configuration For more information, please see KEP: 2400-node-swap

    FieldDescription
    swapBehavior
    SwapBehavior
    (Optional)

    SwapBehavior configures swap memory available to container workloads. May be one of {“LimitedSwap”, “UnlimitedSwap”} defaults to: LimitedSwap

    Monitoring

    (Appears on: ShootSpec)

    Monitoring contains information about the monitoring configuration for the shoot.

    FieldDescription
    alerting
    Alerting
    (Optional)

    Alerting contains information about the alerting configuration for the shoot cluster.

    NamedResourceReference

    (Appears on: ExtensionResourceState, ShootSpec)

    NamedResourceReference is a named reference to a resource.

    FieldDescription
    name
    string

    Name of the resource reference.

    resourceRef
    Kubernetes autoscaling/v1.CrossVersionObjectReference

    ResourceRef is a reference to a resource.

    NamespacedCloudProfileSpec

    (Appears on: NamespacedCloudProfile)

    NamespacedCloudProfileSpec is the specification of a NamespacedCloudProfile.

    FieldDescription
    caBundle
    string
    (Optional)

    CABundle is a certificate bundle which will be installed onto every host machine of shoot cluster targeting this profile.

    kubernetes
    KubernetesSettings
    (Optional)

    Kubernetes contains constraints regarding allowed values of the ‘kubernetes’ block in the Shoot specification.

    machineImages
    []MachineImage
    (Optional)

    MachineImages contains constraints regarding allowed values for machine images in the Shoot specification.

    machineTypes
    []MachineType
    (Optional)

    MachineTypes contains constraints regarding allowed values for machine types in the ‘workers’ block in the Shoot specification.

    regions
    []Region
    (Optional)

    Regions contains constraints regarding allowed values for regions and zones.

    volumeTypes
    []VolumeType
    (Optional)

    VolumeTypes contains constraints regarding allowed values for volume types in the ‘workers’ block in the Shoot specification.

    parent
    CloudProfileReference

    Parent contains a reference to a CloudProfile it inherits from.

    NamespacedCloudProfileStatus

    (Appears on: NamespacedCloudProfile)

    NamespacedCloudProfileStatus holds the most recently observed status of the NamespacedCloudProfile.

    FieldDescription
    cloudProfileSpec
    CloudProfileSpec

    CloudProfile is the most recently generated CloudProfile of the NamespacedCloudProfile.

    observedGeneration
    int64
    (Optional)

    ObservedGeneration is the most recent generation observed for this project.

    Networking

    (Appears on: ShootSpec)

    Networking defines networking parameters for the shoot cluster.

    FieldDescription
    type
    string
    (Optional)

    Type identifies the type of the networking plugin. This field is immutable.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig is the configuration passed to network resource.

    pods
    string
    (Optional)

    Pods is the CIDR of the pod network. This field is immutable.

    nodes
    string
    (Optional)

    Nodes is the CIDR of the entire node network. This field is immutable if the feature gate MutableShootSpecNetworkingNodes is disabled.

    services
    string
    (Optional)

    Services is the CIDR of the service network. This field is immutable.

    ipFamilies
    []IPFamily
    (Optional)

    IPFamilies specifies the IP protocol versions to use for shoot networking. This field is immutable. See https://github.com/gardener/gardener/blob/master/docs/usage/ipv6.md. Defaults to [“IPv4”].

    NginxIngress

    (Appears on: Addons)

    NginxIngress describes configuration values for the nginx-ingress addon.

    FieldDescription
    Addon
    Addon

    (Members of Addon are embedded into this type.)

    loadBalancerSourceRanges
    []string
    (Optional)

    LoadBalancerSourceRanges is list of allowed IP sources for NginxIngress

    config
    map[string]string
    (Optional)

    Config contains custom configuration for the nginx-ingress-controller configuration. See https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/configmap.md#configuration-options

    externalTrafficPolicy
    Kubernetes core/v1.ServiceExternalTrafficPolicy
    (Optional)

    ExternalTrafficPolicy controls the .spec.externalTrafficPolicy value of the load balancer Service exposing the nginx-ingress. Defaults to Cluster.

    NodeLocalDNS

    (Appears on: SystemComponents)

    NodeLocalDNS contains the settings of the node local DNS components running in the data plane of the Shoot cluster.

    FieldDescription
    enabled
    bool

    Enabled indicates whether node local DNS is enabled or not.

    forceTCPToClusterDNS
    bool
    (Optional)

    ForceTCPToClusterDNS indicates whether the connection from the node local DNS to the cluster DNS (Core DNS) will be forced to TCP or not. Default, if unspecified, is to enforce TCP.

    forceTCPToUpstreamDNS
    bool
    (Optional)

    ForceTCPToUpstreamDNS indicates whether the connection from the node local DNS to the upstream DNS (infrastructure DNS) will be forced to TCP or not. Default, if unspecified, is to enforce TCP.

    disableForwardToUpstreamDNS
    bool
    (Optional)

    DisableForwardToUpstreamDNS indicates whether requests from node local DNS to upstream DNS should be disabled. Default, if unspecified, is to forward requests for external domains to upstream DNS

    OIDCConfig

    (Appears on: KubeAPIServerConfig)

    OIDCConfig contains configuration settings for the OIDC provider. Note: Descriptions were taken from the Kubernetes documentation.

    FieldDescription
    caBundle
    string
    (Optional)

    If set, the OpenID server’s certificate will be verified by one of the authorities in the oidc-ca-file, otherwise the host’s root CA set will be used.

    clientAuthentication
    OpenIDConnectClientAuthentication
    (Optional)

    ClientAuthentication can optionally contain client configuration used for kubeconfig generation.

    clientID
    string
    (Optional)

    The client ID for the OpenID Connect client, must be set if oidc-issuer-url is set.

    groupsClaim
    string
    (Optional)

    If provided, the name of a custom OpenID Connect claim for specifying user groups. The claim value is expected to be a string or array of strings. This flag is experimental, please see the authentication documentation for further details.

    groupsPrefix
    string
    (Optional)

    If provided, all groups will be prefixed with this value to prevent conflicts with other authentication strategies.

    issuerURL
    string
    (Optional)

    The URL of the OpenID issuer, only HTTPS scheme will be accepted. If set, it will be used to verify the OIDC JSON Web Token (JWT).

    requiredClaims
    map[string]string
    (Optional)

    key=value pairs that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value.

    signingAlgs
    []string
    (Optional)

    List of allowed JOSE asymmetric signing algorithms. JWTs with a ‘alg’ header value not in this list will be rejected. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1

    usernameClaim
    string
    (Optional)

    The OpenID claim to use as the user name. Note that claims other than the default (‘sub’) is not guaranteed to be unique and immutable. This flag is experimental, please see the authentication documentation for further details. (default “sub”)

    usernamePrefix
    string
    (Optional)

    If provided, all usernames will be prefixed with this value. If not provided, username claims other than ‘email’ are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value ‘-’.

    ObservabilityRotation

    (Appears on: ShootCredentialsRotation)

    ObservabilityRotation contains information about the observability credential rotation.

    FieldDescription
    lastInitiationTime
    Kubernetes meta/v1.Time
    (Optional)

    LastInitiationTime is the most recent time when the observability credential rotation was initiated.

    lastCompletionTime
    Kubernetes meta/v1.Time
    (Optional)

    LastCompletionTime is the most recent time when the observability credential rotation was successfully completed.

    OpenIDConnectClientAuthentication

    (Appears on: OIDCConfig)

    OpenIDConnectClientAuthentication contains configuration for OIDC clients.

    FieldDescription
    extraConfig
    map[string]string
    (Optional)

    Extra configuration added to kubeconfig’s auth-provider. Must not be any of idp-issuer-url, client-id, client-secret, idp-certificate-authority, idp-certificate-authority-data, id-token or refresh-token

    secret
    string
    (Optional)

    The client Secret for the OpenID Connect client.

    ProjectMember

    (Appears on: ProjectSpec)

    ProjectMember is a member of a project.

    FieldDescription
    Subject
    Kubernetes rbac/v1.Subject

    (Members of Subject are embedded into this type.)

    Subject is representing a user name, an email address, or any other identifier of a user, group, or service account that has a certain role.

    role
    string

    Role represents the role of this member. IMPORTANT: Be aware that this field will be removed in the v1 version of this API in favor of the roles list. TODO: Remove this field in favor of the roles list in v1.

    roles
    []string
    (Optional)

    Roles represents the list of roles of this member.

    ProjectPhase (string alias)

    (Appears on: ProjectStatus)

    ProjectPhase is a label for the condition of a project at the current time.

    ProjectSpec

    (Appears on: Project)

    ProjectSpec is the specification of a Project.

    FieldDescription
    createdBy
    Kubernetes rbac/v1.Subject
    (Optional)

    CreatedBy is a subject representing a user name, an email address, or any other identifier of a user who created the project. This field is immutable.

    description
    string
    (Optional)

    Description is a human-readable description of what the project is used for.

    owner
    Kubernetes rbac/v1.Subject
    (Optional)

    Owner is a subject representing a user name, an email address, or any other identifier of a user owning the project. IMPORTANT: Be aware that this field will be removed in the v1 version of this API in favor of the owner role. The only way to change the owner will be by moving the owner role. In this API version the only way to change the owner is to use this field. TODO: Remove this field in favor of the owner role in v1.

    purpose
    string
    (Optional)

    Purpose is a human-readable explanation of the project’s purpose.

    members
    []ProjectMember
    (Optional)

    Members is a list of subjects representing a user name, an email address, or any other identifier of a user, group, or service account that has a certain role.

    namespace
    string
    (Optional)

    Namespace is the name of the namespace that has been created for the Project object. A nil value means that Gardener will determine the name of the namespace. This field is immutable.

    tolerations
    ProjectTolerations
    (Optional)

    Tolerations contains the tolerations for taints on seed clusters.

    dualApprovalForDeletion
    []DualApprovalForDeletion
    (Optional)

    DualApprovalForDeletion contains configuration for the dual approval concept for resource deletion.

    ProjectStatus

    (Appears on: Project)

    ProjectStatus holds the most recently observed status of the project.

    FieldDescription
    observedGeneration
    int64
    (Optional)

    ObservedGeneration is the most recent generation observed for this project.

    phase
    ProjectPhase

    Phase is the current phase of the project.

    staleSinceTimestamp
    Kubernetes meta/v1.Time
    (Optional)

    StaleSinceTimestamp contains the timestamp when the project was first discovered to be stale/unused.

    staleAutoDeleteTimestamp
    Kubernetes meta/v1.Time
    (Optional)

    StaleAutoDeleteTimestamp contains the timestamp when the project will be garbage-collected/automatically deleted because it’s stale/unused.

    lastActivityTimestamp
    Kubernetes meta/v1.Time
    (Optional)

    LastActivityTimestamp contains the timestamp from the last activity performed in this project.

    ProjectTolerations

    (Appears on: ProjectSpec)

    ProjectTolerations contains the tolerations for taints on seed clusters.

    FieldDescription
    defaults
    []Toleration
    (Optional)

    Defaults contains a list of tolerations that are added to the shoots in this project by default.

    whitelist
    []Toleration
    (Optional)

    Whitelist contains a list of tolerations that are allowed to be added to the shoots in this project. Please note that this list may only be added by users having the spec-tolerations-whitelist verb for project resources.

    Provider

    (Appears on: ShootSpec)

    Provider contains provider-specific information that are handed-over to the provider-specific extension controller.

    FieldDescription
    type
    string

    Type is the type of the provider. This field is immutable.

    controlPlaneConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ControlPlaneConfig contains the provider-specific control plane config blob. Please look up the concrete definition in the documentation of your provider extension.

    infrastructureConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    InfrastructureConfig contains the provider-specific infrastructure config blob. Please look up the concrete definition in the documentation of your provider extension.

    workers
    []Worker
    (Optional)

    Workers is a list of worker groups.

    workersSettings
    WorkersSettings
    (Optional)

    WorkersSettings contains settings for all workers.

    ProxyMode (string alias)

    (Appears on: KubeProxyConfig)

    ProxyMode available in Linux platform: ‘userspace’ (older, going to be EOL), ‘iptables’ (newer, faster), ‘ipvs’ (newest, better in performance and scalability). As of now only ‘iptables’ and ‘ipvs’ is supported by Gardener. In Linux platform, if the iptables proxy is selected, regardless of how, but the system’s kernel or iptables versions are insufficient, this always falls back to the userspace proxy. IPVS mode will be enabled when proxy mode is set to ‘ipvs’, and the fall back path is firstly iptables and then userspace.

    QuotaSpec

    (Appears on: Quota)

    QuotaSpec is the specification of a Quota.

    FieldDescription
    clusterLifetimeDays
    int32
    (Optional)

    ClusterLifetimeDays is the lifetime of a Shoot cluster in days before it will be terminated automatically.

    metrics
    Kubernetes core/v1.ResourceList

    Metrics is a list of resources which will be put under constraints.

    scope
    Kubernetes core/v1.ObjectReference

    Scope is the scope of the Quota object, either ‘project’ or ‘secret’. This field is immutable.

    Region

    (Appears on: CloudProfileSpec, NamespacedCloudProfileSpec)

    Region contains certain properties of a region.

    FieldDescription
    name
    string

    Name is a region name.

    zones
    []AvailabilityZone
    (Optional)

    Zones is a list of availability zones in this region.

    labels
    map[string]string
    (Optional)

    Labels is an optional set of key-value pairs that contain certain administrator-controlled labels for this region. It can be used by Gardener administrators/operators to provide additional information about a region, e.g. wrt quality, reliability, access restrictions, etc.

    ResourceData

    (Appears on: ShootStateSpec)

    ResourceData holds the data of a resource referred to by an extension controller state.

    FieldDescription
    CrossVersionObjectReference
    Kubernetes autoscaling/v1.CrossVersionObjectReference

    (Members of CrossVersionObjectReference are embedded into this type.)

    data
    k8s.io/apimachinery/pkg/runtime.RawExtension

    Data of the resource

    ResourceWatchCacheSize

    (Appears on: WatchCacheSizes)

    ResourceWatchCacheSize contains configuration of the API server’s watch cache size for one specific resource.

    FieldDescription
    apiGroup
    string
    (Optional)

    APIGroup is the API group of the resource for which the watch cache size should be configured. An unset value is used to specify the legacy core API (e.g. for secrets).

    resource
    string

    Resource is the name of the resource for which the watch cache size should be configured (in lowercase plural form, e.g. secrets).

    size
    int32

    CacheSize specifies the watch cache size that should be configured for the specified resource.

    SSHAccess

    (Appears on: WorkersSettings)

    SSHAccess contains settings regarding ssh access to the worker nodes.

    FieldDescription
    enabled
    bool

    Enabled indicates whether the SSH access to the worker nodes is ensured to be enabled or disabled in systemd. Defaults to true.

    SchedulingProfile (string alias)

    (Appears on: KubeSchedulerConfig)

    SchedulingProfile is a string alias used for scheduling profile values.

    SecretBindingProvider

    (Appears on: SecretBinding)

    SecretBindingProvider defines the provider type of the SecretBinding.

    FieldDescription
    type
    string

    Type is the type of the provider.

    For backwards compatibility, the field can contain multiple providers separated by a comma. However the usage of single SecretBinding (hence Secret) for different cloud providers is strongly discouraged.

    SeedBackup

    (Appears on: SeedSpec)

    SeedBackup contains the object store configuration for backups for shoot (currently only etcd).

    FieldDescription
    provider
    string

    Provider is a provider name. This field is immutable.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig is the configuration passed to BackupBucket resource.

    region
    string
    (Optional)

    Region is a region name. This field is immutable.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a Secret object containing the cloud provider credentials for the object store where backups should be stored. It should have enough privileges to manipulate the objects as well as buckets.

    SeedDNS

    (Appears on: SeedSpec)

    SeedDNS contains DNS-relevant information about this seed cluster.

    FieldDescription
    provider
    SeedDNSProvider
    (Optional)

    Provider configures a DNSProvider

    SeedDNSProvider

    (Appears on: SeedDNS)

    SeedDNSProvider configures a DNSProvider for Seeds

    FieldDescription
    type
    string

    Type describes the type of the dns-provider, for example aws-route53

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a Secret object containing cloud provider credentials used for registering external domains.

    SeedNetworks

    (Appears on: SeedSpec)

    SeedNetworks contains CIDRs for the pod, service and node networks of a Kubernetes cluster.

    FieldDescription
    nodes
    string
    (Optional)

    Nodes is the CIDR of the node network. This field is immutable.

    pods
    string

    Pods is the CIDR of the pod network. This field is immutable.

    services
    string

    Services is the CIDR of the service network. This field is immutable.

    shootDefaults
    ShootNetworks
    (Optional)

    ShootDefaults contains the default networks CIDRs for shoots.

    blockCIDRs
    []string
    (Optional)

    BlockCIDRs is a list of network addresses that should be blocked for shoot control plane components running in the seed cluster.

    ipFamilies
    []IPFamily
    (Optional)

    IPFamilies specifies the IP protocol versions to use for seed networking. This field is immutable. See https://github.com/gardener/gardener/blob/master/docs/usage/ipv6.md. Defaults to [“IPv4”].

    SeedProvider

    (Appears on: SeedSpec)

    SeedProvider defines the provider-specific information of this Seed cluster.

    FieldDescription
    type
    string

    Type is the name of the provider.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig is the configuration passed to Seed resource.

    region
    string

    Region is a name of a region.

    zones
    []string
    (Optional)

    Zones is the list of availability zones the seed cluster is deployed to.

    SeedSelector

    (Appears on: CloudProfileSpec, ExposureClassScheduling, ShootSpec)

    SeedSelector contains constraints for selecting seed to be usable for shoots using a profile

    FieldDescription
    LabelSelector
    Kubernetes meta/v1.LabelSelector

    (Members of LabelSelector are embedded into this type.)

    (Optional)

    LabelSelector is optional and can be used to select seeds by their label settings

    providerTypes
    []string
    (Optional)

    Providers is optional and can be used by restricting seeds by their provider type. ‘*’ can be used to enable seeds regardless of their provider type.

    SeedSettingDependencyWatchdog

    (Appears on: SeedSettings)

    SeedSettingDependencyWatchdog controls the dependency-watchdog settings for the seed.

    FieldDescription
    weeder
    SeedSettingDependencyWatchdogWeeder
    (Optional)

    Weeder controls the weeder settings for the dependency-watchdog for the seed.

    prober
    SeedSettingDependencyWatchdogProber
    (Optional)

    Prober controls the prober settings for the dependency-watchdog for the seed.

    SeedSettingDependencyWatchdogProber

    (Appears on: SeedSettingDependencyWatchdog)

    SeedSettingDependencyWatchdogProber controls the prober settings for the dependency-watchdog for the seed.

    FieldDescription
    enabled
    bool

    Enabled controls whether the probe controller(prober) of the dependency-watchdog should be enabled. This controller scales down the kube-controller-manager, machine-controller-manager and cluster-autoscaler of shoot clusters in case their respective kube-apiserver is not reachable via its external ingress in order to avoid melt-down situations.

    SeedSettingDependencyWatchdogWeeder

    (Appears on: SeedSettingDependencyWatchdog)

    SeedSettingDependencyWatchdogWeeder controls the weeder settings for the dependency-watchdog for the seed.

    FieldDescription
    enabled
    bool

    Enabled controls whether the endpoint controller(weeder) of the dependency-watchdog should be enabled. This controller helps to alleviate the delay where control plane components remain unavailable by finding the respective pods in CrashLoopBackoff status and restarting them once their dependants become ready and available again.

    SeedSettingExcessCapacityReservation

    (Appears on: SeedSettings)

    SeedSettingExcessCapacityReservation controls the excess capacity reservation for shoot control planes in the seed.

    FieldDescription
    enabled
    bool
    (Optional)

    Enabled controls whether the default excess capacity reservation should be enabled. When not specified, the functionality is enabled.

    configs
    []SeedSettingExcessCapacityReservationConfig
    (Optional)

    Configs configures excess capacity reservation deployments for shoot control planes in the seed.

    SeedSettingExcessCapacityReservationConfig

    (Appears on: SeedSettingExcessCapacityReservation)

    SeedSettingExcessCapacityReservationConfig configures excess capacity reservation deployments for shoot control planes in the seed.

    FieldDescription
    resources
    Kubernetes core/v1.ResourceList

    Resources specify the resource requests and limits of the excess-capacity-reservation pod.

    nodeSelector
    map[string]string
    (Optional)

    NodeSelector specifies the node where the excess-capacity-reservation pod should run.

    tolerations
    []Kubernetes core/v1.Toleration
    (Optional)

    Tolerations specify the tolerations for the the excess-capacity-reservation pod.

    SeedSettingLoadBalancerServices

    (Appears on: SeedSettings)

    SeedSettingLoadBalancerServices controls certain settings for services of type load balancer that are created in the seed.

    FieldDescription
    annotations
    map[string]string
    (Optional)

    Annotations is a map of annotations that will be injected/merged into every load balancer service object.

    externalTrafficPolicy
    Kubernetes core/v1.ServiceExternalTrafficPolicy
    (Optional)

    ExternalTrafficPolicy describes how nodes distribute service traffic they receive on one of the service’s “externally-facing” addresses. Defaults to “Cluster”.

    zones
    []SeedSettingLoadBalancerServicesZones
    (Optional)

    Zones controls settings, which are specific to the single-zone load balancers in a multi-zonal setup. Can be empty for single-zone seeds. Each specified zone has to relate to one of the zones in seed.spec.provider.zones.

    SeedSettingLoadBalancerServicesZones

    (Appears on: SeedSettingLoadBalancerServices)

    SeedSettingLoadBalancerServicesZones controls settings, which are specific to the single-zone load balancers in a multi-zonal setup.

    FieldDescription
    name
    string

    Name is the name of the zone as specified in seed.spec.provider.zones.

    annotations
    map[string]string
    (Optional)

    Annotations is a map of annotations that will be injected/merged into the zone-specific load balancer service object.

    externalTrafficPolicy
    Kubernetes core/v1.ServiceExternalTrafficPolicy
    (Optional)

    ExternalTrafficPolicy describes how nodes distribute service traffic they receive on one of the service’s “externally-facing” addresses. Defaults to “Cluster”.

    SeedSettingScheduling

    (Appears on: SeedSettings)

    SeedSettingScheduling controls settings for scheduling decisions for the seed.

    FieldDescription
    visible
    bool

    Visible controls whether the gardener-scheduler shall consider this seed when scheduling shoots. Invisible seeds are not considered by the scheduler.

    SeedSettingTopologyAwareRouting

    (Appears on: SeedSettings)

    SeedSettingTopologyAwareRouting controls certain settings for topology-aware traffic routing in the seed. See https://github.com/gardener/gardener/blob/master/docs/operations/topology_aware_routing.md.

    FieldDescription
    enabled
    bool

    Enabled controls whether certain Services deployed in the seed cluster should be topology-aware. These Services are etcd-main-client, etcd-events-client, kube-apiserver, gardener-resource-manager and vpa-webhook.

    SeedSettingVerticalPodAutoscaler

    (Appears on: SeedSettings)

    SeedSettingVerticalPodAutoscaler controls certain settings for the vertical pod autoscaler components deployed in the seed.

    FieldDescription
    enabled
    bool

    Enabled controls whether the VPA components shall be deployed into the garden namespace in the seed cluster. It is enabled by default because Gardener heavily relies on a VPA being deployed. You should only disable this if your seed cluster already has another, manually/custom managed VPA deployment.

    SeedSettings

    (Appears on: SeedSpec)

    SeedSettings contains certain settings for this seed cluster.

    FieldDescription
    excessCapacityReservation
    SeedSettingExcessCapacityReservation
    (Optional)

    ExcessCapacityReservation controls the excess capacity reservation for shoot control planes in the seed.

    scheduling
    SeedSettingScheduling
    (Optional)

    Scheduling controls settings for scheduling decisions for the seed.

    loadBalancerServices
    SeedSettingLoadBalancerServices
    (Optional)

    LoadBalancerServices controls certain settings for services of type load balancer that are created in the seed.

    verticalPodAutoscaler
    SeedSettingVerticalPodAutoscaler
    (Optional)

    VerticalPodAutoscaler controls certain settings for the vertical pod autoscaler components deployed in the seed.

    dependencyWatchdog
    SeedSettingDependencyWatchdog
    (Optional)

    DependencyWatchdog controls certain settings for the dependency-watchdog components deployed in the seed.

    topologyAwareRouting
    SeedSettingTopologyAwareRouting
    (Optional)

    TopologyAwareRouting controls certain settings for topology-aware traffic routing in the seed. See https://github.com/gardener/gardener/blob/master/docs/operations/topology_aware_routing.md.

    SeedSpec

    (Appears on: Seed, SeedTemplate)

    SeedSpec is the specification of a Seed.

    FieldDescription
    backup
    SeedBackup
    (Optional)

    Backup holds the object store configuration for the backups of shoot (currently only etcd). If it is not specified, then there won’t be any backups taken for shoots associated with this seed. If backup field is present in seed, then backups of the etcd from shoot control plane will be stored under the configured object store.

    dns
    SeedDNS

    DNS contains DNS-relevant information about this seed cluster.

    networks
    SeedNetworks

    Networks defines the pod, service and worker network of the Seed cluster.

    provider
    SeedProvider

    Provider defines the provider type and region for this Seed cluster.

    taints
    []SeedTaint
    (Optional)

    Taints describes taints on the seed.

    volume
    SeedVolume
    (Optional)

    Volume contains settings for persistentvolumes created in the seed cluster.

    settings
    SeedSettings
    (Optional)

    Settings contains certain settings for this seed cluster.

    ingress
    Ingress
    (Optional)

    Ingress configures Ingress specific settings of the Seed cluster. This field is immutable.

    SeedStatus

    (Appears on: Seed)

    SeedStatus is the status of a Seed.

    FieldDescription
    gardener
    Gardener
    (Optional)

    Gardener holds information about the Gardener which last acted on the Shoot.

    kubernetesVersion
    string
    (Optional)

    KubernetesVersion is the Kubernetes version of the seed cluster.

    conditions
    []Condition
    (Optional)

    Conditions represents the latest available observations of a Seed’s current state.

    observedGeneration
    int64
    (Optional)

    ObservedGeneration is the most recent generation observed for this Seed. It corresponds to the Seed’s generation, which is updated on mutation by the API Server.

    clusterIdentity
    string
    (Optional)

    ClusterIdentity is the identity of the Seed cluster. This field is immutable.

    capacity
    Kubernetes core/v1.ResourceList
    (Optional)

    Capacity represents the total resources of a seed.

    allocatable
    Kubernetes core/v1.ResourceList
    (Optional)

    Allocatable represents the resources of a seed that are available for scheduling. Defaults to Capacity.

    clientCertificateExpirationTimestamp
    Kubernetes meta/v1.Time
    (Optional)

    ClientCertificateExpirationTimestamp is the timestamp at which gardenlet’s client certificate expires.

    lastOperation
    LastOperation
    (Optional)

    LastOperation holds information about the last operation on the Seed.

    SeedTaint

    (Appears on: SeedSpec)

    SeedTaint describes a taint on a seed.

    FieldDescription
    key
    string

    Key is the taint key to be applied to a seed.

    value
    string
    (Optional)

    Value is the taint value corresponding to the taint key.

    SeedTemplate

    SeedTemplate is a template for creating a Seed object.

    FieldDescription
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    SeedSpec
    (Optional)

    Specification of the desired behavior of the Seed.



    backup
    SeedBackup
    (Optional)

    Backup holds the object store configuration for the backups of shoot (currently only etcd). If it is not specified, then there won’t be any backups taken for shoots associated with this seed. If backup field is present in seed, then backups of the etcd from shoot control plane will be stored under the configured object store.

    dns
    SeedDNS

    DNS contains DNS-relevant information about this seed cluster.

    networks
    SeedNetworks

    Networks defines the pod, service and worker network of the Seed cluster.

    provider
    SeedProvider

    Provider defines the provider type and region for this Seed cluster.

    taints
    []SeedTaint
    (Optional)

    Taints describes taints on the seed.

    volume
    SeedVolume
    (Optional)

    Volume contains settings for persistentvolumes created in the seed cluster.

    settings
    SeedSettings
    (Optional)

    Settings contains certain settings for this seed cluster.

    ingress
    Ingress
    (Optional)

    Ingress configures Ingress specific settings of the Seed cluster. This field is immutable.

    SeedVolume

    (Appears on: SeedSpec)

    SeedVolume contains settings for persistentvolumes created in the seed cluster.

    FieldDescription
    minimumSize
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    MinimumSize defines the minimum size that should be used for PVCs in the seed.

    providers
    []SeedVolumeProvider
    (Optional)

    Providers is a list of storage class provisioner types for the seed.

    SeedVolumeProvider

    (Appears on: SeedVolume)

    SeedVolumeProvider is a storage class provisioner type.

    FieldDescription
    purpose
    string

    Purpose is the purpose of this provider.

    name
    string

    Name is the name of the storage class provisioner type.

    ServiceAccountConfig

    (Appears on: KubeAPIServerConfig)

    ServiceAccountConfig is the kube-apiserver configuration for service accounts.

    FieldDescription
    issuer
    string
    (Optional)

    Issuer is the identifier of the service account token issuer. The issuer will assert this identifier in “iss” claim of issued tokens. This value is used to generate new service account tokens. This value is a string or URI. Defaults to URI of the API server.

    extendTokenExpiration
    bool
    (Optional)

    ExtendTokenExpiration turns on projected service account expiration extension during token generation, which helps safe transition from legacy token to bound service account token feature. If this flag is enabled, admission injected tokens would be extended up to 1 year to prevent unexpected failure during transition, ignoring value of service-account-max-token-expiration.

    maxTokenExpiration
    Kubernetes meta/v1.Duration
    (Optional)

    MaxTokenExpiration is the maximum validity duration of a token created by the service account token issuer. If an otherwise valid TokenRequest with a validity duration larger than this value is requested, a token will be issued with a validity duration of this value. This field must be within [30d,90d].

    acceptedIssuers
    []string
    (Optional)

    AcceptedIssuers is an additional set of issuers that are used to determine which service account tokens are accepted. These values are not used to generate new service account tokens. Only useful when service account tokens are also issued by another external system or a change of the current issuer that is used for generating tokens is being performed.

    ServiceAccountKeyRotation

    (Appears on: ShootCredentialsRotation)

    ServiceAccountKeyRotation contains information about the service account key credential rotation.

    FieldDescription
    phase
    CredentialsRotationPhase

    Phase describes the phase of the service account key credential rotation.

    lastCompletionTime
    Kubernetes meta/v1.Time
    (Optional)

    LastCompletionTime is the most recent time when the service account key credential rotation was successfully completed.

    lastInitiationTime
    Kubernetes meta/v1.Time
    (Optional)

    LastInitiationTime is the most recent time when the service account key credential rotation was initiated.

    lastInitiationFinishedTime
    Kubernetes meta/v1.Time
    (Optional)

    LastInitiationFinishedTime is the recent time when the certificate authority credential rotation initiation was completed.

    lastCompletionTriggeredTime
    Kubernetes meta/v1.Time
    (Optional)

    LastCompletionTriggeredTime is the recent time when the certificate authority credential rotation completion was triggered.

    ShootAdvertisedAddress

    (Appears on: ShootStatus)

    ShootAdvertisedAddress contains information for the shoot’s Kube API server.

    FieldDescription
    name
    string

    Name of the advertised address. e.g. external

    url
    string

    The URL of the API Server. e.g. https://api.foo.bar or https://1.2.3.4

    ShootCredentials

    (Appears on: ShootStatus)

    ShootCredentials contains information about the shoot credentials.

    FieldDescription
    rotation
    ShootCredentialsRotation
    (Optional)

    Rotation contains information about the credential rotations.

    ShootCredentialsRotation

    (Appears on: ShootCredentials)

    ShootCredentialsRotation contains information about the rotation of credentials.

    FieldDescription
    certificateAuthorities
    CARotation
    (Optional)

    CertificateAuthorities contains information about the certificate authority credential rotation.

    kubeconfig
    ShootKubeconfigRotation
    (Optional)

    Kubeconfig contains information about the kubeconfig credential rotation.

    sshKeypair
    ShootSSHKeypairRotation
    (Optional)

    SSHKeypair contains information about the ssh-keypair credential rotation.

    observability
    ObservabilityRotation
    (Optional)

    Observability contains information about the observability credential rotation.

    serviceAccountKey
    ServiceAccountKeyRotation
    (Optional)

    ServiceAccountKey contains information about the service account key credential rotation.

    etcdEncryptionKey
    ETCDEncryptionKeyRotation
    (Optional)

    ETCDEncryptionKey contains information about the ETCD encryption key credential rotation.

    ShootKubeconfigRotation

    (Appears on: ShootCredentialsRotation)

    ShootKubeconfigRotation contains information about the kubeconfig credential rotation.

    FieldDescription
    lastInitiationTime
    Kubernetes meta/v1.Time
    (Optional)

    LastInitiationTime is the most recent time when the kubeconfig credential rotation was initiated.

    lastCompletionTime
    Kubernetes meta/v1.Time
    (Optional)

    LastCompletionTime is the most recent time when the kubeconfig credential rotation was successfully completed.

    ShootMachineImage

    (Appears on: Machine)

    ShootMachineImage defines the name and the version of the shoot’s machine image in any environment. Has to be defined in the respective CloudProfile.

    FieldDescription
    name
    string

    Name is the name of the image.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig is the shoot’s individual configuration passed to an extension resource.

    version
    string
    (Optional)

    Version is the version of the shoot’s image. If version is not provided, it will be defaulted to the latest version from the CloudProfile.

    ShootNetworks

    (Appears on: SeedNetworks)

    ShootNetworks contains the default networks CIDRs for shoots.

    FieldDescription
    pods
    string
    (Optional)

    Pods is the CIDR of the pod network.

    services
    string
    (Optional)

    Services is the CIDR of the service network.

    ShootPurpose (string alias)

    (Appears on: ShootSpec)

    ShootPurpose is a type alias for string.

    ShootSSHKeypairRotation

    (Appears on: ShootCredentialsRotation)

    ShootSSHKeypairRotation contains information about the ssh-keypair credential rotation.

    FieldDescription
    lastInitiationTime
    Kubernetes meta/v1.Time
    (Optional)

    LastInitiationTime is the most recent time when the ssh-keypair credential rotation was initiated.

    lastCompletionTime
    Kubernetes meta/v1.Time
    (Optional)

    LastCompletionTime is the most recent time when the ssh-keypair credential rotation was successfully completed.

    ShootSpec

    (Appears on: Shoot, ShootTemplate)

    ShootSpec is the specification of a Shoot.

    FieldDescription
    addons
    Addons
    (Optional)

    Addons contains information about enabled/disabled addons and their configuration.

    cloudProfileName
    string

    CloudProfileName is a name of a CloudProfile object. This field is immutable.

    dns
    DNS
    (Optional)

    DNS contains information about the DNS settings of the Shoot.

    extensions
    []Extension
    (Optional)

    Extensions contain type and provider information for Shoot extensions.

    hibernation
    Hibernation
    (Optional)

    Hibernation contains information whether the Shoot is suspended or not.

    kubernetes
    Kubernetes

    Kubernetes contains the version and configuration settings of the control plane components.

    networking
    Networking
    (Optional)

    Networking contains information about cluster networking such as CNI Plugin type, CIDRs, …etc.

    maintenance
    Maintenance
    (Optional)

    Maintenance contains information about the time window for maintenance operations and which operations should be performed.

    monitoring
    Monitoring
    (Optional)

    Monitoring contains information about custom monitoring configurations for the shoot.

    provider
    Provider

    Provider contains all provider-specific and provider-relevant information.

    purpose
    ShootPurpose
    (Optional)

    Purpose is the purpose class for this cluster.

    region
    string

    Region is a name of a region. This field is immutable.

    secretBindingName
    string
    (Optional)

    SecretBindingName is the name of the a SecretBinding that has a reference to the provider secret. The credentials inside the provider secret will be used to create the shoot in the respective account. This field is immutable.

    seedName
    string
    (Optional)

    SeedName is the name of the seed cluster that runs the control plane of the Shoot.

    seedSelector
    SeedSelector
    (Optional)

    SeedSelector is an optional selector which must match a seed’s labels for the shoot to be scheduled on that seed.

    resources
    []NamedResourceReference
    (Optional)

    Resources holds a list of named resource references that can be referred to in extension configs by their names.

    tolerations
    []Toleration
    (Optional)

    Tolerations contains the tolerations for taints on seed clusters.

    exposureClassName
    string
    (Optional)

    ExposureClassName is the optional name of an exposure class to apply a control plane endpoint exposure strategy. This field is immutable.

    systemComponents
    SystemComponents
    (Optional)

    SystemComponents contains the settings of system components in the control or data plane of the Shoot cluster.

    controlPlane
    ControlPlane
    (Optional)

    ControlPlane contains general settings for the control plane of the shoot.

    schedulerName
    string
    (Optional)

    SchedulerName is the name of the responsible scheduler which schedules the shoot. If not specified, the default scheduler takes over. This field is immutable.

    cloudProfile
    CloudProfileReference
    (Optional)

    CloudProfile contains a reference to a CloudProfile or a NamespacedCloudProfile.

    ShootStateSpec

    (Appears on: ShootState)

    ShootStateSpec is the specification of the ShootState.

    FieldDescription
    gardener
    []GardenerResourceData
    (Optional)

    Gardener holds the data required to generate resources deployed by the gardenlet

    extensions
    []ExtensionResourceState
    (Optional)

    Extensions holds the state of custom resources reconciled by extension controllers in the seed

    resources
    []ResourceData
    (Optional)

    Resources holds the data of resources referred to by extension controller states

    ShootStatus

    (Appears on: Shoot)

    ShootStatus holds the most recently observed status of the Shoot cluster.

    FieldDescription
    conditions
    []Condition
    (Optional)

    Conditions represents the latest available observations of a Shoots’s current state.

    constraints
    []Condition
    (Optional)

    Constraints represents conditions of a Shoot’s current state that constraint some operations on it.

    gardener
    Gardener

    Gardener holds information about the Gardener which last acted on the Shoot.

    hibernated
    bool

    IsHibernated indicates whether the Shoot is currently hibernated.

    lastOperation
    LastOperation
    (Optional)

    LastOperation holds information about the last operation on the Shoot.

    lastErrors
    []LastError
    (Optional)

    LastErrors holds information about the last occurred error(s) during an operation.

    observedGeneration
    int64
    (Optional)

    ObservedGeneration is the most recent generation observed for this Shoot. It corresponds to the Shoot’s generation, which is updated on mutation by the API Server.

    retryCycleStartTime
    Kubernetes meta/v1.Time
    (Optional)

    RetryCycleStartTime is the start time of the last retry cycle (used to determine how often an operation must be retried until we give up).

    seedName
    string
    (Optional)

    SeedName is the name of the seed cluster that runs the control plane of the Shoot. This value is only written after a successful create/reconcile operation. It will be used when control planes are moved between Seeds.

    technicalID
    string

    TechnicalID is the name that is used for creating the Seed namespace, the infrastructure resources, and basically everything that is related to this particular Shoot. This field is immutable.

    uid
    k8s.io/apimachinery/pkg/types.UID

    UID is a unique identifier for the Shoot cluster to avoid portability between Kubernetes clusters. It is used to compute unique hashes. This field is immutable.

    clusterIdentity
    string
    (Optional)

    ClusterIdentity is the identity of the Shoot cluster. This field is immutable.

    advertisedAddresses
    []ShootAdvertisedAddress
    (Optional)

    List of addresses that are relevant to the shoot. These include the Kube API server address and also the service account issuer.

    migrationStartTime
    Kubernetes meta/v1.Time
    (Optional)

    MigrationStartTime is the time when a migration to a different seed was initiated.

    credentials
    ShootCredentials
    (Optional)

    Credentials contains information about the shoot credentials.

    lastHibernationTriggerTime
    Kubernetes meta/v1.Time
    (Optional)

    LastHibernationTriggerTime indicates the last time when the hibernation controller managed to change the hibernation settings of the cluster

    lastMaintenance
    LastMaintenance
    (Optional)

    LastMaintenance holds information about the last maintenance operations on the Shoot.

    encryptedResources
    []string
    (Optional)

    EncryptedResources is the list of resources in the Shoot which are currently encrypted. Secrets are encrypted by default and are not part of the list. See https://github.com/gardener/gardener/blob/master/docs/usage/etcd_encryption_config.md for more details.

    ShootTemplate

    ShootTemplate is a template for creating a Shoot object.

    FieldDescription
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional)

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ShootSpec
    (Optional)

    Specification of the desired behavior of the Shoot.



    addons
    Addons
    (Optional)

    Addons contains information about enabled/disabled addons and their configuration.

    cloudProfileName
    string

    CloudProfileName is a name of a CloudProfile object. This field is immutable.

    dns
    DNS
    (Optional)

    DNS contains information about the DNS settings of the Shoot.

    extensions
    []Extension
    (Optional)

    Extensions contain type and provider information for Shoot extensions.

    hibernation
    Hibernation
    (Optional)

    Hibernation contains information whether the Shoot is suspended or not.

    kubernetes
    Kubernetes

    Kubernetes contains the version and configuration settings of the control plane components.

    networking
    Networking
    (Optional)

    Networking contains information about cluster networking such as CNI Plugin type, CIDRs, …etc.

    maintenance
    Maintenance
    (Optional)

    Maintenance contains information about the time window for maintenance operations and which operations should be performed.

    monitoring
    Monitoring
    (Optional)

    Monitoring contains information about custom monitoring configurations for the shoot.

    provider
    Provider

    Provider contains all provider-specific and provider-relevant information.

    purpose
    ShootPurpose
    (Optional)

    Purpose is the purpose class for this cluster.

    region
    string

    Region is a name of a region. This field is immutable.

    secretBindingName
    string
    (Optional)

    SecretBindingName is the name of the a SecretBinding that has a reference to the provider secret. The credentials inside the provider secret will be used to create the shoot in the respective account. This field is immutable.

    seedName
    string
    (Optional)

    SeedName is the name of the seed cluster that runs the control plane of the Shoot.

    seedSelector
    SeedSelector
    (Optional)

    SeedSelector is an optional selector which must match a seed’s labels for the shoot to be scheduled on that seed.

    resources
    []NamedResourceReference
    (Optional)

    Resources holds a list of named resource references that can be referred to in extension configs by their names.

    tolerations
    []Toleration
    (Optional)

    Tolerations contains the tolerations for taints on seed clusters.

    exposureClassName
    string
    (Optional)

    ExposureClassName is the optional name of an exposure class to apply a control plane endpoint exposure strategy. This field is immutable.

    systemComponents
    SystemComponents
    (Optional)

    SystemComponents contains the settings of system components in the control or data plane of the Shoot cluster.

    controlPlane
    ControlPlane
    (Optional)

    ControlPlane contains general settings for the control plane of the shoot.

    schedulerName
    string
    (Optional)

    SchedulerName is the name of the responsible scheduler which schedules the shoot. If not specified, the default scheduler takes over. This field is immutable.

    cloudProfile
    CloudProfileReference
    (Optional)

    CloudProfile contains a reference to a CloudProfile or a NamespacedCloudProfile.

    SwapBehavior (string alias)

    (Appears on: MemorySwapConfiguration)

    SwapBehavior configures swap memory available to container workloads

    SystemComponents

    (Appears on: ShootSpec)

    SystemComponents contains the settings of system components in the control or data plane of the Shoot cluster.

    FieldDescription
    coreDNS
    CoreDNS
    (Optional)

    CoreDNS contains the settings of the Core DNS components running in the data plane of the Shoot cluster.

    nodeLocalDNS
    NodeLocalDNS
    (Optional)

    NodeLocalDNS contains the settings of the node local DNS components running in the data plane of the Shoot cluster.

    Toleration

    (Appears on: ExposureClassScheduling, ProjectTolerations, ShootSpec)

    Toleration is a toleration for a seed taint.

    FieldDescription
    key
    string

    Key is the toleration key to be applied to a project or shoot.

    value
    string
    (Optional)

    Value is the toleration value corresponding to the toleration key.

    VersionClassification (string alias)

    (Appears on: ExpirableVersion)

    VersionClassification is the logical state of a version.

    VerticalPodAutoscaler

    (Appears on: Kubernetes)

    VerticalPodAutoscaler contains the configuration flags for the Kubernetes vertical pod autoscaler.

    FieldDescription
    enabled
    bool

    Enabled specifies whether the Kubernetes VPA shall be enabled for the shoot cluster.

    evictAfterOOMThreshold
    Kubernetes meta/v1.Duration
    (Optional)

    EvictAfterOOMThreshold defines the threshold that will lead to pod eviction in case it OOMed in less than the given threshold since its start and if it has only one container (default: 10m0s).

    evictionRateBurst
    int32
    (Optional)

    EvictionRateBurst defines the burst of pods that can be evicted (default: 1)

    evictionRateLimit
    float64
    (Optional)

    EvictionRateLimit defines the number of pods that can be evicted per second. A rate limit set to 0 or -1 will disable the rate limiter (default: -1).

    evictionTolerance
    float64
    (Optional)

    EvictionTolerance defines the fraction of replica count that can be evicted for update in case more than one pod can be evicted (default: 0.5).

    recommendationMarginFraction
    float64
    (Optional)

    RecommendationMarginFraction is the fraction of usage added as the safety margin to the recommended request (default: 0.15).

    updaterInterval
    Kubernetes meta/v1.Duration
    (Optional)

    UpdaterInterval is the interval how often the updater should run (default: 1m0s).

    recommenderInterval
    Kubernetes meta/v1.Duration
    (Optional)

    RecommenderInterval is the interval how often metrics should be fetched (default: 1m0s).

    targetCPUPercentile
    float64
    (Optional)

    TargetCPUPercentile is the usage percentile that will be used as a base for CPU target recommendation. Doesn’t affect CPU lower bound, CPU upper bound nor memory recommendations. (default: 0.9)

    Volume

    (Appears on: Worker)

    Volume contains information about the volume type, size, and encryption.

    FieldDescription
    name
    string
    (Optional)

    Name of the volume to make it referencable.

    type
    string
    (Optional)

    Type is the type of the volume.

    size
    string

    VolumeSize is the size of the volume.

    encrypted
    bool
    (Optional)

    Encrypted determines if the volume should be encrypted.

    VolumeType

    (Appears on: CloudProfileSpec, NamespacedCloudProfileSpec)

    VolumeType contains certain properties of a volume type.

    FieldDescription
    class
    string

    Class is the class of the volume type.

    name
    string

    Name is the name of the volume type.

    usable
    bool
    (Optional)

    Usable defines if the volume type can be used for shoot clusters.

    minSize
    k8s.io/apimachinery/pkg/api/resource.Quantity
    (Optional)

    MinSize is the minimal supported storage size.

    WatchCacheSizes

    (Appears on: KubeAPIServerConfig)

    WatchCacheSizes contains configuration of the API server’s watch cache sizes.

    FieldDescription
    default
    int32
    (Optional)

    Default configures the default watch cache size of the kube-apiserver (flag --default-watch-cache-size, defaults to 100). See: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

    resources
    []ResourceWatchCacheSize
    (Optional)

    Resources configures the watch cache size of the kube-apiserver per resource (flag --watch-cache-sizes). See: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

    Worker

    (Appears on: Provider)

    Worker is the base definition of a worker group.

    FieldDescription
    annotations
    map[string]string
    (Optional)

    Annotations is a map of key/value pairs for annotations for all the Node objects in this worker pool.

    caBundle
    string
    (Optional)

    CABundle is a certificate bundle which will be installed onto every machine of this worker pool.

    cri
    CRI
    (Optional)

    CRI contains configurations of CRI support of every machine in the worker pool. Defaults to a CRI with name containerd.

    kubernetes
    WorkerKubernetes
    (Optional)

    Kubernetes contains configuration for Kubernetes components related to this worker pool.

    labels
    map[string]string
    (Optional)

    Labels is a map of key/value pairs for labels for all the Node objects in this worker pool.

    name
    string

    Name is the name of the worker group.

    machine
    Machine

    Machine contains information about the machine type and image.

    maximum
    int32

    Maximum is the maximum number of machines to create. This value is divided by the number of configured zones for a fair distribution.

    minimum
    int32

    Minimum is the minimum number of machines to create. This value is divided by the number of configured zones for a fair distribution.

    maxSurge
    k8s.io/apimachinery/pkg/util/intstr.IntOrString
    (Optional)

    MaxSurge is maximum number of machines that are created during an update. This value is divided by the number of configured zones for a fair distribution.

    maxUnavailable
    k8s.io/apimachinery/pkg/util/intstr.IntOrString
    (Optional)

    MaxUnavailable is the maximum number of machines that can be unavailable during an update. This value is divided by the number of configured zones for a fair distribution.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig is the provider-specific configuration for this worker pool.

    taints
    []Kubernetes core/v1.Taint
    (Optional)

    Taints is a list of taints for all the Node objects in this worker pool.

    volume
    Volume
    (Optional)

    Volume contains information about the volume type and size.

    dataVolumes
    []DataVolume
    (Optional)

    DataVolumes contains a list of additional worker volumes.

    kubeletDataVolumeName
    string
    (Optional)

    KubeletDataVolumeName contains the name of a dataVolume that should be used for storing kubelet state.

    zones
    []string
    (Optional)

    Zones is a list of availability zones that are used to evenly distribute this worker pool. Optional as not every provider may support availability zones.

    systemComponents
    WorkerSystemComponents
    (Optional)

    SystemComponents contains configuration for system components related to this worker pool

    machineControllerManager
    MachineControllerManagerSettings
    (Optional)

    MachineControllerManagerSettings contains configurations for different worker-pools. Eg. MachineDrainTimeout, MachineHealthTimeout.

    sysctls
    map[string]string
    (Optional)

    Sysctls is a map of kernel settings to apply on all machines in this worker pool.

    clusterAutoscaler
    ClusterAutoscalerOptions
    (Optional)

    ClusterAutoscaler contains the cluster autoscaler configurations for the worker pool.

    WorkerKubernetes

    (Appears on: Worker)

    WorkerKubernetes contains configuration for Kubernetes components related to this worker pool.

    FieldDescription
    kubelet
    KubeletConfig
    (Optional)

    Kubelet contains configuration settings for all kubelets of this worker pool. If set, all spec.kubernetes.kubelet settings will be overwritten for this worker pool (no merge of settings).

    version
    string
    (Optional)

    Version is the semantic Kubernetes version to use for the Kubelet in this Worker Group. If not specified the kubelet version is derived from the global shoot cluster kubernetes version. version must be equal or lower than the version of the shoot kubernetes version. Only one minor version difference to other worker groups and global kubernetes version is allowed.

    WorkerSystemComponents

    (Appears on: Worker)

    WorkerSystemComponents contains configuration for system components related to this worker pool

    FieldDescription
    allow
    bool

    Allow determines whether the pool should be allowed to host system components or not (defaults to true)

    WorkersSettings

    (Appears on: Provider)

    WorkersSettings contains settings for all workers.

    FieldDescription
    sshAccess
    SSHAccess
    (Optional)

    SSHAccess contains settings regarding ssh access to the worker nodes.


    Generated with gen-crd-api-reference-docs

    1.3 - Core V1

    Packages:

    core.gardener.cloud/v1

    Package v1 is a version of the API.

    Resource Types:

    ControllerDeployment

    ControllerDeployment contains information about how this controller is deployed.

    FieldDescription
    apiVersion
    string
    core.gardener.cloud/v1
    kind
    string
    ControllerDeployment
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    helm
    HelmControllerDeployment
    (Optional)

    Helm configures that an extension controller is deployed using helm.

    HelmControllerDeployment

    (Appears on: ControllerDeployment)

    HelmControllerDeployment configures how an extension controller is deployed using helm.

    FieldDescription
    rawChart
    []byte
    (Optional)

    RawChart is the base64-encoded, gzip’ed, tar’ed extension controller chart.

    values
    Kubernetes apiextensions/v1.JSON
    (Optional)

    Values are the chart values.


    Generated with gen-crd-api-reference-docs

    1.4 - Extensions

    Packages:

    extensions.gardener.cloud/v1alpha1

    Package v1alpha1 is the v1alpha1 version of the API.

    Resource Types:

    BackupBucket

    BackupBucket is a specification for backup bucket.

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    BackupBucket
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    BackupBucketSpec

    Specification of the BackupBucket. If the object’s deletion timestamp is set, this field is immutable.



    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    region
    string

    Region is the region of this bucket. This field is immutable.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the credentials to access object store.

    status
    BackupBucketStatus
    (Optional)

    BackupEntry

    BackupEntry is a specification for backup Entry.

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    BackupEntry
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    BackupEntrySpec

    Specification of the BackupEntry. If the object’s deletion timestamp is set, this field is immutable.



    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    backupBucketProviderStatus
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    BackupBucketProviderStatus contains the provider status that has been generated by the controller responsible for the BackupBucket resource.

    region
    string

    Region is the region of this Entry. This field is immutable.

    bucketName
    string

    BucketName is the name of backup bucket for this Backup Entry.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the credentials to access object store.

    status
    BackupEntryStatus
    (Optional)

    Bastion

    Bastion is a bastion or jump host that is dynamically created to provide SSH access to shoot nodes.

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    Bastion
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    BastionSpec

    Spec is the specification of this Bastion. If the object’s deletion timestamp is set, this field is immutable.



    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    userData
    []byte

    UserData is the base64-encoded user data for the bastion instance. This should contain code to provision the SSH key on the bastion instance. This field is immutable.

    ingress
    []BastionIngressPolicy

    Ingress controls from where the created bastion host should be reachable.

    status
    BastionStatus
    (Optional)

    Status is the bastion’s status.

    Cluster

    Cluster is a specification for a Cluster resource.

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    Cluster
    metadata
    Kubernetes meta/v1.ObjectMeta
    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ClusterSpec


    cloudProfile
    k8s.io/apimachinery/pkg/runtime.RawExtension

    CloudProfile is a raw extension field that contains the cloudprofile resource referenced by the shoot that has to be reconciled.

    seed
    k8s.io/apimachinery/pkg/runtime.RawExtension

    Seed is a raw extension field that contains the seed resource referenced by the shoot that has to be reconciled.

    shoot
    k8s.io/apimachinery/pkg/runtime.RawExtension

    Shoot is a raw extension field that contains the shoot resource that has to be reconciled.

    ContainerRuntime

    ContainerRuntime is a specification for a container runtime resource.

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    ContainerRuntime
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ContainerRuntimeSpec

    Specification of the ContainerRuntime. If the object’s deletion timestamp is set, this field is immutable.



    binaryPath
    string

    BinaryPath is the Worker’s machine path where container runtime extensions should copy the binaries to.

    workerPool
    ContainerRuntimeWorkerPool

    WorkerPool identifies the worker pool of the Shoot. For each worker pool and type, Gardener deploys a ContainerRuntime CRD.

    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    status
    ContainerRuntimeStatus
    (Optional)

    ControlPlane

    ControlPlane is a specification for a ControlPlane resource.

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    ControlPlane
    metadata
    Kubernetes meta/v1.ObjectMeta
    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ControlPlaneSpec

    Specification of the ControlPlane. If the object’s deletion timestamp is set, this field is immutable.



    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    purpose
    Purpose
    (Optional)

    Purpose contains the data if a cloud provider needs additional components in order to expose the control plane. This field is immutable.

    infrastructureProviderStatus
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    InfrastructureProviderStatus contains the provider status that has been generated by the controller responsible for the Infrastructure resource.

    region
    string

    Region is the region of this control plane. This field is immutable.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the cloud provider specific credentials.

    status
    ControlPlaneStatus
    (Optional)

    DNSRecord

    DNSRecord is a specification for a DNSRecord resource.

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    DNSRecord
    metadata
    Kubernetes meta/v1.ObjectMeta
    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    DNSRecordSpec

    Specification of the DNSRecord. If the object’s deletion timestamp is set, this field is immutable.



    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the cloud provider specific credentials.

    region
    string
    (Optional)

    Region is the region of this DNS record. If not specified, the region specified in SecretRef will be used. If that is also not specified, the extension controller will use its default region.

    zone
    string
    (Optional)

    Zone is the DNS hosted zone of this DNS record. If not specified, it will be determined automatically by getting all hosted zones of the account and searching for the longest zone name that is a suffix of Name.

    name
    string

    Name is the fully qualified domain name, e.g. “api.”. This field is immutable.

    recordType
    DNSRecordType

    RecordType is the DNS record type. Only A, CNAME, and TXT records are currently supported. This field is immutable.

    values
    []string

    Values is a list of IP addresses for A records, a single hostname for CNAME records, or a list of texts for TXT records.

    ttl
    int64
    (Optional)

    TTL is the time to live in seconds. Defaults to 120.

    status
    DNSRecordStatus
    (Optional)

    Extension

    Extension is a specification for a Extension resource.

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    Extension
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    ExtensionSpec

    Specification of the Extension. If the object’s deletion timestamp is set, this field is immutable.



    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    status
    ExtensionStatus
    (Optional)

    Infrastructure

    Infrastructure is a specification for cloud provider infrastructure.

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    Infrastructure
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    InfrastructureSpec

    Specification of the Infrastructure. If the object’s deletion timestamp is set, this field is immutable.



    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    region
    string

    Region is the region of this infrastructure. This field is immutable.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the cloud provider credentials.

    sshPublicKey
    []byte
    (Optional)

    SSHPublicKey is the public SSH key that should be used with this infrastructure.

    status
    InfrastructureStatus
    (Optional)

    Network

    Network is the specification for cluster networking.

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    Network
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    NetworkSpec

    Specification of the Network. If the object’s deletion timestamp is set, this field is immutable.



    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    podCIDR
    string

    PodCIDR defines the CIDR that will be used for pods. This field is immutable.

    serviceCIDR
    string

    ServiceCIDR defines the CIDR that will be used for services. This field is immutable.

    ipFamilies
    []IPFamily
    (Optional)

    IPFamilies specifies the IP protocol versions to use for shoot networking. This field is immutable. See https://github.com/gardener/gardener/blob/master/docs/usage/ipv6.md

    status
    NetworkStatus
    (Optional)

    OperatingSystemConfig

    OperatingSystemConfig is a specification for a OperatingSystemConfig resource

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    OperatingSystemConfig
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    OperatingSystemConfigSpec

    Specification of the OperatingSystemConfig. If the object’s deletion timestamp is set, this field is immutable.



    criConfig
    CRIConfig
    (Optional)

    CRI config is a structure contains configurations of the CRI library

    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    purpose
    OperatingSystemConfigPurpose

    Purpose describes how the result of this OperatingSystemConfig is used by Gardener. Either it gets sent to the Worker extension controller to bootstrap a VM, or it is downloaded by the gardener-node-agent already running on a bootstrapped VM. This field is immutable.

    reloadConfigFilePath
    string
    (Optional)

    ReloadConfigFilePath is the path to the generated operating system configuration. If set, controllers are asked to use it when determining the .status.command of this resource. For example, if for CoreOS the reload-path might be “/var/lib/config”; then the controller shall set .status.command to “/usr/bin/coreos-cloudinit –from-file=/var/lib/config”. Deprecated: This field is deprecated and has no further usage. TODO(rfranzke): Remove this field after v1.95 got released.

    units
    []Unit
    (Optional)

    Units is a list of unit for the operating system configuration (usually, a systemd unit).

    files
    []File
    (Optional)

    Files is a list of files that should get written to the host’s file system.

    status
    OperatingSystemConfigStatus
    (Optional)

    Worker

    Worker is a specification for a Worker resource.

    FieldDescription
    apiVersion
    string
    extensions.gardener.cloud/v1alpha1
    kind
    string
    Worker
    metadata
    Kubernetes meta/v1.ObjectMeta
    (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    WorkerSpec

    Specification of the Worker. If the object’s deletion timestamp is set, this field is immutable.



    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    infrastructureProviderStatus
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    InfrastructureProviderStatus is a raw extension field that contains the provider status that has been generated by the controller responsible for the Infrastructure resource.

    region
    string

    Region is the name of the region where the worker pool should be deployed to. This field is immutable.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the cloud provider specific credentials.

    sshPublicKey
    []byte
    (Optional)

    SSHPublicKey is the public SSH key that should be used with these workers.

    pools
    []WorkerPool

    Pools is a list of worker pools.

    status
    WorkerStatus
    (Optional)

    BackupBucketSpec

    (Appears on: BackupBucket)

    BackupBucketSpec is the spec for an BackupBucket resource.

    FieldDescription
    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    region
    string

    Region is the region of this bucket. This field is immutable.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the credentials to access object store.

    BackupBucketStatus

    (Appears on: BackupBucket)

    BackupBucketStatus is the status for an BackupBucket resource.

    FieldDescription
    DefaultStatus
    DefaultStatus

    (Members of DefaultStatus are embedded into this type.)

    DefaultStatus is a structure containing common fields used by all extension resources.

    generatedSecretRef
    Kubernetes core/v1.SecretReference
    (Optional)

    GeneratedSecretRef is reference to the secret generated by backup bucket, which will have object store specific credentials.

    BackupEntrySpec

    (Appears on: BackupEntry)

    BackupEntrySpec is the spec for an BackupEntry resource.

    FieldDescription
    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    backupBucketProviderStatus
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    BackupBucketProviderStatus contains the provider status that has been generated by the controller responsible for the BackupBucket resource.

    region
    string

    Region is the region of this Entry. This field is immutable.

    bucketName
    string

    BucketName is the name of backup bucket for this Backup Entry.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the credentials to access object store.

    BackupEntryStatus

    (Appears on: BackupEntry)

    BackupEntryStatus is the status for an BackupEntry resource.

    FieldDescription
    DefaultStatus
    DefaultStatus

    (Members of DefaultStatus are embedded into this type.)

    DefaultStatus is a structure containing common fields used by all extension resources.

    BastionIngressPolicy

    (Appears on: BastionSpec)

    BastionIngressPolicy represents an ingress policy for SSH bastion hosts.

    FieldDescription
    ipBlock
    Kubernetes networking/v1.IPBlock

    IPBlock defines an IP block that is allowed to access the bastion.

    BastionSpec

    (Appears on: Bastion)

    BastionSpec contains the specification for an SSH bastion host.

    FieldDescription
    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    userData
    []byte

    UserData is the base64-encoded user data for the bastion instance. This should contain code to provision the SSH key on the bastion instance. This field is immutable.

    ingress
    []BastionIngressPolicy

    Ingress controls from where the created bastion host should be reachable.

    BastionStatus

    (Appears on: Bastion)

    BastionStatus holds the most recently observed status of the Bastion.

    FieldDescription
    DefaultStatus
    DefaultStatus

    (Members of DefaultStatus are embedded into this type.)

    DefaultStatus is a structure containing common fields used by all extension resources.

    ingress
    Kubernetes core/v1.LoadBalancerIngress
    (Optional)

    Ingress is the external IP and/or hostname of the bastion host.

    CRIConfig

    (Appears on: OperatingSystemConfigSpec)

    CRIConfig contains configurations of the CRI library.

    FieldDescription
    name
    CRIName

    Name is a mandatory string containing the name of the CRI library. Supported values are containerd.

    CRIName (string alias)

    (Appears on: CRIConfig)

    CRIName is a type alias for the CRI name string.

    CloudConfig

    (Appears on: OperatingSystemConfigStatus)

    CloudConfig contains the generated output for the given operating system config spec. It contains a reference to a secret as the result may contain confidential data.

    FieldDescription
    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the actual result of the generated cloud config.

    ClusterAutoscalerOptions

    (Appears on: WorkerPool)

    ClusterAutoscalerOptions contains the cluster autoscaler configurations for a worker pool.

    FieldDescription
    scaleDownUtilizationThreshold
    string
    (Optional)

    ScaleDownUtilizationThreshold defines the threshold in fraction (0.0 - 1.0) under which a node is being removed.

    scaleDownGpuUtilizationThreshold
    string
    (Optional)

    ScaleDownGpuUtilizationThreshold defines the threshold in fraction (0.0 - 1.0) of gpu resources under which a node is being removed.

    scaleDownUnneededTime
    Kubernetes meta/v1.Duration
    (Optional)

    ScaleDownUnneededTime defines how long a node should be unneeded before it is eligible for scale down.

    scaleDownUnreadyTime
    Kubernetes meta/v1.Duration
    (Optional)

    ScaleDownUnreadyTime defines how long an unready node should be unneeded before it is eligible for scale down.

    maxNodeProvisionTime
    Kubernetes meta/v1.Duration
    (Optional)

    MaxNodeProvisionTime defines how long cluster autoscaler should wait for a node to be provisioned.

    ClusterSpec

    (Appears on: Cluster)

    ClusterSpec is the spec for a Cluster resource.

    FieldDescription
    cloudProfile
    k8s.io/apimachinery/pkg/runtime.RawExtension

    CloudProfile is a raw extension field that contains the cloudprofile resource referenced by the shoot that has to be reconciled.

    seed
    k8s.io/apimachinery/pkg/runtime.RawExtension

    Seed is a raw extension field that contains the seed resource referenced by the shoot that has to be reconciled.

    shoot
    k8s.io/apimachinery/pkg/runtime.RawExtension

    Shoot is a raw extension field that contains the shoot resource that has to be reconciled.

    ContainerRuntimeSpec

    (Appears on: ContainerRuntime)

    ContainerRuntimeSpec is the spec for a ContainerRuntime resource.

    FieldDescription
    binaryPath
    string

    BinaryPath is the Worker’s machine path where container runtime extensions should copy the binaries to.

    workerPool
    ContainerRuntimeWorkerPool

    WorkerPool identifies the worker pool of the Shoot. For each worker pool and type, Gardener deploys a ContainerRuntime CRD.

    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    ContainerRuntimeStatus

    (Appears on: ContainerRuntime)

    ContainerRuntimeStatus is the status for a ContainerRuntime resource.

    FieldDescription
    DefaultStatus
    DefaultStatus

    (Members of DefaultStatus are embedded into this type.)

    DefaultStatus is a structure containing common fields used by all extension resources.

    ContainerRuntimeWorkerPool

    (Appears on: ContainerRuntimeSpec)

    ContainerRuntimeWorkerPool identifies a Shoot worker pool by its name and selector.

    FieldDescription
    name
    string

    Name specifies the name of the worker pool the container runtime should be available for. This field is immutable.

    selector
    Kubernetes meta/v1.LabelSelector

    Selector is the label selector used by the extension to match the nodes belonging to the worker pool.

    ControlPlaneSpec

    (Appears on: ControlPlane)

    ControlPlaneSpec is the spec of a ControlPlane resource.

    FieldDescription
    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    purpose
    Purpose
    (Optional)

    Purpose contains the data if a cloud provider needs additional components in order to expose the control plane. This field is immutable.

    infrastructureProviderStatus
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    InfrastructureProviderStatus contains the provider status that has been generated by the controller responsible for the Infrastructure resource.

    region
    string

    Region is the region of this control plane. This field is immutable.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the cloud provider specific credentials.

    ControlPlaneStatus

    (Appears on: ControlPlane)

    ControlPlaneStatus is the status of a ControlPlane resource.

    FieldDescription
    DefaultStatus
    DefaultStatus

    (Members of DefaultStatus are embedded into this type.)

    DefaultStatus is a structure containing common fields used by all extension resources.

    DNSRecordSpec

    (Appears on: DNSRecord)

    DNSRecordSpec is the spec of a DNSRecord resource.

    FieldDescription
    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the cloud provider specific credentials.

    region
    string
    (Optional)

    Region is the region of this DNS record. If not specified, the region specified in SecretRef will be used. If that is also not specified, the extension controller will use its default region.

    zone
    string
    (Optional)

    Zone is the DNS hosted zone of this DNS record. If not specified, it will be determined automatically by getting all hosted zones of the account and searching for the longest zone name that is a suffix of Name.

    name
    string

    Name is the fully qualified domain name, e.g. “api.”. This field is immutable.

    recordType
    DNSRecordType

    RecordType is the DNS record type. Only A, CNAME, and TXT records are currently supported. This field is immutable.

    values
    []string

    Values is a list of IP addresses for A records, a single hostname for CNAME records, or a list of texts for TXT records.

    ttl
    int64
    (Optional)

    TTL is the time to live in seconds. Defaults to 120.

    DNSRecordStatus

    (Appears on: DNSRecord)

    DNSRecordStatus is the status of a DNSRecord resource.

    FieldDescription
    DefaultStatus
    DefaultStatus

    (Members of DefaultStatus are embedded into this type.)

    DefaultStatus is a structure containing common fields used by all extension resources.

    zone
    string
    (Optional)

    Zone is the DNS hosted zone of this DNS record.

    DNSRecordType (string alias)

    (Appears on: DNSRecordSpec)

    DNSRecordType is a string alias.

    DataVolume

    (Appears on: WorkerPool)

    DataVolume contains information about a data volume.

    FieldDescription
    name
    string

    Name of the volume to make it referencable.

    type
    string
    (Optional)

    Type is the type of the volume.

    size
    string

    Size is the of the root volume.

    encrypted
    bool
    (Optional)

    Encrypted determines if the volume should be encrypted.

    DefaultSpec

    (Appears on: BackupBucketSpec, BackupEntrySpec, BastionSpec, ContainerRuntimeSpec, ControlPlaneSpec, DNSRecordSpec, ExtensionSpec, InfrastructureSpec, NetworkSpec, OperatingSystemConfigSpec, WorkerSpec)

    DefaultSpec contains common status fields for every extension resource.

    FieldDescription
    type
    string

    Type contains the instance of the resource’s kind.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig is the provider specific configuration.

    DefaultStatus

    (Appears on: BackupBucketStatus, BackupEntryStatus, BastionStatus, ContainerRuntimeStatus, ControlPlaneStatus, DNSRecordStatus, ExtensionStatus, InfrastructureStatus, NetworkStatus, OperatingSystemConfigStatus, WorkerStatus)

    DefaultStatus contains common status fields for every extension resource.

    FieldDescription
    providerStatus
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderStatus contains provider-specific status.

    conditions
    []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition
    (Optional)

    Conditions represents the latest available observations of a Seed’s current state.

    lastError
    github.com/gardener/gardener/pkg/apis/core/v1beta1.LastError
    (Optional)

    LastError holds information about the last occurred error during an operation.

    lastOperation
    github.com/gardener/gardener/pkg/apis/core/v1beta1.LastOperation
    (Optional)

    LastOperation holds information about the last operation on the resource.

    observedGeneration
    int64

    ObservedGeneration is the most recent generation observed for this resource.

    state
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    State can be filled by the operating controller with what ever data it needs.

    resources
    []github.com/gardener/gardener/pkg/apis/core/v1beta1.NamedResourceReference
    (Optional)

    Resources holds a list of named resource references that can be referred to in the state by their names.

    DropIn

    (Appears on: Unit)

    DropIn is a drop-in configuration for a systemd unit.

    FieldDescription
    name
    string

    Name is the name of the drop-in.

    content
    string

    Content is the content of the drop-in.

    ExtensionSpec

    (Appears on: Extension)

    ExtensionSpec is the spec for a Extension resource.

    FieldDescription
    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    ExtensionStatus

    (Appears on: Extension)

    ExtensionStatus is the status for a Extension resource.

    FieldDescription
    DefaultStatus
    DefaultStatus

    (Members of DefaultStatus are embedded into this type.)

    DefaultStatus is a structure containing common fields used by all extension resources.

    File

    (Appears on: OperatingSystemConfigSpec, OperatingSystemConfigStatus)

    File is a file that should get written to the host’s file system. The content can either be inlined or referenced from a secret in the same namespace.

    FieldDescription
    path
    string

    Path is the path of the file system where the file should get written to.

    permissions
    int32
    (Optional)

    Permissions describes with which permissions the file should get written to the file system. Should be defaulted to octal 0644.

    content
    FileContent

    Content describe the file’s content.

    FileCodecID (string alias)

    FileCodecID is the id of a FileCodec for cloud-init scripts.

    FileContent

    (Appears on: File)

    FileContent can either reference a secret or contain inline configuration.

    FieldDescription
    secretRef
    FileContentSecretRef
    (Optional)

    SecretRef is a struct that contains information about the referenced secret.

    inline
    FileContentInline
    (Optional)

    Inline is a struct that contains information about the inlined data.

    transmitUnencoded
    bool
    (Optional)

    TransmitUnencoded set to true will ensure that the os-extension does not encode the file content when sent to the node. This for example can be used to manipulate the clear-text content before it reaches the node.

    imageRef
    FileContentImageRef
    (Optional)

    ImageRef describes a container image which contains a file.

    FileContentImageRef

    (Appears on: FileContent)

    FileContentImageRef describes a container image which contains a file

    FieldDescription
    image
    string

    Image contains the container image repository with tag.

    filePathInImage
    string

    FilePathInImage contains the path in the image to the file that should be extracted.

    FileContentInline

    (Appears on: FileContent)

    FileContentInline contains keys for inlining a file content’s data and encoding.

    FieldDescription
    encoding
    string

    Encoding is the file’s encoding (e.g. base64).

    data
    string

    Data is the file’s data.

    FileContentSecretRef

    (Appears on: FileContent)

    FileContentSecretRef contains keys for referencing a file content’s data from a secret in the same namespace.

    FieldDescription
    name
    string

    Name is the name of the secret.

    dataKey
    string

    DataKey is the key in the secret’s .data field that should be read.

    IPFamily (string alias)

    (Appears on: NetworkSpec)

    IPFamily is a type for specifying an IP protocol version to use in Gardener clusters.

    InfrastructureSpec

    (Appears on: Infrastructure)

    InfrastructureSpec is the spec for an Infrastructure resource.

    FieldDescription
    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    region
    string

    Region is the region of this infrastructure. This field is immutable.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the cloud provider credentials.

    sshPublicKey
    []byte
    (Optional)

    SSHPublicKey is the public SSH key that should be used with this infrastructure.

    InfrastructureStatus

    (Appears on: Infrastructure)

    InfrastructureStatus is the status for an Infrastructure resource.

    FieldDescription
    DefaultStatus
    DefaultStatus

    (Members of DefaultStatus are embedded into this type.)

    DefaultStatus is a structure containing common fields used by all extension resources.

    nodesCIDR
    string
    (Optional)

    NodesCIDR is the CIDR of the node network that was optionally created by the acting extension controller. This might be needed in environments in which the CIDR for the network for the shoot worker node cannot be statically defined in the Shoot resource but must be computed dynamically.

    egressCIDRs
    []string
    (Optional)

    EgressCIDRs is a list of CIDRs used by the shoot as the source IP for egress traffic. For certain environments the egress IPs may not be stable in which case the extension controller may opt to not populate this field.

    MachineDeployment

    (Appears on: WorkerStatus)

    MachineDeployment is a created machine deployment.

    FieldDescription
    name
    string

    Name is the name of the MachineDeployment resource.

    minimum
    int32

    Minimum is the minimum number for this machine deployment.

    maximum
    int32

    Maximum is the maximum number for this machine deployment.

    MachineImage

    (Appears on: WorkerPool)

    MachineImage contains logical information about the name and the version of the machie image that should be used. The logical information must be mapped to the provider-specific information (e.g., AMIs, …) by the provider itself.

    FieldDescription
    name
    string

    Name is the logical name of the machine image.

    version
    string

    Version is the version of the machine image.

    NetworkSpec

    (Appears on: Network)

    NetworkSpec is the spec for an Network resource.

    FieldDescription
    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    podCIDR
    string

    PodCIDR defines the CIDR that will be used for pods. This field is immutable.

    serviceCIDR
    string

    ServiceCIDR defines the CIDR that will be used for services. This field is immutable.

    ipFamilies
    []IPFamily
    (Optional)

    IPFamilies specifies the IP protocol versions to use for shoot networking. This field is immutable. See https://github.com/gardener/gardener/blob/master/docs/usage/ipv6.md

    NetworkStatus

    (Appears on: Network)

    NetworkStatus is the status for an Network resource.

    FieldDescription
    DefaultStatus
    DefaultStatus

    (Members of DefaultStatus are embedded into this type.)

    DefaultStatus is a structure containing common fields used by all extension resources.

    NodeTemplate

    (Appears on: WorkerPool)

    NodeTemplate contains information about the expected node properties.

    FieldDescription
    capacity
    Kubernetes core/v1.ResourceList

    Capacity represents the expected Node capacity.

    Object

    Object is an extension object resource.

    OperatingSystemConfigPurpose (string alias)

    (Appears on: OperatingSystemConfigSpec)

    OperatingSystemConfigPurpose is a string alias.

    OperatingSystemConfigSpec

    (Appears on: OperatingSystemConfig)

    OperatingSystemConfigSpec is the spec for a OperatingSystemConfig resource.

    FieldDescription
    criConfig
    CRIConfig
    (Optional)

    CRI config is a structure contains configurations of the CRI library

    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    purpose
    OperatingSystemConfigPurpose

    Purpose describes how the result of this OperatingSystemConfig is used by Gardener. Either it gets sent to the Worker extension controller to bootstrap a VM, or it is downloaded by the gardener-node-agent already running on a bootstrapped VM. This field is immutable.

    reloadConfigFilePath
    string
    (Optional)

    ReloadConfigFilePath is the path to the generated operating system configuration. If set, controllers are asked to use it when determining the .status.command of this resource. For example, if for CoreOS the reload-path might be “/var/lib/config”; then the controller shall set .status.command to “/usr/bin/coreos-cloudinit –from-file=/var/lib/config”. Deprecated: This field is deprecated and has no further usage. TODO(rfranzke): Remove this field after v1.95 got released.

    units
    []Unit
    (Optional)

    Units is a list of unit for the operating system configuration (usually, a systemd unit).

    files
    []File
    (Optional)

    Files is a list of files that should get written to the host’s file system.

    OperatingSystemConfigStatus

    (Appears on: OperatingSystemConfig)

    OperatingSystemConfigStatus is the status for a OperatingSystemConfig resource.

    FieldDescription
    DefaultStatus
    DefaultStatus

    (Members of DefaultStatus are embedded into this type.)

    DefaultStatus is a structure containing common fields used by all extension resources.

    extensionUnits
    []Unit
    (Optional)

    ExtensionUnits is a list of additional systemd units provided by the extension.

    extensionFiles
    []File
    (Optional)

    ExtensionFiles is a list of additional files provided by the extension.

    cloudConfig
    CloudConfig
    (Optional)

    CloudConfig is a structure for containing the generated output for the given operating system config spec. It contains a reference to a secret as the result may contain confidential data.

    command
    string
    (Optional)

    Command is the command whose execution renews/reloads the cloud config on an existing VM, e.g. “/usr/bin/reload-cloud-config -from-file=”. The is optionally provided by Gardener in the .spec.reloadConfigFilePath field. Deprecated: This field is deprecated and has no further usage. TODO(rfranzke): Remove this field after v1.95 got released.

    units
    []string
    (Optional)

    Units is a list of systemd unit names that are part of the generated Cloud Config and shall be restarted when a new version has been downloaded. Deprecated: This field is deprecated and has no further usage. TODO(rfranzke): Remove this field after v1.95 got released.

    files
    []string
    (Optional)

    Files is a list of file paths that are part of the generated Cloud Config and shall be written to the host’s file system. Deprecated: This field is deprecated and has no further usage. TODO(rfranzke): Remove this field after v1.95 got released.

    Purpose (string alias)

    (Appears on: ControlPlaneSpec)

    Purpose is a string alias.

    Spec

    Spec is the spec section of an Object.

    Status

    Status is the status of an Object.

    Unit

    (Appears on: OperatingSystemConfigSpec, OperatingSystemConfigStatus)

    Unit is a unit for the operating system configuration (usually, a systemd unit).

    FieldDescription
    name
    string

    Name is the name of a unit.

    command
    UnitCommand
    (Optional)

    Command is the unit’s command.

    enable
    bool
    (Optional)

    Enable describes whether the unit is enabled or not.

    content
    string
    (Optional)

    Content is the unit’s content.

    dropIns
    []DropIn
    (Optional)

    DropIns is a list of drop-ins for this unit.

    filePaths
    []string

    FilePaths is a list of files the unit depends on. If any file changes a restart of the dependent unit will be triggered. For each FilePath there must exist a File with matching Path in OperatingSystemConfig.Spec.Files.

    UnitCommand (string alias)

    (Appears on: Unit)

    UnitCommand is a string alias.

    Volume

    (Appears on: WorkerPool)

    Volume contains information about the root disks that should be used for worker pools.

    FieldDescription
    name
    string
    (Optional)

    Name of the volume to make it referencable.

    type
    string
    (Optional)

    Type is the type of the volume.

    size
    string

    Size is the of the root volume.

    encrypted
    bool
    (Optional)

    Encrypted determines if the volume should be encrypted.

    WorkerPool

    (Appears on: WorkerSpec)

    WorkerPool is the definition of a specific worker pool.

    FieldDescription
    machineType
    string

    MachineType contains information about the machine type that should be used for this worker pool.

    maximum
    int32

    Maximum is the maximum size of the worker pool.

    maxSurge
    k8s.io/apimachinery/pkg/util/intstr.IntOrString

    MaxSurge is maximum number of VMs that are created during an update.

    maxUnavailable
    k8s.io/apimachinery/pkg/util/intstr.IntOrString

    MaxUnavailable is the maximum number of VMs that can be unavailable during an update.

    annotations
    map[string]string
    (Optional)

    Annotations is a map of key/value pairs for annotations for all the Node objects in this worker pool.

    labels
    map[string]string
    (Optional)

    Labels is a map of key/value pairs for labels for all the Node objects in this worker pool.

    taints
    []Kubernetes core/v1.Taint
    (Optional)

    Taints is a list of taints for all the Node objects in this worker pool.

    machineImage
    MachineImage

    MachineImage contains logical information about the name and the version of the machie image that should be used. The logical information must be mapped to the provider-specific information (e.g., AMIs, …) by the provider itself.

    minimum
    int32

    Minimum is the minimum size of the worker pool.

    name
    string

    Name is the name of this worker pool.

    providerConfig
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    ProviderConfig is a provider specific configuration for the worker pool.

    userData
    []byte
    (Optional)

    UserData is a base64-encoded string that contains the data that is sent to the provider’s APIs when a new machine/VM that is part of this worker pool shall be spawned. Either this or UserDataSecretRef must be provided. Deprecated: This field will be removed in future release. TODO(rfranzke): Remove this field after v1.100 has been released.

    userDataSecretRef
    Kubernetes core/v1.SecretKeySelector
    (Optional)

    UserDataSecretRef references a Secret and a data key containing the data that is sent to the provider’s APIs when a new machine/VM that is part of this worker pool shall be spawned. Either this or UserData must be provided.

    volume
    Volume
    (Optional)

    Volume contains information about the root disks that should be used for this worker pool.

    dataVolumes
    []DataVolume
    (Optional)

    DataVolumes contains a list of additional worker volumes.

    kubeletDataVolumeName
    string
    (Optional)

    KubeletDataVolumeName contains the name of a dataVolume that should be used for storing kubelet state.

    zones
    []string
    (Optional)

    Zones contains information about availability zones for this worker pool.

    machineControllerManager
    github.com/gardener/gardener/pkg/apis/core/v1beta1.MachineControllerManagerSettings
    (Optional)

    MachineControllerManagerSettings contains configurations for different worker-pools. Eg. MachineDrainTimeout, MachineHealthTimeout.

    kubernetesVersion
    string
    (Optional)

    KubernetesVersion is the kubernetes version in this worker pool

    nodeTemplate
    NodeTemplate
    (Optional)

    NodeTemplate contains resource information of the machine which is used by Cluster Autoscaler to generate nodeTemplate during scaling a nodeGroup from zero

    architecture
    string
    (Optional)

    Architecture is the CPU architecture of the worker pool machines and machine image.

    clusterAutoscaler
    ClusterAutoscalerOptions
    (Optional)

    ClusterAutoscaler contains the cluster autoscaler configurations for the worker pool.

    WorkerSpec

    (Appears on: Worker)

    WorkerSpec is the spec for a Worker resource.

    FieldDescription
    DefaultSpec
    DefaultSpec

    (Members of DefaultSpec are embedded into this type.)

    DefaultSpec is a structure containing common fields used by all extension resources.

    infrastructureProviderStatus
    k8s.io/apimachinery/pkg/runtime.RawExtension
    (Optional)

    InfrastructureProviderStatus is a raw extension field that contains the provider status that has been generated by the controller responsible for the Infrastructure resource.

    region
    string

    Region is the name of the region where the worker pool should be deployed to. This field is immutable.

    secretRef
    Kubernetes core/v1.SecretReference

    SecretRef is a reference to a secret that contains the cloud provider specific credentials.

    sshPublicKey
    []byte
    (Optional)

    SSHPublicKey is the public SSH key that should be used with these workers.

    pools
    []WorkerPool

    Pools is a list of worker pools.

    WorkerStatus

    (Appears on: Worker)

    WorkerStatus is the status for a Worker resource.

    FieldDescription
    DefaultStatus
    DefaultStatus

    (Members of DefaultStatus are embedded into this type.)

    DefaultStatus is a structure containing common fields used by all extension resources.

    machineDeployments
    []MachineDeployment

    MachineDeployments is a list of created machine deployments. It will be used to e.g. configure the cluster-autoscaler properly.

    machineDeploymentsLastUpdateTime
    Kubernetes meta/v1.Time
    (Optional)

    MachineDeploymentsLastUpdateTime is the timestamp when the status.MachineDeployments slice was last updated.


    Generated with gen-crd-api-reference-docs

    1.5 - Operations

    Packages:

    operations.gardener.cloud/v1alpha1

    Package v1alpha1 is a version of the API.

    Resource Types:

    Bastion

    Bastion holds details about an SSH bastion for a shoot cluster.

    FieldDescription
    apiVersion
    string
    operations.gardener.cloud/v1alpha1
    kind
    string
    Bastion
    metadata
    Kubernetes meta/v1.ObjectMeta

    Standard object metadata.

    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    BastionSpec

    Specification of the Bastion.



    shootRef
    Kubernetes core/v1.LocalObjectReference

    ShootRef defines the target shoot for a Bastion. The name field of the ShootRef is immutable.

    seedName
    string
    (Optional)

    SeedName is the name of the seed to which this Bastion is currently scheduled. This field is populated at the beginning of a create/reconcile operation.

    providerType
    string
    (Optional)

    ProviderType is cloud provider used by the referenced Shoot.

    sshPublicKey
    string

    SSHPublicKey is the user’s public key. This field is immutable.

    ingress
    []BastionIngressPolicy

    Ingress controls from where the created bastion host should be reachable.

    status
    BastionStatus
    (Optional)

    Most recently observed status of the Bastion.

    BastionIngressPolicy

    (Appears on: BastionSpec)

    BastionIngressPolicy represents an ingress policy for SSH bastion hosts.

    FieldDescription
    ipBlock
    Kubernetes networking/v1.IPBlock

    IPBlock defines an IP block that is allowed to access the bastion.

    BastionSpec

    (Appears on: Bastion)

    BastionSpec is the specification of a Bastion.

    FieldDescription
    shootRef
    Kubernetes core/v1.LocalObjectReference

    ShootRef defines the target shoot for a Bastion. The name field of the ShootRef is immutable.

    seedName
    string
    (Optional)

    SeedName is the name of the seed to which this Bastion is currently scheduled. This field is populated at the beginning of a create/reconcile operation.

    providerType
    string
    (Optional)

    ProviderType is cloud provider used by the referenced Shoot.

    sshPublicKey
    string

    SSHPublicKey is the user’s public key. This field is immutable.

    ingress
    []BastionIngressPolicy

    Ingress controls from where the created bastion host should be reachable.

    BastionStatus

    (Appears on: Bastion)

    BastionStatus holds the most recently observed status of the Bastion.

    FieldDescription
    ingress
    Kubernetes core/v1.LoadBalancerIngress
    (Optional)

    Ingress holds the public IP and/or hostname of the bastion instance.

    conditions
    []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition
    (Optional)

    Conditions represents the latest available observations of a Bastion’s current state.

    lastHeartbeatTimestamp
    Kubernetes meta/v1.Time
    (Optional)

    LastHeartbeatTimestamp is the time when the bastion was last marked as not to be deleted. When this is set, the ExpirationTimestamp is advanced as well.

    expirationTimestamp
    Kubernetes meta/v1.Time
    (Optional)

    ExpirationTimestamp is the time after which a Bastion is supposed to be garbage collected.

    observedGeneration
    int64
    (Optional)

    ObservedGeneration is the most recent generation observed for this Bastion. It corresponds to the Bastion’s generation, which is updated on mutation by the API Server.


    Generated with gen-crd-api-reference-docs

    1.6 - Operator

    Packages:

    operator.gardener.cloud/v1alpha1

    Package v1alpha1 contains the configuration of the Gardener Operator.

    Resource Types:

      AuditWebhook

      (Appears on: GardenerAPIServerConfig, KubeAPIServerConfig)

      AuditWebhook contains settings related to an audit webhook configuration.

      FieldDescription
      batchMaxSize
      int32
      (Optional)

      BatchMaxSize is the maximum size of a batch.

      kubeconfigSecretName
      string

      KubeconfigSecretName specifies the name of a secret containing the kubeconfig for this webhook.

      version
      string
      (Optional)

      Version is the API version to send and expect from the webhook.

      Authentication

      (Appears on: KubeAPIServerConfig)

      Authentication contains settings related to authentication.

      FieldDescription
      webhook
      AuthenticationWebhook
      (Optional)

      Webhook contains settings related to an authentication webhook configuration.

      AuthenticationWebhook

      (Appears on: Authentication)

      AuthenticationWebhook contains settings related to an authentication webhook configuration.

      FieldDescription
      cacheTTL
      Kubernetes meta/v1.Duration
      (Optional)

      CacheTTL is the duration to cache responses from the webhook authenticator.

      kubeconfigSecretName
      string

      KubeconfigSecretName specifies the name of a secret containing the kubeconfig for this webhook.

      version
      string
      (Optional)

      Version is the API version to send and expect from the webhook.

      Backup

      (Appears on: ETCDMain)

      Backup contains the object store configuration for backups for the virtual garden etcd.

      FieldDescription
      provider
      string

      Provider is a provider name. This field is immutable.

      bucketName
      string

      BucketName is the name of the backup bucket.

      secretRef
      Kubernetes core/v1.LocalObjectReference

      SecretRef is a reference to a Secret object containing the cloud provider credentials for the object store where backups should be stored. It should have enough privileges to manipulate the objects as well as buckets.

      ControlPlane

      (Appears on: VirtualCluster)

      ControlPlane holds information about the general settings for the control plane of the virtual garden cluster.

      FieldDescription
      highAvailability
      HighAvailability
      (Optional)

      HighAvailability holds the configuration settings for high availability settings.

      Credentials

      (Appears on: GardenStatus)

      Credentials contains information about the virtual garden cluster credentials.

      FieldDescription
      rotation
      CredentialsRotation
      (Optional)

      Rotation contains information about the credential rotations.

      CredentialsRotation

      (Appears on: Credentials)

      CredentialsRotation contains information about the rotation of credentials.

      FieldDescription
      certificateAuthorities
      github.com/gardener/gardener/pkg/apis/core/v1beta1.CARotation
      (Optional)

      CertificateAuthorities contains information about the certificate authority credential rotation.

      serviceAccountKey
      github.com/gardener/gardener/pkg/apis/core/v1beta1.ServiceAccountKeyRotation
      (Optional)

      ServiceAccountKey contains information about the service account key credential rotation.

      etcdEncryptionKey
      github.com/gardener/gardener/pkg/apis/core/v1beta1.ETCDEncryptionKeyRotation
      (Optional)

      ETCDEncryptionKey contains information about the ETCD encryption key credential rotation.

      observability
      github.com/gardener/gardener/pkg/apis/core/v1beta1.ObservabilityRotation
      (Optional)

      Observability contains information about the observability credential rotation.

      DNS

      (Appears on: VirtualCluster)

      DNS holds information about DNS settings.

      FieldDescription
      domains
      []string
      (Optional)

      Domains are the external domains of the virtual garden cluster. The first given domain in this list is immutable.

      DashboardGitHub

      (Appears on: GardenerDashboardConfig)

      DashboardGitHub contains configuration for the GitHub ticketing feature.

      FieldDescription
      apiURL
      string

      APIURL is the URL to the GitHub API.

      organisation
      string

      Organisation is the name of the GitHub organisation.

      repository
      string

      Repository is the name of the GitHub repository.

      secretRef
      Kubernetes core/v1.LocalObjectReference

      SecretRef is the reference to a secret in the garden namespace containing the GitHub credentials.

      pollInterval
      Kubernetes meta/v1.Duration
      (Optional)

      PollInterval is the interval of how often the GitHub API is polled for issue updates. This field is used as a fallback mechanism to ensure state synchronization, even when there is a GitHub webhook configuration. If a webhook event is missed or not successfully delivered, the polling will help catch up on any missed updates. If this field is not provided and there is no ‘webhookSecret’ key in the referenced secret, it will be implicitly defaulted to 15m.

      DashboardOIDC

      (Appears on: GardenerDashboardConfig)

      DashboardOIDC contains configuration for the OIDC settings.

      FieldDescription
      sessionLifetime
      Kubernetes meta/v1.Duration
      (Optional)

      SessionLifetime is the maximum duration of a session.

      additionalScopes
      []string
      (Optional)

      AdditionalScopes is the list of additional OIDC scopes.

      secretRef
      Kubernetes core/v1.LocalObjectReference

      SecretRef is the reference to a secret in the garden namespace containing the OIDC client ID and secret for the dashboard.

      DashboardTerminal

      (Appears on: GardenerDashboardConfig)

      DashboardTerminal contains configuration for the terminal settings.

      FieldDescription
      container
      DashboardTerminalContainer

      Container contains configuration for the dashboard terminal container.

      allowedHosts
      []string
      (Optional)

      AllowedHosts should consist of permitted hostnames (without the scheme) for terminal connections. It is important to consider that the usage of wildcards follows the rules defined by the content security policy. ‘.seed.local.gardener.cloud’, or ‘.other-seeds.local.gardener.cloud’. For more information, see https://github.com/gardener/dashboard/blob/master/docs/operations/webterminals.md#allowlist-for-hosts.

      DashboardTerminalContainer

      (Appears on: DashboardTerminal)

      DashboardTerminalContainer contains configuration for the dashboard terminal container.

      FieldDescription
      image
      string

      Image is the container image for the dashboard terminal container.

      description
      string
      (Optional)

      Description is a description for the dashboard terminal container with hints for the user.

      ETCD

      (Appears on: VirtualCluster)

      ETCD contains configuration for the etcds of the virtual garden cluster.

      FieldDescription
      main
      ETCDMain
      (Optional)

      Main contains configuration for the main etcd.

      events
      ETCDEvents
      (Optional)

      Events contains configuration for the events etcd.

      ETCDEvents

      (Appears on: ETCD)

      ETCDEvents contains configuration for the events etcd.

      FieldDescription
      storage
      Storage
      (Optional)

      Storage contains storage configuration.

      ETCDMain

      (Appears on: ETCD)

      ETCDMain contains configuration for the main etcd.

      FieldDescription
      backup
      Backup
      (Optional)

      Backup contains the object store configuration for backups for the virtual garden etcd.

      storage
      Storage
      (Optional)

      Storage contains storage configuration.

      Garden

      Garden describes a list of gardens.

      FieldDescription
      metadata
      Kubernetes meta/v1.ObjectMeta

      Standard object metadata.

      Refer to the Kubernetes API documentation for the fields of the metadata field.
      spec
      GardenSpec

      Spec contains the specification of this garden.



      runtimeCluster
      RuntimeCluster

      RuntimeCluster contains configuration for the runtime cluster.

      virtualCluster
      VirtualCluster

      VirtualCluster contains configuration for the virtual cluster.

      status
      GardenStatus

      Status contains the status of this garden.

      GardenSpec

      (Appears on: Garden)

      GardenSpec contains the specification of a garden environment.

      FieldDescription
      runtimeCluster
      RuntimeCluster

      RuntimeCluster contains configuration for the runtime cluster.

      virtualCluster
      VirtualCluster

      VirtualCluster contains configuration for the virtual cluster.

      GardenStatus

      (Appears on: Garden)

      GardenStatus is the status of a garden environment.

      FieldDescription
      gardener
      github.com/gardener/gardener/pkg/apis/core/v1beta1.Gardener
      (Optional)

      Gardener holds information about the Gardener which last acted on the Garden.

      conditions
      []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition

      Conditions is a list of conditions.

      lastOperation
      github.com/gardener/gardener/pkg/apis/core/v1beta1.LastOperation
      (Optional)

      LastOperation holds information about the last operation on the Garden.

      observedGeneration
      int64

      ObservedGeneration is the most recent generation observed for this resource.

      credentials
      Credentials
      (Optional)

      Credentials contains information about the virtual garden cluster credentials.

      encryptedResources
      []string
      (Optional)

      EncryptedResources is the list of resources which are currently encrypted in the virtual garden by the virtual kube-apiserver. Resources which are encrypted by default will not appear here. See https://github.com/gardener/gardener/blob/master/docs/concepts/operator.md#etcd-encryption-config for more details.

      Gardener

      (Appears on: VirtualCluster)

      Gardener contains the configuration settings for the Gardener components.

      FieldDescription
      clusterIdentity
      string

      ClusterIdentity is the identity of the garden cluster. This field is immutable.

      gardenerAPIServer
      GardenerAPIServerConfig
      (Optional)

      APIServer contains configuration settings for the gardener-apiserver.

      gardenerAdmissionController
      GardenerAdmissionControllerConfig
      (Optional)

      AdmissionController contains configuration settings for the gardener-admission-controller.

      gardenerControllerManager
      GardenerControllerManagerConfig
      (Optional)

      ControllerManager contains configuration settings for the gardener-controller-manager.

      gardenerScheduler
      GardenerSchedulerConfig
      (Optional)

      Scheduler contains configuration settings for the gardener-scheduler.

      gardenerDashboard
      GardenerDashboardConfig
      (Optional)

      Dashboard contains configuration settings for the gardener-dashboard.

      GardenerAPIServerConfig

      (Appears on: Gardener)

      GardenerAPIServerConfig contains configuration settings for the gardener-apiserver.

      FieldDescription
      KubernetesConfig
      github.com/gardener/gardener/pkg/apis/core/v1beta1.KubernetesConfig

      (Members of KubernetesConfig are embedded into this type.)

      admissionPlugins
      []github.com/gardener/gardener/pkg/apis/core/v1beta1.AdmissionPlugin
      (Optional)

      AdmissionPlugins contains the list of user-defined admission plugins (additional to those managed by Gardener), and, if desired, the corresponding configuration.

      auditConfig
      github.com/gardener/gardener/pkg/apis/core/v1beta1.AuditConfig
      (Optional)

      AuditConfig contains configuration settings for the audit of the kube-apiserver.

      auditWebhook
      AuditWebhook
      (Optional)

      AuditWebhook contains settings related to an audit webhook configuration.

      logging
      github.com/gardener/gardener/pkg/apis/core/v1beta1.APIServerLogging
      (Optional)

      Logging contains configuration for the log level and HTTP access logs.

      requests
      github.com/gardener/gardener/pkg/apis/core/v1beta1.APIServerRequests
      (Optional)

      Requests contains configuration for request-specific settings for the kube-apiserver.

      watchCacheSizes
      github.com/gardener/gardener/pkg/apis/core/v1beta1.WatchCacheSizes
      (Optional)

      WatchCacheSizes contains configuration of the API server’s watch cache sizes. Configuring these flags might be useful for large-scale Garden clusters with a lot of parallel update requests and a lot of watching controllers (e.g. large ManagedSeed clusters). When the API server’s watch cache’s capacity is too small to cope with the amount of update requests and watchers for a particular resource, it might happen that controller watches are permanently stopped with too old resource version errors. Starting from kubernetes v1.19, the API server’s watch cache size is adapted dynamically and setting the watch cache size flags will have no effect, except when setting it to 0 (which disables the watch cache).

      encryptionConfig
      github.com/gardener/gardener/pkg/apis/core/v1beta1.EncryptionConfig
      (Optional)

      EncryptionConfig contains customizable encryption configuration of the Gardener API server.

      GardenerAdmissionControllerConfig

      (Appears on: Gardener)

      GardenerAdmissionControllerConfig contains configuration settings for the gardener-admission-controller.

      FieldDescription
      logLevel
      string
      (Optional)

      LogLevel is the configured log level for the gardener-admission-controller. Must be one of [info,debug,error]. Defaults to info.

      resourceAdmissionConfiguration
      ResourceAdmissionConfiguration
      (Optional)

      ResourceAdmissionConfiguration is the configuration for resource size restrictions for arbitrary Group-Version-Kinds.

      GardenerControllerManagerConfig

      (Appears on: Gardener)

      GardenerControllerManagerConfig contains configuration settings for the gardener-controller-manager.

      FieldDescription
      KubernetesConfig
      github.com/gardener/gardener/pkg/apis/core/v1beta1.KubernetesConfig

      (Members of KubernetesConfig are embedded into this type.)

      defaultProjectQuotas
      []ProjectQuotaConfiguration
      (Optional)

      DefaultProjectQuotas is the default configuration matching projects are set up with if a quota is not already specified.

      logLevel
      string
      (Optional)

      LogLevel is the configured log level for the gardener-controller-manager. Must be one of [info,debug,error]. Defaults to info.

      GardenerDashboardConfig

      (Appears on: Gardener)

      GardenerDashboardConfig contains configuration settings for the gardener-dashboard.

      FieldDescription
      enableTokenLogin
      bool
      (Optional)

      EnableTokenLogin specifies whether it is possible to log into the dashboard with a JWT token. If disabled, OIDC must be configured.

      frontendConfigMapRef
      Kubernetes core/v1.LocalObjectReference
      (Optional)

      FrontendConfigMapRef is the reference to a ConfigMap in the garden namespace containing the frontend configuration.

      assetsConfigMapRef
      Kubernetes core/v1.LocalObjectReference
      (Optional)

      AssetsConfigMapRef is the reference to a ConfigMap in the garden namespace containing the assets (logos/icons).

      gitHub
      DashboardGitHub
      (Optional)

      GitHub contains configuration for the GitHub ticketing feature.

      logLevel
      string
      (Optional)

      LogLevel is the configured log level. Must be one of [trace,debug,info,warn,error]. Defaults to info.

      oidcConfig
      DashboardOIDC
      (Optional)

      OIDC contains configuration for the OIDC provider. This field must be provided when EnableTokenLogin is false.

      terminal
      DashboardTerminal
      (Optional)

      Terminal contains configuration for the terminal settings.

      GardenerSchedulerConfig

      (Appears on: Gardener)

      GardenerSchedulerConfig contains configuration settings for the gardener-scheduler.

      FieldDescription
      KubernetesConfig
      github.com/gardener/gardener/pkg/apis/core/v1beta1.KubernetesConfig

      (Members of KubernetesConfig are embedded into this type.)

      logLevel
      string
      (Optional)

      LogLevel is the configured log level for the gardener-scheduler. Must be one of [info,debug,error]. Defaults to info.

      GroupResource

      (Appears on: KubeAPIServerConfig)

      GroupResource contains a list of resources which should be stored in etcd-events instead of etcd-main.

      FieldDescription
      group
      string

      Group is the API group name.

      resource
      string

      Resource is the resource name.

      HighAvailability

      (Appears on: ControlPlane)

      HighAvailability specifies the configuration settings for high availability for a resource.

      Ingress

      (Appears on: RuntimeCluster)

      Ingress configures the Ingress specific settings of the runtime cluster.

      FieldDescription
      domains
      []string
      (Optional)

      Domains specify the ingress domains of the cluster pointing to the ingress controller endpoint. They will be used to construct ingress URLs for system applications running in runtime cluster.

      controller
      github.com/gardener/gardener/pkg/apis/core/v1beta1.IngressController

      Controller configures a Gardener managed Ingress Controller listening on the ingressDomain.

      KubeAPIServerConfig

      (Appears on: Kubernetes)

      KubeAPIServerConfig contains configuration settings for the kube-apiserver.

      FieldDescription
      KubeAPIServerConfig
      github.com/gardener/gardener/pkg/apis/core/v1beta1.KubeAPIServerConfig

      (Members of KubeAPIServerConfig are embedded into this type.)

      (Optional)

      KubeAPIServerConfig contains all configuration values not specific to the virtual garden cluster.

      auditWebhook
      AuditWebhook
      (Optional)

      AuditWebhook contains settings related to an audit webhook configuration.

      authentication
      Authentication
      (Optional)

      Authentication contains settings related to authentication.

      resourcesToStoreInETCDEvents
      []GroupResource
      (Optional)

      ResourcesToStoreInETCDEvents contains a list of resources which should be stored in etcd-events instead of etcd-main. The ‘events’ resource is always stored in etcd-events. Note that adding or removing resources from this list will not migrate them automatically from the etcd-main to etcd-events or vice versa.

      sni
      SNI
      (Optional)

      SNI contains configuration options for the TLS SNI settings.

      KubeControllerManagerConfig

      (Appears on: Kubernetes)

      KubeControllerManagerConfig contains configuration settings for the kube-controller-manager.

      FieldDescription
      KubeControllerManagerConfig
      github.com/gardener/gardener/pkg/apis/core/v1beta1.KubeControllerManagerConfig

      (Members of KubeControllerManagerConfig are embedded into this type.)

      (Optional)

      KubeControllerManagerConfig contains all configuration values not specific to the virtual garden cluster.

      certificateSigningDuration
      Kubernetes meta/v1.Duration
      (Optional)

      CertificateSigningDuration is the maximum length of duration signed certificates will be given. Individual CSRs may request shorter certs by setting spec.expirationSeconds.

      Kubernetes

      (Appears on: VirtualCluster)

      Kubernetes contains the version and configuration options for the Kubernetes components of the virtual garden cluster.

      FieldDescription
      kubeAPIServer
      KubeAPIServerConfig
      (Optional)

      KubeAPIServer contains configuration settings for the kube-apiserver.

      kubeControllerManager
      KubeControllerManagerConfig
      (Optional)

      KubeControllerManager contains configuration settings for the kube-controller-manager.

      version
      string

      Version is the semantic Kubernetes version to use for the virtual garden cluster.

      Maintenance

      (Appears on: VirtualCluster)

      Maintenance contains information about the time window for maintenance operations.

      FieldDescription
      timeWindow
      github.com/gardener/gardener/pkg/apis/core/v1beta1.MaintenanceTimeWindow

      TimeWindow contains information about the time window for maintenance operations.

      Networking

      (Appears on: VirtualCluster)

      Networking defines networking parameters for the virtual garden cluster.

      FieldDescription
      services
      string

      Services is the CIDR of the service network. This field is immutable.

      ProjectQuotaConfiguration

      (Appears on: GardenerControllerManagerConfig)

      ProjectQuotaConfiguration defines quota configurations.

      FieldDescription
      config
      k8s.io/apimachinery/pkg/runtime.RawExtension

      Config is the quota specification used for the project set-up. Only v1.ResourceQuota resources are supported.

      projectSelector
      Kubernetes meta/v1.LabelSelector
      (Optional)

      ProjectSelector is an optional setting to select the projects considered for quotas. Defaults to empty LabelSelector, which matches all projects.

      Provider

      (Appears on: RuntimeCluster)

      Provider defines the provider-specific information for this cluster.

      FieldDescription
      zones
      []string
      (Optional)

      Zones is the list of availability zones the cluster is deployed to.

      ResourceAdmissionConfiguration

      (Appears on: GardenerAdmissionControllerConfig)

      ResourceAdmissionConfiguration contains settings about arbitrary kinds and the size each resource should have at most.

      FieldDescription
      limits
      []ResourceLimit

      Limits contains configuration for resources which are subjected to size limitations.

      unrestrictedSubjects
      []Kubernetes rbac/v1.Subject
      (Optional)

      UnrestrictedSubjects contains references to users, groups, or service accounts which aren’t subjected to any resource size limit.

      operationMode
      ResourceAdmissionWebhookMode
      (Optional)

      OperationMode specifies the mode the webhooks operates in. Allowed values are “block” and “log”. Defaults to “block”.

      ResourceAdmissionWebhookMode (string alias)

      (Appears on: ResourceAdmissionConfiguration)

      ResourceAdmissionWebhookMode is an alias type for the resource admission webhook mode.

      ResourceLimit

      (Appears on: ResourceAdmissionConfiguration)

      ResourceLimit contains settings about a kind and the size each resource should have at most.

      FieldDescription
      apiGroups
      []string
      (Optional)

      APIGroups is the name of the APIGroup that contains the limited resource. WildcardAll represents all groups.

      apiVersions
      []string
      (Optional)

      APIVersions is the version of the resource. WildcardAll represents all versions.

      resources
      []string

      Resources is the name of the resource this rule applies to. WildcardAll represents all resources.

      size
      k8s.io/apimachinery/pkg/api/resource.Quantity

      Size specifies the imposed limit.

      RuntimeCluster

      (Appears on: GardenSpec)

      RuntimeCluster contains configuration for the runtime cluster.

      FieldDescription
      ingress
      Ingress

      Ingress configures Ingress specific settings for the Garden cluster.

      networking
      RuntimeNetworking

      Networking defines the networking configuration of the runtime cluster.

      provider
      Provider

      Provider defines the provider-specific information for this cluster.

      settings
      Settings
      (Optional)

      Settings contains certain settings for this cluster.

      volume
      Volume
      (Optional)

      Volume contains settings for persistent volumes created in the runtime cluster.

      RuntimeNetworking

      (Appears on: RuntimeCluster)

      RuntimeNetworking defines the networking configuration of the runtime cluster.

      FieldDescription
      nodes
      string
      (Optional)

      Nodes is the CIDR of the node network. This field is immutable.

      pods
      string

      Pods is the CIDR of the pod network. This field is immutable.

      services
      string

      Services is the CIDR of the service network. This field is immutable.

      blockCIDRs
      []string
      (Optional)

      BlockCIDRs is a list of network addresses that should be blocked.

      SNI

      (Appears on: KubeAPIServerConfig)

      SNI contains configuration options for the TLS SNI settings.

      FieldDescription
      secretName
      string

      SecretName is the name of a secret containing the TLS certificate and private key.

      domainPatterns
      []string
      (Optional)

      DomainPatterns is a list of fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names.

      SettingLoadBalancerServices

      (Appears on: Settings)

      SettingLoadBalancerServices controls certain settings for services of type load balancer that are created in the runtime cluster.

      FieldDescription
      annotations
      map[string]string
      (Optional)

      Annotations is a map of annotations that will be injected/merged into every load balancer service object.

      SettingTopologyAwareRouting

      (Appears on: Settings)

      SettingTopologyAwareRouting controls certain settings for topology-aware traffic routing in the cluster. See https://github.com/gardener/gardener/blob/master/docs/operations/topology_aware_routing.md.

      FieldDescription
      enabled
      bool

      Enabled controls whether certain Services deployed in the cluster should be topology-aware. These Services are virtual-garden-etcd-main-client, virtual-garden-etcd-events-client and virtual-garden-kube-apiserver. Additionally, other components that are deployed to the runtime cluster via other means can read this field and according to its value enable/disable topology-aware routing for their Services.

      SettingVerticalPodAutoscaler

      (Appears on: Settings)

      SettingVerticalPodAutoscaler controls certain settings for the vertical pod autoscaler components deployed in the seed.

      FieldDescription
      enabled
      bool
      (Optional)

      Enabled controls whether the VPA components shall be deployed into this cluster. It is true by default because the operator (and Gardener) heavily rely on a VPA being deployed. You should only disable this if your runtime cluster already has another, manually/custom managed VPA deployment. If this is not the case, but you still disable it, then reconciliation will fail.

      Settings

      (Appears on: RuntimeCluster)

      Settings contains certain settings for this cluster.

      FieldDescription
      loadBalancerServices
      SettingLoadBalancerServices
      (Optional)

      LoadBalancerServices controls certain settings for services of type load balancer that are created in the runtime cluster.

      verticalPodAutoscaler
      SettingVerticalPodAutoscaler
      (Optional)

      VerticalPodAutoscaler controls certain settings for the vertical pod autoscaler components deployed in the cluster.

      topologyAwareRouting
      SettingTopologyAwareRouting
      (Optional)

      TopologyAwareRouting controls certain settings for topology-aware traffic routing in the cluster. See https://github.com/gardener/gardener/blob/master/docs/operations/topology_aware_routing.md.

      Storage

      (Appears on: ETCDEvents, ETCDMain)

      Storage contains storage configuration.

      FieldDescription
      capacity
      k8s.io/apimachinery/pkg/api/resource.Quantity
      (Optional)

      Capacity is the storage capacity for the volumes.

      className
      string
      (Optional)

      ClassName is the name of a storage class.

      VirtualCluster

      (Appears on: GardenSpec)

      VirtualCluster contains configuration for the virtual cluster.

      FieldDescription
      controlPlane
      ControlPlane
      (Optional)

      ControlPlane holds information about the general settings for the control plane of the virtual cluster.

      dns
      DNS

      DNS holds information about DNS settings.

      etcd
      ETCD
      (Optional)

      ETCD contains configuration for the etcds of the virtual garden cluster.

      gardener
      Gardener

      Gardener contains the configuration options for the Gardener control plane components.

      kubernetes
      Kubernetes

      Kubernetes contains the version and configuration options for the Kubernetes components of the virtual garden cluster.

      maintenance
      Maintenance

      Maintenance contains information about the time window for maintenance operations.

      networking
      Networking

      Networking contains information about cluster networking such as CIDRs, etc.

      Volume

      (Appears on: RuntimeCluster)

      Volume contains settings for persistent volumes created in the runtime cluster.

      FieldDescription
      minimumSize
      k8s.io/apimachinery/pkg/api/resource.Quantity
      (Optional)

      MinimumSize defines the minimum size that should be used for PVCs in the runtime cluster.


      Generated with gen-crd-api-reference-docs

      1.7 - Provider Local

      Packages:

      local.provider.extensions.gardener.cloud/v1alpha1

      Package v1alpha1 contains the local provider API resources.

      Resource Types:

      CloudProfileConfig

      CloudProfileConfig contains provider-specific configuration that is embedded into Gardener’s CloudProfile resource.

      FieldDescription
      apiVersion
      string
      local.provider.extensions.gardener.cloud/v1alpha1
      kind
      string
      CloudProfileConfig
      machineImages
      []MachineImages

      MachineImages is the list of machine images that are understood by the controller. It maps logical names and versions to provider-specific identifiers.

      WorkerStatus

      WorkerStatus contains information about created worker resources.

      FieldDescription
      apiVersion
      string
      local.provider.extensions.gardener.cloud/v1alpha1
      kind
      string
      WorkerStatus
      machineImages
      []MachineImage
      (Optional)

      MachineImages is a list of machine images that have been used in this worker. Usually, the extension controller gets the mapping from name/version to the provider-specific machine image data from the CloudProfile. However, if a version that is still in use gets removed from this componentconfig it cannot reconcile anymore existing Worker resources that are still using this version. Hence, it stores the used versions in the provider status to ensure reconciliation is possible.

      MachineImage

      (Appears on: WorkerStatus)

      MachineImage is a mapping from logical names and versions to provider-specific machine image data.

      FieldDescription
      name
      string

      Name is the logical name of the machine image.

      version
      string

      Version is the logical version of the machine image.

      image
      string

      Image is the image for the machine image.

      MachineImageVersion

      (Appears on: MachineImages)

      MachineImageVersion contains a version and a provider-specific identifier.

      FieldDescription
      version
      string

      Version is the version of the image.

      image
      string

      Image is the image for the machine image.

      MachineImages

      (Appears on: CloudProfileConfig)

      MachineImages is a mapping from logical names and versions to provider-specific identifiers.

      FieldDescription
      name
      string

      Name is the logical name of the machine image.

      versions
      []MachineImageVersion

      Versions contains versions and a provider-specific identifier.


      Generated with gen-crd-api-reference-docs

      1.8 - Resources

      Packages:

      resources.gardener.cloud/v1alpha1

      Package v1alpha1 contains the configuration of the Gardener Resource Manager.

      Resource Types:

        ManagedResource

        ManagedResource describes a list of managed resources.

        FieldDescription
        metadata
        Kubernetes meta/v1.ObjectMeta

        Standard object metadata.

        Refer to the Kubernetes API documentation for the fields of the metadata field.
        spec
        ManagedResourceSpec

        Spec contains the specification of this managed resource.



        class
        string
        (Optional)

        Class holds the resource class used to control the responsibility for multiple resource manager instances

        secretRefs
        []Kubernetes core/v1.LocalObjectReference

        SecretRefs is a list of secret references.

        injectLabels
        map[string]string
        (Optional)

        InjectLabels injects the provided labels into every resource that is part of the referenced secrets.

        forceOverwriteLabels
        bool
        (Optional)

        ForceOverwriteLabels specifies that all existing labels should be overwritten. Defaults to false.

        forceOverwriteAnnotations
        bool
        (Optional)

        ForceOverwriteAnnotations specifies that all existing annotations should be overwritten. Defaults to false.

        keepObjects
        bool
        (Optional)

        KeepObjects specifies whether the objects should be kept although the managed resource has already been deleted. Defaults to false.

        equivalences
        [][]k8s.io/apimachinery/pkg/apis/meta/v1.GroupKind
        (Optional)

        Equivalences specifies possible group/kind equivalences for objects.

        deletePersistentVolumeClaims
        bool
        (Optional)

        DeletePersistentVolumeClaims specifies if PersistentVolumeClaims created by StatefulSets, which are managed by this resource, should also be deleted when the corresponding StatefulSet is deleted (defaults to false).

        status
        ManagedResourceStatus

        Status contains the status of this managed resource.

        ManagedResourceSpec

        (Appears on: ManagedResource)

        ManagedResourceSpec contains the specification of this managed resource.

        FieldDescription
        class
        string
        (Optional)

        Class holds the resource class used to control the responsibility for multiple resource manager instances

        secretRefs
        []Kubernetes core/v1.LocalObjectReference

        SecretRefs is a list of secret references.

        injectLabels
        map[string]string
        (Optional)

        InjectLabels injects the provided labels into every resource that is part of the referenced secrets.

        forceOverwriteLabels
        bool
        (Optional)

        ForceOverwriteLabels specifies that all existing labels should be overwritten. Defaults to false.

        forceOverwriteAnnotations
        bool
        (Optional)

        ForceOverwriteAnnotations specifies that all existing annotations should be overwritten. Defaults to false.

        keepObjects
        bool
        (Optional)

        KeepObjects specifies whether the objects should be kept although the managed resource has already been deleted. Defaults to false.

        equivalences
        [][]k8s.io/apimachinery/pkg/apis/meta/v1.GroupKind
        (Optional)

        Equivalences specifies possible group/kind equivalences for objects.

        deletePersistentVolumeClaims
        bool
        (Optional)

        DeletePersistentVolumeClaims specifies if PersistentVolumeClaims created by StatefulSets, which are managed by this resource, should also be deleted when the corresponding StatefulSet is deleted (defaults to false).

        ManagedResourceStatus

        (Appears on: ManagedResource)

        ManagedResourceStatus is the status of a managed resource.

        FieldDescription
        conditions
        []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition
        observedGeneration
        int64

        ObservedGeneration is the most recent generation observed for this resource.

        resources
        []ObjectReference
        (Optional)

        Resources is a list of objects that have been created.

        secretsDataChecksum
        string
        (Optional)

        SecretsDataChecksum is the checksum of referenced secrets data.

        ObjectReference

        (Appears on: ManagedResourceStatus)

        ObjectReference is a reference to another object.

        FieldDescription
        ObjectReference
        Kubernetes core/v1.ObjectReference

        (Members of ObjectReference are embedded into this type.)

        labels
        map[string]string

        Labels is a map of labels that were used during last update of the resource.

        annotations
        map[string]string

        Annotations is a map of annotations that were used during last update of the resource.


        Generated with gen-crd-api-reference-docs

        1.9 - Security

        Packages:

        security.gardener.cloud/v1alpha1

        Package v1alpha1 is a version of the API.

        Resource Types:

        CredentialsBinding

        CredentialsBinding represents a binding to credentials in the same or another namespace.

        FieldDescription
        apiVersion
        string
        security.gardener.cloud/v1alpha1
        kind
        string
        CredentialsBinding
        metadata
        Kubernetes meta/v1.ObjectMeta
        (Optional)

        Standard object metadata.

        Refer to the Kubernetes API documentation for the fields of the metadata field.
        provider
        CredentialsBindingProvider

        Provider defines the provider type of the CredentialsBinding. This field is immutable.

        credentialsRef
        Kubernetes core/v1.ObjectReference

        CredentialsRef is a reference to a resource holding the credentials. Accepted resources are core/v1.Secret and security.gardener.cloud/v1alpha1.WorkloadIdentity

        quotas
        []Kubernetes core/v1.ObjectReference
        (Optional)

        Quotas is a list of references to Quota objects in the same or another namespace. This field is immutable.

        CredentialsBindingProvider

        (Appears on: CredentialsBinding)

        CredentialsBindingProvider defines the provider type of the CredentialsBinding.

        FieldDescription
        type
        string

        Type is the type of the provider.


        Generated with gen-crd-api-reference-docs

        1.10 - Seedmanagement

        Packages:

        seedmanagement.gardener.cloud/v1alpha1

        Package v1alpha1 is a version of the API.

        Resource Types:

        ManagedSeed

        ManagedSeed represents a Shoot that is registered as Seed.

        FieldDescription
        apiVersion
        string
        seedmanagement.gardener.cloud/v1alpha1
        kind
        string
        ManagedSeed
        metadata
        Kubernetes meta/v1.ObjectMeta
        (Optional)

        Standard object metadata.

        Refer to the Kubernetes API documentation for the fields of the metadata field.
        spec
        ManagedSeedSpec
        (Optional)

        Specification of the ManagedSeed.



        shoot
        Shoot
        (Optional)

        Shoot references a Shoot that should be registered as Seed. This field is immutable.

        gardenlet
        Gardenlet
        (Optional)

        Gardenlet specifies that the ManagedSeed controller should deploy a gardenlet into the cluster with the given deployment parameters and GardenletConfiguration.

        status
        ManagedSeedStatus
        (Optional)

        Most recently observed status of the ManagedSeed.

        ManagedSeedSet

        ManagedSeedSet represents a set of identical ManagedSeeds.

        FieldDescription
        apiVersion
        string
        seedmanagement.gardener.cloud/v1alpha1
        kind
        string
        ManagedSeedSet
        metadata
        Kubernetes meta/v1.ObjectMeta
        (Optional)

        Standard object metadata.

        Refer to the Kubernetes API documentation for the fields of the metadata field.
        spec
        ManagedSeedSetSpec
        (Optional)

        Spec defines the desired identities of ManagedSeeds and Shoots in this set.



        replicas
        int32
        (Optional)

        Replicas is the desired number of replicas of the given Template. Defaults to 1.

        selector
        Kubernetes meta/v1.LabelSelector

        Selector is a label query over ManagedSeeds and Shoots that should match the replica count. It must match the ManagedSeeds and Shoots template’s labels. This field is immutable.

        template
        ManagedSeedTemplate

        Template describes the ManagedSeed that will be created if insufficient replicas are detected. Each ManagedSeed created / updated by the ManagedSeedSet will fulfill this template.

        shootTemplate
        github.com/gardener/gardener/pkg/apis/core/v1beta1.ShootTemplate

        ShootTemplate describes the Shoot that will be created if insufficient replicas are detected for hosting the corresponding ManagedSeed. Each Shoot created / updated by the ManagedSeedSet will fulfill this template.

        updateStrategy
        UpdateStrategy
        (Optional)

        UpdateStrategy specifies the UpdateStrategy that will be employed to update ManagedSeeds / Shoots in the ManagedSeedSet when a revision is made to Template / ShootTemplate.

        revisionHistoryLimit
        int32
        (Optional)

        RevisionHistoryLimit is the maximum number of revisions that will be maintained in the ManagedSeedSet’s revision history. Defaults to 10. This field is immutable.

        status
        ManagedSeedSetStatus
        (Optional)

        Status is the current status of ManagedSeeds and Shoots in this ManagedSeedSet.

        Bootstrap (string alias)

        (Appears on: Gardenlet)

        Bootstrap describes a mechanism for bootstrapping gardenlet connection to the Garden cluster.

        Gardenlet

        (Appears on: ManagedSeedSpec)

        Gardenlet specifies gardenlet deployment parameters and the GardenletConfiguration used to configure gardenlet.

        FieldDescription
        deployment
        GardenletDeployment
        (Optional)

        Deployment specifies certain gardenlet deployment parameters, such as the number of replicas, the image, etc.

        config
        k8s.io/apimachinery/pkg/runtime.RawExtension
        (Optional)

        Config is the GardenletConfiguration used to configure gardenlet.

        bootstrap
        Bootstrap
        (Optional)

        Bootstrap is the mechanism that should be used for bootstrapping gardenlet connection to the Garden cluster. One of ServiceAccount, BootstrapToken, None. If set to ServiceAccount or BootstrapToken, a service account or a bootstrap token will be created in the garden cluster and used to compute the bootstrap kubeconfig. If set to None, the gardenClientConnection.kubeconfig field will be used to connect to the Garden cluster. Defaults to BootstrapToken. This field is immutable.

        mergeWithParent
        bool
        (Optional)

        MergeWithParent specifies whether the GardenletConfiguration of the parent gardenlet should be merged with the specified GardenletConfiguration. Defaults to true. This field is immutable.

        GardenletDeployment

        (Appears on: Gardenlet)

        GardenletDeployment specifies certain gardenlet deployment parameters, such as the number of replicas, the image, etc.

        FieldDescription
        replicaCount
        int32
        (Optional)

        ReplicaCount is the number of gardenlet replicas. Defaults to 2.

        revisionHistoryLimit
        int32
        (Optional)

        RevisionHistoryLimit is the number of old gardenlet ReplicaSets to retain to allow rollback. Defaults to 2.

        serviceAccountName
        string
        (Optional)

        ServiceAccountName is the name of the ServiceAccount to use to run gardenlet pods.

        image
        Image
        (Optional)

        Image is the gardenlet container image.

        resources
        Kubernetes core/v1.ResourceRequirements
        (Optional)

        Resources are the compute resources required by the gardenlet container.

        podLabels
        map[string]string
        (Optional)

        PodLabels are the labels on gardenlet pods.

        podAnnotations
        map[string]string
        (Optional)

        PodAnnotations are the annotations on gardenlet pods.

        additionalVolumes
        []Kubernetes core/v1.Volume
        (Optional)

        AdditionalVolumes is the list of additional volumes that should be mounted by gardenlet containers.

        additionalVolumeMounts
        []Kubernetes core/v1.VolumeMount
        (Optional)

        AdditionalVolumeMounts is the list of additional pod volumes to mount into the gardenlet container’s filesystem.

        env
        []Kubernetes core/v1.EnvVar
        (Optional)

        Env is the list of environment variables to set in the gardenlet container.

        vpa
        bool
        (Optional)

        VPA specifies whether to enable VPA for gardenlet. Defaults to true.

        Image

        (Appears on: GardenletDeployment)

        Image specifies container image parameters.

        FieldDescription
        repository
        string
        (Optional)

        Repository is the image repository.

        tag
        string
        (Optional)

        Tag is the image tag.

        pullPolicy
        Kubernetes core/v1.PullPolicy
        (Optional)

        PullPolicy is the image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if latest tag is specified, or IfNotPresent otherwise.

        ManagedSeedSetSpec

        (Appears on: ManagedSeedSet)

        ManagedSeedSetSpec is the specification of a ManagedSeedSet.

        FieldDescription
        replicas
        int32
        (Optional)

        Replicas is the desired number of replicas of the given Template. Defaults to 1.

        selector
        Kubernetes meta/v1.LabelSelector

        Selector is a label query over ManagedSeeds and Shoots that should match the replica count. It must match the ManagedSeeds and Shoots template’s labels. This field is immutable.

        template
        ManagedSeedTemplate

        Template describes the ManagedSeed that will be created if insufficient replicas are detected. Each ManagedSeed created / updated by the ManagedSeedSet will fulfill this template.

        shootTemplate
        github.com/gardener/gardener/pkg/apis/core/v1beta1.ShootTemplate

        ShootTemplate describes the Shoot that will be created if insufficient replicas are detected for hosting the corresponding ManagedSeed. Each Shoot created / updated by the ManagedSeedSet will fulfill this template.

        updateStrategy
        UpdateStrategy
        (Optional)

        UpdateStrategy specifies the UpdateStrategy that will be employed to update ManagedSeeds / Shoots in the ManagedSeedSet when a revision is made to Template / ShootTemplate.

        revisionHistoryLimit
        int32
        (Optional)

        RevisionHistoryLimit is the maximum number of revisions that will be maintained in the ManagedSeedSet’s revision history. Defaults to 10. This field is immutable.

        ManagedSeedSetStatus

        (Appears on: ManagedSeedSet)

        ManagedSeedSetStatus represents the current state of a ManagedSeedSet.

        FieldDescription
        observedGeneration
        int64

        ObservedGeneration is the most recent generation observed for this ManagedSeedSet. It corresponds to the ManagedSeedSet’s generation, which is updated on mutation by the API Server.

        replicas
        int32

        Replicas is the number of replicas (ManagedSeeds and their corresponding Shoots) created by the ManagedSeedSet controller.

        readyReplicas
        int32

        ReadyReplicas is the number of ManagedSeeds created by the ManagedSeedSet controller that have a Ready Condition.

        nextReplicaNumber
        int32

        NextReplicaNumber is the ordinal number that will be assigned to the next replica of the ManagedSeedSet.

        currentReplicas
        int32

        CurrentReplicas is the number of ManagedSeeds created by the ManagedSeedSet controller from the ManagedSeedSet version indicated by CurrentRevision.

        updatedReplicas
        int32

        UpdatedReplicas is the number of ManagedSeeds created by the ManagedSeedSet controller from the ManagedSeedSet version indicated by UpdateRevision.

        currentRevision
        string

        CurrentRevision, if not empty, indicates the version of the ManagedSeedSet used to generate ManagedSeeds with smaller ordinal numbers during updates.

        updateRevision
        string

        UpdateRevision, if not empty, indicates the version of the ManagedSeedSet used to generate ManagedSeeds with larger ordinal numbers during updates

        collisionCount
        int32
        (Optional)

        CollisionCount is the count of hash collisions for the ManagedSeedSet. The ManagedSeedSet controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest ControllerRevision.

        conditions
        []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition
        (Optional)

        Conditions represents the latest available observations of a ManagedSeedSet’s current state.

        pendingReplica
        PendingReplica
        (Optional)

        PendingReplica, if not empty, indicates the replica that is currently pending creation, update, or deletion. This replica is in a state that requires the controller to wait for it to change before advancing to the next replica.

        ManagedSeedSpec

        (Appears on: ManagedSeed, ManagedSeedTemplate)

        ManagedSeedSpec is the specification of a ManagedSeed.

        FieldDescription
        shoot
        Shoot
        (Optional)

        Shoot references a Shoot that should be registered as Seed. This field is immutable.

        gardenlet
        Gardenlet
        (Optional)

        Gardenlet specifies that the ManagedSeed controller should deploy a gardenlet into the cluster with the given deployment parameters and GardenletConfiguration.

        ManagedSeedStatus

        (Appears on: ManagedSeed)

        ManagedSeedStatus is the status of a ManagedSeed.

        FieldDescription
        conditions
        []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition
        (Optional)

        Conditions represents the latest available observations of a ManagedSeed’s current state.

        observedGeneration
        int64

        ObservedGeneration is the most recent generation observed for this ManagedSeed. It corresponds to the ManagedSeed’s generation, which is updated on mutation by the API Server.

        ManagedSeedTemplate

        (Appears on: ManagedSeedSetSpec)

        ManagedSeedTemplate is a template for creating a ManagedSeed object.

        FieldDescription
        metadata
        Kubernetes meta/v1.ObjectMeta
        (Optional)

        Standard object metadata.

        Refer to the Kubernetes API documentation for the fields of the metadata field.
        spec
        ManagedSeedSpec
        (Optional)

        Specification of the desired behavior of the ManagedSeed.



        shoot
        Shoot
        (Optional)

        Shoot references a Shoot that should be registered as Seed. This field is immutable.

        gardenlet
        Gardenlet
        (Optional)

        Gardenlet specifies that the ManagedSeed controller should deploy a gardenlet into the cluster with the given deployment parameters and GardenletConfiguration.

        PendingReplica

        (Appears on: ManagedSeedSetStatus)

        PendingReplica contains information about a replica that is currently pending creation, update, or deletion.

        FieldDescription
        name
        string

        Name is the replica name.

        reason
        PendingReplicaReason

        Reason is the reason for the replica to be pending.

        since
        Kubernetes meta/v1.Time

        Since is the moment in time since the replica is pending with the specified reason.

        retries
        int32
        (Optional)

        Retries is the number of times the shoot operation (reconcile or delete) has been retried after having failed. Only applicable if Reason is ShootReconciling or ShootDeleting.

        PendingReplicaReason (string alias)

        (Appears on: PendingReplica)

        PendingReplicaReason is a string enumeration type that enumerates all possible reasons for a replica to be pending.

        RollingUpdateStrategy

        (Appears on: UpdateStrategy)

        RollingUpdateStrategy is used to communicate parameters for RollingUpdateStrategyType.

        FieldDescription
        partition
        int32
        (Optional)

        Partition indicates the ordinal at which the ManagedSeedSet should be partitioned. Defaults to 0.

        Shoot

        (Appears on: ManagedSeedSpec)

        Shoot identifies the Shoot that should be registered as Seed.

        FieldDescription
        name
        string

        Name is the name of the Shoot that will be registered as Seed.

        UpdateStrategy

        (Appears on: ManagedSeedSetSpec)

        UpdateStrategy specifies the strategy that the ManagedSeedSet controller will use to perform updates. It includes any additional parameters necessary to perform the update for the indicated strategy.

        FieldDescription
        type
        UpdateStrategyType
        (Optional)

        Type indicates the type of the UpdateStrategy. Defaults to RollingUpdate.

        rollingUpdate
        RollingUpdateStrategy
        (Optional)

        RollingUpdate is used to communicate parameters when Type is RollingUpdateStrategyType.

        UpdateStrategyType (string alias)

        (Appears on: UpdateStrategy)

        UpdateStrategyType is a string enumeration type that enumerates all possible update strategies for the ManagedSeedSet controller.


        Generated with gen-crd-api-reference-docs

        1.11 - Settings

        Packages:

        settings.gardener.cloud/v1alpha1

        Package v1alpha1 is a version of the API.

        Resource Types:

        ClusterOpenIDConnectPreset

        ClusterOpenIDConnectPreset is a OpenID Connect configuration that is applied to a Shoot objects cluster-wide.

        FieldDescription
        apiVersion
        string
        settings.gardener.cloud/v1alpha1
        kind
        string
        ClusterOpenIDConnectPreset
        metadata
        Kubernetes meta/v1.ObjectMeta

        Standard object metadata.

        Refer to the Kubernetes API documentation for the fields of the metadata field.
        spec
        ClusterOpenIDConnectPresetSpec

        Spec is the specification of this OpenIDConnect preset.



        OpenIDConnectPresetSpec
        OpenIDConnectPresetSpec

        (Members of OpenIDConnectPresetSpec are embedded into this type.)

        projectSelector
        Kubernetes meta/v1.LabelSelector
        (Optional)

        Project decides whether to apply the configuration if the Shoot is in a specific Project matching the label selector. Use the selector only if the OIDC Preset is opt-in, because end users may skip the admission by setting the labels. Defaults to the empty LabelSelector, which matches everything.

        OpenIDConnectPreset

        OpenIDConnectPreset is a OpenID Connect configuration that is applied to a Shoot in a namespace.

        FieldDescription
        apiVersion
        string
        settings.gardener.cloud/v1alpha1
        kind
        string
        OpenIDConnectPreset
        metadata
        Kubernetes meta/v1.ObjectMeta

        Standard object metadata.

        Refer to the Kubernetes API documentation for the fields of the metadata field.
        spec
        OpenIDConnectPresetSpec

        Spec is the specification of this OpenIDConnect preset.



        server
        KubeAPIServerOpenIDConnect

        Server contains the kube-apiserver’s OpenID Connect configuration. This configuration is not overwriting any existing OpenID Connect configuration already set on the Shoot object.

        client
        OpenIDConnectClientAuthentication
        (Optional)

        Client contains the configuration used for client OIDC authentication of Shoot clusters. This configuration is not overwriting any existing OpenID Connect client authentication already set on the Shoot object.

        shootSelector
        Kubernetes meta/v1.LabelSelector
        (Optional)

        ShootSelector decides whether to apply the configuration if the Shoot has matching labels. Use the selector only if the OIDC Preset is opt-in, because end users may skip the admission by setting the labels. Default to the empty LabelSelector, which matches everything.

        weight
        int32

        Weight associated with matching the corresponding preset, in the range 1-100. Required.

        ClusterOpenIDConnectPresetSpec

        (Appears on: ClusterOpenIDConnectPreset)

        ClusterOpenIDConnectPresetSpec contains the OpenIDConnect specification and project selector matching Shoots in Projects.

        FieldDescription
        OpenIDConnectPresetSpec
        OpenIDConnectPresetSpec

        (Members of OpenIDConnectPresetSpec are embedded into this type.)

        projectSelector
        Kubernetes meta/v1.LabelSelector
        (Optional)

        Project decides whether to apply the configuration if the Shoot is in a specific Project matching the label selector. Use the selector only if the OIDC Preset is opt-in, because end users may skip the admission by setting the labels. Defaults to the empty LabelSelector, which matches everything.

        KubeAPIServerOpenIDConnect

        (Appears on: OpenIDConnectPresetSpec)

        KubeAPIServerOpenIDConnect contains configuration settings for the OIDC provider. Note: Descriptions were taken from the Kubernetes documentation.

        FieldDescription
        caBundle
        string
        (Optional)

        If set, the OpenID server’s certificate will be verified by one of the authorities in the oidc-ca-file, otherwise the host’s root CA set will be used.

        clientID
        string

        The client ID for the OpenID Connect client. Required.

        groupsClaim
        string
        (Optional)

        If provided, the name of a custom OpenID Connect claim for specifying user groups. The claim value is expected to be a string or array of strings. This field is experimental, please see the authentication documentation for further details.

        groupsPrefix
        string
        (Optional)

        If provided, all groups will be prefixed with this value to prevent conflicts with other authentication strategies.

        issuerURL
        string

        The URL of the OpenID issuer, only HTTPS scheme will be accepted. If set, it will be used to verify the OIDC JSON Web Token (JWT). Required.

        requiredClaims
        map[string]string
        (Optional)

        key=value pairs that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value.

        signingAlgs
        []string
        (Optional)

        List of allowed JOSE asymmetric signing algorithms. JWTs with a ‘alg’ header value not in this list will be rejected. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1 Defaults to [RS256]

        usernameClaim
        string
        (Optional)

        The OpenID claim to use as the user name. Note that claims other than the default (‘sub’) is not guaranteed to be unique and immutable. This field is experimental, please see the authentication documentation for further details. Defaults to “sub”.

        usernamePrefix
        string
        (Optional)

        If provided, all usernames will be prefixed with this value. If not provided, username claims other than ‘email’ are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value ‘-’.

        OpenIDConnectClientAuthentication

        (Appears on: OpenIDConnectPresetSpec)

        OpenIDConnectClientAuthentication contains configuration for OIDC clients.

        FieldDescription
        secret
        string
        (Optional)

        The client Secret for the OpenID Connect client.

        extraConfig
        map[string]string
        (Optional)

        Extra configuration added to kubeconfig’s auth-provider. Must not be any of idp-issuer-url, client-id, client-secret, idp-certificate-authority, idp-certificate-authority-data, id-token or refresh-token

        OpenIDConnectPresetSpec

        (Appears on: OpenIDConnectPreset, ClusterOpenIDConnectPresetSpec)

        OpenIDConnectPresetSpec contains the Shoot selector for which a specific OpenID Connect configuration is applied.

        FieldDescription
        server
        KubeAPIServerOpenIDConnect

        Server contains the kube-apiserver’s OpenID Connect configuration. This configuration is not overwriting any existing OpenID Connect configuration already set on the Shoot object.

        client
        OpenIDConnectClientAuthentication
        (Optional)

        Client contains the configuration used for client OIDC authentication of Shoot clusters. This configuration is not overwriting any existing OpenID Connect client authentication already set on the Shoot object.

        shootSelector
        Kubernetes meta/v1.LabelSelector
        (Optional)

        ShootSelector decides whether to apply the configuration if the Shoot has matching labels. Use the selector only if the OIDC Preset is opt-in, because end users may skip the admission by setting the labels. Default to the empty LabelSelector, which matches everything.

        weight
        int32

        Weight associated with matching the corresponding preset, in the range 1-100. Required.


        Generated with gen-crd-api-reference-docs

        2 - Concepts

        2.1 - APIServer Admission Plugins

        A list of all gardener managed admission plugins together with their responsibilities

        Overview

        Similar to the kube-apiserver, the gardener-apiserver comes with a few in-tree managed admission plugins. If you want to get an overview of the what and why of admission plugins then this document might be a good start.

        This document lists all existing admission plugins with a short explanation of what it is responsible for.

        ClusterOpenIDConnectPreset, OpenIDConnectPreset

        (both enabled by default)

        These admission controllers react on CREATE operations for Shoots. If the Shoot does not specify any OIDC configuration (.spec.kubernetes.kubeAPIServer.oidcConfig=nil), then it tries to find a matching ClusterOpenIDConnectPreset or OpenIDConnectPreset, respectively. If there are multiple matches, then the one with the highest weight “wins”. In this case, the admission controller will default the OIDC configuration in the Shoot.

        ControllerRegistrationResources

        (enabled by default)

        This admission controller reacts on CREATE and UPDATE operations for ControllerRegistrations. It validates that there exists only one ControllerRegistration in the system that is primarily responsible for a given kind/type resource combination. This prevents misconfiguration by the Gardener administrator/operator.

        CustomVerbAuthorizer

        (enabled by default)

        This admission controller reacts on CREATE and UPDATE operations for Projects. It validates whether the user is bound to a RBAC role with the modify-spec-tolerations-whitelist verb in case the user tries to change the .spec.tolerations.whitelist field of the respective Project resource. Usually, regular project members are not bound to this custom verb, allowing the Gardener administrator to manage certain toleration whitelists on Project basis.

        DeletionConfirmation

        (enabled by default)

        This admission controller reacts on DELETE operations for Projects, Shoots, and ShootStates. It validates that the respective resource is annotated with a deletion confirmation annotation, namely confirmation.gardener.cloud/deletion=true. Only if this annotation is present it allows the DELETE operation to pass. This prevents users from accidental/undesired deletions. In addition, it applies the “four-eyes principle for deletion” concept if the Project is configured accordingly. Find all information about it in this document.

        Furthermore, this admission controller reacts on CREATE or UPDATE operations for Shoots. It makes sure that the deletion.gardener.cloud/confirmed-by annotation is properly maintained in case the Shoot deletion is confirmed with above mentioned annotation.

        ExposureClass

        (enabled by default)

        This admission controller reacts on Create operations for Shoots. It mutates Shoot resources which have an ExposureClass referenced by merging both their shootSelectors and/or tolerations into the Shoot resource.

        ExtensionValidator

        (enabled by default)

        This admission controller reacts on CREATE and UPDATE operations for BackupEntrys, BackupBuckets, Seeds, and Shoots. For all the various extension types in the specifications of these objects, it validates whether there exists a ControllerRegistration in the system that is primarily responsible for the stated extension type(s). This prevents misconfigurations that would otherwise allow users to create such resources with extension types that don’t exist in the cluster, effectively leading to failing reconciliation loops.

        ExtensionLabels

        (enabled by default)

        This admission controller reacts on CREATE and UPDATE operations for BackupBuckets, BackupEntrys, CloudProfiles, Seeds, SecretBindings and Shoots. For all the various extension types in the specifications of these objects, it adds a corresponding label in the resource. This would allow extension admission webhooks to filter out the resources they are responsible for and ignore all others. This label is of the form <extension-type>.extensions.gardener.cloud/<extension-name> : "true". For example, an extension label for provider extension type aws, looks like provider.extensions.gardener.cloud/aws : "true".

        ProjectValidator

        (enabled by default)

        This admission controller reacts on CREATE operations for Projects. It prevents creating Projects with a non-empty .spec.namespace if the value in .spec.namespace does not start with garden-.

        ⚠️ This admission plugin will be removed in a future release and its business logic will be incorporated into the static validation of the gardener-apiserver.

        ResourceQuota

        (enabled by default)

        This admission controller enables object count ResourceQuotas for Gardener resources, e.g. Shoots, SecretBindings, Projects, etc.

        ⚠️ In addition to this admission plugin, the ResourceQuota controller must be enabled for the Kube-Controller-Manager of your Garden cluster.

        ResourceReferenceManager

        (enabled by default)

        This admission controller reacts on CREATE and UPDATE operations for CloudProfiles, Projects, SecretBindings, Seeds, and Shoots. Generally, it checks whether referred resources stated in the specifications of these objects exist in the system (e.g., if a referenced Secret exists). However, it also has some special behaviours for certain resources:

        • CloudProfiles: It rejects removing Kubernetes or machine image versions if there is at least one Shoot that refers to them.
        • Projects: It sets the .spec.createdBy field for newly created Project resources, and defaults the .spec.owner field in case it is empty (to the same value of .spec.createdBy).
        • Shoots: It sets the gardener.cloud/created-by=<username> annotation for newly created Shoot resources.

        SeedValidator

        (enabled by default)

        This admission controller reacts on DELETE operations for Seeds. Rejects the deletion if Shoot(s) reference the seed cluster.

        ShootDNS

        (enabled by default)

        This admission controller reacts on CREATE and UPDATE operations for Shoots. It tries to assign a default domain to the Shoot. It also validates the DNS configuration (.spec.dns) for shoots.

        ShootNodeLocalDNSEnabledByDefault

        (disabled by default)

        This admission controller reacts on CREATE operations for Shoots. If enabled, it will enable node local dns within the shoot cluster (for more information, see NodeLocalDNS Configuration) by setting spec.systemComponents.nodeLocalDNS.enabled=true for newly created Shoots. Already existing Shoots and new Shoots that explicitly disable node local dns (spec.systemComponents.nodeLocalDNS.enabled=false) will not be affected by this admission plugin.

        ShootQuotaValidator

        (enabled by default)

        This admission controller reacts on CREATE and UPDATE operations for Shoots. It validates the resource consumption declared in the specification against applicable Quota resources. Only if the applicable Quota resources admit the configured resources in the Shoot then it allows the request. Applicable Quotas are referred in the SecretBinding that is used by the Shoot.

        ShootResourceReservation

        (enabled by default)

        This admission controller reacts on CREATE and UPDATE operations for Shoots. It injects the Kubernetes.Kubelet.KubeReserved setting for kubelet either as global setting for a shoot or on a per worker pool basis. If the admission configuration (see this example) for the ShootResourceReservation plugin contains useGKEFormula: false (the default), then it sets a static default resource reservation for the shoot.

        If useGKEFormula: true is set, then the plugin injects resource reservations based on the machine type similar to GKE’s formula for resource reservation into each worker pool. Already existing resource reservations are not modified; this also means that resource reservations are not automatically updated if the machine type for a worker pool is changed. If a shoot contains global resource reservations, then no per worker pool resource reservations are injected.

        ShootVPAEnabledByDefault

        (disabled by default)

        This admission controller reacts on CREATE operations for Shoots. If enabled, it will enable the managed VerticalPodAutoscaler components (for more information, see Vertical Pod Auto-Scaling) by setting spec.kubernetes.verticalPodAutoscaler.enabled=true for newly created Shoots. Already existing Shoots and new Shoots that explicitly disable VPA (spec.kubernetes.verticalPodAutoscaler.enabled=false) will not be affected by this admission plugin.

        ShootTolerationRestriction

        (enabled by default)

        This admission controller reacts on CREATE and UPDATE operations for Shoots. It validates the .spec.tolerations used in Shoots against the whitelist of its Project, or against the whitelist configured in the admission controller’s configuration, respectively. Additionally, it defaults the .spec.tolerations in Shoots with those configured in its Project, and those configured in the admission controller’s configuration, respectively.

        ShootValidator

        (enabled by default)

        This admission controller reacts on CREATE, UPDATE and DELETE operations for Shoots. It validates certain configurations in the specification against the referred CloudProfile (e.g., machine images, machine types, used Kubernetes version, …). Generally, it performs validations that cannot be handled by the static API validation due to their dynamic nature (e.g., when something needs to be checked against referred resources). Additionally, it takes over certain defaulting tasks (e.g., default machine image for worker pools, default Kubernetes version).

        ShootManagedSeed

        (enabled by default)

        This admission controller reacts on UPDATE and DELETE operations for Shoots. It validates certain configuration values in the specification that are specific to ManagedSeeds (e.g. the nginx-addon of the Shoot has to be disabled, the Shoot VPA has to be enabled). It rejects the deletion if the Shoot is referred to by a ManagedSeed.

        ManagedSeedValidator

        (enabled by default)

        This admission controller reacts on CREATE and UPDATE operations for ManagedSeedss. It validates certain configuration values in the specification against the referred Shoot, for example Seed provider, network ranges, DNS domain, etc. Similar to ShootValidator, it performs validations that cannot be handled by the static API validation due to their dynamic nature. Additionally, it performs certain defaulting tasks, making sure that configuration values that are not specified are defaulted to the values of the referred Shoot, for example Seed provider, network ranges, DNS domain, etc.

        ManagedSeedShoot

        (enabled by default)

        This admission controller reacts on DELETE operations for ManagedSeeds. It rejects the deletion if there are Shoots that are scheduled onto the Seed that is registered by the ManagedSeed.

        ShootDNSRewriting

        (disabled by default)

        This admission controller reacts on CREATE operations for Shoots. If enabled, it adds a set of common suffixes configured in its admission plugin configuration to the Shoot (spec.systemComponents.coreDNS.rewriting.commonSuffixes) (for more information, see DNS Search Path Optimization). Already existing Shoots will not be affected by this admission plugin.

        NamespacedCloudProfileValidator

        (enabled by default)

        This admission controller reacts on CREATE and UPDATE operations for NamespacedCloudProfiles. It primarily validates if the referenced parent CloudProfile exists in the system. In addition, the admission controller ensures that the NamespacedCloudProfile only configures new machine types, and does not overwrite those from the parent CloudProfile.

        2.2 - Architecture

        The concepts behind the Gardener architecture

        Official Definition - What is Kubernetes?

        “Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.”

        Introduction - Basic Principle

        The foundation of the Gardener (providing Kubernetes Clusters as a Service) is Kubernetes itself, because Kubernetes is the go-to solution to manage software in the Cloud, even when it’s Kubernetes itself (see also OpenStack which is provisioned more and more on top of Kubernetes as well).

        While self-hosting, meaning to run Kubernetes components inside Kubernetes, is a popular topic in the community, we apply a special pattern catering to the needs of our cloud platform to provision hundreds or even thousands of clusters. We take a so-called “seed” cluster and seed the control plane (such as the API server, scheduler, controllers, etcd persistence and others) of an end-user cluster, which we call “shoot” cluster, as pods into the “seed” cluster. That means that one “seed” cluster, of which we will have one per IaaS and region, hosts the control planes of multiple “shoot” clusters. That allows us to avoid dedicated hardware/virtual machines for the “shoot” cluster control planes. We simply put the control plane into pods/containers and since the “seed” cluster watches them, they can be deployed with a replica count of 1 and only need to be scaled out when the control plane gets under pressure, but no longer for HA reasons. At the same time, the deployments get simpler (standard Kubernetes deployment) and easier to update (standard Kubernetes rolling update). The actual “shoot” cluster consists only of the worker nodes (no control plane) and therefore the users may get full administrative access to their clusters.

        Setting The Scene - Components and Procedure

        We provide a central operator UI, which we call the “Gardener Dashboard”. It talks to a dedicated cluster, which we call the “Garden” cluster, and uses custom resources managed by an aggregated API server (one of the general extension concepts of Kubernetes) to represent “shoot” clusters. In this “Garden” cluster runs the “Gardener”, which is basically a Kubernetes controller that watches the custom resources and acts upon them, i.e. creates, updates/modifies, or deletes “shoot” clusters. The creation follows basically these steps:

        • Create a namespace in the “seed” cluster for the “shoot” cluster, which will host the “shoot” cluster control plane.
        • Generate secrets and credentials, which the worker nodes will need to talk to the control plane.
        • Create the infrastructure (using Terraform), which basically consists out of the network setup.
        • Deploy the “shoot” cluster control plane into the “shoot” namespace in the “seed” cluster, containing the “machine-controller-manager” pod.
        • Create machine CRDs in the “seed” cluster, describing the configuration and the number of worker machines for the “shoot” (the machine-controller-manager watches the CRDs and creates virtual machines out of it).
        • Wait for the “shoot” cluster API server to become responsive (pods will be scheduled, persistent volumes and load balancers are created by Kubernetes via the respective cloud provider).
        • Finally, we deploy kube-system daemons like kube-proxy and further add-ons like the dashboard into the “shoot” cluster and the cluster becomes active.

        Overview Architecture Diagram

        Gardener Overview Architecture Diagram

        Detailed Architecture Diagram

        Gardener Detailed Architecture Diagram

        Note: The kubelet, as well as the pods inside the “shoot” cluster, talks through the front-door (load balancer IP; public Internet) to its “shoot” cluster API server running in the “seed” cluster. The reverse communication from the API server to the pod, service, and node networks happens through a VPN connection that we deploy into the “seed” and “shoot” clusters.

        2.3 - Backup and Restore

        Understand the etcd backup and restore capabilities of Gardener

        Overview

        Kubernetes uses etcd as the key-value store for its resource definitions. Gardener supports the backup and restore of etcd. It is the responsibility of the shoot owners to backup the workload data.

        Gardener uses an etcd-backup-restore component to backup the etcd backing the Shoot cluster regularly and restore it in case of disaster. It is deployed as sidecar via etcd-druid. This doc mainly focuses on the backup and restore configuration used by Gardener when deploying these components. For more details on the design and internal implementation details, please refer to GEP-06 and the documentation on individual repositories.

        Bucket Provisioning

        Refer to the backup bucket extension document to find out details about configuring the backup bucket.

        Backup Policy

        etcd-backup-restore supports full snapshot and delta snapshots over full snapshot. In Gardener, this configuration is currently hard-coded to the following parameters:

        • Full Snapshot schedule:
          • Daily, 24hr interval.
          • For each Shoot, the schedule time in a day is randomized based on the configured Shoot maintenance window.
        • Delta Snapshot schedule:
          • At 5min interval.
          • If aggregated events size since last snapshot goes beyond 100Mib.
        • Backup History / Garbage backup deletion policy:
          • Gardener configures backup restore to have Exponential garbage collection policy.
          • As per policy, the following backups are retained:
            • All full backups and delta backups for the previous hour.
            • Latest full snapshot of each previous hour for the day.
            • Latest full snapshot of each previous day for 7 days.
            • Latest full snapshot of the previous 4 weeks.
          • Garbage Collection is configured at 12hr interval.
        • Listing:
          • Gardener doesn’t have any API to list out the backups.
          • To find the backups list, an admin can checkout the BackupEntry resource associated with the Shoot which holds the bucket and prefix details on the object store.

        Restoration

        The restoration process of etcd is automated through the etcd-backup-restore component from the latest snapshot. Gardener doesn’t support Point-In-Time-Recovery (PITR) of etcd. In case of an etcd disaster, the etcd is recovered from the latest backup automatically. For further details, please refer the Restoration topic. Post restoration of etcd, the Shoot reconciliation loop brings the cluster back to its previous state.

        Again, the Shoot owner is responsible for maintaining the backup/restore of his workload. Gardener only takes care of the cluster’s etcd.

        2.4 - Cluster API

        Understand the evolution of the Gardener API and its relation to the Cluster API

        Relation Between Gardener API and Cluster API (SIG Cluster Lifecycle)

        In essence, the Cluster API harmonizes how to get to clusters, while Gardener goes one step further and also harmonizes the clusters themselves. The Cluster API delegates the specifics to so-called providers for infrastructures or control planes via specific CR(D)s, while Gardener only has one cluster CR(D). Different Cluster API providers, e.g. for AWS, Azure, GCP, etc., give you vastly different Kubernetes clusters. In contrast, Gardener gives you the exact same clusters with the exact same K8s version, operating system, control plane configuration like for API server or kubelet, add-ons like overlay network, HPA/VPA, DNS and certificate controllers, ingress and network policy controllers, control plane monitoring and logging stacks, down to the behavior of update procedures, auto-scaling, self-healing, etc., on all supported infrastructures. These homogeneous clusters are an essential goal for Gardener, as its main purpose is to simplify operations for teams that need to develop and ship software on Kubernetes clusters on a plethora of infrastructures (a.k.a. multi-cloud).

        Incidentally, Gardener influenced the Machine API in the Cluster API with its Machine Controller Manager and was the first to adopt it. You can find more information on that in the joint SIG Cluster Lifecycle KubeCon talk where @hardikdr from our Gardener team in India spoke.

        That means that we follow the Cluster API with great interest and are active members. It was completely overhauled from v1alpha1 to v1alpha2. But because v1alpha2 made too many assumptions about the bring-up of masters and was enforcing master machine operations (for more information, see The Cluster API Book: “As of v1alpha2, Machine-Based is the only control plane type that Cluster API supports”), services that managed their control planes differently like GKE or Gardener couldn’t adopt it (e.g. Google only supports v1alpha1). In 2020 v1alpha3 was introduced and made it possible (again) to integrate managed services like GKE or Gardener. The mapping from the Gardener API to the Cluster API is mostly syntactic.

        To wrap it up, while the Cluster API knows about clusters, it doesn’t know about their make-up. With Gardener, we wanted to go beyond that and harmonize the make-up of the clusters themselves and make them homogeneous across all supported infrastructures. Gardener can therefore deliver homogeneous clusters with exactly the same configuration and behavior on all infrastructures (see also Gardener’s coverage in the official conformance test grid).

        With Cluster API v1alpha3 and the support for declarative control plane management, it has became possible (again) to enable Kubernetes managed services like GKE or Gardener. We would be more than happy if the community would be interested to contribute a Gardener control plane provider.

        2.5 - etcd

        How Gardener uses the etcd key-value store

        etcd - Key-Value Store for Kubernetes

        etcd is a strongly consistent key-value store and the most prevalent choice for the Kubernetes persistence layer. All API cluster objects like Pods, Deployments, Secrets, etc., are stored in etcd, which makes it an essential part of a Kubernetes control plane.

        Garden or Shoot Cluster Persistence

        Each garden or shoot cluster gets its very own persistence for the control plane. It runs in the shoot namespace on the respective seed cluster (or in the garden namespace in the garden cluster, respectively). Concretely, there are two etcd instances per shoot cluster, which the kube-apiserver is configured to use in the following way:

        • etcd-main

        A store that contains all “cluster critical” or “long-term” objects. These object kinds are typically considered for a backup to prevent any data loss.

        • etcd-events

        A store that contains all Event objects (events.k8s.io) of a cluster. Events usually have a short retention period and occur frequently, but are not essential for a disaster recovery.

        The setup above prevents both, the critical etcd-main is not flooded by Kubernetes Events, as well as backup space is not occupied by non-critical data. This separation saves time and resources.

        etcd Operator

        Configuring, maintaining, and health-checking etcd is outsourced to a dedicated operator called etcd Druid. When a gardenlet reconciles a Shoot resource or a gardener-operator reconciles a Garden resource, they manage an Etcd resource in the seed or garden cluster, containing necessary information (backup information, defragmentation schedule, resources, etc.). etcd-druid needs to manage the lifecycle of the desired etcd instance (today main or events). Likewise, when the Shoot or Garden is deleted, gardenlet or gardener-operator deletes the Etcd resources and etcd Druid takes care of cleaning up all related objects, e.g. the backing StatefulSets.

        Backup

        If Seeds specify backups for etcd (example), then Gardener and the respective provider extensions are responsible for creating a bucket on the cloud provider’s side (modelled through a BackupBucket resource). The bucket stores backups of Shoots scheduled on that Seed. Furthermore, Gardener creates a BackupEntry, which subdivides the bucket and thus makes it possible to store backups of multiple shoot clusters.

        How long backups are stored in the bucket after a shoot has been deleted depends on the configured retention period in the Seed resource. Please see this example configuration for more information.

        For Gardens specifying backups for etcd (example), the bucket must be pre-created externally and provided via the Garden specification.

        Both etcd instances are configured to run with a special backup-restore sidecar. It takes care about regularly backing up etcd data and restoring it in case of data loss (in the main etcd only). The sidecar also performs defragmentation and other house-keeping tasks. More information can be found in the component’s GitHub repository.

        Housekeeping

        etcd maintenance tasks must be performed from time to time in order to re-gain database storage and to ensure the system’s reliability. The backup-restore sidecar takes care about this job as well.

        For both Shoots and Gardens, a random time within the shoot’s maintenance time is chosen for scheduling these tasks.

        2.6 - Gardener Admission Controller

        Functions and list of handlers for the Gardener Admission Controller

        Overview

        While the Gardener API server works with admission plugins to validate and mutate resources belonging to Gardener related API groups, e.g. core.gardener.cloud, the same is needed for resources belonging to non-Gardener API groups as well, e.g. secrets in the core API group. Therefore, the Gardener Admission Controller runs a http(s) server with the following handlers which serve as validating/mutating endpoints for admission webhooks. It is also used to serve http(s) handlers for authorization webhooks.

        Admission Webhook Handlers

        This section describes the admission webhook handlers that are currently served.

        Admission Plugin Secret Validator

        In Shoot, AdmissionPlugin can have reference to other files. This validation handler validates the referred admission plugin secret and ensures that the secret always contains the required data kubeconfig.

        Kubeconfig Secret Validator

        Malicious Kubeconfigs applied by end users may cause a leakage of sensitive data. This handler checks if the incoming request contains a Kubernetes secret with a .data.kubeconfig field and denies the request if the Kubeconfig structure violates Gardener’s security standards.

        Namespace Validator

        Namespaces are the backing entities of Gardener projects in which shoot cluster objects reside. This validation handler protects active namespaces against premature deletion requests. Therefore, it denies deletion requests if a namespace still contains shoot clusters or if it belongs to a non-deleting Gardener project (without .metadata.deletionTimestamp).

        Resource Size Validator

        Since users directly apply Kubernetes native objects to the Garden cluster, it also involves the risk of being vulnerable to DoS attacks because these resources are continuously watched and read by controllers. One example is the creation of shoot resources with large annotation values (up to 256 kB per value), which can cause severe out-of-memory issues for the gardenlet component. Vertical autoscaling can help to mitigate such situations, but we cannot expect to scale infinitely, and thus need means to block the attack itself.

        The Resource Size Validator checks arbitrary incoming admission requests against a configured maximum size for the resource’s group-version-kind combination. It denies the request if the object exceeds the quota.

        Example for Gardener Admission Controller configuration:

        server:
          resourceAdmissionConfiguration:
            limits:
            - apiGroups: ["core.gardener.cloud"]
              apiVersions: ["*"]
              resources: ["shoots"]
              size: 100k
            - apiGroups: [""]
              apiVersions: ["v1"]
              resources: ["secrets"]
              size: 100k
            unrestrictedSubjects:
            - kind: Group
              name: gardener.cloud:system:seeds
              apiGroup: rbac.authorization.k8s.io
         #  - kind: User
         #    name: admin
         #    apiGroup: rbac.authorization.k8s.io
         #  - kind: ServiceAccount
         #    name: "*"
         #    namespace: garden
         #    apiGroup: ""
            operationMode: block #log
        

        With the configuration above, the Resource Size Validator denies requests for shoots with Gardener’s core API group which exceed a size of 100 kB. The same is done for Kubernetes secrets.

        As this feature is meant to protect the system from malicious requests sent by users, it is recommended to exclude trusted groups, users or service accounts from the size restriction via resourceAdmissionConfiguration.unrestrictedSubjects. For example, the backing user for the gardenlet should always be capable of changing the shoot resource instead of being blocked due to size restrictions. This is because the gardenlet itself occasionally changes the shoot specification, labels or annotations, and might violate the quota if the existing resource is already close to the quota boundary. Also, operators are supposed to be trusted users and subjecting them to a size limitation can inhibit important operational tasks. Wildcard ("*") in subject name is supported.

        Size limitations depend on the individual Gardener setup and choosing the wrong values can affect the availability of your Gardener service. resourceAdmissionConfiguration.operationMode allows to control if a violating request is actually denied (default) or only logged. It’s recommended to start with log, check the logs for exceeding requests, adjust the limits if necessary and finally switch to block.

        SeedRestriction

        Please refer to Scoped API Access for Gardenlets for more information.

        Authorization Webhook Handlers

        This section describes the authorization webhook handlers that are currently served.

        SeedAuthorization

        Please refer to Scoped API Access for Gardenlets for more information.

        2.7 - Gardener API Server

        Understand the Gardener API server extension and the resources it exposes

        Overview

        The Gardener API server is a Kubernetes-native extension based on its aggregation layer. It is registered via an APIService object and designed to run inside a Kubernetes cluster whose API it wants to extend.

        After registration, it exposes the following resources:

        CloudProfiles

        CloudProfiles are resources that describe a specific environment of an underlying infrastructure provider, e.g. AWS, Azure, etc. Each shoot has to reference a CloudProfile to declare the environment it should be created in. In a CloudProfile, the gardener operator specifies certain constraints like available machine types, regions, which Kubernetes versions they want to offer, etc. End-users can read CloudProfiles to see these values, but only operators can change the content or create/delete them. When a shoot is created or updated, then an admission plugin checks that only allowed values are used via the referenced CloudProfile.

        Additionally, a CloudProfile may contain a providerConfig, which is a special configuration dedicated for the infrastructure provider. Gardener does not evaluate or understand this config, but extension controllers might need it for declaration of provider-specific constraints, or global settings.

        Please see this example manifest and consult the documentation of your provider extension controller to get information about its providerConfig.

        NamespacedCloudProfiles

        In addition to CloudProfiles, NamespacedCloudProfiles exist to enable project level CloudProfiles. Please view GEP-25 for additional information. This feature is currently under development and not ready for productive use. At the moment, only the necessary APIs and validations exist to allow for extensions to adapt to the new NamespacedCloudProfile resource.

        InternalSecrets

        End-users can read and/or write Secrets in their project namespaces in the garden cluster. This prevents Gardener components from storing such “Gardener-internal” secrets in the respective project namespace. InternalSecrets are resources that contain shoot or project-related secrets that are “Gardener-internal”, i.e., secrets used and managed by the system that end-users don’t have access to. InternalSecrets are defined like plain Kubernetes Secrets, behave exactly like them, and can be used in the same manners. The only difference is, that the InternalSecret resource is a dedicated API resource (exposed by gardener-apiserver). This allows separating access to “normal” secrets and internal secrets by the usual RBAC means.

        Gardener uses an InternalSecret per Shoot for syncing the client CA to the project namespace in the garden cluster (named <shoot-name>.ca-client). The shoots/adminkubeconfig subresource signs short-lived client certificates by retrieving the CA from the InternalSecret.

        Operators should configure gardener-apiserver to encrypt the internalsecrets.core.gardener.cloud resource in etcd.

        Please see this example manifest.

        Seeds

        Seeds are resources that represent seed clusters. Gardener does not care about how a seed cluster got created - the only requirement is that it is of at least Kubernetes v1.25 and passes the Kubernetes conformance tests. The Gardener operator has to either deploy the gardenlet into the cluster they want to use as seed (recommended, then the gardenlet will create the Seed object itself after bootstrapping) or provide the kubeconfig to the cluster inside a secret (that is referenced by the Seed resource) and create the Seed resource themselves.

        Please see this, this, and optionally this example manifests.

        Shoot Quotas

        To allow end-users not having their dedicated infrastructure account to try out Gardener, the operator can register an account owned by them that they allow to be used for trial clusters. Trial clusters can be put under quota so that they don’t consume too many resources (resulting in costs) and that one user cannot consume all resources on their own. These clusters are automatically terminated after a specified time, but end-users may extend the lifetime manually if needed.

        Please see this example manifest.

        Projects

        The first thing before creating a shoot cluster is to create a Project. A project is used to group multiple shoot clusters together. End-users can invite colleagues to the project to enable collaboration, and they can either make them admin or viewer. After an end-user has created a project, they will get a dedicated namespace in the garden cluster for all their shoots.

        Please see this example manifest.

        SecretBindings

        Now that the end-user has a namespace the next step is registering their infrastructure provider account.

        Please see this example manifest and consult the documentation of the extension controller for the respective infrastructure provider to get information about which keys are required in this secret.

        After the secret has been created, the end-user has to create a special SecretBinding resource that binds this secret. Later, when creating shoot clusters, they will reference such binding.

        Please see this example manifest.

        Shoots

        Shoot cluster contain various settings that influence how end-user Kubernetes clusters will look like in the end. As Gardener heavily relies on extension controllers for operating system configuration, networking, and infrastructure specifics, the end-user has the possibility (and responsibility) to provide these provider-specific configurations as well. Such configurations are not evaluated by Gardener (because it doesn’t know/understand them), but they are only transported to the respective extension controller.

        ⚠️ This means that any configuration issues/mistake on the end-user side that relates to a provider-specific flag or setting cannot be caught during the update request itself but only later during the reconciliation (unless a validator webhook has been registered in the garden cluster by an operator).

        Please see this example manifest and consult the documentation of the provider extension controller to get information about its spec.provider.controlPlaneConfig, .spec.provider.infrastructureConfig, and .spec.provider.workers[].providerConfig.

        (Cluster)OpenIDConnectPresets

        Please see this separate documentation file.

        Overview Data Model

        Gardener Overview Data Model

        2.8 - Gardener Controller Manager

        Understand where the gardener-controller-manager runs and its functionalities

        Overview

        The gardener-controller-manager (often referred to as “GCM”) is a component that runs next to the Gardener API server, similar to the Kubernetes Controller Manager. It runs several controllers that do not require talking to any seed or shoot cluster. Also, as of today, it exposes an HTTP server that is serving several health check endpoints and metrics.

        This document explains the various functionalities of the gardener-controller-manager and their purpose.

        Controllers

        Bastion Controller

        Bastion resources have a limited lifetime which can be extended up to a certain amount by performing a heartbeat on them. The Bastion controller is responsible for deleting expired or rotten Bastions.

        • “expired” means a Bastion has exceeded its status.expirationTimestamp.
        • “rotten” means a Bastion is older than the configured maxLifetime.

        The maxLifetime defaults to 24 hours and is an option in the BastionControllerConfiguration which is part of gardener-controller-managers ControllerManagerControllerConfiguration, see the example config file for details.

        The controller also deletes Bastions in case the referenced Shoot:

        • no longer exists
        • is marked for deletion (i.e., have a non-nil .metadata.deletionTimestamp)
        • was migrated to another seed (i.e., Shoot.spec.seedName is different than Bastion.spec.seedName).

        The deletion of Bastions triggers the gardenlet to perform the necessary cleanups in the Seed cluster, so some time can pass between deletion and the Bastion actually disappearing. Clients like gardenctl are advised to not re-use Bastions whose deletion timestamp has been set already.

        Refer to GEP-15 for more information on the lifecycle of Bastion resources.

        CertificateSigningRequest Controller

        After the gardenlet gets deployed on the Seed cluster, it needs to establish itself as a trusted party to communicate with the Gardener API server. It runs through a bootstrap flow similar to the kubelet bootstrap process.

        On startup, the gardenlet uses a kubeconfig with a bootstrap token which authenticates it as being part of the system:bootstrappers group. This kubeconfig is used to create a CertificateSigningRequest (CSR) against the Gardener API server.

        The controller in gardener-controller-manager checks whether the CertificateSigningRequest has the expected organization, common name and usages which the gardenlet would request.

        It only auto-approves the CSR if the client making the request is allowed to “create” the certificatesigningrequests/seedclient subresource. Clients with the system:bootstrappers group are bound to the gardener.cloud:system:seed-bootstrapper ClusterRole, hence, they have such privileges. As the bootstrap kubeconfig for the gardenlet contains a bootstrap token which is authenticated as being part of the systems:bootstrappers group, its created CSR gets auto-approved.

        CloudProfile Controller

        CloudProfiles are essential when it comes to reconciling Shoots since they contain constraints (like valid machine types, Kubernetes versions, or machine images) and sometimes also some global configuration for the respective environment (typically via provider-specific configuration in .spec.providerConfig).

        Consequently, to ensure that CloudProfiles in-use are always present in the system until the last referring Shoot gets deleted, the controller adds a finalizer which is only released when there is no Shoot referencing the CloudProfile anymore.

        ControllerDeployment Controller

        Extensions are registered in the garden cluster via ControllerRegistration and deployment of respective extensions are specified via ControllerDeployment. For more info refer to Registering Extension Controllers.

        This controller ensures that ControllerDeployment in-use always exists until the last ControllerRegistration referencing them gets deleted. The controller adds a finalizer which is only released when there is no ControllerRegistration referencing the ControllerDeployment anymore.

        ControllerRegistration Controller

        The ControllerRegistration controller makes sure that the required Gardener Extensions specified by the ControllerRegistration resources are present in the seed clusters. It also takes care of the creation and deletion of ControllerInstallation objects for a given seed cluster. The controller has three reconciliation loops.

        “Main” Reconciler

        This reconciliation loop watches the Seed objects and determines which ControllerRegistrations are required for them and reconciles the corresponding ControllerInstallation resources to reach the determined state. To begin with, it computes the kind/type combinations of extensions required for the seed. For this, the controller examines a live list of ControllerRegistrations, ControllerInstallations, BackupBuckets, BackupEntrys, Shoots, and Secrets from the garden cluster. For example, it examines the shoots running on the seed and deducts the kind/type, like Infrastructure/gcp. The seed (seed.spec.provider.type) and DNS (seed.spec.dns.provider.type) provider types are considered when calculating the list of required ControllerRegistrations, as well. It also decides whether they should always be deployed based on the .spec.deployment.policy. For the configuration options, please see this section.

        Based on these required combinations, each of them are mapped to ControllerRegistration objects and then to their corresponding ControllerInstallation objects (if existing). The controller then creates or updates the required ControllerInstallation objects for the given seed. It also deletes every existing ControllerInstallation whose referenced ControllerRegistration is not part of the required list. For example, if the shoots in the seed are no longer using the DNS provider aws-route53, then the controller proceeds to delete the respective ControllerInstallation object.

        "ControllerRegistration Finalizer" Reconciler

        This reconciliation loop watches the ControllerRegistration resource and adds finalizers to it when they are created. In case a deletion request comes in for the resource, i.e., if a .metadata.deletionTimestamp is set, it actively scans for a ControllerInstallation resource using this ControllerRegistration, and decides whether the deletion can be allowed. In case no related ControllerInstallation is present, it removes the finalizer and marks it for deletion.

        "Seed Finalizer" Reconciler

        This loop also watches the Seed object and adds finalizers to it at creation. If a .metadata.deletionTimestamp is set for the seed, then the controller checks for existing ControllerInstallation objects which reference this seed. If no such objects exist, then it removes the finalizer and allows the deletion.

        “Extension ClusterRole” Reconciler

        This reconciler watches two resources in the garden cluster:

        • ClusterRoles labelled with authorization.gardener.cloud/custom-extensions-permissions=true
        • ServiceAccounts in seed namespaces matching the selector provided via the authorization.gardener.cloud/extensions-serviceaccount-selector annotation of such ClusterRoles.

        Its core task is to maintain a ClusterRoleBinding resource referencing the respective ClusterRole. This gets bound to all ServiceAccounts in seed namespaces whose labels match the selector provided via the authorization.gardener.cloud/extensions-serviceaccount-selector annotation of such ClusterRoles.

        You can read more about the purpose of this reconciler in this document.

        Event Controller

        With the Gardener Event Controller, you can prolong the lifespan of events related to Shoot clusters. This is an optional controller which will become active once you provide the below mentioned configuration.

        All events in K8s are deleted after a configurable time-to-live (controlled via a kube-apiserver argument called --event-ttl (defaulting to 1 hour)). The need to prolong the time-to-live for Shoot cluster events frequently arises when debugging customer issues on live systems. This controller leaves events involving Shoots untouched, while deleting all other events after a configured time. In order to activate it, provide the following configuration:

        • concurrentSyncs: The amount of goroutines scheduled for reconciling events.
        • ttlNonShootEvents: When an event reaches this time-to-live it gets deleted unless it is a Shoot-related event (defaults to 1h, equivalent to the event-ttl default).

        ⚠️ In addition, you should also configure the --event-ttl for the kube-apiserver to define an upper-limit of how long Shoot-related events should be stored. The --event-ttl should be larger than the ttlNonShootEvents or this controller will have no effect.

        ExposureClass Controller

        ExposureClass abstracts the ability to expose a Shoot clusters control plane in certain network environments (e.g. corporate networks, DMZ, internet) on all Seeds or a subset of the Seeds. For more information, see ExposureClasses.

        Consequently, to ensure that ExposureClasses in-use are always present in the system until the last referring Shoot gets deleted, the controller adds a finalizer which is only released when there is no Shoot referencing the ExposureClass anymore.

        ManagedSeedSet Controller

        ManagedSeedSet objects maintain a stable set of replicas of ManagedSeeds, i.e. they guarantee the availability of a specified number of identical ManagedSeeds on an equal number of identical Shoots. The ManagedSeedSet controller creates and deletes ManagedSeeds and Shoots in response to changes to the replicas and selector fields. For more information, refer to the ManagedSeedSet proposal document.

        1. The reconciler first gets all the replicas of the given ManagedSeedSet in the ManagedSeedSet’s namespace and with the matching selector. Each replica is a struct that contains a ManagedSeed, its corresponding Seed and Shoot objects.
        2. Then the pending replica is retrieved, if it exists.
        3. Next it determines the ready, postponed, and deletable replicas.
          • A replica is considered ready when a Seed owned by a ManagedSeed has been registered either directly or by deploying gardenlet into a Shoot, the Seed is Ready and the Shoot’s status is Healthy.
          • If a replica is not ready and it is not pending, i.e. it is not specified in the ManagedSeed’s status.pendingReplica field, then it is added to the postponed replicas.
          • A replica is deletable if it has no scheduled Shoots and the replica’s Shoot and ManagedSeed do not have the seedmanagement.gardener.cloud/protect-from-deletion annotation.
        4. Finally, it checks the actual and target replica counts. If the actual count is less than the target count, the controller scales up the replicas by creating new replicas to match the desired target count. If the actual count is more than the target, the controller deletes replicas to match the desired count. Before scale-out or scale-in, the controller first reconciles the pending replica (there can always only be one) and makes sure the replica is ready before moving on to the next one.
          • Scale-out(actual count < target count)
            • During the scale-out phase, the controller first creates the Shoot object from the ManagedSeedSet’s spec.shootTemplate field and adds the replica to the status.pendingReplica of the ManagedSeedSet.
            • For the subsequent reconciliation steps, the controller makes sure that the pending replica is ready before proceeding to the next replica. Once the Shoot is created successfully, the ManagedSeed object is created from the ManagedSeedSet’s spec.template. The ManagedSeed object is reconciled by the ManagedSeed controller and a Seed object is created for the replica. Once the replica’s Seed becomes ready and the Shoot becomes healthy, the replica also becomes ready.
          • Scale-in(actual count > target count)
            • During the scale-in phase, the controller first determines the replica that can be deleted. From the deletable replicas, it chooses the one with the lowest priority and deletes it. Priority is determined in the following order:
              • First, compare replica statuses. Replicas with “less advanced” status are considered lower priority. For example, a replica with StatusShootReconciling status has a lower value than a replica with StatusShootReconciled status. Hence, in this case, a replica with a StatusShootReconciling status will have lower priority and will be considered for deletion.
              • Then, the replicas are compared with the readiness of their Seeds. Replicas with non-ready Seeds are considered lower priority.
              • Then, the replicas are compared with the health statuses of their Shoots. Replicas with “worse” statuses are considered lower priority.
              • Finally, the replica ordinals are compared. Replicas with lower ordinals are considered lower priority.

        Quota Controller

        Quota object limits the resources consumed by shoot clusters either per provider secret or per project/namespace.

        Consequently, to ensure that Quotas in-use are always present in the system until the last SecretBinding that references them gets deleted, the controller adds a finalizer which is only released when there is no SecretBinding referencing the Quota anymore.

        Project Controller

        There are multiple controllers responsible for different aspects of Project objects. Please also refer to the Project documentation.

        “Main” Reconciler

        This reconciler manages a dedicated Namespace for each Project. The namespace name can either be specified explicitly in .spec.namespace (must be prefixed with garden-) or it will be determined by the controller. If .spec.namespace is set, it tries to create it. If it already exists, it tries to adopt it. This will only succeed if the Namespace was previously labeled with gardener.cloud/role=project and project.gardener.cloud/name=<project-name>. This is to prevent end-users from being able to adopt arbitrary namespaces and escalate their privileges, e.g. the kube-system namespace.

        After the namespace was created/adopted, the controller creates several ClusterRoles and ClusterRoleBindings that allow the project members to access related resources based on their roles. These RBAC resources are prefixed with gardener.cloud:system:project{-member,-viewer}:<project-name>. Gardener administrators and extension developers can define their own roles. For more information, see Extending Project Roles for more information.

        In addition, operators can configure the Project controller to maintain a default ResourceQuota for project namespaces. Quotas can especially limit the creation of user facing resources, e.g. Shoots, SecretBindings, Secrets and thus protect the garden cluster from massive resource exhaustion but also enable operators to align quotas with respective enterprise policies.

        ⚠️ Gardener itself is not exempted from configured quotas. For example, Gardener creates Secrets for every shoot cluster in the project namespace and at the same time increases the available quota count. Please mind this additional resource consumption.

        The controller configuration provides a template section controllers.project.quotas where such a ResourceQuota (see the example below) can be deposited.

        controllers:
          project:
            quotas:
            - config:
                apiVersion: v1
                kind: ResourceQuota
                spec:
                  hard:
                    count/shoots.core.gardener.cloud: "100"
                    count/secretbindings.core.gardener.cloud: "10"
                    count/secrets: "800"
              projectSelector: {}
        

        The Project controller takes the specified config and creates a ResourceQuota with the name gardener in the project namespace. If a ResourceQuota resource with the name gardener already exists, the controller will only update fields in spec.hard which are unavailable at that time. This is done to configure a default Quota in all projects but to allow manual quota increases as the projects’ demands increase. spec.hard fields in the ResourceQuota object that are not present in the configuration are removed from the object. Labels and annotations on the ResourceQuota config get merged with the respective fields on existing ResourceQuotas. An optional projectSelector narrows down the amount of projects that are equipped with the given config. If multiple configs match for a project, then only the first match in the list is applied to the project namespace.

        The .status.phase of the Project resources is set to Ready or Failed by the reconciler to indicate whether the reconciliation loop was performed successfully. Also, it generates Events to provide further information about its operations.

        When a Project is marked for deletion, the controller ensures that there are no Shoots left in the project namespace. Once all Shoots are gone, the Namespace and Project are released.

        “Stale Projects” Reconciler

        As Gardener is a large-scale Kubernetes as a Service, it is designed for being used by a large amount of end-users. Over time, it is likely to happen that some of the hundreds or thousands of Project resources are no longer actively used.

        Gardener offers the “stale projects” reconciler which will take care of identifying such stale projects, marking them with a “warning”, and eventually deleting them after a certain time period. This reconciler is enabled by default and works as follows:

        1. Projects are considered as “stale”/not actively used when all of the following conditions apply: The namespace associated with the Project does not have any…
          1. Shoot resources.
          2. BackupEntry resources.
          3. Secret resources that are referenced by a SecretBinding that is in use by a Shoot (not necessarily in the same namespace).
          4. Quota resources that are referenced by a SecretBinding that is in use by a Shoot (not necessarily in the same namespace).
          5. The time period when the project was used for the last time (status.lastActivityTimestamp) is longer than the configured minimumLifetimeDays

        If a project is considered “stale”, then its .status.staleSinceTimestamp will be set to the time when it was first detected to be stale. If it gets actively used again, this timestamp will be removed. After some time, the .status.staleAutoDeleteTimestamp will be set to a timestamp after which Gardener will auto-delete the Project resource if it still is not actively used.

        The component configuration of the gardener-controller-manager offers to configure the following options:

        • minimumLifetimeDays: Don’t consider newly created Projects as “stale” too early to give people/end-users some time to onboard and get familiar with the system. The “stale project” reconciler won’t set any timestamp for Projects younger than minimumLifetimeDays. When you change this value, then projects marked as “stale” may be no longer marked as “stale” in case they are young enough, or vice versa.
        • staleGracePeriodDays: Don’t compute auto-delete timestamps for stale Projects that are unused for less than staleGracePeriodDays. This is to not unnecessarily make people/end-users nervous “just because” they haven’t actively used their Project for a given amount of time. When you change this value, then already assigned auto-delete timestamps may be removed if the new grace period is not yet exceeded.
        • staleExpirationTimeDays: Expiration time after which stale Projects are finally auto-deleted (after .status.staleSinceTimestamp). If this value is changed and an auto-delete timestamp got already assigned to the projects, then the new value will only take effect if it’s increased. Hence, decreasing the staleExpirationTimeDays will not decrease already assigned auto-delete timestamps.

        Gardener administrators/operators can exclude specific Projects from the stale check by annotating the related Namespace resource with project.gardener.cloud/skip-stale-check=true.

        “Activity” Reconciler

        Since the other two reconcilers are unable to actively monitor the relevant objects that are used in a Project (Shoot, Secret, etc.), there could be a situation where the user creates and deletes objects in a short period of time. In that case, the Stale Project Reconciler could not see that there was any activity on that project and it will still mark it as a Stale, even though it is actively used.

        The Project Activity Reconciler is implemented to take care of such cases. An event handler will notify the reconciler for any activity and then it will update the status.lastActivityTimestamp. This update will also trigger the Stale Project Reconciler.

        SecretBinding Controller

        SecretBindings reference Secrets and Quotas and are themselves referenced by Shoots. The controller adds finalizers to the referenced objects to ensure they don’t get deleted while still being referenced. Similarly, to ensure that SecretBindings in-use are always present in the system until the last referring Shoot gets deleted, the controller adds a finalizer which is only released when there is no Shoot referencing the SecretBinding anymore.

        Referenced Secrets will also be labeled with provider.shoot.gardener.cloud/<type>=true, where <type> is the value of the .provider.type of the SecretBinding. Also, all referenced Secrets, as well as Quotas, will be labeled with reference.gardener.cloud/secretbinding=true to allow for easily filtering for objects referenced by SecretBindings.

        Seed Controller

        The Seed controller in the gardener-controller-manager reconciles Seed objects with the help of the following reconcilers.

        “Main” Reconciler

        This reconciliation loop takes care of seed related operations in the garden cluster. When a new Seed object is created, the reconciler creates a new Namespace in the garden cluster seed-<seed-name>. Namespaces dedicated to single seed clusters allow us to segregate access permissions i.e., a gardenlet must not have permissions to access objects in all Namespaces in the garden cluster. There are objects in a Garden environment which are created once by the operator e.g., default domain secret, alerting credentials, and are required for operations happening in the gardenlet. Therefore, we not only need a seed specific Namespace but also a copy of these “shared” objects.

        The “main” reconciler takes care about this replication:

        KindNamespaceLabel Selector
        Secretgardengardener.cloud/role

        “Backup Buckets Check” Reconciler

        Every time a BackupBucket object is created or updated, the referenced Seed object is enqueued for reconciliation. It’s the reconciler’s task to check the status subresource of all existing BackupBuckets that reference this Seed. If at least one BackupBucket has .status.lastError != nil, the BackupBucketsReady condition on the Seed will be set to False, and consequently the Seed is considered as NotReady. If the SeedBackupBucketsCheckControllerConfiguration (which is part of gardener-controller-managers component configuration) contains a conditionThreshold for the BackupBucketsReady, the condition will instead first be set to Progressing and eventually to False once the conditionThreshold expires. See the example config file for details. Once the BackupBucket is healthy again, the seed will be re-queued and the condition will turn true.

        “Extensions Check” Reconciler

        This reconciler reconciles Seed objects and checks whether all ControllerInstallations referencing them are in a healthy state. Concretely, all three conditions Valid, Installed, and Healthy must have status True and the Progressing condition must have status False. Based on this check, it maintains the ExtensionsReady condition in the respective Seed’s .status.conditions list.

        “Lifecycle” Reconciler

        The “Lifecycle” reconciler processes Seed objects which are enqueued every 10 seconds in order to check if the responsible gardenlet is still responding and operable. Therefore, it checks renewals via Lease objects of the seed in the garden cluster which are renewed regularly by the gardenlet.

        In case a Lease is not renewed for the configured amount in config.controllers.seed.monitorPeriod.duration:

        1. The reconciler assumes that the gardenlet stopped operating and updates the GardenletReady condition to Unknown.
        2. Additionally, the conditions and constraints of all Shoot resources scheduled on the affected seed are set to Unknown as well, because a striking gardenlet won’t be able to maintain these conditions any more.
        3. If the gardenlet’s client certificate has expired (identified based on the .status.clientCertificateExpirationTimestamp field in the Seed resource) and if it is managed by a ManagedSeed, then this will be triggered for a reconciliation. This will trigger the bootstrapping process again and allows gardenlets to obtain a fresh client certificate.

        Shoot Controller

        “Conditions” Reconciler

        In case the reconciled Shoot is registered via a ManagedSeed as a seed cluster, this reconciler merges the conditions in the respective Seed’s .status.conditions into the .status.conditions of the Shoot. This is to provide a holistic view on the status of the registered seed cluster by just looking at the Shoot resource.

        “Hibernation” Reconciler

        This reconciler is responsible for hibernating or awakening shoot clusters based on the schedules defined in their .spec.hibernation.schedules. It ignores failed Shoots and those marked for deletion.

        “Maintenance” Reconciler

        This reconciler is responsible for maintaining shoot clusters based on the time window defined in their .spec.maintenance.timeWindow. It might auto-update the Kubernetes version or the operating system versions specified in the worker pools (.spec.provider.workers). It could also add some operation or task annotations. For more information, see Shoot Maintenance.

        “Quota” Reconciler

        This reconciler might auto-delete shoot clusters in case their referenced SecretBinding is itself referencing a Quota with .spec.clusterLifetimeDays != nil. If the shoot cluster is older than the configured lifetime, then it gets deleted. It maintains the expiration time of the Shoot in the value of the shoot.gardener.cloud/expiration-timestamp annotation. This annotation might be overridden, however only by at most twice the value of the .spec.clusterLifetimeDays.

        “Reference” Reconciler

        Shoot objects may specify references to other objects in the garden cluster which are required for certain features. For example, users can configure various DNS providers via .spec.dns.providers and usually need to refer to a corresponding Secret with valid DNS provider credentials inside. Such objects need a special protection against deletion requests as long as they are still being referenced by one or multiple shoots.

        Therefore, this reconciler checks Shoots for referenced objects and adds the finalizer gardener.cloud/reference-protection to their .metadata.finalizers list. The reconciled Shoot also gets this finalizer to enable a proper garbage collection in case the gardener-controller-manager is offline at the moment of an incoming deletion request. When an object is not actively referenced anymore because the Shoot specification has changed or all related shoots were deleted (are in deletion), the controller will remove the added finalizer again so that the object can safely be deleted or garbage collected.

        This reconciler inspects the following references:

        • DNS provider secrets (.spec.dns.provider)
        • Audit policy configmaps (.spec.kubernetes.kubeAPIServer.auditConfig.auditPolicy.configMapRef)

        Further checks might be added in the future.

        “Retry” Reconciler

        This reconciler is responsible for retrying certain failed Shoots. Currently, the reconciler retries only failed Shoots with an error code ERR_INFRA_RATE_LIMITS_EXCEEDED. See Shoot Status for more details.

        “Status Label” Reconciler

        This reconciler is responsible for maintaining the shoot.gardener.cloud/status label on Shoots. See Shoot Status for more details.

        2.9 - Gardener Node Agent

        How Gardener bootstraps machines into worker nodes and how it installs and maintains gardener-managed node-specific components

        Overview

        The goal of the gardener-node-agent is to bootstrap a machine into a worker node and maintain node-specific components, which run on the node and are unmanaged by Kubernetes (e.g. the kubelet service, systemd units, …).

        It effectively is a Kubernetes controller deployed onto the worker node.

        Architecture and Basic Design

        Design

        This figure visualizes the overall architecture of the gardener-node-agent. On the left side, it starts with an OperatingSystemConfig resource (OSC) with a corresponding worker pool specific cloud-config-<worker-pool> secret being passed by reference through the userdata to a machine by the machine-controller-manager (MCM).

        On the right side, the cloud-config secret will be extracted and used by the gardener-node-agent after being installed. Details on this can be found in the next section.

        Finally, the gardener-node-agent runs a systemd service watching on secret resources located in the kube-system namespace like our cloud-config secret that contains the OperatingSystemConfig. When gardener-node-agent applies the OSC, it installs the kubelet + configuration on the worker node.

        Installation and Bootstrapping

        This section describes how the gardener-node-agent is initially installed onto the worker node.

        In the beginning, there is a very small bash script called gardener-node-init.sh, which will be copied to /var/lib/gardener-node-agent/init.sh on the node with cloud-init data. This script’s sole purpose is downloading and starting the gardener-node-agent. The binary artifact is extracted from an OCI artifact and lives at /opt/bin/gardener-node-agent.

        Along with the init script, a configuration for the gardener-node-agent is carried over to the worker node at /var/lib/gardener-node-agent/config.yaml. This configuration contains things like the shoot’s kube-apiserver endpoint, the according certificates to communicate with it, and controller configuration.

        In a bootstrapping phase, the gardener-node-agent sets itself up as a systemd service. It also executes tasks that need to be executed before any other components are installed, e.g. formatting the data device for the kubelet.

        Controllers

        This section describes the controllers in more details.

        Lease Controller

        This controller creates a Lease for gardener-node-agent in kube-system namespace of the shoot cluster. Each instance of gardener-node-agent creates its own Lease when its corresponding Node was created. It renews the Lease resource every 10 seconds. This indicates a heartbeat to the external world.

        Node Controller

        This controller watches the Node object for the machine it runs on. The correct Node is identified based on the hostname of the machine (Nodes have the kubernetes.io/hostname label). Whenever the worker.gardener.cloud/restart-systemd-services annotation changes, the controller performs the desired changes by restarting the specified systemd unit files. See also this document for more information. After restarting all units, the annotation is removed.

        ℹ️ When the gardener-node-agent systemd service itself is requested to be restarted, the annotation is removed first to ensure it does not restart itself indefinitely.

        Operating System Config Controller

        This controller contains the main logic of gardener-node-agent. It watches Secrets whose data map contains the OperatingSystemConfig which consists of all systemd units and files that are relevant for the node configuration. Amongst others, a prominent example is the configuration file for kubelet and its unit file for the kubelet.service.

        The controller decodes the configuration and computes the files and units that have changed since its last reconciliation. It writes or update the files and units to the file system, removes no longer needed files and units, reloads the systemd daemon, and starts or stops the units accordingly.

        After successful reconciliation, it persists the just applied OperatingSystemConfig into a file on the host. This file will be used for future reconciliations to compute file/unit changes.

        The controller also maintains two annotations on the Node:

        • worker.gardener.cloud/kubernetes-version, describing the version of the installed kubelet.
        • checksum/cloud-config-data, describing the checksum of the applied OperatingSystemConfig (used in future reconciliations to determine whether it needs to reconcile, and to report that this node is up-to-date).

        Token Controller

        This controller watches the access token Secrets in the kube-system namespace configured via the gardener-node-agent’s component configuration (.controllers.token.syncConfigs[] field). Whenever the .data.token field changes, it writes the new content to a file on the configured path on the host file system. This mechanism is used to download its own access token for the shoot cluster, but also the access tokens of other systemd components (e.g., valitail). Since the underlying client is based on k8s.io/client-go and the kubeconfig points to this token file, it is dynamically reloaded without the necessity of explicit configuration or code changes. This procedure ensures that the most up-to-date tokens are always present on the host and used by the gardener-node-agent and the other systemd components.

        Reasoning

        The gardener-node-agent is a replacement for what was called the cloud-config-downloader and the cloud-config-executor, both written in bash. The gardener-node-agent implements this functionality as a regular controller and feels more uniform in terms of maintenance.

        With the new architecture we gain a lot, let’s describe the most important gains here.

        Developer Productivity

        Since the Gardener community develops in Go day by day, writing business logic in bash is difficult, hard to maintain, almost impossible to test. Getting rid of almost all bash scripts which are currently in use for this very important part of the cluster creation process will enhance the speed of adding new features and removing bugs.

        Speed

        Until now, the cloud-config-downloader runs in a loop every 60s to check if something changed on the shoot which requires modifications on the worker node. This produces a lot of unneeded traffic on the API server and wastes time, it will sometimes take up to 60s until a desired modification is started on the worker node. By writing a “real” Kubernetes controller, we can watch for the Node, the OSC in the Secret, and the shoot-access token in the secret. If any of these object changed, and only then, the required action will take effect immediately. This will speed up operations and will reduce the load on the API server of the shoot especially for large clusters.

        Scalability

        The cloud-config-downloader adds a random wait time before restarting the kubelet in case the kubelet was updated or a configuration change was made to it. This is required to reduce the load on the API server and the traffic on the internet uplink. It also reduces the overall downtime of the services in the cluster because every kubelet restart transforms a node for several seconds into NotReady state which potentially interrupts service availability.

        Decision was made to keep the existing jitter mechanism which calculates the kubelet-download-and-restart-delay-seconds on the controller itself.

        Correctness

        The configuration of the cloud-config-downloader is actually done by placing a file for every configuration item on the disk on the worker node. This was done because parsing the content of a single file and using this as a value in bash reduces to something like VALUE=$(cat /the/path/to/the/file). Simple, but it lacks validation, type safety and whatnot. With the gardener-node-agent we introduce a new API which is then stored in the gardener-node-agent secret and stored on disk in a single YAML file for comparison with the previous known state. This brings all benefits of type safe configuration. Because actual and previous configuration are compared, removed files and units are also removed and stopped on the worker if removed from the OSC.

        Availability

        Previously, the cloud-config-downloader simply restarted the systemd units on every change to the OSC, regardless which of the services changed. The gardener-node-agent first checks which systemd unit was changed, and will only restart these. This will prevent unneeded kubelet restarts.

        2.10 - Gardener Operator

        Understand the component responsible for the garden cluster environment and its various features

        Overview

        The gardener-operator is responsible for the garden cluster environment. Without this component, users must deploy ETCD, the Gardener control plane, etc., manually and with separate mechanisms (not maintained in this repository). This is quite unfortunate since this requires separate tooling, processes, etc. A lot of production- and enterprise-grade features were built into Gardener for managing the seed and shoot clusters, so it makes sense to re-use them as much as possible also for the garden cluster.

        Deployment

        There is a Helm chart which can be used to deploy the gardener-operator. Once deployed and ready, you can create a Garden resource. Note that there can only be one Garden resource per system at a time.

        ℹ️ Similar to seed clusters, garden runtime clusters require a VPA, see this section. By default, gardener-operator deploys the VPA components. However, when there already is a VPA available, then set .spec.runtimeCluster.settings.verticalPodAutoscaler.enabled=false in the Garden resource.

        Garden Resources

        Please find an exemplary Garden resource here.

        Configuration For Runtime Cluster

        Settings

        The Garden resource offers a few settings that are used to control the behaviour of gardener-operator in the runtime cluster. This section provides an overview over the available settings in .spec.runtimeCluster.settings:

        Load Balancer Services

        gardener-operator deploys Istio and relevant resources to the runtime cluster in order to expose the virtual-garden-kube-apiserver service (similar to how the kube-apiservers of shoot clusters are exposed). In most cases, the cloud-controller-manager (responsible for managing these load balancers on the respective underlying infrastructure) supports certain customization and settings via annotations. This document provides a good overview and many examples.

        By setting the .spec.runtimeCluster.settings.loadBalancerServices.annotations field the Gardener administrator can specify a list of annotations which will be injected into the Services of type LoadBalancer.

        Vertical Pod Autoscaler

        gardener-operator heavily relies on the Kubernetes vertical-pod-autoscaler component. By default, the Garden controller deploys the VPA components into the garden namespace of the respective runtime cluster. In case you want to manage the VPA deployment on your own or have a custom one, then you might want to disable the automatic deployment of gardener-operator. Otherwise, you might end up with two VPAs which will cause erratic behaviour. By setting the .spec.runtimeCluster.settings.verticalPodAutoscaler.enabled=false you can disable the automatic deployment.

        ⚠️ In any case, there must be a VPA available for your runtime cluster. Using a runtime cluster without VPA is not supported.

        Topology-Aware Traffic Routing

        Refer to the Topology-Aware Traffic Routing documentation as this document contains the documentation for the topology-aware routing setting for the garden runtime cluster.

        Volumes

        It is possible to define the minimum size for PersistentVolumeClaims in the runtime cluster created by gardener-operator via the .spec.runtimeCluster.volume.minimumSize field. This can be relevant in case the runtime cluster runs on an infrastructure that does only support disks of at least a certain size.

        Configuration For Virtual Cluster

        ETCD Encryption Config

        The spec.virtualCluster.kubernetes.kubeAPIServer.encryptionConfig field in the Garden API allows operators to customize encryption configurations for the kube-apiserver of the virtual cluster. It provides options to specify additional resources for encryption. Similarly spec.virtualCluster.gardener.gardenerAPIServer.encryptionConfig field allows operators to customize encryption configurations for the gardener-apiserver.

        • The resources field can be used to specify resources that should be encrypted in addition to secrets. Secrets are always encrypted for the kube-apiserver. For the gardener-apiserver, the following resources are always encrypted:
          • controllerdeployments.core.gardener.cloud
          • controllerregistrations.core.gardener.cloud
          • internalsecrets.core.gardener.cloud
          • shootstates.core.gardener.cloud
        • Adding an item to any of the lists will cause patch requests for all the resources of that kind to encrypt them in the etcd. See Encrypting Confidential Data at Rest for more details.
        • Removing an item from any of these lists will cause patch requests for all the resources of that type to decrypt and rewrite the resource as plain text. See Decrypt Confidential Data that is Already Encrypted at Rest for more details.

        ℹ️ Note that configuring encryption for a custom resource for the kube-apiserver is only supported for Kubernetes versions >= 1.26.

        Controllers

        As of today, the gardener-operator only has two controllers which are now described in more detail.

        Garden Controller

        The Garden controller in the operator reconciles Garden objects with the help of the following reconcilers.

        Main Reconciler

        The reconciler first generates a general CA certificate which is valid for ~30d and auto-rotated when 80% of its lifetime is reached. Afterwards, it brings up the so-called “garden system components”. The gardener-resource-manager is deployed first since its ManagedResource controller will be used to bring up the remainders.

        Other system components are:

        • runtime garden system resources (PriorityClasses for the workload resources)
        • virtual garden system resources (RBAC rules)
        • Vertical Pod Autoscaler (if enabled via .spec.runtimeCluster.settings.verticalPodAutoscaler.enabled=true in the Garden)
        • HVPA Controller (when HVPA feature gate is enabled)
        • ETCD Druid
        • Istio

        As soon as all system components are up, the reconciler deploys the virtual garden cluster. It comprises out of two ETCDs (one “main” etcd, one “events” etcd) which are managed by ETCD Druid via druid.gardener.cloud/v1alpha1.Etcd custom resources. The whole management works similar to how it works for Shoots, so you can take a look at this document for more information in general.

        The virtual garden control plane components are:

        • virtual-garden-etcd-main
        • virtual-garden-etcd-events
        • virtual-garden-kube-apiserver
        • virtual-garden-kube-controller-manager
        • virtual-garden-gardener-resource-manager

        If the .spec.virtualCluster.controlPlane.highAvailability={} is set then these components will be deployed in a “highly available” mode. For ETCD, this means that there will be 3 replicas each. This works similar like for Shoots (see this document) except for the fact that there is no failure tolerance type configurability. The gardener-resource-manager’s HighAvailabilityConfig webhook makes sure that all pods with multiple replicas are spread on nodes, and if there are at least two zones in .spec.runtimeCluster.provider.zones then they also get spread across availability zones.

        If once set, removing .spec.virtualCluster.controlPlane.highAvailability again is not supported.

        The virtual-garden-kube-apiserver Deployment is exposed via Istio, similar to how the kube-apiservers of shoot clusters are exposed.

        Similar to the Shoot API, the version of the virtual garden cluster is controlled via .spec.virtualCluster.kubernetes.version. Likewise, specific configuration for the control plane components can be provided in the same section, e.g. via .spec.virtualCluster.kubernetes.kubeAPIServer for the kube-apiserver or .spec.virtualCluster.kubernetes.kubeControllerManager for the kube-controller-manager.

        The kube-controller-manager only runs a few controllers that are necessary in the scenario of the virtual garden. Most prominently, the serviceaccount-token controller is unconditionally disabled. Hence, the usage of static ServiceAccount secrets is not supported generally. Instead, the TokenRequest API should be used. Third-party components that need to communicate with the virtual cluster can leverage the gardener-resource-manager’s TokenRequestor controller and the generic kubeconfig, just like it works for Shoots. Please note, that this functionality is restricted to the garden namespace. The current Secret name of the generic kubeconfig can be found in the annotations (key: generic-token-kubeconfig.secret.gardener.cloud/name) of the Garden resource.

        For the virtual cluster, it is essential to provide at least one DNS domain via .spec.virtualCluster.dns.domains. The respective DNS records are not managed by gardener-operator and should be created manually. They should point to the load balancer IP of the istio-ingressgateway Service in namespace virtual-garden-istio-ingress. The DNS records must be prefixed with both gardener. and api. for all domains in .spec.virtualCluster.dns.domains.

        The first DNS domain in this list is used for the server in the kubeconfig, and for configuring the --external-hostname flag of the API server.

        Apart from the control plane components of the virtual cluster, the reconcile also deploys the control plane components of Gardener. gardener-apiserver reuses the same ETCDs like the virtual-garden-kube-apiserver, so all data related to the “the garden cluster” is stored together and “isolated” from ETCD data related to the runtime cluster. This drastically simplifies backup and restore capabilities (e.g., moving the virtual garden cluster from one runtime cluster to another).

        The Gardener control plane components are:

        • gardener-apiserver
        • gardener-admission-controller
        • gardener-controller-manager
        • gardener-scheduler

        Besides those, the optional Gardener Dashboard (and the controller for web terminals) can also get deployed when .spec.virtualCluster.gardener.gardenerDashboard (or .spec.virtualCluster.gardener.gardenerDashboard.terminal, respectively) is set. You can read more about it and its configuration in this section.

        The reconciler also manages a few observability-related components (more planned as part of GEP-19):

        • fluent-operator
        • fluent-bit
        • gardener-metrics-exporter
        • kube-state-metrics
        • plutono
        • vali
        • prometheus-operator
        • alertmanager-garden (read more here)
        • prometheus-garden (read more here)
        • prometheus-longterm (read more here)
        • blackbox-exporter

        It is also mandatory to provide an IPv4 CIDR for the service network of the virtual cluster via .spec.virtualCluster.networking.services. This range is used by the API server to compute the cluster IPs of Services.

        The controller maintains the .status.lastOperation which indicates the status of an operation.

        Gardener Dashboard

        .spec.virtualCluster.gardener.gardenerDashboard serves a few configuration options for the dashboard. This section highlights the most prominent fields:

        • oidcConfig: The general OIDC configuration is part of .spec.virtualCluster.kubernetes.kubeAPIServer.oidcConfig. This section allows you to define a few specific settings for the dashboard. sessionLifetime is the duration after which a session is terminated (i.e., after which a user is automatically logged out). additionalScopes allows to extend the list of scopes of the JWT token that are to be recognized. You must reference a Secret in the garden namespace containing the client ID/secret for the dashboard:
          apiVersion: v1
          kind: Secret
          metadata:
            name: gardener-dashboard-oidc
            namespace: garden
          type: Opaque
          stringData:
            client_id: <secret>
            client_secret: <secret>
          
        • enableTokenLogin: This is enabled by default and allows logging into the dashboard with a JWT token. You can disable it in case you want to only allow OIDC-based login. However, at least one of the both login methods must be enabled.
        • frontendConfigMapRef: Reference a ConfigMap in the garden namespace containing the frontend configuration in the data with key frontend-config.yaml, for example
          apiVersion: v1
          kind: ConfigMap
          metadata:
            name: gardener-dashboard-frontend
            namespace: garden
          data:
            frontend-config.yaml: |
              helpMenuItems:
              - title: Homepage
                icon: mdi-file-document
                url: https://gardener.cloud    
          
          Please take a look at this file to get an idea of which values are configurable. This configuration can also include branding, themes, and colors. Read more about it here. Assets (logos/icons) are configured in a separate ConfigMap, see below.
        • assetsConfigMapRef: Reference a ConfigMap in the garden namespace containing the assets, for example
          apiVersion: v1
          kind: ConfigMap
          metadata:
            name: gardener-dashboard-assets
            namespace: garden
          binaryData:
            favicon-16x16.png: base64(favicon-16x16.png)
            favicon-32x32.png: base64(favicon-32x32.png)
            favicon-96x96.png: base64(favicon-96x96.png)
            favicon.ico: base64(favicon.ico)
            logo.svg: base64(logo.svg)
          
          Note that the assets must be provided base64-encoded, hence binaryData (instead of data) must be used. Please take a look at this file to get more information.
        • gitHub: You can connect a GitHub repository that can be used to create issues for shoot clusters in the cluster details page. You have to reference a Secret in the garden namespace that contains the GitHub credentials, for example:
          apiVersion: v1
          kind: Secret
          metadata:
            name: gardener-dashboard-github
            namespace: garden
          type: Opaque
          stringData:
            # This is for GitHub token authentication:
            authentication.token: <secret>
            # Alternatively, this is for GitHub app authentication:
            authentication.appId: <secret>
            authentication.clientId: <secret>
            authentication.clientSecret: <secret>
            authentication.installationId: <secret>
            authentication.privateKey: <secret>
            # This is the webhook secret, see explanation below
            webhookSecret: <secret>
          
          Note that you can also set up a GitHub webhook to the dashboard such that it receives updates when somebody changes the GitHub issue. The webhookSecret field is the secret that you enter in GitHub in the webhook configuration. The dashboard uses it to verify that received traffic is indeed originated from GitHub. If you don’t want to set up such webhook, or if the dashboard is not reachable by the GitHub webhook (e.g., in restricted environments) you can also configure gitHub.pollInterval. It is the interval of how often the GitHub API is polled for issue updates. This field is used as a fallback mechanism to ensure state synchronization, even when there is a GitHub webhook configuration. If a webhook event is missed or not successfully delivered, the polling will help catch up on any missed updates. If this field is not provided and there is no webhookSecret key in the referenced secret, it will be implicitly defaulted to 15m. The dashboard will use this to regularly poll the GitHub API for updates on issues.
        • terminal: This enables the web terminal feature, read more about it here. When set, the terminal-controller-manager will be deployed to the runtime cluster. The allowedHosts field is explained here. The container section allows you to specify a container image and a description that should be used for the web terminals.
        Observability
        Garden Prometheus

        gardener-operator deploys a Prometheus instance in the garden namespace (called “Garden Prometheus”) which fetches metrics and data from garden system components, cAdvisors, the virtual cluster control plane, and the Seeds’ aggregate Prometheus instances. Its purpose is to provide an entrypoint for operators when debugging issues with components running in the garden cluster. It also serves as the top-level aggregator of metering across a Gardener landscape.

        To extend the configuration of the Garden Prometheus, you can create the prometheus-operator’s custom resources and label them with prometheus=garden, for example:

        apiVersion: monitoring.coreos.com/v1
        kind: ServiceMonitor
        metadata:
          labels:
            prometheus: garden
          name: garden-my-component
          namespace: garden
        spec:
          selector:
            matchLabels:
              app: my-component
          endpoints:
          - metricRelabelings:
            - action: keep
              regex: ^(metric1|metric2|...)$
              sourceLabels:
              - __name__
            port: metrics
        
        Long-Term Prometheus

        gardener-operator deploys another Prometheus instance in the garden namespace (called “Long-Term Prometheus”) which federates metrics from Garden Prometheus. Its purpose is to store those with a longer retention than Garden Prometheus would. It is not possible to define different retention periods for different metrics in Prometheus, hence, using another Prometheus instance is the only option. This Long-term Prometheus also has an additional Cortex sidecar container for caching some queries to achieve faster processing times.

        To extend the configuration of the Long-term Prometheus, you can create the prometheus-operator’s custom resources and label them with prometheus=longterm, for example:

        apiVersion: monitoring.coreos.com/v1
        kind: ServiceMonitor
        metadata:
          labels:
            prometheus: longterm
          name: longterm-my-component
          namespace: garden
        spec:
          selector:
            matchLabels:
              app: my-component
          endpoints:
          - metricRelabelings:
            - action: keep
              regex: ^(metric1|metric2|...)$
              sourceLabels:
              - __name__
            port: metrics
        
        Alertmanager

        By default, the alertmanager-garden deployed by gardener-operator does not come with any configuration. It is the responsibility of the human operators to design and provide it. This can be done by creating monitoring.coreos.com/v1alpha1.AlertmanagerConfig resources labeled with alertmanager=garden (read more about them here), for example:

        apiVersion: monitoring.coreos.com/v1alpha1
        kind: AlertmanagerConfig
        metadata:
          name: config
          namespace: garden
          labels:
            alertmanager: garden
        spec:
          route:
            receiver: dev-null
            groupBy:
            - alertname
            - landscape
            routes:
            - continue: true
              groupWait: 3m
              groupInterval: 5m
              repeatInterval: 12h
              routes:
              - receiver: ops
                matchers:
                - name: severity
                  value: warning
                  matchType: =
                - name: topology
                  value: garden
                  matchType: =
          receivers:
          - name: dev-null
          - name: ops
            slackConfigs:
            - apiURL: https://<slack-api-url>
              channel: <channel-name>
              username: Gardener-Alertmanager
              iconEmoji: ":alert:"
              title: "[{{ .Status | toUpper }}] Gardener Alert(s)"
              text: "{{ range .Alerts }}*{{ .Annotations.summary }} ({{ .Status }})*\n{{ .Annotations.description }}\n\n{{ end }}"
              sendResolved: true
        
        Plutono

        A Plutono instance is deployed by gardener-operator into the garden namespace for visualizing monitoring metrics and logs via dashboards. In order to provide custom dashboards, create a ConfigMap in the garden namespace labelled with dashboard.monitoring.gardener.cloud/garden=true that contains the respective JSON documents, for example:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          labels:
            dashboard.monitoring.gardener.cloud/garden: "true"
          name: my-custom-dashboard
          namespace: garden
        data:
          my-custom-dashboard.json: <dashboard-JSON-document>
        

        Care Reconciler

        This reconciler performs four “care” actions related to Gardens.

        It maintains the following conditions:

        • VirtualGardenAPIServerAvailable: The /healthz endpoint of the garden’s virtual-garden-kube-apiserver is called and considered healthy when it responds with 200 OK.
        • RuntimeComponentsHealthy: The conditions of the ManagedResources applied to the runtime cluster are checked (e.g., ResourcesApplied).
        • VirtualComponentsHealthy: The virtual components are considered healthy when the respective Deployments (for example virtual-garden-kube-apiserver,virtual-garden-kube-controller-manager), and Etcds (for example virtual-garden-etcd-main) exist and are healthy. Additionally, the conditions of the ManagedResources applied to the virtual cluster are checked (e.g., ResourcesApplied).
        • ObservabilityComponentsHealthy: This condition is considered healthy when the respective Deployments (for example plutono) and StatefulSets (for example prometheus, vali) exist and are healthy.

        If all checks for a certain condition are succeeded, then its status will be set to True. Otherwise, it will be set to False or Progressing.

        If at least one check fails and there is threshold configuration for the conditions (in .controllers.gardenCare.conditionThresholds), then the status will be set:

        • to Progressing if it was True before.
        • to Progressing if it was Progressing before and the lastUpdateTime of the condition does not exceed the configured threshold duration yet.
        • to False if it was Progressing before and the lastUpdateTime of the condition exceeds the configured threshold duration.

        The condition thresholds can be used to prevent reporting issues too early just because there is a rollout or a short disruption. Only if the unhealthiness persists for at least the configured threshold duration, then the issues will be reported (by setting the status to False).

        In order to compute the condition statuses, this reconciler considers ManagedResources (in the garden and istio-system namespace) and their status, see this document for more information. The following table explains which ManagedResources are considered for which condition type:

        Condition TypeManagedResources are considered when
        RuntimeComponentsHealthy.spec.class=seed and care.gardener.cloud/condition-type label either unset, or set to RuntimeComponentsHealthy
        VirtualComponentsHealthy.spec.class unset or care.gardener.cloud/condition-type label set to VirtualComponentsHealthy
        ObservabilityComponentsHealthycare.gardener.cloud/condition-type label set to ObservabilityComponentsHealthy

        Reference Reconciler

        Garden objects may specify references to other objects in the Garden cluster which are required for certain features. For example, operators can configure a secret for ETCD backup via .spec.virtualCluster.etcd.main.backup.secretRef.name or an audit policy ConfigMap via .spec.virtualCluster.kubernetes.kubeAPIServer.auditConfig.auditPolicy.configMapRef.name. Such objects need a special protection against deletion requests as long as they are still being referenced by the Garden.

        Therefore, this reconciler checks Gardens for referenced objects and adds the finalizer gardener.cloud/reference-protection to their .metadata.finalizers list. The reconciled Garden also gets this finalizer to enable a proper garbage collection in case the gardener-operator is offline at the moment of an incoming deletion request. When an object is not actively referenced anymore because the Garden specification has changed is in deletion, the controller will remove the added finalizer again so that the object can safely be deleted or garbage collected.

        This reconciler inspects the following references:

        • ETCD backup Secrets (.spec.virtualCluster.etcd.main.backup.secretRef)
        • Admission plugin kubeconfig Secrets (.spec.virtualCluster.kubernetes.kubeAPIServer.admissionPlugins[].kubeconfigSecretName and .spec.virtualCluster.gardener.gardenerAPIServer.admissionPlugins[].kubeconfigSecretName)
        • Authentication webhook kubeconfig Secrets (.spec.virtualCluster.kubernetes.kubeAPIServer.authentication.webhook.kubeconfigSecretName)
        • Audit webhook kubeconfig Secrets (.spec.virtualCluster.kubernetes.kubeAPIServer.auditWebhook.kubeconfigSecretName and .spec.virtualCluster.gardener.gardenerAPIServer.auditWebhook.kubeconfigSecretName)
        • SNI Secrets (.spec.virtualCluster.kubernetes.kubeAPIServer.sni.secretName)
        • Audit policy ConfigMaps (.spec.virtualCluster.kubernetes.kubeAPIServer.auditConfig.auditPolicy.configMapRef.name and .spec.virtualCluster.gardener.gardenerAPIServer.auditConfig.auditPolicy.configMapRef.name)

        Further checks might be added in the future.

        Controller Registrar controller

        This controller registers controllers, which need to be installed in two contexts. If the Garden cluster is at the same time used as a Seed cluster, the gardener-operator will start these controllers. If the Garden cluster is separate from the Seed cluster, the controllers will be started by gardenlet.

        Currently, this applies to two controllers:

        The registration happens as soon as the Garden resource is created. It contains the networking information of the garden runtime cluster which is required configuration for the NetworkPolicy controller.

        Webhooks

        As of today, the gardener-operator only has one webhook handler which is now described in more detail.

        Validation

        This webhook handler validates CREATE/UPDATE/DELETE operations on Garden resources. Simple validation is performed via standard CRD validation. However, more advanced validation is hard to express via these means and is performed by this webhook handler.

        Furthermore, for deletion requests, it is validated that the Garden is annotated with a deletion confirmation annotation, namely confirmation.gardener.cloud/deletion=true. Only if this annotation is present it allows the DELETE operation to pass. This prevents users from accidental/undesired deletions.

        Another validation is to check that there is only one Garden resource at a time. It prevents creating a second Garden when there is already one in the system.

        Defaulting

        This webhook handler mutates the Garden resource on CREATE/UPDATE/DELETE operations. Simple defaulting is performed via standard CRD defaulting. However, more advanced defaulting is hard to express via these means and is performed by this webhook handler.

        Using Garden Runtime Cluster As Seed Cluster

        In production scenarios, you probably wouldn’t use the Kubernetes cluster running gardener-operator and the Gardener control plane (called “runtime cluster”) as seed cluster at the same time. However, such setup is technically possible and might simplify certain situations (e.g., development, evaluation, …).

        If the runtime cluster is a seed cluster at the same time, gardenlet’s Seed controller will not manage the components which were already deployed (and reconciled) by gardener-operator. As of today, this applies to:

        • gardener-resource-manager
        • vpa-{admission-controller,recommender,updater}
        • hvpa-controller (when HVPA feature gate is enabled)
        • etcd-druid
        • istio control-plane
        • nginx-ingress-controller

        Those components are so-called “seed system components”. In addition, there are a few observability components:

        • fluent-operator
        • fluent-bit
        • vali
        • plutono
        • kube-state-metrics
        • prometheus-operator

        As all of these components are managed by gardener-operator in this scenario, the gardenlet just skips them.

        ℹ️ There is no need to configure anything - the gardenlet will automatically detect when its seed cluster is the garden runtime cluster at the same time.

        ⚠️ Note that such setup requires that you upgrade the versions of gardener-operator and gardenlet in lock-step. Otherwise, you might experience unexpected behaviour or issues with your seed or shoot clusters.

        Credentials Rotation

        The credentials rotation works in the same way as it does for Shoot resources, i.e. there are gardener.cloud/operation annotation values for starting or completing the rotation procedures.

        For certificate authorities, gardener-operator generates one which is automatically rotated roughly each month (ca-garden-runtime) and several CAs which are NOT automatically rotated but only on demand.

        🚨 Hence, it is the responsibility of the (human) operator to regularly perform the credentials rotation.

        Please refer to this document for more details. As of today, gardener-operator only creates the following types of credentials (i.e., some sections of the document don’t apply for Gardens and can be ignored):

        • certificate authorities (and related server and client certificates)
        • ETCD encryption key
        • observability password for Plutono
        • ServiceAccount token signing key

        ⚠️ Rotation of static ServiceAccount secrets is not supported since the kube-controller-manager does not enable the serviceaccount-token controller.

        When the ServiceAccount token signing key rotation is in Preparing phase, then gardener-operator annotates all Seeds with gardener.cloud/operation=renew-garden-access-secrets. This causes gardenlet to populate new ServiceAccount tokens for the garden cluster to all extensions, which are now signed with the new signing key. Read more about it here.

        Similarly, when the CA certificate rotation is in Preparing phase, then gardener-operator annotates all Seeds with gardener.cloud/operation=renew-kubeconfig. This causes gardenlet to request a new client certificate for its garden cluster kubeconfig, which is now signed with the new client CA, and which also contains the new CA bundle for the server certificate verification. Read more about it here.

        Migrating an Existing Gardener Landscape to gardener-operator

        Since gardener-operator was only developed in 2023, six years after the Gardener project initiation, most users probably already have an existing Gardener landscape. The most prominent installation procedure is garden-setup, however experience shows that most community members have developed their own tooling for managing the garden cluster and the Gardener control plane components.

        Consequently, providing a general migration guide is not possible since the detailed steps vary heavily based on how the components were set up previously. As a result, this section can only highlight the most important caveats and things to know, while the concrete migration steps must be figured out individually based on the existing installation.

        Please test your migration procedure thoroughly. Note that in some cases it can be easier to set up a fresh landscape with gardener-operator, restore the ETCD data, switch the DNS records, and issue new credentials for all clients.

        Please make sure that you configure all your desired fields in the Garden resource.

        ETCD

        gardener-operator leverages etcd-druid for managing the virtual-garden-etcd-main and virtual-garden-etcd-events, similar to how shoot cluster control planes are handled. The PersistentVolumeClaim names differ slightly - for virtual-garden-etcd-events it’s virtual-garden-etcd-events-virtual-garden-etcd-events-0, while for virtual-garden-etcd-main it’s main-virtual-garden-etcd-virtual-garden-etcd-main-0. The easiest approach for the migration is to make your existing ETCD volumes follow the same naming scheme. Alternatively, backup your data, let gardener-operator take over ETCD, and then restore your data to the new volume.

        The backup bucket must be created separately, and its name as well as the respective credentials must be provided via the Garden resource in .spec.virtualCluster.etcd.main.backup.

        virtual-garden-kube-apiserver Deployment

        gardener-operator deploys a virtual-garden-kube-apiserver into the runtime cluster. This virtual-garden-kube-apiserver spans a new cluster, called the virtual cluster. There are a few certificates and other credentials that should not change during the migration. You have to prepare the environment accordingly by leveraging the secret’s manager capabilities.

        • The existing Cluster CA Secret should be labeled with secrets-manager-use-data-for-name=ca.
        • The existing Client CA Secret should be labeled with secrets-manager-use-data-for-name=ca-client.
        • The existing Front Proxy CA Secret should be labeled with secrets-manager-use-data-for-name=ca-front-proxy.
        • The existing Service Account Signing Key Secret should be labeled with secrets-manager-use-data-for-name=service-account-key.
        • The existing ETCD Encryption Key Secret should be labeled with secrets-manager-use-data-for-name=kube-apiserver-etcd-encryption-key.

        virtual-garden-kube-apiserver Exposure

        The virtual-garden-kube-apiserver is exposed via a dedicated istio-ingressgateway deployed to namespace virtual-garden-istio-ingress. The virtual-garden-kube-apiserver Service in the garden namespace is only of type ClusterIP. Consequently, DNS records for this API server must target the load balancer IP of the istio-ingressgateway.

        Virtual Garden Kubeconfig

        gardener-operator does not generate any static token or likewise for access to the virtual cluster. Ideally, human users access it via OIDC only. Alternatively, you can create an auto-rotated token that you can use for automation like CI/CD pipelines:

        apiVersion: v1
        kind: Secret
        type: Opaque
        metadata:
          name: shoot-access-virtual-garden
          namespace: garden
          labels:
            resources.gardener.cloud/purpose: token-requestor
            resources.gardener.cloud/class: shoot
          annotations:
            serviceaccount.resources.gardener.cloud/name: virtual-garden-user
            serviceaccount.resources.gardener.cloud/namespace: kube-system
            serviceaccount.resources.gardener.cloud/token-expiration-duration: 3h
        ---
        apiVersion: v1
        kind: Secret
        metadata:
          name: managedresource-virtual-garden-access
          namespace: garden
        type: Opaque
        stringData:
          clusterrolebinding____gardener.cloud.virtual-garden-access.yaml: |
            apiVersion: rbac.authorization.k8s.io/v1
            kind: ClusterRoleBinding
            metadata:
              name: gardener.cloud.sap:virtual-garden
            roleRef:
              apiGroup: rbac.authorization.k8s.io
              kind: ClusterRole
              name: cluster-admin
            subjects:
            - kind: ServiceAccount
              name: virtual-garden-user
              namespace: kube-system    
        ---
        apiVersion: resources.gardener.cloud/v1alpha1
        kind: ManagedResource
        metadata:
          name: virtual-garden-access
          namespace: garden
        spec:
          secretRefs:
          - name: managedresource-virtual-garden-access
        

        The shoot-access-virtual-garden Secret will get a .data.token field which can be used to authenticate against the virtual garden cluster. See also this document for more information about the TokenRequestor.

        gardener-apiserver

        Similar to the virtual-garden-kube-apiserver, the gardener-apiserver also uses a few certificates and other credentials that should not change during the migration. Again, you have to prepare the environment accordingly by leveraging the secret’s manager capabilities.

        • The existing ETCD Encryption Key Secret should be labeled with secrets-manager-use-data-for-name=gardener-apiserver-etcd-encryption-key.

        Also note that gardener-operator manages the Service and Endpoints resources for the gardener-apiserver in the virtual cluster within the kube-system namespace (garden-setup uses the garden namespace).

        Local Development

        The easiest setup is using a local KinD cluster and the Skaffold based approach to deploy and develop the gardener-operator.

        Setting Up the KinD Cluster (runtime cluster)

        make kind-operator-up
        

        This command sets up a new KinD cluster named gardener-local and stores the kubeconfig in the ./example/gardener-local/kind/operator/kubeconfig file.

        It might be helpful to copy this file to $HOME/.kube/config, since you will need to target this KinD cluster multiple times. Alternatively, make sure to set your KUBECONFIG environment variable to ./example/gardener-local/kind/operator/kubeconfig for all future steps via export KUBECONFIG=$PWD/example/gardener-local/kind/operator/kubeconfig.

        All the following steps assume that you are using this kubeconfig.

        Setting Up Gardener Operator

        make operator-up
        

        This will first build the base images (which might take a bit if you do it for the first time). Afterwards, the Gardener Operator resources will be deployed into the cluster.

        Developing Gardener Operator (Optional)

        make operator-dev
        

        This is similar to make operator-up but additionally starts a skaffold dev loop. After the initial deployment, skaffold starts watching source files. Once it has detected changes, press any key to trigger a new build and deployment of the changed components.

        Debugging Gardener Operator (Optional)

        make operator-debug
        

        This is similar to make gardener-debug but for Gardener Operator component. Please check Debugging Gardener for details.

        Creating a Garden

        In order to create a garden, just run:

        kubectl apply -f example/operator/20-garden.yaml
        

        You can wait for the Garden to be ready by running:

        ./hack/usage/wait-for.sh garden local Reconciled
        

        Alternatively, you can run kubectl get garden and wait for the RECONCILED status to reach True:

        NAME     RECONCILED    AGE
        garden   Progressing   1s
        

        (Optional): Instead of creating above Garden resource manually, you could execute the e2e tests by running:

        make test-e2e-local-operator
        

        Accessing the Virtual Garden Cluster

        ⚠️ Please note that in this setup, the virtual garden cluster is not accessible by default when you download the kubeconfig and try to communicate with it. The reason is that your host most probably cannot resolve the DNS name of the cluster. Hence, if you want to access the virtual garden cluster, you have to run the following command which will extend your /etc/hosts file with the required information to make the DNS names resolvable:

        cat <<EOF | sudo tee -a /etc/hosts
        
        # Manually created to access local Gardener virtual garden cluster.
        # TODO: Remove this again when the virtual garden cluster access is no longer required.
        127.0.0.1 api.virtual-garden.local.gardener.cloud
        EOF
        

        To access the virtual garden, you can acquire a kubeconfig by

        kubectl -n garden get secret gardener -o jsonpath={.data.kubeconfig} | base64 -d > /tmp/virtual-garden-kubeconfig
        kubectl --kubeconfig /tmp/virtual-garden-kubeconfig get namespaces
        

        Note that this kubeconfig uses a token that has validity of 12h only, hence it might expire and causing you to re-download the kubeconfig.

        Deleting the Garden

        ./hack/usage/delete garden local
        

        Tear Down the Gardener Operator Environment

        make operator-down
        make kind-operator-down
        

        2.11 - Gardener Resource Manager

        Set of controllers with different responsibilities running once per seed and once per shoot

        Overview

        Initially, the gardener-resource-manager was a project similar to the kube-addon-manager. It manages Kubernetes resources in a target cluster which means that it creates, updates, and deletes them. Also, it makes sure that manual modifications to these resources are reconciled back to the desired state.

        In the Gardener project we were using the kube-addon-manager since more than two years. While we have progressed with our extensibility story (moving cloud providers out-of-tree), we had decided that the kube-addon-manager is no longer suitable for this use-case. The problem with it is that it needs to have its managed resources on its file system. This requires storing the resources in ConfigMaps or Secrets and mounting them to the kube-addon-manager pod during deployment time. The gardener-resource-manager uses CustomResourceDefinitions which allows to dynamically add, change, and remove resources with immediate action and without the need to reconfigure the volume mounts/restarting the pod.

        Meanwhile, the gardener-resource-manager has evolved to a more generic component comprising several controllers and webhook handlers. It is deployed by gardenlet once per seed (in the garden namespace) and once per shoot (in the respective shoot namespaces in the seed).

        Component Configuration

        Similar to other Gardener components, the gardener-resource-manager uses a so-called component configuration file. It allows specifying certain central settings like log level and formatting, client connection configuration, server ports and bind addresses, etc. In addition, controllers and webhooks can be configured and sometimes even disabled.

        Note that the very basic ManagedResource and health controllers cannot be disabled.

        You can find an example configuration file here.

        Controllers

        ManagedResource Controller

        This controller watches custom objects called ManagedResources in the resources.gardener.cloud/v1alpha1 API group. These objects contain references to secrets, which itself contain the resources to be managed. The reason why a Secret is used to store the resources is that they could contain confidential information like credentials.

        ---
        apiVersion: v1
        kind: Secret
        metadata:
          name: managedresource-example1
          namespace: default
        type: Opaque
        data:
          objects.yaml: YXBpVmVyc2lvbjogdjEKa2luZDogQ29uZmlnTWFwCm1ldGFkYXRhOgogIG5hbWU6IHRlc3QtMTIzNAogIG5hbWVzcGFjZTogZGVmYXVsdAotLS0KYXBpVmVyc2lvbjogdjEKa2luZDogQ29uZmlnTWFwCm1ldGFkYXRhOgogIG5hbWU6IHRlc3QtNTY3OAogIG5hbWVzcGFjZTogZGVmYXVsdAo=
            # apiVersion: v1
            # kind: ConfigMap
            # metadata:
            #   name: test-1234
            #   namespace: default
            # ---
            # apiVersion: v1
            # kind: ConfigMap
            # metadata:
            #   name: test-5678
            #   namespace: default
        ---
        apiVersion: resources.gardener.cloud/v1alpha1
        kind: ManagedResource
        metadata:
          name: example
          namespace: default
        spec:
          secretRefs:
          - name: managedresource-example1
        

        In the above example, the controller creates two ConfigMaps in the default namespace. When a user is manually modifying them, they will be reconciled back to the desired state stored in the managedresource-example secret.

        It is also possible to inject labels into all the resources:

        ---
        apiVersion: v1
        kind: Secret
        metadata:
          name: managedresource-example2
          namespace: default
        type: Opaque
        data:
          other-objects.yaml: YXBpVmVyc2lvbjogYXBwcy92MSAjIGZvciB2ZXJzaW9ucyBiZWZvcmUgMS45LjAgdXNlIGFwcHMvdjFiZXRhMgpraW5kOiBEZXBsb3ltZW50Cm1ldGFkYXRhOgogIG5hbWU6IG5naW54LWRlcGxveW1lbnQKc3BlYzoKICBzZWxlY3RvcjoKICAgIG1hdGNoTGFiZWxzOgogICAgICBhcHA6IG5naW54CiAgcmVwbGljYXM6IDIgIyB0ZWxscyBkZXBsb3ltZW50IHRvIHJ1biAyIHBvZHMgbWF0Y2hpbmcgdGhlIHRlbXBsYXRlCiAgdGVtcGxhdGU6CiAgICBtZXRhZGF0YToKICAgICAgbGFiZWxzOgogICAgICAgIGFwcDogbmdpbngKICAgIHNwZWM6CiAgICAgIGNvbnRhaW5lcnM6CiAgICAgIC0gbmFtZTogbmdpbngKICAgICAgICBpbWFnZTogbmdpbng6MS43LjkKICAgICAgICBwb3J0czoKICAgICAgICAtIGNvbnRhaW5lclBvcnQ6IDgwCg==
            # apiVersion: apps/v1
            # kind: Deployment
            # metadata:
            #   name: nginx-deployment
            # spec:
            #   selector:
            #     matchLabels:
            #       app: nginx
            #   replicas: 2 # tells deployment to run 2 pods matching the template
            #   template:
            #     metadata:
            #       labels:
            #         app: nginx
            #     spec:
            #       containers:
            #       - name: nginx
            #         image: nginx:1.7.9
            #         ports:
            #         - containerPort: 80
        
        ---
        apiVersion: resources.gardener.cloud/v1alpha1
        kind: ManagedResource
        metadata:
          name: example
          namespace: default
        spec:
          secretRefs:
          - name: managedresource-example2
          injectLabels:
            foo: bar
        

        In this example, the label foo=bar will be injected into the Deployment, as well as into all created ReplicaSets and Pods.

        Preventing Reconciliations

        If a ManagedResource is annotated with resources.gardener.cloud/ignore=true, then it will be skipped entirely by the controller (no reconciliations or deletions of managed resources at all). However, when the ManagedResource itself is deleted (for example when a shoot is deleted), then the annotation is not respected and all resources will be deleted as usual. This feature can be helpful to temporarily patch/change resources managed as part of such ManagedResource. Condition checks will be skipped for such ManagedResources.

        Modes

        The gardener-resource-manager can manage a resource in the following supported modes:

        • Ignore
          • The corresponding resource is removed from the ManagedResource status (.status.resources). No action is performed on the cluster.
          • The resource is no longer “managed” (updated or deleted).
          • The primary use case is a migration of a resource from one ManagedResource to another one.

        The mode for a resource can be specified with the resources.gardener.cloud/mode annotation. The annotation should be specified in the encoded resource manifest in the Secret that is referenced by the ManagedResource.

        Resource Class and Reconcilation Scope

        By default, the gardener-resource-manager controller watches for ManagedResources in all namespaces. The .sourceClientConnection.namespace field in the component configuration restricts the watch to ManagedResources in a single namespace only. Note that this setting also affects all other controllers and webhooks since it’s a central configuration.

        A ManagedResource has an optional .spec.class field that allows it to indicate that it belongs to a given class of resources. The .controllers.resourceClass field in the component configuration restricts the watch to ManagedResources with the given .spec.class. A default class is assumed if no class is specified.

        For instance, the gardener-resource-manager which is deployed in the Shoot’s control plane namespace in the Seed does not specify a .spec.class and watches only for resources in the control plane namespace by specifying it in the .sourceClientConnection.namespace field.

        If the .spec.class changes this means that the resources have to be handled by a different Gardener Resource Manager. That is achieved by:

        1. Cleaning all referenced resources by the Gardener Resource Manager that was responsible for the old class in its target cluster.
        2. Creating all referenced resources by the Gardener Resource Manager that is responsible for the new class in its target cluster.

        Conditions

        A ManagedResource has a ManagedResourceStatus, which has an array of Conditions. Conditions currently include:

        ConditionDescription
        ResourcesAppliedTrue if all resources are applied to the target cluster
        ResourcesHealthyTrue if all resources are present and healthy
        ResourcesProgressingFalse if all resources have been fully rolled out

        ResourcesApplied may be False when:

        • the resource apiVersion is not known to the target cluster
        • the resource spec is invalid (for example the label value does not match the required regex for it)

        ResourcesHealthy may be False when:

        • the resource is not found
        • the resource is a Deployment and the Deployment does not have the minimum availability.

        ResourcesProgressing may be True when:

        • a Deployment, StatefulSet or DaemonSet has not been fully rolled out yet, i.e. not all replicas have been updated with the latest changes to spec.template.
        • there are still old Pods belonging to an older ReplicaSet of a Deployment which are not terminated yet.

        Each Kubernetes resources has different notion for being healthy. For example, a Deployment is considered healthy if the controller observed its current revision and if the number of updated replicas is equal to the number of replicas.

        The following status.conditions section describes a healthy ManagedResource:

        conditions:
        - lastTransitionTime: "2022-05-03T10:55:39Z"
          lastUpdateTime: "2022-05-03T10:55:39Z"
          message: All resources are healthy.
          reason: ResourcesHealthy
          status: "True"
          type: ResourcesHealthy
        - lastTransitionTime: "2022-05-03T10:55:36Z"
          lastUpdateTime: "2022-05-03T10:55:36Z"
          message: All resources have been fully rolled out.
          reason: ResourcesRolledOut
          status: "False"
          type: ResourcesProgressing
        - lastTransitionTime: "2022-05-03T10:55:18Z"
          lastUpdateTime: "2022-05-03T10:55:18Z"
          message: All resources are applied.
          reason: ApplySucceeded
          status: "True"
          type: ResourcesApplied
        

        Ignoring Updates

        In some cases, it is not desirable to update or re-apply some of the cluster components (for example, if customization is required or needs to be applied by the end-user). For these resources, the annotation “resources.gardener.cloud/ignore” needs to be set to “true” or a truthy value (Truthy values are “1”, “t”, “T”, “true”, “TRUE”, “True”) in the corresponding managed resource secrets. This can be done from the components that create the managed resource secrets, for example Gardener extensions or Gardener. Once this is done, the resource will be initially created and later ignored during reconciliation.

        Finalizing Deletion of Resources After Grace Period

        When a ManagedResource is deleted, the controller deletes all managed resources from the target cluster. In case the resources still have entries in their .metadata.finalizers[] list, they will remain stuck in the system until another entity removes the finalizers. If you want the controller to forcefully finalize the deletion after some grace period (i.e., setting .metadata.finalizers=null), you can annotate the managed resources with resources.gardener.cloud/finalize-deletion-after=<duration>, e.g., resources.gardener.cloud/finalize-deletion-after=1h.

        Preserving replicas or resources in Workload Resources

        The objects which are part of the ManagedResource can be annotated with:

        • resources.gardener.cloud/preserve-replicas=true in case the .spec.replicas field of workload resources like Deployments, StatefulSets, etc., shall be preserved during updates.
        • resources.gardener.cloud/preserve-resources=true in case the .spec.containers[*].resources fields of all containers of workload resources like Deployments, StatefulSets, etc., shall be preserved during updates.

        This can be useful if there are non-standard horizontal/vertical auto-scaling mechanisms in place. Standard mechanisms like HorizontalPodAutoscaler or VerticalPodAutoscaler will be auto-recognized by gardener-resource-manager, i.e., in such cases the annotations are not needed.

        Origin

        All the objects managed by the resource manager get a dedicated annotation resources.gardener.cloud/origin describing the ManagedResource object that describes this object. The default format is <namespace>/<objectname>.

        In multi-cluster scenarios (the ManagedResource objects are maintained in a cluster different from the one the described objects are managed), it might be useful to include the cluster identity, as well.

        This can be enforced by setting the .controllers.clusterID field in the component configuration. Here, several possibilities are supported:

        • given a direct value: use this as id for the source cluster.
        • <cluster>: read the cluster identity from a cluster-identity config map in the kube-system namespace (attribute cluster-identity). This is automatically maintained in all clusters managed or involved in a gardener landscape.
        • <default>: try to read the cluster identity from the config map. If not found, no identity is used.
        • empty string: no cluster identity is used (completely cluster local scenarios).

        By default, cluster id is not used. If cluster id is specified, the format is <cluster id>:<namespace>/<objectname>.

        In addition to the origin annotation, all objects managed by the resource manager get a dedicated label resources.gardener.cloud/managed-by. This label can be used to describe these objects with a selector. By default it is set to “gardener”, but this can be overwritten by setting the .conrollers.managedResources.managedByLabelValue field in the component configuration.

        health Controller

        This controller processes ManagedResources that were reconciled by the main ManagedResource Controller at least once. Its main job is to perform checks for maintaining the well known conditions ResourcesHealthy and ResourcesProgressing.

        Progressing Checks

        In Kubernetes, applied changes must usually be rolled out first, e.g. when changing the base image in a Deployment. Progressing checks detect ongoing roll-outs and report them in the ResourcesProgressing condition of the corresponding ManagedResource.

        The following object kinds are considered for progressing checks:

        Health Checks

        gardener-resource-manager can evaluate the health of specific resources, often by consulting their conditions. Health check results are regularly updated in the ResourcesHealthy condition of the corresponding ManagedResource.

        The following object kinds are considered for health checks:

        Skipping Health Check

        If a resource owned by a ManagedResource is annotated with resources.gardener.cloud/skip-health-check=true, then the resource will be skipped during health checks by the health controller. The ManagedResource conditions will not reflect the health condition of this resource anymore. The ResourcesProgressing condition will also be set to False.

        Garbage Collector For Immutable ConfigMaps/Secrets

        In Kubernetes, workload resources (e.g., Pods) can mount ConfigMaps or Secrets or reference them via environment variables in containers. Typically, when the content of such a ConfigMap/Secret gets changed, then the respective workload is usually not dynamically reloading the configuration, i.e., a restart is required. The most commonly used approach is probably having the so-called checksum annotations in the pod template, which makes Kubernetes recreate the pod if the checksum changes. However, it has the downside that old, still running versions of the workload might not be able to properly work with the already updated content in the ConfigMap/Secret, potentially causing application outages.

        In order to protect users from such outages (and also to improve the performance of the cluster), the Kubernetes community provides the “immutable ConfigMaps/Secrets feature”. Enabling immutability requires ConfigMaps/Secrets to have unique names. Having unique names requires the client to delete ConfigMaps/Secrets no longer in use.

        In order to provide a similarly lightweight experience for clients (compared to the well-established checksum annotation approach), the gardener-resource-manager features an optional garbage collector controller (disabled by default). The purpose of this controller is cleaning up such immutable ConfigMaps/Secrets if they are no longer in use.

        How Does the Garbage Collector Work?

        The following algorithm is implemented in the GC controller:

        1. List all ConfigMaps and Secrets labeled with resources.gardener.cloud/garbage-collectable-reference=true.
        2. List all Deployments, StatefulSets, DaemonSets, Jobs, CronJobs, Pods, ManagedResources and for each of them:
          • iterate over the .metadata.annotations and for each of them:
            • If the annotation key follows the reference.resources.gardener.cloud/{configmap,secret}-<hash> scheme and the value equals <name>, then consider it as “in-use”.
        3. Delete all ConfigMaps and Secrets not considered as “in-use”.

        Consequently, clients need to:

        1. Create immutable ConfigMaps/Secrets with unique names (e.g., a checksum suffix based on the .data).

        2. Label such ConfigMaps/Secrets with resources.gardener.cloud/garbage-collectable-reference=true.

        3. Annotate their workload resources with reference.resources.gardener.cloud/{configmap,secret}-<hash>=<name> for all ConfigMaps/Secrets used by the containers of the respective Pods.

          ⚠️ Add such annotations to .metadata.annotations, as well as to all templates of other resources (e.g., .spec.template.metadata.annotations in Deployments or .spec.jobTemplate.metadata.annotations and .spec.jobTemplate.spec.template.metadata.annotations for CronJobs. This ensures that the GC controller does not unintentionally consider ConfigMaps/Secrets as “not in use” just because there isn’t a Pod referencing them anymore (e.g., they could still be used by a Deployment scaled down to 0).

        ℹ️ For the last step, there is a helper function InjectAnnotations in the pkg/controller/garbagecollector/references, which you can use for your convenience.

        Example:

        ---
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: test-1234
          namespace: default
          labels:
            resources.gardener.cloud/garbage-collectable-reference: "true"
        ---
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: test-5678
          namespace: default
          labels:
            resources.gardener.cloud/garbage-collectable-reference: "true"
        ---
        apiVersion: v1
        kind: Pod
        metadata:
          name: example
          namespace: default
          annotations:
            reference.resources.gardener.cloud/configmap-82a3537f: test-5678
        spec:
          containers:
          - name: nginx
            image: nginx:1.14.2
            terminationGracePeriodSeconds: 2
        

        The GC controller would delete the ConfigMap/test-1234 because it is considered as not “in-use”.

        ℹ️ If the GC controller is activated then the ManagedResource controller will no longer delete ConfigMaps/Secrets having the above label.

        How to Activate the Garbage Collector?

        The GC controller can be activated by setting the .controllers.garbageCollector.enabled field to true in the component configuration.

        TokenInvalidator Controller

        The Kubernetes community is slowly transitioning from static ServiceAccount token Secrets to ServiceAccount Token Volume Projection. Typically, when you create a ServiceAccount

        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: default
        

        then the serviceaccount-token controller (part of kube-controller-manager) auto-generates a Secret with a static token:

        apiVersion: v1
        kind: Secret
        metadata:
           annotations:
              kubernetes.io/service-account.name: default
              kubernetes.io/service-account.uid: 86e98645-2e05-11e9-863a-b2d4d086dd5a)
           name: default-token-ntxs9
        type: kubernetes.io/service-account-token
        data:
           ca.crt: base64(cluster-ca-cert)
           namespace: base64(namespace)
           token: base64(static-jwt-token)
        

        Unfortunately, when using ServiceAccount Token Volume Projection in a Pod, this static token is actually not used at all:

        apiVersion: v1
        kind: Pod
        metadata:
          name: nginx
        spec:
          serviceAccountName: default
          containers:
          - image: nginx
            name: nginx
            volumeMounts:
            - mountPath: /var/run/secrets/tokens
              name: token
          volumes:
          - name: token
            projected:
              sources:
              - serviceAccountToken:
                  path: token
                  expirationSeconds: 7200
        

        While the Pod is now using an expiring and auto-rotated token, the static token is still generated and valid.

        There is neither a way of preventing kube-controller-manager to generate such static tokens, nor a way to proactively remove or invalidate them:

        Disabling the serviceaccount-token controller is an option, however, especially in the Gardener context it may either break end-users or it may not even be possible to control such settings. Also, even if a future Kubernetes version supports native configuration of the above behaviour, Gardener still supports older versions which won’t get such features but need a solution as well.

        This is where the TokenInvalidator comes into play: Since it is not possible to prevent kube-controller-manager from generating static ServiceAccount Secrets, the TokenInvalidator is, as its name suggests, just invalidating these tokens. It considers all such Secrets belonging to ServiceAccounts with .automountServiceAccountToken=false. By default, all namespaces in the target cluster are watched, however, this can be configured by specifying the .targetClientConnection.namespace field in the component configuration. Note that this setting also affects all other controllers and webhooks since it’s a central configuration.

        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: my-serviceaccount
        automountServiceAccountToken: false
        

        This will result in a static ServiceAccount token secret whose token value is invalid:

        apiVersion: v1
        kind: Secret
        metadata:
          annotations:
            kubernetes.io/service-account.name: my-serviceaccount
            kubernetes.io/service-account.uid: 86e98645-2e05-11e9-863a-b2d4d086dd5a
          name: my-serviceaccount-token-ntxs9
        type: kubernetes.io/service-account-token
        data:
          ca.crt: base64(cluster-ca-cert)
          namespace: base64(namespace)
          token: AAAA
        

        Any attempt to regenerate the token or creating a new such secret will again make the component invalidating it.

        You can opt-out of this behaviour for ServiceAccounts setting .automountServiceAccountToken=false by labeling them with token-invalidator.resources.gardener.cloud/skip=true.

        In order to enable the TokenInvalidator you have to set both .controllers.tokenValidator.enabled=true and .webhooks.tokenValidator.enabled=true in the component configuration.

        The below graphic shows an overview of the Token Invalidator for Service account secrets in the Shoot cluster. image

        TokenRequestor Controller

        This controller provides the service to create and auto-renew tokens via the TokenRequest API.

        It provides a functionality similar to the kubelet’s Service Account Token Volume Projection. It was created to handle the special case of issuing tokens to pods that run in a different cluster than the API server they communicate with (hence, using the native token volume projection feature is not possible).

        The controller differentiates between source cluster and target cluster. The source cluster hosts the gardener-resource-manager pod. Secrets in this cluster are watched and modified by the controller. The target cluster can be configured to point to another cluster. The existence of ServiceAccounts are ensured and token requests are issued against the target. When the gardener-resource-manager is deployed next to the Shoot’s controlplane in the Seed, the source cluster is the Seed while the target cluster points to the Shoot.

        Reconciliation Loop

        This controller reconciles Secrets in all namespaces in the source cluster with the label: resources.gardener.cloud/purpose=token-requestor. See this YAML file for an example of the secret.

        The controller ensures a ServiceAccount exists in the target cluster as specified in the annotations of the Secret in the source cluster:

        serviceaccount.resources.gardener.cloud/name: <sa-name>
        serviceaccount.resources.gardener.cloud/namespace: <sa-namespace>
        

        You can optionally annotate the Secret with serviceaccount.resources.gardener.cloud/labels, e.g. serviceaccount.resources.gardener.cloud/labels={"some":"labels","foo":"bar"}. This will make the ServiceAccount getting labelled accordingly.

        The requested tokens will act with the privileges which are assigned to this ServiceAccount.

        The controller will then request a token via the TokenRequest API and populate it into the .data.token field to the Secret in the source cluster.

        Alternatively, the client can provide a raw kubeconfig (in YAML or JSON format) via the Secret’s .data.kubeconfig field. The controller will then populate the requested token in the kubeconfig for the user used in the .current-context. For example, if .data.kubeconfig is

        apiVersion: v1
        clusters:
        - cluster:
            certificate-authority-data: AAAA
            server: some-server-url
          name: shoot--foo--bar
        contexts:
        - context:
            cluster: shoot--foo--bar
            user: shoot--foo--bar-token
          name: shoot--foo--bar
        current-context: shoot--foo--bar
        kind: Config
        preferences: {}
        users:
        - name: shoot--foo--bar-token
          user:
            token: ""
        

        then the .users[0].user.token field of the kubeconfig will be updated accordingly.

        The controller also adds an annotation to the Secret to keep track when to renew the token before it expires. By default, the tokens are issued to expire after 12 hours. The expiration time can be set with the following annotation:

        serviceaccount.resources.gardener.cloud/token-expiration-duration: 6h
        

        It automatically renews once 80% of the lifetime is reached, or after 24h.

        Optionally, the controller can also populate the token into a Secret in the target cluster. This can be requested by annotating the Secret in the source cluster with:

        token-requestor.resources.gardener.cloud/target-secret-name: "foo"
        token-requestor.resources.gardener.cloud/target-secret-namespace: "bar"
        

        Overall, the TokenRequestor controller provides credentials with limited lifetime (JWT tokens) used by Shoot control plane components running in the Seed to talk to the Shoot API Server. Please see the graphic below:

        image

        ℹ️ Generally, the controller can run with multiple instances in different components. For example, gardener-resource-manager might run the TokenRequestor controller, but gardenlet might run it, too. In order to differentiate which instance of the controller is responsible for a Secret, it can be labeled with resources.gardener.cloud/class=<class>. The <class> must be configured in the respective controller, otherwise it will be responsible for all Secrets no matter whether they have the label or not.

        Kubelet Server CertificateSigningRequest Approver

        Gardener configures the kubelets such that they request two certificates via the CertificateSigningRequest API:

        1. client certificate for communicating with the kube-apiserver
        2. server certificate for serving its HTTPS server

        For client certificates, the kubernetes.io/kube-apiserver-client-kubelet signer is used (see Certificate Signing Requests for more details). The kube-controller-manager’s csrapprover controller is responsible for auto-approving such CertificateSigningRequests so that the respective certificates can be issued.

        For server certificates, the kubernetes.io/kubelet-serving signer is used. Unfortunately, the kube-controller-manager is not able to auto-approve such CertificateSigningRequests (see kubernetes/kubernetes#73356 for details).

        That’s the motivation for having this controller as part of gardener-resource-manager. It watches CertificateSigningRequests with the kubernetes.io/kubelet-serving signer and auto-approves them when all the following conditions are met:

        • The .spec.username is prefixed with system:node:.
        • There must be at least one DNS name or IP address as part of the certificate SANs.
        • The common name in the CSR must match the .spec.username.
        • The organization in the CSR must only contain system:nodes.
        • There must be a Node object with the same name in the shoot cluster.
        • There must be exactly one Machine for the node in the seed cluster.
        • The DNS names part of the SANs must be equal to all .status.addresses[] of type Hostname in the Node.
        • The IP addresses part of the SANs must be equal to all .status.addresses[] of type InternalIP in the Node.

        If any one of these requirements is violated, the CertificateSigningRequest will be denied. Otherwise, once approved, the kube-controller-manager’s csrsigner controller will issue the requested certificate.

        NetworkPolicy Controller

        This controller reconciles Services with a non-empty .spec.podSelector. It creates two NetworkPolicys for each port in the .spec.ports[] list. For example:

        apiVersion: v1
        kind: Service
        metadata:
          name: gardener-resource-manager
          namespace: a
        spec:
          selector:
            app: gardener-resource-manager
          ports:
          - name: server
            port: 443
            protocol: TCP
            targetPort: 10250
        

        leads to

        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          annotations:
            gardener.cloud/description: Allows ingress TCP traffic to port 10250 for pods
              selected by the a/gardener-resource-manager service selector from pods running
              in namespace a labeled with map[networking.resources.gardener.cloud/to-gardener-resource-manager-tcp-10250:allowed].
          name: ingress-to-gardener-resource-manager-tcp-10250
          namespace: a
        spec:
          ingress:
          - from:
            - podSelector:
                matchLabels:
                  networking.resources.gardener.cloud/to-gardener-resource-manager-tcp-10250: allowed
            ports:
            - port: 10250
              protocol: TCP
          podSelector:
            matchLabels:
              app: gardener-resource-manager
          policyTypes:
          - Ingress
        ---
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          annotations:
            gardener.cloud/description: Allows egress TCP traffic to port 10250 from pods
              running in namespace a labeled with map[networking.resources.gardener.cloud/to-gardener-resource-manager-tcp-10250:allowed]
              to pods selected by the a/gardener-resource-manager service selector.
          name: egress-to-gardener-resource-manager-tcp-10250
          namespace: a
        spec:
          egress:
          - to:
            - podSelector:
                matchLabels:
                  app: gardener-resource-manager
            ports:
            - port: 10250
              protocol: TCP
          podSelector:
            matchLabels:
              networking.resources.gardener.cloud/to-gardener-resource-manager-tcp-10250: allowed
          policyTypes:
          - Egress
        

        A component that initiates the connection to gardener-resource-manager’s tcp/10250 port can now be labeled with networking.resources.gardener.cloud/to-gardener-resource-manager-tcp-10250=allowed. That’s all this component needs to do - it does not need to create any NetworkPolicys itself.

        Cross-Namespace Communication

        Apart from this “simple” case where both communicating components run in the same namespace a, there is also the cross-namespace communication case. With above example, let’s say there are components running in another namespace b, and they would like to initiate the communication with gardener-resource-manager in a. To cover this scenario, the Service can be annotated with networking.resources.gardener.cloud/namespace-selectors='[{"matchLabels":{"kubernetes.io/metadata.name":"b"}}]'.

        Note that you can specify multiple namespace selectors in this annotation which are OR-ed.

        This will make the controller create additional NetworkPolicys as follows:

        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          annotations:
            gardener.cloud/description: Allows ingress TCP traffic to port 10250 for pods selected
              by the a/gardener-resource-manager service selector from pods running in namespace b
              labeled with map[networking.resources.gardener.cloud/to-a-gardener-resource-manager-tcp-10250:allowed].
          name: ingress-to-gardener-resource-manager-tcp-10250-from-b
          namespace: a
        spec:
          ingress:
          - from:
            - namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: b
              podSelector:
                matchLabels:
                  networking.resources.gardener.cloud/to-a-gardener-resource-manager-tcp-10250: allowed
            ports:
            - port: 10250
              protocol: TCP
          podSelector:
            matchLabels:
              app: gardener-resource-manager
          policyTypes:
          - Ingress
        ---
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          annotations:
            gardener.cloud/description: Allows egress TCP traffic to port 10250 from pods running in
              namespace b labeled with map[networking.resources.gardener.cloud/to-a-gardener-resource-manager-tcp-10250:allowed]
              to pods selected by the a/gardener-resource-manager service selector.
          name: egress-to-a-gardener-resource-manager-tcp-10250
          namespace: b
        spec:
          egress:
          - to:
            - namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: a
              podSelector:
                matchLabels:
                  app: gardener-resource-manager
            ports:
            - port: 10250
              protocol: TCP
          podSelector:
            matchLabels:
              networking.resources.gardener.cloud/to-a-gardener-resource-manager-tcp-10250: allowed
          policyTypes:
          - Egress
        

        The components in namespace b now need to be labeled with networking.resources.gardener.cloud/to-a-gardener-resource-manager-tcp-10250=allowed, but that’s already it.

        Obviously, this approach also works for namespace selectors different from kubernetes.io/metadata.name to cover scenarios where the namespace name is not known upfront or where multiple namespaces with a similar label are relevant. The controller creates two dedicated policies for each namespace matching the selectors.

        Service Targets In Multiple Namespaces

        Finally, let’s say there is a Service called example which exists in different namespaces whose names are not static (e.g., foo-1, foo-2), and a component in namespace bar wants to initiate connections with all of them.

        The example Services in these namespaces can now be annotated with networking.resources.gardener.cloud/namespace-selectors='[{"matchLabels":{"kubernetes.io/metadata.name":"bar"}}]'. As a consequence, the component in namespace bar now needs to be labeled with networking.resources.gardener.cloud/to-foo-1-example-tcp-8080=allowed, networking.resources.gardener.cloud/to-foo-2-example-tcp-8080=allowed, etc. This approach does not work in practice, however, since the namespace names are neither static nor known upfront.

        To overcome this, it is possible to specify an alias for the concrete namespace in the pod label selector via the networking.resources.gardener.cloud/pod-label-selector-namespace-alias annotation.

        In above case, the example Service in the foo-* namespaces could be annotated with networking.resources.gardener.cloud/pod-label-selector-namespace-alias=all-foos. This would modify the label selector in all NetworkPolicys related to cross-namespace communication, i.e. instead of networking.resources.gardener.cloud/to-foo-{1,2,...}-example-tcp-8080=allowed, networking.resources.gardener.cloud/to-all-foos-example-tcp-8080=allowed would be used. Now the component in namespace bar only needs this single label and is able to talk to all such Services in the different namespaces.

        Real-world examples for this scenario are the kube-apiserver Service (which exists in all shoot namespaces), or the istio-ingressgateway Service (which exists in all istio-ingress* namespaces). In both cases, the names of the namespaces are not statically known and depend on user input.

        Overwriting The Pod Selector Label

        For a component which initiates the connection to many other components, it’s sometimes impractical to specify all the respective labels in its pod template. For example, let’s say a component foo talks to bar{0..9} on ports tcp/808{0..9}. foo would need to have the ten networking.resources.gardener.cloud/to-bar{0..9}-tcp-808{0..9}=allowed labels.

        As an alternative and to simplify this, it is also possible to annotate the targeted Services with networking.resources.gardener.cloud/from-<some-alias>-allowed-ports. For our example, <some-alias> could be all-bars.

        As a result, component foo just needs to have the label networking.resources.gardener.cloud/to-all-bars=allowed instead of all the other ten explicit labels.

        ⚠️ Note that this also requires to specify the list of allowed container ports as annotation value since the pod selector label will no longer be specific for a dedicated service/port. For our example, the Service for barX with X in {0..9} needs to be annotated with networking.resources.gardener.cloud/from-all-bars-allowed-ports=[{"port":808X,"protocol":"TCP"}] in addition.

        Real-world examples for this scenario are the Prometheis in seed clusters which initiate the communication to a lot of components in order to scrape their metrics. Another example is the kube-apiserver which initiates the communication to webhook servers (potentially of extension components that are not known by Gardener itself).

        Ingress From Everywhere

        All above scenarios are about components initiating connections to some targets. However, some components also receive incoming traffic from sources outside the cluster. This traffic requires adequate ingress policies so that it can be allowed.

        To cover this scenario, the Service can be annotated with networking.resources.gardener.cloud/from-world-to-ports=[{"port":"10250","protocol":"TCP"}]. As a result, the controller creates the following NetworkPolicy:

        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          name: ingress-to-gardener-resource-manager-from-world
          namespace: a
        spec:
          ingress:
          - from:
            - namespaceSelector: {}
              podSelector: {}
            - ipBlock:
                cidr: 0.0.0.0/0
            - ipBlock:
                cidr: ::/0
            ports:
            - port: 10250
              protocol: TCP
          podSelector:
            matchLabels:
              app: gardener-resource-manager
          policyTypes:
          - Ingress
        

        The respective pods don’t need any additional labels. If the annotation’s value is empty ([]) then all ports are allowed.

        Services Exposed via Ingress Resources

        The controller can optionally be configured to watch Ingress resources by specifying the pod and namespace selectors for the Ingress controller. If this information is provided, it automatically creates NetworkPolicy resources allowing the respective ingress/egress traffic for the backends exposed by the Ingresses. This way, neither custom NetworkPolicys nor custom labels must be provided.

        The needed configuration is part of the component configuration:

        controllers:
          networkPolicy:
            enabled: true
            concurrentSyncs: 5
          # namespaceSelectors:
          # - matchLabels:
          #     kubernetes.io/metadata.name: default
            ingressControllerSelector:
              namespace: default
              podSelector:
                matchLabels:
                  foo: bar
        

        As an example, let’s assume that above gardener-resource-manager Service was exposed via the following Ingress resource:

        apiVersion: networking.k8s.io/v1
        kind: Ingress
        metadata:
          name: gardener-resource-manager
          namespace: a
        spec:
          rules:
          - host: grm.foo.example.com
            http:
              paths:
              - backend:
                  service:
                    name: gardener-resource-manager
                    port:
                      number: 443
                path: /
                pathType: Prefix
        

        As a result, the controller would automatically create the following NetworkPolicys:

        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          annotations:
            gardener.cloud/description: Allows ingress TCP traffic to port 10250 for pods
              selected by the a/gardener-resource-manager service selector from ingress controller
              pods running in the default namespace labeled with map[foo:bar].
          name: ingress-to-gardener-resource-manager-tcp-10250-from-ingress-controller
          namespace: a
        spec:
          ingress:
          - from:
            - podSelector:
                matchLabels:
                  foo: bar
              namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: default
            ports:
            - port: 10250
              protocol: TCP
          podSelector:
            matchLabels:
              app: gardener-resource-manager
          policyTypes:
          - Ingress
        ---
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          annotations:
            gardener.cloud/description: Allows egress TCP traffic to port 10250 from pods
              running in the default namespace labeled with map[foo:bar] to pods selected by
              the a/gardener-resource-manager service selector.
          name: egress-to-a-gardener-resource-manager-tcp-10250-from-ingress-controller
          namespace: default
        spec:
          egress:
          - to:
            - podSelector:
                matchLabels:
                  app: gardener-resource-manager
              namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: a
            ports:
            - port: 10250
              protocol: TCP
          podSelector:
            matchLabels:
              foo: bar
          policyTypes:
          - Egress
        

        ℹ️ Note that Ingress resources reference the service port while NetworkPolicys reference the target port/container port. The controller automatically translates this when reconciling the NetworkPolicy resources.

        Node Controller

        Critical Components Controller

        Gardenlet configures kubelet of shoot worker nodes to register the Node object with the node.gardener.cloud/critical-components-not-ready taint (effect NoSchedule). This controller watches newly created Node objects in the shoot cluster and removes the taint once all node-critical components are scheduled and ready. If the controller finds node-critical components that are not scheduled or not ready yet, it checks the Node again after the duration configured in ResourceManagerConfiguration.controllers.node.backoff Please refer to the feature documentation or proposal issue for more details.

        Node Agent Reconciliation Delay Controller

        This controller computes a reconciliation delay per node by using a simple linear mapping approach based on the index of the nodes in the list of all nodes in the shoot cluster. This approach ensures that the delays of all instances of gardener-node-agent are distributed evenly.

        The minimum and maximum delays can be configured, but they are defaulted to 0s and 5m, respectively.

        This approach works well as long as the number of nodes in the cluster is not higher than the configured maximum delay in seconds. In this case, the delay is still computed linearly, however, the more nodes exist in the cluster, the closer the delay times become (which might be of limited use then). Consider increasing the maximum delay by annotating the Shoot with shoot.gardener.cloud/cloud-config-execution-max-delay-seconds=<value>. The highest possible value is 1800.

        The controller adds the node-agent.gardener.cloud/reconciliation-delay annotation to nodes whose value is read by the node-agents.

        Webhooks

        Mutating Webhooks

        High Availability Config

        This webhook is used to conveniently apply the configuration to make components deployed to seed or shoot clusters highly available. The details and scenarios are described in High Availability Of Deployed Components.

        The webhook reacts on creation/update of Deployments, StatefulSets, HorizontalPodAutoscalers and HVPAs in namespaces labeled with high-availability-config.resources.gardener.cloud/consider=true.

        The webhook performs the following actions:

        1. The .spec.replicas (or spec.minReplicas respectively) field is mutated based on the high-availability-config.resources.gardener.cloud/type label of the resource and the high-availability-config.resources.gardener.cloud/failure-tolerance-type annotation of the namespace:

          Failure Tolerance Type ➡️
          /
          ⬇️ Component Type️ ️
          unsetemptynon-empty
          controller212
          server222
          • The replica count values can be overwritten by the high-availability-config.resources.gardener.cloud/replicas annotation.
          • It does NOT mutate the replicas when:
            • the replicas are already set to 0 (hibernation case), or
            • when the resource is scaled horizontally by HorizontalPodAutoscaler or Hvpa, and the current replica count is higher than what was computed above.
        2. When the high-availability-config.resources.gardener.cloud/zones annotation is NOT empty and either the high-availability-config.resources.gardener.cloud/failure-tolerance-type annotation is set or the high-availability-config.resources.gardener.cloud/zone-pinning annotation is set to true, then it adds a node affinity to the pod template spec:

          spec:
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: topology.kubernetes.io/zone
                      operator: In
                      values:
                      - <zone1>
                    # - ...
          

          This ensures that all pods are pinned to only nodes in exactly those concrete zones.

        3. Topology Spread Constraints are added to the pod template spec when the .spec.replicas are greater than 1. When the high-availability-config.resources.gardener.cloud/zones annotation …

          • … contains only one zone, then the following is added:

            spec:
              topologySpreadConstraints:
              - topologyKey: kubernetes.io/hostname
                maxSkew: 1
                whenUnsatisfiable: ScheduleAnyway
                labelSelector: ...
            

            This ensures that the (multiple) pods are scheduled across nodes on best-effort basis.

          • … contains at least two zones, then the following is added:

            spec:
              topologySpreadConstraints:
              - topologyKey: kubernetes.io/hostname
                maxSkew: 1
                whenUnsatisfiable: ScheduleAnyway
                labelSelector: ...
              - topologyKey: topology.kubernetes.io/zone
                minDomains: 2 # lower value of max replicas or number of zones
                maxSkew: 1
                whenUnsatisfiable: DoNotSchedule
                labelSelector: ...
            

            This enforces that the (multiple) pods are scheduled across zones. It circumvents a known limitation in Kubernetes for clusters < 1.26 (ref kubernetes/kubernetes#109364. In case the number of replicas is larger than twice the number of zones, then the maxSkew=2 for the second spread constraints. The minDomains calculation is based on whatever value is lower - (maximum) replicas or number of zones. This is the number of minimum domains required to schedule pods in a highly available manner.

          Independent on the number of zones, when one of the following conditions is true, then the field whenUnsatisfiable is set to DoNotSchedule for the constraint with topologyKey=kubernetes.io/hostname (which enforces the node-spread):

          • The high-availability-config.resources.gardener.cloud/host-spread annotation is set to true.
          • The high-availability-config.resources.gardener.cloud/failure-tolerance-type annotation is set and NOT empty.
        4. Adds default tolerations for taint-based evictions:

          Tolerations for taints node.kubernetes.io/not-ready and node.kubernetes.io/unreachable are added to the handled Deployment and StatefulSet if their podTemplates do not already specify them. The TolerationSeconds are taken from the respective configuration section of the webhook’s configuration (see example)).

          We consider fine-tuned values for those tolerations a matter of high-availability because they often help to reduce recovery times in case of node or zone outages, also see High-Availability Best Practices. In addition, this webhook handling helps to set defaults for many but not all workload components in a cluster. For instance, Gardener can use this webhook to set defaults for nearly every component in seed clusters but only for the system components in shoot clusters. Any customer workload remains unchanged.

        Kubernetes Service Host Injection

        By default, when Pods are created, Kubernetes implicitly injects the KUBERNETES_SERVICE_HOST environment variable into all containers. The value of this variable points it to the default Kubernetes service (i.e., kubernetes.default.svc.cluster.local). This allows pods to conveniently talk to the API server of their cluster.

        In shoot clusters, this network path involves the apiserver-proxy DaemonSet which eventually forwards the traffic to the API server. Hence, it results in additional network hop.

        The purpose of this webhook is to explicitly inject the KUBERNETES_SERVICE_HOST environment variable into all containers and setting its value to the FQDN of the API server. This way, the additional network hop is avoided.

        Auto-Mounting Projected ServiceAccount Tokens

        When this webhook is activated, then it automatically injects projected ServiceAccount token volumes into Pods and all its containers if all of the following preconditions are fulfilled:

        1. The Pod is NOT labeled with projected-token-mount.resources.gardener.cloud/skip=true.
        2. The Pod’s .spec.serviceAccountName field is NOT empty and NOT set to default.
        3. The ServiceAccount specified in the Pod’s .spec.serviceAccountName sets .automountServiceAccountToken=false.
        4. The Pod’s .spec.volumes[] DO NOT already contain a volume with a name prefixed with kube-api-access-.

        The projected volume will look as follows:

        spec:
          volumes:
          - name: kube-api-access-gardener
            projected:
              defaultMode: 420
              sources:
              - serviceAccountToken:
                  expirationSeconds: 43200
                  path: token
              - configMap:
                  items:
                  - key: ca.crt
                    path: ca.crt
                  name: kube-root-ca.crt
              - downwardAPI:
                  items:
                  - fieldRef:
                      apiVersion: v1
                      fieldPath: metadata.namespace
                    path: namespace
        

        The expirationSeconds are defaulted to 12h and can be overwritten with the .webhooks.projectedTokenMount.expirationSeconds field in the component configuration, or with the projected-token-mount.resources.gardener.cloud/expiration-seconds annotation on a Pod resource.

        The volume will be mounted into all containers specified in the Pod to the path /var/run/secrets/kubernetes.io/serviceaccount. This is the default location where client libraries expect to find the tokens and mimics the upstream ServiceAccount admission plugin. See Managing Service Accounts for more information.

        Overall, this webhook is used to inject projected service account tokens into pods running in the Shoot and the Seed cluster. Hence, it is served from the Seed GRM and each Shoot GRM. Please find an overview below for pods deployed in the Shoot cluster:

        image

        Pod Topology Spread Constraints

        When this webhook is enabled, then it mimics the topologyKey feature for Topology Spread Constraints (TSC) on the label pod-template-hash. Concretely, when a pod is labelled with pod-template-hash, the handler of this webhook extends any topology spread constraint in the pod:

        metadata:
          labels:
            pod-template-hash: 123abc
        spec:
          topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                pod-template-hash: 123abc # added by webhook
        

        The procedure circumvents a known limitation with TSCs which leads to imbalanced deployments after rolling updates. Gardener enables this webhook to schedule pods of deployments across nodes and zones.

        Please note that the gardener-resource-manager itself as well as pods labelled with topology-spread-constraints.resources.gardener.cloud/skip are excluded from any mutations.

        System Components Webhook

        If enabled, this webhook handles scheduling concerns for system components Pods (except those managed by DaemonSets). The following tasks are performed by this webhook:

        1. Add pod.spec.nodeSelector as given in the webhook configuration.
        2. Add pod.spec.tolerations as given in the webhook configuration.
        3. Add pod.spec.tolerations for any existing nodes matching the node selector given in the webhook configuration. Known taints and tolerations used for taint based evictions are disregarded.

        Gardener enables this webhook for kube-system and kubernetes-dashboard namespaces in shoot clusters, selecting Pods being labelled with resources.gardener.cloud/managed-by: gardener. It adds a configuration, so that Pods will get the worker.gardener.cloud/system-components: true node selector (step 1) as well as tolerate any custom taint (step 2) that is added to system component worker nodes (shoot.spec.provider.workers[].systemComponents.allow: true). In addition, the webhook merges these tolerations with the ones required for at that time available system component Nodes in the cluster (step 3). Both is required to ensure system component Pods can be scheduled or executed during an active shoot reconciliation that is happening due to any modifications to shoot.spec.provider.workers[].taints, e.g. Pods must be scheduled while there are still Nodes not having the updated taint configuration.

        You can opt-out of this behaviour for Pods by labeling them with system-components-config.resources.gardener.cloud/skip=true.

        EndpointSlice Hints

        This webhook mutates EndpointSlices. For each endpoint in the EndpointSlice, it sets the endpoint’s hints to the endpoint’s zone.

        apiVersion: discovery.k8s.io/v1
        kind: EndpointSlice
        metadata:
          name: example-hints
        endpoints:
        - addresses:
          - "10.1.2.3"
          conditions:
            ready: true
          hostname: pod-1
          zone: zone-a
          hints:
            forZones:
            - name: "zone-a" # added by webhook
        - addresses:
          - "10.1.2.4"
          conditions:
            ready: true
          hostname: pod-2
          zone: zone-b
          hints:
            forZones:
            - name: "zone-b" # added by webhook
        

        The webhook aims to circumvent issues with the Kubernetes TopologyAwareHints feature that currently does not allow to achieve a deterministic topology-aware traffic routing. For more details, see the following issue kubernetes/kubernetes#113731 that describes drawbacks of the TopologyAwareHints feature for our use case. If the above-mentioned issue gets resolved and there is a native support for deterministic topology-aware traffic routing in Kubernetes, then this webhook can be dropped in favor of the native Kubernetes feature.

        Validating Webhooks

        Unconfirmed Deletion Prevention For Custom Resources And Definitions

        As part of Gardener’s extensibility concepts, a lot of CustomResourceDefinitions are deployed to the seed clusters that serve as extension points for provider-specific controllers. For example, the Infrastructure CRD triggers the provider extension to prepare the IaaS infrastructure of the underlying cloud provider for a to-be-created shoot cluster. Consequently, these extension CRDs have a lot of power and control large portions of the end-user’s shoot cluster. Accidental or undesired deletions of those resource can cause tremendous and hard-to-recover-from outages and should be prevented.

        When this webhook is activated, it reacts for CustomResourceDefinitions and most of the custom resources in the extensions.gardener.cloud/v1alpha1 API group. It also reacts for the druid.gardener.cloud/v1alpha1.Etcd resources.

        The webhook prevents DELETE requests for those CustomResourceDefinitions labeled with gardener.cloud/deletion-protected=true, and for all mentioned custom resources if they were not previously annotated with the confirmation.gardener.cloud/deletion=true. This prevents that undesired kubectl delete <...> requests are accepted.

        Extension Resource Validation

        When this webhook is activated, it reacts for most of the custom resources in the extensions.gardener.cloud/v1alpha1 API group. It also reacts for the druid.gardener.cloud/v1alpha1.Etcd resources.

        The webhook validates the resources specifications for CREATE and UPDATE requests.

        2.12 - Gardener Scheduler

        Understand the configuration and flow of the controller that assigns a seed cluster to newly created shoots

        Overview

        The Gardener Scheduler is in essence a controller that watches newly created shoots and assigns a seed cluster to them. Conceptually, the task of the Gardener Scheduler is very similar to the task of the Kubernetes Scheduler: finding a seed for a shoot instead of a node for a pod.

        Either the scheduling strategy or the shoot cluster purpose hereby determines how the scheduler is operating. The following sections explain the configuration and flow in greater detail.

        Why Is the Gardener Scheduler Needed?

        1. Decoupling

        Previously, an admission plugin in the Gardener API server conducted the scheduling decisions. This implies changes to the API server whenever adjustments of the scheduling are needed. Decoupling the API server and the scheduler comes with greater flexibility to develop these components independently.

        2. Extensibility

        It should be possible to easily extend and tweak the scheduler in the future. Possibly, similar to the Kubernetes scheduler, hooks could be provided which influence the scheduling decisions. It should be also possible to completely replace the standard Gardener Scheduler with a custom implementation.

        Algorithm Overview

        The following sequence describes the steps involved to determine a seed candidate:

        1. Determine usable seeds with “usable” defined as follows:
          • no .metadata.deletionTimestamp
          • .spec.settings.scheduling.visible is true
          • .status.lastOperation is not nil
          • conditions GardenletReady, BackupBucketsReady (if available) are true
        2. Filter seeds:
          • matching .spec.seedSelector in CloudProfile used by the Shoot
          • matching .spec.seedSelector in Shoot
          • having no network intersection with the Shoot’s networks (due to the VPN connectivity between seeds and shoots their networks must be disjoint)
          • whose taints (.spec.taints) are tolerated by the Shoot (.spec.tolerations)
          • whose capacity for shoots would not be exceeded if the shoot is scheduled onto the seed, see Ensuring seeds capacity for shoots is not exceeded
          • which have at least three zones in .spec.provider.zones if shoot requests a high available control plane with failure tolerance type zone.
        3. Apply active strategy e.g., Minimal Distance strategy
        4. Choose least utilized seed, i.e., the one with the least number of shoot control planes, will be the winner and written to the .spec.seedName field of the Shoot.

        In order to put the scheduling decision into effect, the scheduler sends an update request for the Shoot resource to the API server. After validation, the gardener-apiserver updates the Shoot to have the spec.seedName field set. Subsequently, the gardenlet picks up and starts to create the cluster on the specified seed.

        Configuration

        The Gardener Scheduler configuration has to be supplied on startup. It is a mandatory and also the only available flag. This yaml file holds an example scheduler configuration.

        Most of the configuration options are the same as in the Gardener Controller Manager (leader election, client connection, …). However, the Gardener Scheduler on the other hand does not need a TLS configuration, because there are currently no webhooks configurable.

        Strategies

        The scheduling strategy is defined in the candidateDeterminationStrategy of the scheduler’s configuration and can have the possible values SameRegion and MinimalDistance. The SameRegion strategy is the default strategy.

        Same Region strategy

        The Gardener Scheduler reads the spec.provider.type and .spec.region fields from the Shoot resource. It tries to find a seed that has the identical .spec.provider.type and .spec.provider.region fields set. If it cannot find a suitable seed, it adds an event to the shoot stating that it is unschedulable.

        Minimal Distance strategy

        The Gardener Scheduler tries to find a valid seed with minimal distance to the shoot’s intended region. Distances are configured via ConfigMap(s), usually per cloud provider in a Gardener landscape. The configuration is structured like this:

        • It refers to one or multiple CloudProfiles via annotation scheduling.gardener.cloud/cloudprofiles.
        • It contains the declaration as region-config via label scheduling.gardener.cloud/purpose.
        • If a CloudProfile is referred by multiple ConfigMaps, only the first one is considered.
        • The data fields configure actual distances, where key relates to the Shoot region and value contains distances to Seed regions.
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: <name>
          namespace: garden
          annotations:
            scheduling.gardener.cloud/cloudprofiles: cloudprofile-name-1{,optional-cloudprofile-name-2,...}
          labels:
            scheduling.gardener.cloud/purpose: region-config
        data:
          region-1: |
            region-2: 10
            region-3: 20
            ...    
          region-2: |
            region-1: 10
            region-3: 10
            ...    
        

        Gardener provider extensions for public cloud providers usually have an example weight ConfigMap in their repositories. We suggest to check them out before defining your own data.

        If a valid seed candidate cannot be found after consulting the distance configuration, the scheduler will fall back to the Levenshtein distance to find the closest region. Therefore, the region name is split into a base name and an orientation. Possible orientations are north, south, east, west and central. The distance then is twice the Levenshtein distance of the region’s base name plus a correction value based on the orientation and the provider.

        If the orientations of shoot and seed candidate match, the correction value is 0, if they differ it is 2 and if either the seed’s or the shoot’s region does not have an orientation it is 1. If the provider differs, the correction value is additionally incremented by 2.

        Because of this, a matching region with a matching provider is always preferred.

        Special handling based on shoot cluster purpose

        Every shoot cluster can have a purpose that describes what the cluster is used for, and also influences how the cluster is setup (see Shoot Cluster Purpose for more information).

        In case the shoot has the testing purpose, then the scheduler only reads the .spec.provider.type from the Shoot resource and tries to find a Seed that has the identical .spec.provider.type. The region does not matter, i.e., testing shoots may also be scheduled on a seed in a complete different region if it is better for balancing the whole Gardener system.

        shoots/binding Subresource

        The shoots/binding subresource is used to bind a Shoot to a Seed. On creation of a shoot cluster/s, the scheduler updates the binding automatically if an appropriate seed cluster is available. Only an operator with the necessary RBAC can update this binding manually. This can be done by changing the .spec.seedName of the shoot. However, if a different seed is already assigned to the shoot, this will trigger a control-plane migration. For required steps, please see Triggering the Migration.

        spec.schedulerName Field in the Shoot Specification

        Similar to the spec.schedulerName field in Pods, the Shoot specification has an optional .spec.schedulerName field. If this field is set on creation, only the scheduler which relates to the configured name is responsible for scheduling the shoot. The default-scheduler name is reserved for the default scheduler of Gardener. Affected Shoots will remain in Pending state if the mentioned scheduler is not present in the landscape.

        spec.seedName Field in the Shoot Specification

        Similar to the .spec.nodeName field in Pods, the Shoot specification has an optional .spec.seedName field. If this field is set on creation, the shoot will be scheduled to this seed. However, this field can only be set by users having RBAC for the shoots/binding subresource. If this field is not set, the scheduler will assign a suitable seed automatically and populate this field with the seed name.

        seedSelector Field in the Shoot Specification

        Similar to the .spec.nodeSelector field in Pods, the Shoot specification has an optional .spec.seedSelector field. It allows the user to provide a label selector that must match the labels of the Seeds in order to be scheduled to one of them. The labels on the Seeds are usually controlled by Gardener administrators/operators - end users cannot add arbitrary labels themselves. If provided, the Gardener Scheduler will only consider as “suitable” those seeds whose labels match those provided in the .spec.seedSelector of the Shoot.

        By default, only seeds with the same provider as the shoot are selected. By adding a providerTypes field to the seedSelector, a dedicated set of possible providers (* means all provider types) can be selected.

        Ensuring a Seed’s Capacity for Shoots Is Not Exceeded

        Seeds have a practical limit of how many shoots they can accommodate. Exceeding this limit is undesirable, as the system performance will be noticeably impacted. Therefore, the scheduler ensures that a seed’s capacity for shoots is not exceeded by taking into account a maximum number of shoots that can be scheduled onto a seed.

        This mechanism works as follows:

        • The gardenlet is configured with certain resources and their total capacity (and, for certain resources, the amount reserved for Gardener), see /example/20-componentconfig-gardenlet.yaml. Currently, the only such resource is the maximum number of shoots that can be scheduled onto a seed.
        • The gardenlet seed controller updates the capacity and allocatable fields in the Seed status with the capacity of each resource and how much of it is actually available to be consumed by shoots. The allocatable value of a resource is equal to capacity minus reserved.
        • When scheduling shoots, the scheduler filters out all candidate seeds whose allocatable capacity for shoots would be exceeded if the shoot is scheduled onto the seed.

        Failure to Determine a Suitable Seed

        In case the scheduler fails to find a suitable seed, the operation is being retried with exponential backoff. The reason for the failure will be reported in the Shoot’s .status.lastOperation field as well as a Kubernetes event (which can be retrieved via kubectl -n <namespace> describe shoot <shoot-name>).

        Current Limitation / Future Plans

        • Azure unfortunately has a geographically non-hierarchical naming pattern and does not start with the continent. This is the reason why we will exchange the implementation of the MinimalDistance strategy with a more suitable one in the future.

        2.13 - gardenlet

        Understand how the gardenlet, the primary “agent” on every seed cluster, works and learn more about the different Gardener components

        Overview

        Gardener is implemented using the operator pattern: It uses custom controllers that act on our own custom resources, and apply Kubernetes principles to manage clusters instead of containers. Following this analogy, you can recognize components of the Gardener architecture as well-known Kubernetes components, for example, shoot clusters can be compared with pods, and seed clusters can be seen as worker nodes.

        The following Gardener components play a similar role as the corresponding components in the Kubernetes architecture:

        Gardener ComponentKubernetes Component
        gardener-apiserverkube-apiserver
        gardener-controller-managerkube-controller-manager
        gardener-schedulerkube-scheduler
        gardenletkubelet

        Similar to how the kube-scheduler of Kubernetes finds an appropriate node for newly created pods, the gardener-scheduler of Gardener finds an appropriate seed cluster to host the control plane for newly ordered clusters. By providing multiple seed clusters for a region or provider, and distributing the workload, Gardener also reduces the blast radius of potential issues.

        Kubernetes runs a primary “agent” on every node, the kubelet, which is responsible for managing pods and containers on its particular node. Decentralizing the responsibility to the kubelet has the advantage that the overall system is scalable. Gardener achieves the same for cluster management by using a gardenlet as а primary “agent” on every seed cluster, and is only responsible for shoot clusters located in its particular seed cluster:

        Counterparts in the Gardener Architecture and the Kubernetes Architecture

        The gardener-controller-manager has controllers to manage resources of the Gardener API. However, instead of letting the gardener-controller-manager talk directly to seed clusters or shoot clusters, the responsibility isn’t only delegated to the gardenlet, but also managed using a reversed control flow: It’s up to the gardenlet to contact the Gardener API server, for example, to share a status for its managed seed clusters.

        Reversing the control flow allows placing seed clusters or shoot clusters behind firewalls without the necessity of direct access via VPN tunnels anymore.

        Reversed Control Flow Using a gardenlet

        TLS Bootstrapping

        Kubernetes doesn’t manage worker nodes itself, and it’s also not responsible for the lifecycle of the kubelet running on the workers. Similarly, Gardener doesn’t manage seed clusters itself, so it is also not responsible for the lifecycle of the gardenlet running on the seeds. As a consequence, both the gardenlet and the kubelet need to prepare a trusted connection to the Gardener API server and the Kubernetes API server correspondingly.

        To prepare a trusted connection between the gardenlet and the Gardener API server, the gardenlet initializes a bootstrapping process after you deployed it into your seed clusters:

        1. The gardenlet starts up with a bootstrap kubeconfig having a bootstrap token that allows to create CertificateSigningRequest (CSR) resources.

        2. After the CSR is signed, the gardenlet downloads the created client certificate, creates a new kubeconfig with it, and stores it inside a Secret in the seed cluster.

        3. The gardenlet deletes the bootstrap kubeconfig secret, and starts up with its new kubeconfig.

        4. The gardenlet starts normal operation.

        The gardener-controller-manager runs a control loop that automatically signs CSRs created by gardenlets.

        The gardenlet bootstrapping process is based on the kubelet bootstrapping process. More information: Kubelet’s TLS bootstrapping.

        If you don’t want to run this bootstrap process, you can create a kubeconfig pointing to the garden cluster for the gardenlet yourself, and use the field gardenClientConnection.kubeconfig in the gardenlet configuration to share it with the gardenlet.

        gardenlet Certificate Rotation

        The certificate used to authenticate the gardenlet against the API server has a certain validity based on the configuration of the garden cluster (--cluster-signing-duration flag of the kube-controller-manager (default 1y)).

        You can also configure the validity for the client certificate by specifying .gardenClientConnection.kubeconfigValidity.validity in the gardenlet’s component configuration. Note that changing this value will only take effect when the kubeconfig is rotated again (it is not picked up immediately). The minimum validity is 10m (that’s what is enforced by the CertificateSigningRequest API in Kubernetes which is used by the gardenlet).

        By default, after about 70-90% of the validity has expired, the gardenlet tries to automatically replace the current certificate with a new one (certificate rotation).

        You can change these boundaries by specifying .gardenClientConnection.kubeconfigValidity.autoRotationJitterPercentage{Min,Max} in the gardenlet’s component configuration.

        To use a certificate rotation, you need to specify the secret to store the kubeconfig with the rotated certificate in the field .gardenClientConnection.kubeconfigSecret of the gardenlet component configuration.

        Rotate Certificates Using Bootstrap kubeconfig

        If the gardenlet created the certificate during the initial TLS Bootstrapping using the Bootstrap kubeconfig, certificates can be rotated automatically. The same control loop in the gardener-controller-manager that signs the CSRs during the initial TLS Bootstrapping also automatically signs the CSR during a certificate rotation.

        ℹ️ You can trigger an immediate renewal by annotating the Secret in the seed cluster stated in the .gardenClientConnection.kubeconfigSecret field with gardener.cloud/operation=renew. Within 10s, gardenlet detects this and terminates itself to request new credentials. After it has booted up again, gardenlet will issue a new certificate independent of the remaining validity of the existing one.

        ℹ️ Alternatively, annotate the respective Seed with gardener.cloud/operation=renew-kubeconfig. This will make gardenlet annotate its own kubeconfig secret with gardener.cloud/operation=renew and triggers the process described in the previous paragraph.

        Rotate Certificates Using Custom kubeconfig

        When trying to rotate a custom certificate that wasn’t created by gardenlet as part of the TLS Bootstrap, the x509 certificate’s Subject field needs to conform to the following:

        • the Common Name (CN) is prefixed with gardener.cloud:system:seed:
        • the Organization (O) equals gardener.cloud:system:seeds

        Otherwise, the gardener-controller-manager doesn’t automatically sign the CSR. In this case, an external component or user needs to approve the CSR manually, for example, using the command kubectl certificate approve seed-csr-<...>). If that doesn’t happen within 15 minutes, the gardenlet repeats the process and creates another CSR.

        Configuring the Seed to Work with gardenlet

        The gardenlet works with a single seed, which must be configured in the GardenletConfiguration under .seedConfig. This must be a copy of the Seed resource, for example:

        apiVersion: gardenlet.config.gardener.cloud/v1alpha1
        kind: GardenletConfiguration
        seedConfig:
          metadata:
            name: my-seed
          spec:
            provider:
              type: aws
            # ...
            settings:
              scheduling:
                visible: true
        

        (see this yaml file for a more complete example)

        On startup, gardenlet registers a Seed resource using the given template in the seedConfig if it’s not present already.

        Component Configuration

        In the component configuration for the gardenlet, it’s possible to define:

        • settings for the Kubernetes clients interacting with the various clusters
        • settings for the controllers inside the gardenlet
        • settings for leader election and log levels, feature gates, and seed selection or seed configuration.

        More information: Example gardenlet Component Configuration.

        Heartbeats

        Similar to how Kubernetes uses Lease objects for node heart beats (see KEP), the gardenlet is using Lease objects for heart beats of the seed cluster. Every two seconds, the gardenlet checks that the seed cluster’s /healthz endpoint returns HTTP status code 200. If that is the case, the gardenlet renews the lease in the Garden cluster in the gardener-system-seed-lease namespace and updates the GardenletReady condition in the status.conditions field of the Seed resource. For more information, see this section.

        Similar to the node-lifecycle-controller inside the kube-controller-manager, the gardener-controller-manager features a seed-lifecycle-controller that sets the GardenletReady condition to Unknown in case the gardenlet fails to renew the lease. As a consequence, the gardener-scheduler doesn’t consider this seed cluster for newly created shoot clusters anymore.

        /healthz Endpoint

        The gardenlet includes an HTTP server that serves a /healthz endpoint. It’s used as a liveness probe in the Deployment of the gardenlet. If the gardenlet fails to renew its lease, then the endpoint returns 500 Internal Server Error, otherwise it returns 200 OK.

        Please note that the /healthz only indicates whether the gardenlet could successfully probe the Seed’s API server and renew the lease with the Garden cluster. It does not show that the Gardener extension API server (with the Gardener resource groups) is available. However, the gardenlet is designed to withstand such connection outages and retries until the connection is reestablished.

        Controllers

        The gardenlet consists out of several controllers which are now described in more detail.

        BackupBucket Controller

        The BackupBucket controller reconciles those core.gardener.cloud/v1beta1.BackupBucket resources whose .spec.seedName value is equal to the name of the Seed the respective gardenlet is responsible for. A core.gardener.cloud/v1beta1.BackupBucket resource is created by the Seed controller if .spec.backup is defined in the Seed.

        The controller adds finalizers to the BackupBucket and the secret mentioned in the .spec.secretRef of the BackupBucket. The controller also copies this secret to the seed cluster. Additionally, it creates an extensions.gardener.cloud/v1alpha1.BackupBucket resource (non-namespaced) in the seed cluster and waits until the responsible extension controller reconciles it (see Contract: BackupBucket Resource for more details). The status from the reconciliation is reported in the .status.lastOperation field. Once the extension resource is ready and the .status.generatedSecretRef is set by the extension controller, the gardenlet copies the referenced secret to the garden namespace in the garden cluster. An owner reference to the core.gardener.cloud/v1beta1.BackupBucket is added to this secret.

        If the core.gardener.cloud/v1beta1.BackupBucket is deleted, the controller deletes the generated secret in the garden cluster and the extensions.gardener.cloud/v1alpha1.BackupBucket resource in the seed cluster and it waits for the respective extension controller to remove its finalizers from the extensions.gardener.cloud/v1alpha1.BackupBucket. Then it deletes the secret in the seed cluster and finally removes the finalizers from the core.gardener.cloud/v1beta1.BackupBucket and the referred secret.

        BackupEntry Controller

        The BackupEntry controller reconciles those core.gardener.cloud/v1beta1.BackupEntry resources whose .spec.seedName value is equal to the name of a Seed the respective gardenlet is responsible for. Those resources are created by the Shoot controller (only if backup is enabled for the respective Seed) and there is exactly one BackupEntry per Shoot.

        The controller creates an extensions.gardener.cloud/v1alpha1.BackupEntry resource (non-namespaced) in the seed cluster and waits until the responsible extension controller reconciled it (see Contract: BackupEntry Resource for more details). The status is populated in the .status.lastOperation field.

        The core.gardener.cloud/v1beta1.BackupEntry resource has an owner reference pointing to the corresponding Shoot. Hence, if the Shoot is deleted, the BackupEntry resource also gets deleted. In this case, the controller deletes the extensions.gardener.cloud/v1alpha1.BackupEntry resource in the seed cluster and waits until the responsible extension controller has deleted it. Afterwards, the finalizer of the core.gardener.cloud/v1beta1.BackupEntry resource is released so that it finally disappears from the system.

        If the spec.seedName and .status.seedName of the core.gardener.cloud/v1beta1.BackupEntry are different, the controller will migrate it by annotating the extensions.gardener.cloud/v1alpha1.BackupEntry in the Source Seed with gardener.cloud/operation: migrate, waiting for it to be migrated successfully and eventually deleting it from the Source Seed cluster. Afterwards, the controller will recreate the extensions.gardener.cloud/v1alpha1.BackupEntry in the Destination Seed, annotate it with gardener.cloud/operation: restore and wait for the restore operation to finish. For more details about control plane migration, please read Shoot Control Plane Migration.

        Keep Backup for Deleted Shoots

        In some scenarios it might be beneficial to not immediately delete the BackupEntrys (and with them, the etcd backup) for deleted Shoots.

        In this case you can configure the .controllers.backupEntry.deletionGracePeriodHours field in the component configuration of the gardenlet. For example, if you set it to 48, then the BackupEntrys for deleted Shoots will only be deleted 48 hours after the Shoot was deleted.

        Additionally, you can limit the shoot purposes for which this applies by setting .controllers.backupEntry.deletionGracePeriodShootPurposes[]. For example, if you set it to [production] then only the BackupEntrys for Shoots with .spec.purpose=production will be deleted after the configured grace period. All others will be deleted immediately after the Shoot deletion.

        In case a BackupEntry is scheduled for future deletion but you want to delete it immediately, add the annotation backupentry.core.gardener.cloud/force-deletion=true.

        Bastion Controller

        The Bastion controller reconciles those operations.gardener.cloud/v1alpha1.Bastion resources whose .spec.seedName value is equal to the name of a Seed the respective gardenlet is responsible for.

        The controller creates an extensions.gardener.cloud/v1alpha1.Bastion resource in the seed cluster in the shoot namespace with the same name as operations.gardener.cloud/v1alpha1.Bastion. Then it waits until the responsible extension controller has reconciled it (see Contract: Bastion Resource for more details). The status is populated in the .status.conditions and .status.ingress fields.

        During the deletion of operations.gardener.cloud/v1alpha1.Bastion resources, the controller first sets the Ready condition to False and then deletes the extensions.gardener.cloud/v1alpha1.Bastion resource in the seed cluster. Once this resource is gone, the finalizer of the operations.gardener.cloud/v1alpha1.Bastion resource is released, so it finally disappears from the system.

        ControllerInstallation Controller

        The ControllerInstallation controller in the gardenlet reconciles ControllerInstallation objects with the help of the following reconcilers.

        “Main” Reconciler

        This reconciler is responsible for ControllerInstallations referencing a ControllerDeployment whose type=helm.

        For each ControllerInstallation, it creates a namespace on the seed cluster named extension-<controller-installation-name>. Then, it creates a generic garden kubeconfig and garden access secret for the extension for accessing the garden cluster.

        After that, it unpacks the Helm chart tarball in the ControllerDeployments .providerConfig.chart field and deploys the rendered resources to the seed cluster. The Helm chart values in .providerConfig.values will be used and extended with some information about the Gardener environment and the seed cluster:

        gardener:
          version: <gardenlet-version>
          garden:
            clusterIdentity: <identity-of-garden-cluster>
            genericKubeconfigSecretName: <secret-name>
          gardenlet:
            featureGates:
              Foo: true
              Bar: false
              # ...
          seed:
            name: <seed-name>
            clusterIdentity: <identity-of-seed-cluster>
            annotations: <seed-annotations>
            labels: <seed-labels>
            spec: <seed-specification>
        

        As of today, there are a few more fields in .gardener.seed, but it is recommended to use the .gardener.seed.spec if the Helm chart needs more information about the seed configuration.

        The rendered chart will be deployed via a ManagedResource created in the garden namespace of the seed cluster. It is labeled with controllerinstallation-name=<name> so that one can easily find the owning ControllerInstallation for an existing ManagedResource.

        The reconciler maintains the Installed condition of the ControllerInstallation and sets it to False if the rendering or deployment fails.

        “Care” Reconciler

        This reconciler reconciles ControllerInstallation objects and checks whether they are in a healthy state. It checks the .status.conditions of the backing ManagedResource created in the garden namespace of the seed cluster.

        • If the ResourcesApplied condition of the ManagedResource is True, then the Installed condition of the ControllerInstallation will be set to True.
        • If the ResourcesHealthy condition of the ManagedResource is True, then the Healthy condition of the ControllerInstallation will be set to True.
        • If the ResourcesProgressing condition of the ManagedResource is True, then the Progressing condition of the ControllerInstallation will be set to True.

        A ControllerInstallation is considered “healthy” if Applied=Healthy=True and Progressing=False.

        “Required” Reconciler

        This reconciler watches all resources in the extensions.gardener.cloud API group in the seed cluster. It is responsible for maintaining the Required condition on ControllerInstallations. Concretely, when there is at least one extension resource in the seed cluster a ControllerInstallation is responsible for, then the status of the Required condition will be True. If there are no extension resources anymore, its status will be False.

        This condition is taken into account by the ControllerRegistration controller part of gardener-controller-manager when it computes which extensions have to be deployed to which seed cluster. See Gardener Controller Manager for more details.

        VPAEvictionRequirements Controller

        The VPAEvictionRequirements controller in the gardenlet reconciles VerticalPodAutoscaler objects labeled with autoscaling.gardener.cloud/eviction-requirements: managed-by-controller. It manages the EvictionRequirements on a VPA object, which are used to restrict when and how a Pod can be evicted to apply a new resource recommendation. Specifically, the following actions will be taken for the respective label and annotation configuration:

        • If the VPA has the label eviction-requirements.autoscaling.gardener.cloud/downscale-restriction: never, an EvictionRequirement is added to the VPA object that allows evictions for upscaling only
        • If the VPA has the label eviction-requirements.autoscaling.gardener.cloud/downscale-restriction: in-maintenance-window-only, the same EvictionRequirement is added to the VPA object when the Shoot is currently outside of its maintenance window. When the Shoot is inside its maintenance window, the EvictionRequirement is removed. Information about the Shoot maintenance window times are stored in the annotation shoot.gardener.cloud/maintenance-window on the VPA

        ManagedSeed Controller

        The ManagedSeed controller in the gardenlet reconciles ManagedSeed that refers to Shoot scheduled on Seed the gardenlet is responsible for. Additionally, the controller monitors Seeds, which are owned by ManagedSeeds for which the gardenlet is responsible.

        On ManagedSeed reconciliation, the controller first waits for the referenced Shoot to undergo a reconciliation process. Once the Shoot is successfully reconciled, the controller sets the ShootReconciled status of the ManagedSeed to true. Then, it creates garden namespace within the target Shoot cluster. The controller also manages secrets related to Seeds, such as the backup and kubeconfig secrets. It ensures that these secrets are created and updated according to the ManagedSeed spec. Finally, it deploys the gardenlet within the specified Shoot cluster which registers the Seed cluster.

        On ManagedSeed deletion, the controller first deletes the corresponding Seed that was originally created by the controller. Subsequently, it deletes the gardenlet instance within the Shoot cluster. The controller also ensures the deletion of related Seed secrets. Finally, the dedicated garden namespace within the Shoot cluster is deleted.

        NetworkPolicy Controller

        The NetworkPolicy controller reconciles NetworkPolicys in all relevant namespaces in the seed cluster and provides so-called “general” policies for access to the runtime cluster’s API server, DNS, public networks, etc.

        The controller resolves the IP address of the Kubernetes service in the default namespace and creates an egress NetworkPolicys for it.

        For more details about NetworkPolicys in Gardener, please see NetworkPolicys In Garden, Seed, Shoot Clusters.

        Seed Controller

        The Seed controller in the gardenlet reconciles Seed objects with the help of the following reconcilers.

        “Main Reconciler”

        This reconciler is responsible for managing the seed’s system components. Those comprise CA certificates, the various CustomResourceDefinitions, the logging and monitoring stacks, and few central components like gardener-resource-manager, etcd-druid, istio, etc.

        The reconciler also deploys a BackupBucket resource in the garden cluster in case the Seed's .spec.backup is set. It also checks whether the seed cluster’s Kubernetes version is at least the minimum supported version and errors in case this constraint is not met.

        This reconciler maintains the .status.lastOperation field, i.e. it sets it:

        • to state=Progressing before it executes its reconciliation flow.
        • to state=Error in case an error occurs.
        • to state=Succeeded in case the reconciliation succeeded.

        “Care” Reconciler

        This reconciler checks whether the seed system components (deployed by the “main” reconciler) are healthy. It checks the .status.conditions of the backing ManagedResource created in the garden namespace of the seed cluster. A ManagedResource is considered “healthy” if the conditions ResourcesApplied=ResourcesHealthy=True and ResourcesProgressing=False.

        If all ManagedResources are healthy, then the SeedSystemComponentsHealthy condition of the Seed will be set to True. Otherwise, it will be set to False.

        If at least one ManagedResource is unhealthy and there is threshold configuration for the conditions (in .controllers.seedCare.conditionThresholds), then the status of the SeedSystemComponentsHealthy condition will be set:

        • to Progressing if it was True before.
        • to Progressing if it was Progressing before and the lastUpdateTime of the condition does not exceed the configured threshold duration yet.
        • to False if it was Progressing before and the lastUpdateTime of the condition exceeds the configured threshold duration.

        The condition thresholds can be used to prevent reporting issues too early just because there is a rollout or a short disruption. Only if the unhealthiness persists for at least the configured threshold duration, then the issues will be reported (by setting the status to False).

        In order to compute the condition statuses, this reconciler considers ManagedResources (in the garden and istio-system namespace) and their status, see this document for more information. The following table explains which ManagedResources are considered for which condition type:

        Condition TypeManagedResources are considered when
        SeedSystemComponentsHealthy.spec.class is set

        “Lease” Reconciler

        This reconciler checks whether the connection to the seed cluster’s /healthz endpoint works. If this succeeds, then it renews a Lease resource in the garden cluster’s gardener-system-seed-lease namespace. This indicates a heartbeat to the external world, and internally the gardenlet sets its health status to true. In addition, the GardenletReady condition in the status of the Seed is set to True. The whole process is similar to what the kubelet does to report heartbeats for its Node resource and its KubeletReady condition. For more information, see this section.

        If the connection to the /healthz endpoint or the update of the Lease fails, then the internal health status of gardenlet is set to false. Also, this internal health status is set to false automatically after some time, in case the controller gets stuck for whatever reason. This internal health status is available via the gardenlet’s /healthz endpoint and is used for the livenessProbe in the gardenlet pod.

        Shoot Controller

        The Shoot controller in the gardenlet reconciles Shoot objects with the help of the following reconcilers.

        “Main” Reconciler

        This reconciler is responsible for managing all shoot cluster components and implements the core logic for creating, updating, hibernating, deleting, and migrating shoot clusters. It is also responsible for syncing the Cluster cluster to the seed cluster before and after each successful shoot reconciliation.

        The main reconciliation logic is performed in 3 different task flows dedicated to specific operation types:

        • reconcile (operations: create, reconcile, restore): this is the main flow responsible for creation and regular reconciliation of shoots. Hibernating a shoot also triggers this flow. It is also used for restoration of the shoot control plane on the new seed (second half of a Control Plane Migration)
        • migrate: this flow is triggered when spec.seedName specifies a different seed than status.seedName. It performs the first half of the Control Plane Migration, i.e., a backup (migrate operation) of all control plane components followed by a “shallow delete”.
        • delete: this flow is triggered when the shoot’s deletionTimestamp is set, i.e., when it is deleted.

        The gardenlet takes special care to prevent unnecessary shoot reconciliations. This is important for several reasons, e.g., to not overload the seed API servers and to not exhaust infrastructure rate limits too fast. The gardenlet performs shoot reconciliations according to the following rules:

        • If status.observedGeneration is less than metadata.generation: this is the case, e.g., when the spec was changed, a manual reconciliation operation was triggered, or the shoot was deleted.
        • If the last operation was not successful.
        • If the shoot is in a failed state, the gardenlet does not perform any reconciliation on the shoot (unless the retry operation was triggered). However, it syncs the Cluster resource to the seed in order to inform the extension controllers about the failed state.
        • Regular reconciliations are performed with every GardenletConfiguration.controllers.shoot.syncPeriod (defaults to 1h).
        • Shoot reconciliations are not performed if the assigned seed cluster is not healthy or has not been reconciled by the current gardenlet version yet (determined by the Seed.status.gardener section). This is done to make sure that shoots are reconciled with fully rolled out seed system components after a Gardener upgrade. Otherwise, the gardenlet might perform operations of the new version that doesn’t match the old version of the deployed seed system components, which might lead to unspecified behavior.

        There are a few special cases that overwrite or confine how often and under which circumstances periodic shoot reconciliations are performed:

        • In case the gardenlet config allows it (controllers.shoot.respectSyncPeriodOverwrite, disabled by default), the sync period for a shoot can be increased individually by setting the shoot.gardener.cloud/sync-period annotation. This is always allowed for shoots in the garden namespace. Shoots are not reconciled with a higher frequency than specified in GardenletConfiguration.controllers.shoot.syncPeriod.
        • In case the gardenlet config allows it (controllers.shoot.respectSyncPeriodOverwrite, disabled by default), shoots can be marked as “ignored” by setting the shoot.gardener.cloud/ignore annotation. In this case, the gardenlet does not perform any reconciliation for the shoot.
        • In case GardenletConfiguration.controllers.shoot.reconcileInMaintenanceOnly is enabled (disabled by default), the gardenlet performs regular shoot reconciliations only once in the respective maintenance time window (GardenletConfiguration.controllers.shoot.syncPeriod is ignored). The gardenlet randomly distributes shoot reconciliations over the maintenance time window to avoid high bursts of reconciliations (see Shoot Maintenance).
        • In case Shoot.spec.maintenance.confineSpecUpdateRollout is enabled (disabled by default), changes to the shoot specification are not rolled out immediately but only during the respective maintenance time window (see Shoot Maintenance).

        “Care” Reconciler

        This reconciler performs three “care” actions related to Shoots.

        Conditions

        It maintains the following conditions:

        • APIServerAvailable: The /healthz endpoint of the shoot’s kube-apiserver is called and considered healthy when it responds with 200 OK.
        • ControlPlaneHealthy: The control plane is considered healthy when the respective Deployments (for example kube-apiserver,kube-controller-manager), and Etcds (for example etcd-main) exist and are healthy.
        • ObservabilityComponentsHealthy: This condition is considered healthy when the respective Deployments (for example plutono) and StatefulSets (for example prometheus,vali) exist and are healthy.
        • EveryNodeReady: The conditions of the worker nodes are checked (e.g., Ready, MemoryPressure). Also, it’s checked whether the Kubernetes version of the installed kubelet matches the desired version specified in the Shoot resource.
        • SystemComponentsHealthy: The conditions of the ManagedResources are checked (e.g., ResourcesApplied). Also, it is verified whether the VPN tunnel connection is established (which is required for the kube-apiserver to communicate with the worker nodes).

        Sometimes, ManagedResources can have both Healthy and Progressing conditions set to True (e.g., when a DaemonSet rolls out one-by-one on a large cluster with many nodes) while this is not reflected in the Shoot status. In order to catch issues where the rollout gets stuck, one can set .controllers.shootCare.managedResourceProgressingThreshold in the gardenlet’s component configuration. If the Progressing condition is still True for more than the configured duration, the SystemComponentsHealthy condition in the Shoot is set to False, eventually.

        Each condition can optionally also have error codes in order to indicate which type of issue was detected (see Shoot Status for more details).

        Apart from the above, extension controllers can also contribute to the status or error codes of these conditions (see Contributing to Shoot Health Status Conditions for more details).

        If all checks for a certain conditions are succeeded, then its status will be set to True. Otherwise, it will be set to False.

        If at least one check fails and there is threshold configuration for the conditions (in .controllers.seedCare.conditionThresholds), then the status will be set:

        • to Progressing if it was True before.
        • to Progressing if it was Progressing before and the lastUpdateTime of the condition does not exceed the configured threshold duration yet.
        • to False if it was Progressing before and the lastUpdateTime of the condition exceeds the configured threshold duration.

        The condition thresholds can be used to prevent reporting issues too early just because there is a rollout or a short disruption. Only if the unhealthiness persists for at least the configured threshold duration, then the issues will be reported (by setting the status to False).

        Besides directly checking the status of Deployments, Etcds, StatefulSets in the shoot namespace, this reconciler also considers ManagedResources (in the shoot namespace) and their status in order to compute the condition statuses, see this document for more information. The following table explains which ManagedResources are considered for which condition type:

        Condition TypeManagedResources are considered when
        ControlPlaneHealthy.spec.class=seed and care.gardener.cloud/condition-type label either unset, or set to ControlPlaneHealthy
        ObservabilityComponentsHealthycare.gardener.cloud/condition-type label set to ObservabilityComponentsHealthy
        SystemComponentsHealthy.spec.class unset or care.gardener.cloud/condition-type label set to SystemComponentsHealthy
        Constraints And Automatic Webhook Remediation

        Please see Shoot Status for more details.

        Garbage Collection

        Stale pods in the shoot namespace in the seed cluster and in the kube-system namespace in the shoot cluster are deleted. A pod is considered stale when:

        • it was terminated with reason Evicted.
        • it was terminated with reason starting with OutOf (e.g., OutOfCpu).
        • it is stuck in termination (i.e., if its deletionTimestamp is more than 5m ago).

        “State” Reconciler

        This reconciler periodically (default: every 6h) performs backups of the state of Shoot clusters and persists them into ShootState resources into the same namespace as the Shoots in the garden cluster. It is only started in case the gardenlet is responsible for an unmanaged Seed, i.e. a Seed which is not backed by a seedmanagement.gardener.cloud/v1alpha1.ManagedSeed object. Alternatively, it can be disabled by setting the concurrentSyncs=0 for the controller in the gardenlet’s component configuration.

        Please refer to GEP-22: Improved Usage of the ShootState API for all information.

        TokenRequestor Controller

        The gardenlet uses an instance of the TokenRequestor controller which initially was developed in the context of the gardener-resource-manager, please read this document for further information.

        gardenlet uses it for requesting tokens for components running in the seed cluster that need to communicate with the garden cluster. The mechanism works the same way as for shoot control plane components running in the seed which need to communicate with the shoot cluster. However, gardenlet’s instance of the TokenRequestor controller is restricted to Secrets labeled with resources.gardener.cloud/class=garden. Furthermore, it doesn’t respect the serviceaccount.resources.gardener.cloud/namespace annotation. Instead, it always uses the seed’s namespace in the garden cluster for managing ServiceAccounts and their tokens.

        Managed Seeds

        Gardener users can use shoot clusters as seed clusters, so-called “managed seeds” (aka “shooted seeds”), by creating ManagedSeed resources. By default, the gardenlet that manages this shoot cluster then automatically creates a clone of itself with the same version and the same configuration that it currently has. Then it deploys the gardenlet clone into the managed seed cluster.

        For more information, see Register Shoot as Seed.

        Migrating from Previous Gardener Versions

        If your Gardener version doesn’t support gardenlets yet, no special migration is required, but the following prerequisites must be met:

        • Your Gardener version is at least 0.31 before upgrading to v1.
        • You have to make sure that your garden cluster is exposed in a way that it’s reachable from all your seed clusters.

        With previous Gardener versions, you had deployed the Gardener Helm chart (incorporating the API server, controller-manager, and scheduler). With v1, this stays the same, but you now have to deploy the gardenlet Helm chart as well into all of your seeds (if they aren’t managed, as mentioned earlier).

        See Deploy a gardenlet for all instructions.

        3 - Extensions

        Extensibility Overview

        Initially, everything was developed in-tree in the Gardener project. All cloud providers and the configuration for all the supported operating systems were released together with the Gardener core itself. But as the project grew, it got more and more difficult to add new providers and maintain the existing code base. As a consequence and in order to become agile and flexible again, we proposed GEP-1 (Gardener Enhancement Proposal). The document describes an out-of-tree extension architecture that keeps the Gardener core logic independent of provider-specific knowledge (similar to what Kubernetes has achieved with out-of-tree cloud providers or with CSI volume plugins).

        Basic Concepts

        Gardener keeps running in the “garden cluster” and implements the core logic of shoot cluster reconciliation / deletion. Extensions are Kubernetes controllers themselves (like Gardener) and run in the seed clusters. As usual, we try to use Kubernetes wherever applicable. We rely on Kubernetes extension concepts in order to enable extensibility for Gardener. The main ideas of GEP-1 are the following:

        1. During the shoot reconciliation process, Gardener will write CRDs into the seed cluster that are watched and managed by the extension controllers. They will reconcile (based on the .spec) and report whether everything went well or errors occurred in the CRD’s .status field.

        2. Gardener keeps deploying the provider-independent control plane components (etcd, kube-apiserver, etc.). However, some of these components might still need little customization by providers, e.g., additional configuration, flags, etc. In this case, the extension controllers register webhooks in order to manipulate the manifests.

        Example 1:

        Gardener creates a new AWS shoot cluster and requires the preparation of infrastructure in order to proceed (networks, security groups, etc.). It writes the following CRD into the seed cluster:

        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: Infrastructure
        metadata:
          name: infrastructure
          namespace: shoot--core--aws-01
        spec:
          type: aws
          providerConfig:
            apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
            kind: InfrastructureConfig
            networks:
              vpc:
                cidr: 10.250.0.0/16
              internal:
              - 10.250.112.0/22
              public:
              - 10.250.96.0/22
              workers:
              - 10.250.0.0/19
            zones:
            - eu-west-1a
          dns:
            apiserver: api.aws-01.core.example.com
          region: eu-west-1
          secretRef:
            name: my-aws-credentials
          sshPublicKey: |
                base64(key)
        

        Please note that the .spec.providerConfig is a raw blob and not evaluated or known in any way by Gardener. Instead, it was specified by the user (in the Shoot resource) and just “forwarded” to the extension controller. Only the AWS controller understands this configuration and will now start provisioning/reconciling the infrastructure. It reports in the .status field the result:

        status:
          observedGeneration: ...
          state: ...
          lastError: ..
          lastOperation: ...
          providerStatus:
            apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
            kind: InfrastructureStatus
            vpc:
              id: vpc-1234
              subnets:
              - id: subnet-acbd1234
                name: workers
                zone: eu-west-1
              securityGroups:
              - id: sg-xyz12345
                name: workers
            iam:
              nodesRoleARN: <some-arn>
              instanceProfileName: foo
            ec2:
              keyName: bar
        

        Gardener waits until the .status.lastOperation / .status.lastError indicates that the operation reached a final state and either continuous with the next step, or stops and reports the potential error. The extension-specific output in .status.providerStatus is - similar to .spec.providerConfig - not evaluated, and simply forwarded to CRDs in subsequent steps.

        Example 2:

        Gardener deploys the control plane components into the seed cluster, e.g. the kube-controller-manager deployment with the following flags:

        apiVersion: apps/v1
        kind: Deployment
        ...
        spec:
          template:
            spec:
              containers:
              - command:
                - /usr/local/bin/kube-controller-manager
                - --allocate-node-cidrs=true
                - --attach-detach-reconcile-sync-period=1m0s
                - --controllers=*,bootstrapsigner,tokencleaner
                - --cluster-cidr=100.96.0.0/11
                - --cluster-name=shoot--core--aws-01
                - --cluster-signing-cert-file=/srv/kubernetes/ca/ca.crt
                - --cluster-signing-key-file=/srv/kubernetes/ca/ca.key
                - --concurrent-deployment-syncs=10
                - --concurrent-replicaset-syncs=10
        ...
        

        The AWS controller requires some additional flags in order to make the cluster functional. It needs to provide a Kubernetes cloud-config and also some cloud-specific flags. Consequently, it registers a MutatingWebhookConfiguration on Deployments and adds these flags to the container:

                - --cloud-provider=external
                - --external-cloud-volume-plugin=aws
                - --cloud-config=/etc/kubernetes/cloudprovider/cloudprovider.conf
        

        Of course, it would have needed to create a ConfigMap containing the cloud config and to add the proper volume and volumeMounts to the manifest as well.

        (Please note for this special example: The Kubernetes community is also working on making the kube-controller-manager provider-independent. However, there will most probably be still components other than the kube-controller-manager which need to be adapted by extensions.)

        If you are interested in writing an extension, or generally in digging deeper to find out the nitty-gritty details of the extension concepts, please read GEP-1. We are truly looking forward to your feedback!

        Current Status

        Meanwhile, the out-of-tree extension architecture of Gardener is in place and has been productively validated. We are tracking all internal and external extensions of Gardener in the Gardener Extensions Library repo.

        3.1 - Access to the Garden Cluster for Extensions

        Access to the Garden Cluster for Extensions

        Extensions that are installed on seed clusters via a ControllerInstallation can simply read the kubeconfig file specified by the GARDEN_KUBECONFIG environment variable to create a garden cluster client. With this, they use a short-lived token (valid for 12h) associated with a dedicated ServiceAccount in the seed-<seed-name> namespace to securely access the garden cluster. The used ServiceAccounts are granted permissions in the garden cluster similar to gardenlet clients.

        Background

        Historically, gardenlet has been the only component running in the seed cluster that has access to both the seed cluster and the garden cluster. Accordingly, extensions running on the seed cluster didn’t have access to the garden cluster.

        Starting from Gardener v1.74.0, there is a new mechanism for components running on seed clusters to get access to the garden cluster. For this, gardenlet runs an instance of the TokenRequestor for requesting tokens that can be used to communicate with the garden cluster.

        Using Gardenlet-Managed Garden Access

        By default, extensions are equipped with secure access to the garden cluster using a dedicated ServiceAccount without requiring any additional action. They can simply read the file specified by the GARDEN_KUBECONFIG and construct a garden client with it.

        When installing a ControllerInstallation, gardenlet creates two secrets in the installation’s namespace: a generic garden kubeconfig (generic-garden-kubeconfig-<hash>) and a garden access secret (garden-access-extension). Note that the ServiceAccount created based on this access secret will be created in the respective seed-* namespace in the garden cluster and labelled with controllerregistration.core.gardener.cloud/name=<name>.

        Additionally, gardenlet injects volume, volumeMounts, and two environment variables into all (init) containers in all objects in the apps and batch API groups:

        • GARDEN_KUBECONFIG: points to the path where the generic garden kubeconfig is mounted.
        • SEED_NAME: set to the name of the Seed where the extension is installed. This is useful for restricting watches in the garden cluster to relevant objects.

        If an object already contains the GARDEN_KUBECONFIG environment variable, it is not overwritten and injection of volume and volumeMounts is skipped.

        For example, a Deployment deployed via a ControllerInstallation will be mutated as follows:

        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: gardener-extension-provider-local
          annotations:
            reference.resources.gardener.cloud/secret-795f7ca6: garden-access-extension
            reference.resources.gardener.cloud/secret-d5f5a834: generic-garden-kubeconfig-81fb3a88
        spec:
          template:
            metadata:
              annotations:
                reference.resources.gardener.cloud/secret-795f7ca6: garden-access-extension
                reference.resources.gardener.cloud/secret-d5f5a834: generic-garden-kubeconfig-81fb3a88
            spec:
              containers:
              - name: gardener-extension-provider-local
                env:
                - name: GARDEN_KUBECONFIG
                  value: /var/run/secrets/gardener.cloud/garden/generic-kubeconfig/kubeconfig
                - name: SEED_NAME
                  value: local
                volumeMounts:
                - mountPath: /var/run/secrets/gardener.cloud/garden/generic-kubeconfig
                  name: garden-kubeconfig
                  readOnly: true
              volumes:
              - name: garden-kubeconfig
                projected:
                  defaultMode: 420
                  sources:
                  - secret:
                      items:
                      - key: kubeconfig
                        path: kubeconfig
                      name: generic-garden-kubeconfig-81fb3a88
                      optional: false
                  - secret:
                      items:
                      - key: token
                        path: token
                      name: garden-access-extension
                      optional: false
        

        The generic garden kubeconfig will look like this:

        apiVersion: v1
        kind: Config
        clusters:
        - cluster:
            certificate-authority-data: LS0t...
            server: https://garden.local.gardener.cloud:6443
          name: garden
        contexts:
        - context:
            cluster: garden
            user: extension
          name: garden
        current-context: garden
        users:
        - name: extension
          user:
            tokenFile: /var/run/secrets/gardener.cloud/garden/generic-kubeconfig/token
        

        Manually Requesting a Token for the Garden Cluster

        Seed components that need to communicate with the garden cluster can request a token in the garden cluster by creating a garden access secret. This secret has to be labelled with resources.gardener.cloud/purpose=token-requestor and resources.gardener.cloud/class=garden, e.g.:

        apiVersion: v1
        kind: Secret
        metadata:
          name: garden-access-example
          namespace: example
          labels:
            resources.gardener.cloud/purpose: token-requestor
            resources.gardener.cloud/class: garden
          annotations:
            serviceaccount.resources.gardener.cloud/name: example
        type: Opaque
        

        This will instruct gardenlet to create a new ServiceAccount named example in its own seed-<seed-name> namespace in the garden cluster, request a token for it, and populate the token in the secret’s data under the token key.

        Permissions in the Garden Cluster

        Both the SeedAuthorizer and the SeedRestriction plugin handle extensions clients and generally grant the same permissions in the garden cluster to them as to gardenlet clients. With this, extensions are restricted to work with objects in the garden cluster that are related to seed they are running one just like gardenlet. Note that if the plugins are not enabled, extension clients are only granted read access to global resources like CloudProfiles (this is granted to all authenticated users). There are a few exceptions to the granted permissions as documented here.

        Additional Permissions

        If an extension needs access to additional resources in the garden cluster (e.g., extension-specific custom resources), permissions need to be granted via the usual RBAC means. Let’s consider the following example: An extension requires the privileges to create authorization.k8s.io/v1.SubjectAccessReviews (which is not covered by the “default” permissions mentioned above). This requires a human Gardener operator to create a ClusterRole in the garden cluster with the needed rules:

        apiVersion: rbac.authorization.k8s.io/v1
        kind: ClusterRole
        metadata:
          name: extension-create-subjectaccessreviews
          annotations:
            authorization.gardener.cloud/extensions-serviceaccount-selector: '{"matchLabels":{"controllerregistration.core.gardener.cloud/name":"<extension-name>"}}'
          labels:
            authorization.gardener.cloud/custom-extensions-permissions: "true"
        rules:
        - apiGroups:
          - authorization.k8s.io
          resources:
          - subjectaccessreviews
          verbs:
          - create
        

        Note the label authorization.gardener.cloud/extensions-serviceaccount-selector which contains a label selector for ServiceAccounts.

        There is a controller part of gardener-controller-manager which takes care of maintaining the respective ClusterRoleBinding resources. It binds all ServiceAccounts in the seed namespaces in the garden cluster (i.e., all extension clients) whose labels match. You can read more about this controller here.

        Custom Permissions

        If an extension wants to create a dedicated ServiceAccount for accessing the garden cluster without automatically inheriting all permissions of the gardenlet, it first needs to create a garden access secret in its extension namespace in the seed cluster:

        apiVersion: v1
        kind: Secret
        metadata:
          name: my-custom-component
          namespace: <extension-namespace>
          labels:
            resources.gardener.cloud/purpose: token-requestor
            resources.gardener.cloud/class: garden
          annotations:
            serviceaccount.resources.gardener.cloud/name: my-custom-component-extension-foo
            serviceaccount.resources.gardener.cloud/labels: '{"foo":"bar}'
        type: Opaque
        

        ❗️️Do not prefix the service account name with extension- to prevent inheriting the gardenlet permissions! It is still recommended to add the extension name (e.g., as a suffix) for easier identification where this ServiceAccount comes from.

        Next, you can follow the same approach described above. However, the authorization.gardener.cloud/extensions-serviceaccount-selector annotation should not contain controllerregistration.core.gardener.cloud/name=<extension-name> but rather custom labels, e.g. foo=bar.

        This way, the created ServiceAccount will only get the permissions of above ClusterRole and nothing else.

        Renewing All Garden Access Secrets

        Operators can trigger an automatic renewal of all garden access secrets in a given Seed and their requested ServiceAccount tokens, e.g., when rotating the garden cluster’s ServiceAccount signing key. For this, the Seed has to be annotated with gardener.cloud/operation=renew-garden-access-secrets.

        3.2 - Admission

        Extension Admission

        The extensions are expected to validate their respective resources for their extension specific configurations, when the resources are newly created or updated. For example, provider extensions would validate spec.provider.infrastructureConfig and spec.provider.controlPlaneConfig in the Shoot resource and spec.providerConfig in the CloudProfile resource, networking extensions would validate spec.networking.providerConfig in the Shoot resource. As best practice, the validation should be performed only if there is a change in the spec of the resource. Please find an exemplary implementation in the gardener/gardener-extension-provider-aws repository.

        When a resource is newly created or updated, Gardener adds an extension label for all the extension types referenced in the spec of the resource. This label is of the form <extension-type>.extensions.gardener.cloud/<extension-name> : "true". For example, an extension label for a provider extension type aws looks like provider.extensions.gardener.cloud/aws : "true". The extensions should add object selectors in their admission webhooks for these labels, to filter out the objects they are responsible for. At present, these labels are added to BackupEntrys, BackupBuckets, CloudProfiles, Seeds, SecretBindings and Shoots. Please see the types_constants.go file for the full list of extension labels.

        3.3 - BackupBucket

        Contract: BackupBucket Resource

        The Gardener project features a sub-project called etcd-backup-restore to take periodic backups of etcd backing Shoot clusters. It demands the bucket (or its equivalent in different object store providers) to be created and configured externally with appropriate credentials. The BackupBucket resource takes this responsibility in Gardener.

        Before introducing the BackupBucket extension resource, Gardener was using Terraform in order to create and manage these provider-specific resources (e.g., see AWS Backup). Now, Gardener commissions an external, provider-specific controller to take over this task. You can also refer to backupInfra proposal documentation to get an idea about how the transition was done and understand the resource in a broader scope.

        What Is the Scope of a Bucket?

        A bucket will be provisioned per Seed. So, a backup of every Shoot created on that Seed will be stored under a different shoot specific prefix under the bucket. For the backup of the Shoot rescheduled on different Seed, it will continue to use the same bucket.

        What Is the Lifespan of a BackupBucket?

        The bucket associated with BackupBucket will be created at the creation of the Seed. And as per current implementation, it will also be deleted on deletion of the Seed, if there isn’t any BackupEntry resource associated with it.

        In the future, we plan to introduce a schedule for BackupBucket - the deletion logic for the BackupBucket resource, which will reschedule it on different available Seeds on deletion or failure of a health check for the currently associated seed. In that case, the BackupBucket will be deleted only if there isn’t any schedulable Seed available and there isn’t any associated BackupEntry resource.

        What Needs to Be Implemented to Support a New Infrastructure Provider?

        As part of the seed flow, Gardener will create a special CRD in the seed cluster that needs to be reconciled by an extension controller, for example:

        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: BackupBucket
        metadata:
          name: foo
        spec:
          type: azure
          providerConfig:
            <some-optional-provider-specific-backupbucket-configuration>
          region: eu-west-1
          secretRef:
            name: backupprovider
            namespace: shoot--foo--bar
        

        The .spec.secretRef contains a reference to the provider secret pointing to the account that shall be used to create the needed resources. This provider secret will be configured by the Gardener operator in the Seed resource and propagated over there by the seed controller.

        After your controller has created the required bucket, if required, it generates the secret to access the objects in the bucket and put a reference to it in status. This secret is supposed to be used by Gardener, or eventually a BackupEntry resource and etcd-backup-restore component, to backup the etcd.

        In order to support a new infrastructure provider, you need to write a controller that watches all BackupBuckets with .spec.type=<my-provider-name>. You can take a look at the below referenced example implementation for the Azure provider.

        References and Additional Resources

        3.4 - BackupEntry

        Contract: BackupEntry Resource

        The Gardener project features a sub-project called etcd-backup-restore to take periodic backups of etcd backing Shoot clusters. It demands the bucket (or its equivalent in different object store providers) access credentials to be created and configured externally with appropriate credentials. The BackupEntry resource takes this responsibility in Gardener to provide this information by creating a secret specific to the component.

        That being said, the core motivation for introducing this resource was to support retention of backups post deletion of Shoot. The etcd-backup-restore components take responsibility of garbage collecting old backups out of the defined period. Once a shoot is deleted, we need to persist the backups for few days. Hence, Gardener uses the BackupEntry resource for this housekeeping work post deletion of a Shoot. The BackupEntry resource is responsible for shoot specific prefix under referred bucket.

        Before introducing the BackupEntry extension resource, Gardener was using Terraform in order to create and manage these provider-specific resources (e.g., see AWS Backup). Now, Gardener commissions an external, provider-specific controller to take over this task. You can also refer to backupInfra proposal documentation to get idea about how the transition was done and understand the resource in broader scope.

        What Is the Lifespan of a BackupEntry?

        The bucket associated with BackupEntry will be created by using a BackupBucket resource. The BackupEntry resource will be created as a part of the Shoot creation. But resources might continue to exist post deletion of a Shoot (see gardenlet for more details).

        What Needs to be Implemented to Support a New Infrastructure Provider?

        As part of the shoot flow, Gardener will create a special CRD in the seed cluster that needs to be reconciled by an extension controller, for example:

        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: BackupEntry
        metadata:
          name: shoot--foo--bar
        spec:
          type: azure
          providerConfig:
            <some-optional-provider-specific-backup-bucket-configuration>
          backupBucketProviderStatus:
            <some-optional-provider-specific-backup-bucket-status>
          region: eu-west-1
          bucketName: foo
          secretRef:
            name: backupprovider
            namespace: shoot--foo--bar
        

        The .spec.secretRef contains a reference to the provider secret pointing to the account that shall be used to create the needed resources. This provider secret will be propagated from the BackupBucket resource by the shoot controller.

        Your controller is supposed to create the etcd-backup secret in the control plane namespace of a shoot. This secret is supposed to be used by Gardener or eventually by the etcd-backup-restore component to backup the etcd. The controller implementation should clean up the objects created under the shoot specific prefix in the bucket equivalent to the name of the BackupEntry resource.

        In order to support a new infrastructure provider, you need to write a controller that watches all the BackupBuckets with .spec.type=<my-provider-name>. You can take a look at the below referenced example implementation for the Azure provider.

        References and Additional Resources

        3.5 - Bastion

        Contract: Bastion Resource

        The Gardener project allows users to connect to Shoot worker nodes via SSH. As nodes are usually firewalled and not directly accessible from the public internet, GEP-15 introduced the concept of “Bastions”. A bastion is a dedicated server that only serves to allow SSH ingress to the worker nodes.

        Bastion resources contain the user’s public SSH key and IP address, in order to provision the server accordingly: The public key is put onto the Bastion and SSH ingress is only authorized for the given IP address (in fact, it’s not a single IP address, but a set of IP ranges, however for most purposes a single IP is be used).

        What Is the Lifespan of a Bastion?

        Once a Bastion has been created in the garden, it will be replicated to the appropriate seed cluster, where a controller then reconciles a server and firewall rules etc., on the cloud provider used by the target Shoot. When the Bastion is ready (i.e. has a public IP), that IP is stored in the Bastion’s status and from there it is picked up by the garden cluster and gardenctl eventually.

        To make multiple SSH sessions possible, the existence of the Bastion is not directly tied to the execution of gardenctl: users can exit out of gardenctl and use ssh manually to connect to the bastion and worker nodes.

        However, Bastions have an expiry date, after which they will be garbage collected.

        When SSH access is set to false for the Shoot in the workers settings (see Shoot Worker Nodes Settings), Bastion resources are deleted during Shoot reconciliation and new Bastions are prevented from being created.

        What Needs to Be Implemented to Support a New Infrastructure Provider?

        As part of the shoot flow, Gardener will create a special CRD in the seed cluster that needs to be reconciled by an extension controller, for example:

        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: Bastion
        metadata:
          name: mybastion
          namespace: shoot--foo--bar
        spec:
          type: aws
          # userData is base64-encoded cloud provider user data; this contains the
          # user's SSH key
          userData: IyEvYmluL2Jhc2ggL....Nlcgo=
          ingress:
            - ipBlock:
                cidr: 192.88.99.0/32 # this is most likely the user's IP address
        

        Your controller is supposed to create a new instance at the given cloud provider, firewall it to only allow SSH (TCP port 22) from the given IP blocks, and then configure the firewall for the worker nodes to allow SSH from the bastion instance. When a Bastion is deleted, all these changes need to be reverted.

        Implementation Details

        ConfigValidator Interface

        For bastion controllers, the generic Reconciler also delegates to a ConfigValidator interface that contains a single Validate method. This method is called by the generic Reconciler at the beginning of every reconciliation, and can be implemented by the extension to validate the .spec.providerConfig part of the Bastion resource with the respective cloud provider, typically the existence and validity of cloud provider resources such as VPCs, images, etc.

        The Validate method returns a list of errors. If this list is non-empty, the generic Reconciler will fail with an error. This error will have the error code ERR_CONFIGURATION_PROBLEM, unless there is at least one error in the list that has its ErrorType field set to field.ErrorTypeInternal.

        References and Additional Resources

        3.6 - CA Rotation

        CA Rotation in Extensions

        GEP-18 proposes adding support for automated rotation of Shoot cluster certificate authorities (CAs). This document outlines all the requirements that Gardener extensions need to fulfill in order to support the CA rotation feature.

        Requirements for Shoot Cluster CA Rotation

        • Extensions must not rely on static CA Secret names managed by the gardenlet, because their names are changing during CA rotation.
        • Extensions cannot issue or use client certificates for authenticating against shoot API servers. Instead, they should use short-lived auto-rotated ServiceAccount tokens via gardener-resource-manager’s TokenRequestor. Also see Conventions and TokenRequestor documents.
        • Extensions need to generate dedicated CAs for signing server certificates (e.g. cloud-controller-manager). There should be one CA per controller and purpose in order to bind the lifecycle to the reconciliation cycle of the respective object for which it is created.
        • CAs managed by extensions should be rotated in lock-step with the shoot cluster CA. When the user triggers a rotation, the gardenlet writes phase and initiation time to Shoot.status.credentials.rotation.certificateAuthorities.{phase,lastInitiationTime}. See GEP-18 for a detailed description on what needs to happen in each phase. Extensions can retrieve this information from Cluster.shoot.status.

        Utilities for Secrets Management

        In order to fulfill the requirements listed above, extension controllers can reuse the SecretsManager that the gardenlet uses to manage all shoot cluster CAs, certificates, and other secrets as well. It implements the core logic for managing secrets that need to be rotated, auto-renewed, etc.

        Additionally, there are utilities for reusing SecretsManager in extension controllers. They already implement the above requirements based on the Cluster resource and allow focusing on the extension controllers’ business logic.

        For example, a simple SecretsManager usage in an extension controller could look like this:

        const (
          // identity for SecretsManager instance in ControlPlane controller
          identity = "provider-foo-controlplane"
          // secret config name of the dedicated CA
          caControlPlaneName = "ca-provider-foo-controlplane"
        )
        
        func Reconcile() {
          var (
            cluster *extensionscontroller.Cluster
            client  client.Client
        
            // define wanted secrets with options
            secretConfigs = []extensionssecretsmanager.SecretConfigWithOptions{
              {
                // dedicated CA for ControlPlane controller
                Config: &secretutils.CertificateSecretConfig{
                  Name:       caControlPlaneName,
                  CommonName: "ca-provider-foo-controlplane",
                  CertType:   secretutils.CACert,
                },
                // persist CA so that it gets restored on control plane migration
                Options: []secretsmanager.GenerateOption{secretsmanager.Persist()},
              },
              {
                // server cert for control plane component
                Config: &secretutils.CertificateSecretConfig{
                  Name:       "cloud-controller-manager",
                  CommonName: "cloud-controller-manager",
                  DNSNames:   kutil.DNSNamesForService("cloud-controller-manager", namespace),
                  CertType:   secretutils.ServerCert,
                },
                // sign with our dedicated CA
                Options: []secretsmanager.GenerateOption{secretsmanager.SignedByCA(caControlPlaneName)},
              },
            }
          )
        
          // initialize SecretsManager based on Cluster object
          sm, err := extensionssecretsmanager.SecretsManagerForCluster(ctx, logger.WithName("secretsmanager"), clock.RealClock{}, client, cluster, identity, secretConfigs)
          
          // generate all wanted secrets (first CAs, then the rest)
          secrets, err := extensionssecretsmanager.GenerateAllSecrets(ctx, sm, secretConfigs)
        
          // cleanup any secrets that are not needed any more (e.g. after rotation)
          err = sm.Cleanup(ctx)
        }
        

        Please pay attention to the following points:

        • There should be one SecretsManager identity per controller (and purpose if applicable) in order to prevent conflicts between different instances. E.g., there should be different identities for Infrastructrue, Worker controller, etc., and the ControlPlane controller should use dedicated SecretsManager identities per purpose (e.g. provider-foo-controlplane and provider-foo-controlplane-exposure).
        • All other points in Reusing the SecretsManager in Other Components.

        3.7 - Cluster

        Cluster Resource

        As part of the extensibility epic, a lot of responsibility that was previously taken over by Gardener directly has now been shifted to extension controllers running in the seed clusters. These extensions often serve a well-defined purpose, e.g. the management of DNS records, infrastructure, etc. We have introduced a couple of extension CRDs in the seeds whose specification is written by Gardener, and which are acted up by the extensions.

        However, the extensions sometimes require more information that is not directly part of the specification. One example of that is the GCP infrastructure controller which needs to know the shoot’s pod and service network. Another example is the Azure infrastructure controller which requires some information out of the CloudProfile resource. The problem is that Gardener does not know which extension requires which information so that it can write it into their specific CRDs.

        In order to deal with this problem we have introduced the Cluster extension resource. This CRD is written into the seeds, however, it does not contain a status, so it is not expected that something acts upon it. Instead, you can treat it like a ConfigMap which contains data that might be interesting for you. In the context of Gardener, seeds and shoots, and extensibility the Cluster resource contains the CloudProfile, Seed, and Shoot manifest. Extension controllers can take whatever information they want out of it that might help completing their individual tasks.

        ---
        
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: Cluster
        metadata:
          name: shoot--foo--bar
        spec:
          cloudProfile:
            apiVersion: core.gardener.cloud/v1beta1
            kind: CloudProfile
            ...
          seed:
            apiVersion: core.gardener.cloud/v1beta1
            kind: Seed
            ...
          shoot:
            apiVersion: core.gardener.cloud/v1beta1
            kind: Shoot
            ...
        

        The resource is written by Gardener before it starts the reconciliation flow of the shoot.

        ⚠️ All Gardener components use the core.gardener.cloud/v1beta1 version, i.e., the Cluster resource will contain the objects in this version.

        Important Information that Should Be Taken into Account

        There are some fields in the Shoot specification that might be interesting to take into account.

        • .spec.hibernation.enabled={true,false}: Extension controllers might want to behave differently if the shoot is hibernated or not (probably they might want to scale down their control plane components, for example).
        • .status.lastOperation.state=Failed: If Gardener sets the shoot’s last operation state to Failed, it means that Gardener won’t automatically retry to finish the reconciliation/deletion flow because an error occurred that could not be resolved within the last 24h (default). In this case, end-users are expected to manually re-trigger the reconciliation flow in case they want Gardener to try again. Extension controllers are expected to follow the same principle. This means they have to read the shoot state out of the Cluster resource.

        Extension Resources Not Associated with a Shoot

        In some cases, Gardener may create extension resources that are not associated with a shoot, but are needed to support some functionality internal to Gardener. Such resources will be created in the garden namespace of a seed cluster.

        For example, if the managed ingress controller is active on the seed, Gardener will create a DNSRecord resource(s) in the garden namespace of the seed cluster for the ingress DNS record.

        Extension controllers that may be expected to reconcile extension resources in the garden namespace should make sure that they can tolerate the absence of a cluster resource. This means that they should not attempt to read the cluster resource in such cases, or if they do they should ignore the “not found” error.

        References and Additional Resources

        3.8 - ContainerRuntime

        Gardener Container Runtime Extension

        At the lowest layers of a Kubernetes node is the software that, among other things, starts and stops containers. It is called “Container Runtime”. The most widely known container runtime is Docker, but it is not alone in this space. In fact, the container runtime space has been rapidly evolving.

        Kubernetes supports different container runtimes using Container Runtime Interface (CRI) – a plugin interface which enables kubelet to use a wide variety of container runtimes.

        Gardener supports creation of Worker machines using CRI. For more information, see CRI Support.

        Motivation

        Prior to the Container Runtime Extensibility concept, Gardener used Docker as the only container runtime to use in shoot worker machines. Because of the wide variety of different container runtimes offering multiple important features (for example, enhanced security concepts), it is important to enable end users to use other container runtimes as well.

        The ContainerRuntime Extension Resource

        Here is what a typical ContainerRuntime resource would look like:

        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: ContainerRuntime
        metadata:
          name: my-container-runtime
        spec:
          binaryPath: /var/bin/containerruntimes
          type: gvisor
          workerPool:
            name: worker-ubuntu
            selector:
              matchLabels:
                worker.gardener.cloud/pool: worker-ubuntu
        

        Gardener deploys one ContainerRuntime resource per worker pool per CRI. To exemplify this, consider a Shoot having two worker pools (worker-one, worker-two) using containerd as the CRI as well as gvisor and kata as enabled container runtimes. Gardener would deploy four ContainerRuntime resources. For worker-one: one ContainerRuntime for type gvisor and one for type kata. The same resource are being deployed for worker-two.

        Supporting a New Container Runtime Provider

        To add support for another container runtime (e.g., gvisor, kata-containers), a container runtime extension controller needs to be implemented. It should support Gardener’s supported CRI plugins.

        The container runtime extension should install the necessary resources into the shoot cluster (e.g., RuntimeClasses), and it should copy the runtime binaries to the relevant worker machines in path: spec.binaryPath. Gardener labels the shoot nodes according to the CRI configured: worker.gardener.cloud/cri-name=<value> (e.g worker.gardener.cloud/cri-name=containerd) and multiple labels for each of the container runtimes configured for the shoot Worker machine: containerruntime.worker.gardener.cloud/<container-runtime-type-value>=true (e.g containerruntime.worker.gardener.cloud/gvisor=true). The way to install the binaries is by creating a daemon set which copies the binaries from an image in a docker registry to the relevant labeled Worker’s nodes (avoid downloading binaries from the internet to also cater with isolated environments).

        For additional reference, please have a look at the runtime-gvsior provider extension, which provides more information on how to configure the necessary charts, as well as the actuators required to reconcile container runtime inside the Shoot cluster to the desired state.

        3.9 - ControllerRegistration

        Registering Extension Controllers

        Extensions are registered in the garden cluster via ControllerRegistration resources. Deployment for respective extensions are specified via ControllerDeployment resources. Gardener evaluates the registrations and deployments and creates ControllerInstallation resources which describe the request “please install this controller X to this seed Y”.

        Similar to how CloudProfile or Seed resources get into the system, the Gardener administrator must deploy the ControllerRegistration and ControllerDeployment resources (this does not happen automatically in any way - the administrator decides which extensions shall be enabled).

        The specification mainly describes which of Gardener’s extension CRDs are managed, for example:

        apiVersion: core.gardener.cloud/v1
        kind: ControllerDeployment
        metadata:
          name: os-gardenlinux
        helm:
          rawChart: H4sIFAAAAAAA/yk... # base64-encoded, gzip'ed, tar'ed extension controller chart
          values:
            foo: bar
        ---
        apiVersion: core.gardener.cloud/v1beta1
        kind: ControllerRegistration
        metadata:
          name: os-gardenlinux
        spec:
          deployment:
            deploymentRefs:
            - name: os-gardenlinux
          resources:
          - kind: OperatingSystemConfig
            type: gardenlinux
            primary: true
        

        This information tells Gardener that there is an extension controller that can handle OperatingSystemConfig resources of type gardenlinux. A reference to the shown ControllerDeployment specifies how the deployment of the extension controller is accomplished.

        Also, it specifies that this controller is the primary one responsible for the lifecycle of the OperatingSystemConfig resource. Setting primary to false would allow to register additional, secondary controllers that may also watch/react on the OperatingSystemConfig/coreos resources, however, only the primary controller may change/update the main status of the extension object (that are used to “communicate” with the gardenlet). Particularly, only the primary controller may set .status.lastOperation, .status.lastError, .status.observedGeneration, and .status.state. Secondary controllers may contribute to the .status.conditions[] if they like, of course.

        Secondary controllers might be helpful in scenarios where additional tasks need to be completed which are not part of the reconciliation logic of the primary controller but separated out into a dedicated extension.

        ⚠️ There must be exactly one primary controller for every registered kind/type combination. Also, please note that the primary field cannot be changed after creation of the ControllerRegistration.

        Deploying Extension Controllers

        Submitting the above ControllerDeployment and ControllerRegistration will create a ControllerInstallation resource:

        apiVersion: core.gardener.cloud/v1beta1
        kind: ControllerInstallation
        metadata:
          name: os-gardenlinux
        spec:
          deploymentRef:
            name: os-gardenlinux
          registrationRef:
            name: os-gardenlinux
          seedRef:
            name: aws-eu1
        

        This resource expresses that Gardener requires the os-gardenlinux extension controller to run on the aws-eu1 seed cluster.

        gardener-controller-manager automatically determines which extension is required on which seed cluster and will only create ControllerInstallation objects for those. Also, it will automatically delete ControllerInstallations referencing extension controllers that are no longer required on a seed (e.g., because all shoots on it have been deleted). There are additional configuration options, please see the Deployment Configuration Options section. After gardener-controller-manager has written the ControllerInstallation resource, gardenlet picks it up and installs the controller on the respective Seed using the referenced ControllerDeployment.

        It is sufficient to create a Helm chart and deploy it together with some static configuration values. For this, operators have to provide the deployment information in the ControllerDeployment.helm section:

        ...
        helm:
          rawChart: H4sIFAAAAAAA/yk...
          values:
            foo: bar
        

        You can check out hack/generate-controller-registration.yaml for generating a ControllerDeployment including a controller helm chart.

        If ControllerDeployment.helm is specified, gardenlet decodes the provided Helm chart (.helm.rawChart) and deploys it with the provided static configuration (.helm.values). The chart and the values can be updated at any time - Gardener will recognize it and re-trigger the deployment process.

        In order to allow extensions to get information about the garden and the seed cluster, gardenlet mixes in certain properties into the values (root level) of every deployed Helm chart:

        gardener:
          version: <gardener-version>
          garden:
            clusterIdentity: <uuid-of-gardener-installation>
            genericKubeconfigSecretName: <generic-garden-kubeconfig-secret-name>
          seed:
            name:             <seed-name>
            clusterIdentity:  <seed-cluster-identity>
            annotations:      <seed-annotations>
            labels:           <seed-labels>
            provider:         <seed-provider-type>
            region:           <seed-region>
            volumeProvider:   <seed-first-volume-provider>
            volumeProviders:  <seed-volume-providers>
            ingressDomain:    <seed-ingress-domain>
            protected:        <seed-protected-taint>
            visible:          <seed-visible-setting>
            taints:           <seed-taints>
            networks:         <seed-networks>
            blockCIDRs:       <seed-networks-blockCIDRs>
            spec:             <seed-spec>
          gardenlet:
            featureGates: <gardenlet-feature-gates>
        

        Extensions can use this information in their Helm chart in case they require knowledge about the garden and the seed environment. The list might be extended in the future.

        gardenlet reports whether the extension controller has been installed successfully and running in the ControllerInstallation status:

        status:
          conditions:
          - lastTransitionTime: "2024-05-16T13:04:16Z"
            lastUpdateTime: "2024-05-16T13:04:16Z"
            message: The controller running in the seed cluster is healthy.
            reason: ControllerHealthy
            status: "True"
            type: Healthy
          - lastTransitionTime: "2024-05-16T13:04:06Z"
            lastUpdateTime: "2024-05-16T13:04:06Z"
            message: The controller was successfully installed in the seed cluster.
            reason: InstallationSuccessful
            status: "True"
            type: Installed
          - lastTransitionTime: "2024-05-16T13:04:16Z"
            lastUpdateTime: "2024-05-16T13:04:16Z"
            message: The controller has been rolled out successfully.
            reason: ControllerRolledOut
            status: "False"
            type: Progressing
          - lastTransitionTime: "2024-05-16T13:03:39Z"
            lastUpdateTime: "2024-05-16T13:03:39Z"
            message: chart could be rendered successfully.
            reason: RegistrationValid
            status: "True"
            type: Valid
        

        Deployment Configuration Options

        The .spec.deployment resource allows to configure a deployment policy. There are the following policies:

        • OnDemand (default): Gardener will demand the deployment and deletion of the extension controller to/from seed clusters dynamically. It will automatically determine (based on other resources like Shoots) whether it is required and decide accordingly.
        • Always: Gardener will demand the deployment of the extension controller to seed clusters independent of whether it is actually required or not. This might be helpful if you want to add a new component/controller to all seed clusters by default. Another use-case is to minimize the durations until extension controllers get deployed and ready in case you have highly fluctuating seed clusters.
        • AlwaysExceptNoShoots: Similar to Always, but if the seed does not have any shoots, then the extension is not being deployed. It will be deleted from a seed after the last shoot has been removed from it.

        Also, the .spec.deployment.seedSelector allows to specify a label selector for seed clusters. Only if it matches the labels of a seed, then it will be deployed to it. Please note that a seed selector can only be specified for secondary controllers (primary=false for all .spec.resources[]).

        Extensions in the Garden Cluster Itself

        The Shoot resource itself will contain some provider-specific data blobs. As a result, some extensions might also want to run in the garden cluster, e.g., to provide ValidatingWebhookConfigurations for validating the correctness of their provider-specific blobs:

        apiVersion: core.gardener.cloud/v1beta1
        kind: Shoot
        metadata:
          name: johndoe-aws
          namespace: garden-dev
        spec:
          ...
          cloud:
            type: aws
            region: eu-west-1
            providerConfig:
              apiVersion: aws.cloud.gardener.cloud/v1alpha1
              kind: InfrastructureConfig
              networks:
                vpc: # specify either 'id' or 'cidr'
                # id: vpc-123456
                  cidr: 10.250.0.0/16
                internal:
                - 10.250.112.0/22
                public:
                - 10.250.96.0/22
                workers:
                - 10.250.0.0/19
              zones:
              - eu-west-1a
        ...
        

        In the above example, Gardener itself does not understand the AWS-specific provider configuration for the infrastructure. However, if this part of the Shoot resource should be validated, then you should run an AWS-specific component in the garden cluster that registers a webhook. You can do it similarly if you want to default some fields of a resource (by using a MutatingWebhookConfiguration).

        Again, similar to how Gardener is deployed to the garden cluster, these components must be deployed and managed by the Gardener administrator.

        Extension Resource Configurations

        The Extension resource allows injecting arbitrary steps into the shoot reconciliation flow that are unknown to Gardener. Hence, it is slightly special and allows further configuration when registering it:

        apiVersion: core.gardener.cloud/v1beta1
        kind: ControllerRegistration
        metadata:
          name: extension-foo
        spec:
          resources:
          - kind: Extension
            type: foo
            primary: true
            globallyEnabled: true
            reconcileTimeout: 30s
            lifecycle:
              reconcile: AfterKubeAPIServer
              delete: BeforeKubeAPIServer
              migrate: BeforeKubeAPIServer
        

        The globallyEnabled=true option specifies that the Extension/foo object shall be created by default for all shoots (unless they opted out by setting .spec.extensions[].enabled=false in the Shoot spec).

        The reconcileTimeout tells Gardener how long it should wait during its shoot reconciliation flow for the Extension/foo’s reconciliation to finish.

        Extension Lifecycle

        The lifecycle field tells Gardener when to perform a certain action on the Extension resource during the reconciliation flows. If omitted, then the default behaviour will be applied. Please find more information on the defaults in the explanation below. Possible values for each control flow are AfterKubeAPIServer, BeforeKubeAPIServer, and AfterWorker. Let’s take the following configuration and explain it.

            ...
            lifecycle:
              reconcile: AfterKubeAPIServer
              delete: BeforeKubeAPIServer
              migrate: BeforeKubeAPIServer
        
        • reconcile: AfterKubeAPIServer means that the extension resource will be reconciled after the successful reconciliation of the kube-apiserver during shoot reconciliation. This is also the default behaviour if this value is not specified. During shoot hibernation, the opposite rule is applied, meaning that in this case the reconciliation of the extension will happen before the kube-apiserver is scaled to 0 replicas. On the other hand, if the extension needs to be reconciled before the kube-apiserver and scaled down after it, then the value BeforeKubeAPIServer should be used.
        • delete: BeforeKubeAPIServer means that the extension resource will be deleted before the kube-apiserver is destroyed during shoot deletion. This is the default behaviour if this value is not specified.
        • migrate: BeforeKubeAPIServer means that the extension resource will be migrated before the kube-apiserver is destroyed in the source cluster during control plane migration. This is the default behaviour if this value is not specified. The restoration of the control plane follows the reconciliation control flow.

        The lifecycle value AfterWorker is only available during reconcile. When specified, the extension resource will be reconciled after the workers are deployed. This is useful for extensions that want to deploy a workload in the shoot control plane and want to wait for the workload to run and get ready on a node. During shoot creation the extension will start its reconciliation before the first workers have joined the cluster, they will become available at some later point.

        3.10 - ControlPlane

        Contract: ControlPlane Resource

        Most Kubernetes clusters require a cloud-controller-manager or CSI drivers in order to work properly. Before introducing the ControlPlane extension resource Gardener was having several different Helm charts for the cloud-controller-manager deployments for the various providers. Now, Gardener commissions an external, provider-specific controller to take over this task.

        Which control plane resources are required?

        As mentioned in the controlplane customization webhooks document, Gardener shall not deploy any cloud-controller-manager or any other provider-specific component. Instead, it creates a ControlPlane CRD that should be picked up by provider extensions. Its purpose is to trigger the deployment of such provider-specific components in the shoot namespace in the seed cluster.

        What needs to be implemented to support a new infrastructure provider?

        As part of the shoot flow Gardener will create a special CRD in the seed cluster that needs to be reconciled by an extension controller, for example:

        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: ControlPlane
        metadata:
          name: control-plane
          namespace: shoot--foo--bar
        spec:
          type: openstack
          region: europe-west1
          secretRef:
            name: cloudprovider
            namespace: shoot--foo--bar
          providerConfig:
            apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1
            kind: ControlPlaneConfig
            loadBalancerProvider: provider
            zone: eu-1a
            cloudControllerManager:
              featureGates:
                CustomResourceValidation: true
          infrastructureProviderStatus:
            apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1
            kind: InfrastructureStatus
            networks:
              floatingPool:
                id: vpc-1234
              subnets:
              - purpose: nodes
                id: subnetid
        

        The .spec.secretRef contains a reference to the provider secret pointing to the account that shall be used for the shoot cluster. However, the most important section is the .spec.providerConfig and the .spec.infrastructureProviderStatus. The first one contains an embedded declaration of the provider specific configuration for the control plane (that cannot be known by Gardener itself). You are responsible for designing how this configuration looks like. Gardener does not evaluate it but just copies this part from what has been provided by the end-user in the Shoot resource. The second one contains the output of the Infrastructure resource (that might be relevant for the CCM config).

        In order to support a new control plane provider, you need to write a controller that watches all ControlPlanes with .spec.type=<my-provider-name>. You can take a look at the below referenced example implementation for the Alicloud provider.

        The control plane controller as part of the ControlPlane reconciliation often deploys resources (e.g. pods/deployments) into the Shoot namespace in the Seed as part of its ControlPlane reconciliation loop. Because the namespace contains network policies that per default deny all ingress and egress traffic, the pods may need to have proper labels matching to the selectors of the network policies in order to allow the required network traffic. Otherwise, they won’t be allowed to talk to certain other components (e.g., the kube-apiserver of the shoot). For more information, see NetworkPolicys In Garden, Seed, Shoot Clusters.

        Non-Provider Specific Information Required for Infrastructure Creation

        Most providers might require further information that is not provider specific but already part of the shoot resource. One example for this is the GCP control plane controller, which needs the Kubernetes version of the shoot cluster (because it already uses the in-tree Kubernetes cloud-controller-manager). As Gardener cannot know which information is required by providers, it simply mirrors the Shoot, Seed, and CloudProfile resources into the seed. They are part of the Cluster extension resource and can be used to extract information that is not part of the Infrastructure resource itself.

        References and Additional Resources

        3.11 - ControlPlane Exposure

        Contract: ControlPlane Resource with Purpose exposure

        Some Kubernetes clusters require an additional deployments required by the seed cloud provider in order to work properly, e.g. AWS Load Balancer Readvertiser. Before using ControlPlane resources with purpose exposure, Gardener was having different Helm charts for the deployments for the various providers. Now, Gardener commissions an external, provider-specific controller to take over this task.

        Which control plane resources are required?

        As mentioned in the controlplane document, Gardener shall not deploy any other provider-specific component. Instead, it creates a ControlPlane CRD with purpose exposure that should be picked up by provider extensions. Its purpose is to trigger the deployment of such provider-specific components in the shoot namespace in the seed cluster that are needed to expose the kube-apiserver.

        The shoot cluster’s kube-apiserver are exposed via a Service of type LoadBalancer from the shoot provider (you may run the control plane of an Azure shoot in a GCP seed). It’s the seed provider extension controller that should act on the ControlPlane resources with purpose exposure.

        If SNI is enabled, then the Service from above is of type ClusterIP and Gardner will not create ControlPlane resources with purpose exposure.

        What needs to be implemented to support a new infrastructure provider?

        As part of the shoot flow, Gardener will create a special CRD in the seed cluster that needs to be reconciled by an extension controller, for example:

        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: ControlPlane
        metadata:
          name: control-plane-exposure
          namespace: shoot--foo--bar
        spec:
          type: aws
          purpose: exposure
          region: europe-west1
          secretRef:
            name: cloudprovider
            namespace: shoot--foo--bar
        

        The .spec.secretRef contains a reference to the provider secret pointing to the account that shall be used for the shoot cluster. It is most likely not needed, however, still added for some potential corner cases. If you don’t need it, then just ignore it. The .spec.region contains the region of the seed cluster.

        In order to support a control plane provider with purpose exposure, you need to write a controller or expand the existing controlplane controller that watches all ControlPlanes with .spec.type=<my-provider-name> and purpose exposure. You can take a look at the below referenced example implementation for the AWS provider.

        Non-Provider Specific Information Required for Infrastructure Creation

        Most providers might require further information that is not provider specific but already part of the shoot resource. As Gardener cannot know which information is required by providers, it simply mirrors the Shoot, Seed, and CloudProfile resources into the seed. They are part of the Cluster extension resource and can be used to extract information.

        References and Additional Resources

        3.12 - ControlPlane Webhooks

        ControlPlane Customization Webhooks

        Gardener creates the Shoot controlplane in several steps of the Shoot flow. At different point of this flow, it:

        • Deploys standard controlplane components such as kube-apiserver, kube-controller-manager, and kube-scheduler by creating the corresponding deployments, services, and other resources in the Shoot namespace.
        • Initiates the deployment of custom controlplane components by ControlPlane controllers by creating a ControlPlane resource in the Shoot namespace.

        In order to apply any provider-specific changes to the configuration provided by Gardener for the standard controlplane components, cloud extension providers can install mutating admission webhooks for the resources created by Gardener in the Shoot namespace.

        What needs to be implemented to support a new cloud provider?

        In order to support a new cloud provider, you should install “controlplane” mutating webhooks for any of the following resources:

        • Deployment with name kube-apiserver, kube-controller-manager, or kube-scheduler
        • Service with name kube-apiserver
        • OperatingSystemConfig with any name, and purpose reconcile

        See Contract Specification for more details on the contract that Gardener and webhooks should adhere to regarding the content of the above resources.

        You can install 3 different kinds of controlplane webhooks:

        • Shoot, or controlplane webhooks apply changes needed by the Shoot cloud provider, for example the --cloud-provider command line flag of kube-apiserver and kube-controller-manager. Such webhooks should only operate on Shoot namespaces labeled with shoot.gardener.cloud/provider=<provider>.
        • Seed, or controlplaneexposure webhooks apply changes needed by the Seed cloud provider, for example annotations on the kube-apiserver service to ensure cloud-specific load balancers are correctly provisioned for a service of type LoadBalancer. Such webhooks should only operate on Shoot namespaces labeled with seed.gardener.cloud/provider=<provider>.

        The labels shoot.gardener.cloud/provider and seed.gardener.cloud/provider are added by Gardener when it creates the Shoot namespace.

        Contract Specification

        This section specifies the contract that Gardener and webhooks should adhere to in order to ensure smooth interoperability. Note that this contract can’t be specified formally and is therefore easy to violate, especially by Gardener. The Gardener team will nevertheless do its best to adhere to this contract in the future and to ensure via additional measures (tests, validations) that it’s not unintentionally broken. If it needs to be changed intentionally, this can only happen after proper communication has taken place to ensure that the affected provider webhooks could be adapted to work with the new version of the contract.

        Note: The contract described below may not necessarily be what Gardener does currently (as of May 2019). Rather, it reflects the target state after changes for Gardener extensibility have been introduced.

        kube-apiserver

        To deploy kube-apiserver, Gardener shall create a deployment and a service both named kube-apiserver in the Shoot namespace. They can be mutated by webhooks to apply any provider-specific changes to the standard configuration provided by Gardener.

        The pod template of the kube-apiserver deployment shall contain a container named kube-apiserver.

        The command field of the kube-apiserver container shall contain the kube-apiserver command line. It shall contain a number of provider-independent flags that should be ignored by webhooks, such as:

        • admission plugins (--enable-admission-plugins, --disable-admission-plugins)
        • secure communications (--etcd-cafile, --etcd-certfile, --etcd-keyfile, …)
        • audit log (--audit-log-*)
        • ports (--secure-port)

        The kube-apiserver command line shall not contain any provider-specific flags, such as:

        • --cloud-provider
        • --cloud-config

        These flags can be added by webhooks if needed.

        The kube-apiserver command line may contain a number of additional provider-independent flags. In general, webhooks should ignore these unless they are known to interfere with the desired kube-apiserver behavior for the specific provider. Among the flags to be considered are:

        • --endpoint-reconciler-type
        • --advertise-address
        • --feature-gates

        Gardener uses SNI to expose the apiserver. In this case, Gardener will label the kube-apiserver’s Deployment with core.gardener.cloud/apiserver-exposure: gardener-managed label (deprecated, the label will no longer be added as of v1.80) and expects that the --endpoint-reconciler-type and --advertise-address flags are not modified.

        The --enable-admission-plugins flag may contain admission plugins that are not compatible with CSI plugins such as PersistentVolumeLabel. Webhooks should therefore ensure that such admission plugins are either explicitly enabled (if CSI plugins are not used) or disabled (otherwise).

        The env field of the kube-apiserver container shall not contain any provider-specific environment variables (so it will be empty). If any provider-specific environment variables are needed, they should be added by webhooks.

        The volumes field of the pod template of the kube-apiserver deployment, and respectively the volumeMounts field of the kube-apiserver container shall not contain any provider-specific Secret or ConfigMap resources. If such resources should be mounted as volumes, this should be done by webhooks.

        The kube-apiserver Service may be of type LoadBalancer, but shall not contain any provider-specific annotations that may be needed to actually provision a load balancer resource in the Seed provider’s cloud. If any such annotations are needed, they should be added by webhooks (typically controlplaneexposure webhooks).

        The kube-apiserver Service will be of type ClusterIP. In this case, Gardener will label this Service with core.gardener.cloud/apiserver-exposure: gardener-managed label (deprecated, the label will no longer be added as of v1.80) and expects that no mutations happen.

        kube-controller-manager

        To deploy kube-controller-manager, Gardener shall create a deployment named kube-controller-manager in the Shoot namespace. It can be mutated by webhooks to apply any provider-specific changes to the standard configuration provided by Gardener.

        The pod template of the kube-controller-manager deployment shall contain a container named kube-controller-manager.

        The command field of the kube-controller-manager container shall contain the kube-controller-manager command line. It shall contain a number of provider-independent flags that should be ignored by webhooks, such as:

        • --kubeconfig, --authentication-kubeconfig, --authorization-kubeconfig
        • --leader-elect
        • secure communications (--tls-cert-file, --tls-private-key-file, …)
        • cluster CIDR and identity (--cluster-cidr, --cluster-name)
        • sync settings (--concurrent-deployment-syncs, --concurrent-replicaset-syncs)
        • horizontal pod autoscaler (--horizontal-pod-autoscaler-*)
        • ports (--port, --secure-port)

        The kube-controller-manager command line shall not contain any provider-specific flags, such as:

        • --cloud-provider
        • --cloud-config
        • --configure-cloud-routes
        • --external-cloud-volume-plugin

        These flags can be added by webhooks if needed.

        The kube-controller-manager command line may contain a number of additional provider-independent flags. In general, webhooks should ignore these unless they are known to interfere with the desired kube-controller-manager behavior for the specific provider. Among the flags to be considered are:

        • --feature-gates

        The env field of the kube-controller-manager container shall not contain any provider-specific environment variables (so it will be empty). If any provider-specific environment variables are needed, they should be added by webhooks.

        The volumes field of the pod template of the kube-controller-manager deployment, and respectively the volumeMounts field of the kube-controller-manager container shall not contain any provider-specific Secret or ConfigMap resources. If such resources should be mounted as volumes, this should be done by webhooks.

        kube-scheduler

        To deploy kube-scheduler, Gardener shall create a deployment named kube-scheduler in the Shoot namespace. It can be mutated by webhooks to apply any provider-specific changes to the standard configuration provided by Gardener.

        The pod template of the kube-scheduler deployment shall contain a container named kube-scheduler.

        The command field of the kube-scheduler container shall contain the kube-scheduler command line. It shall contain a number of provider-independent flags that should be ignored by webhooks, such as:

        • --config
        • --authentication-kubeconfig, --authorization-kubeconfig
        • secure communications (--tls-cert-file, --tls-private-key-file, …)
        • ports (--port, --secure-port)

        The kube-scheduler command line may contain additional provider-independent flags. In general, webhooks should ignore these unless they are known to interfere with the desired kube-controller-manager behavior for the specific provider. Among the flags to be considered are:

        • --feature-gates

        The kube-scheduler command line can’t contain provider-specific flags, and it makes no sense to specify provider-specific environment variables or mount provider-specific Secret or ConfigMap resources as volumes.

        etcd-main and etcd-events

        To deploy etcd, Gardener shall create 2 Etcd named etcd-main and etcd-events in the Shoot namespace. They can be mutated by webhooks to apply any provider-specific changes to the standard configuration provided by Gardener.

        Gardener shall configure the Etcd resource completely to set up an etcd cluster which uses the default storage class of the seed cluster.

        cloud-controller-manager

        Gardener shall not deploy a cloud-controller-manager. If it is needed, it should be added by a ControlPlane controller

        CSI Controllers

        Gardener shall not deploy a CSI controller. If it is needed, it should be added by a ControlPlane controller

        kubelet

        To specify the kubelet configuration, Gardener shall create a OperatingSystemConfig resource with any name and purpose reconcile in the Shoot namespace. It can therefore also be mutated by webhooks to apply any provider-specific changes to the standard configuration provided by Gardener. Gardener may write multiple such resources with different type to the same Shoot namespaces if multiple OSs are used.

        The OSC resource shall contain a unit named kubelet.service, containing the corresponding systemd unit configuration file. The [Service] section of this file shall contain a single ExecStart option having the kubelet command line as its value.

        The OSC resource shall contain a file with path /var/lib/kubelet/config/kubelet, which contains a KubeletConfiguration resource in YAML format. Most of the flags that can be specified in the kubelet command line can alternatively be specified as options in this configuration as well.

        The kubelet command line shall contain a number of provider-independent flags that should be ignored by webhooks, such as:

        • --config
        • --bootstrap-kubeconfig, --kubeconfig
        • --network-plugin (and, if it equals cni, also --cni-bin-dir and --cni-conf-dir)
        • --node-labels

        The kubelet command line shall not contain any provider-specific flags, such as:

        • --cloud-provider
        • --cloud-config
        • --provider-id

        These flags can be added by webhooks if needed.

        The kubelet command line / configuration may contain a number of additional provider-independent flags / options. In general, webhooks should ignore these unless they are known to interfere with the desired kubelet behavior for the specific provider. Among the flags / options to be considered are:

        • --enable-controller-attach-detach (enableControllerAttachDetach) - should be set to true if CSI plugins are used, but in general can also be ignored since its default value is also true, and this should work both with and without CSI plugins.
        • --feature-gates (featureGates) - should contain a list of specific feature gates if CSI plugins are used. If CSI plugins are not used, the corresponding feature gates can be ignored since enabling them should not harm in any way.

        3.13 - Conventions

        General Conventions

        All the extensions that are registered to Gardener are deployed to the seed clusters on which they are required (also see ControllerRegistration).

        Some of these extensions might need to create global resources in the seed (e.g., ClusterRoles), i.e., it’s important to have a naming scheme to avoid conflicts as it cannot be checked or validated upfront that two extensions don’t use the same names.

        Consequently, this page should help answering some general questions that might come up when it comes to developing an extension.

        PriorityClasses

        Extensions are not supposed to create and use self-defined PriorityClasses. Instead, they can and should rely on well-known PriorityClasses managed by gardenlet.

        High Availability of Deployed Components

        Extensions might deploy components via Deployments, StatefulSets, etc., as part of the shoot control plane, or the seed or shoot system components. In case a seed or shoot cluster is highly available, there are various failure tolerance types. For more information, see Highly Available Shoot Control Plane. Accordingly, the replicas, topologySpreadConstraints or affinity settings of the deployed components might need to be adapted.

        Instead of doing this one-by-one for each and every component, extensions can rely on a mutating webhook provided by Gardener. Please refer to High Availability of Deployed Components for details.

        To reduce costs and to improve the network traffic latency in multi-zone clusters, extensions can make a Service topology-aware. Please refer to this document for details.

        Is there a naming scheme for (global) resources?

        As there is no formal process to validate non-existence of conflicts between two extensions, please follow these naming schemes when creating resources (especially, when creating global resources, but it’s in general a good idea for most created resources):

        The resource name should be prefixed with extensions.gardener.cloud:<extension-type>-<extension-name>:<resource-name>, for example:

        • extensions.gardener.cloud:provider-aws:some-controller-manager
        • extensions.gardener.cloud:extension-certificate-service:cert-broker

        How to create resources in the shoot cluster?

        Some extensions might not only create resources in the seed cluster itself but also in the shoot cluster. Usually, every extension comes with a ServiceAccount and the required RBAC permissions when it gets installed to the seed. However, there are no credentials for the shoot for every extension.

        Extensions are supposed to use ManagedResources to manage resources in shoot clusters. gardenlet deploys gardener-resource-manager instances into all shoot control planes, that will reconcile ManagedResources without a specified class (spec.class=null) in shoot clusters. Mind that Gardener acts on ManagedResources with the origin=gardener label. In order to prevent unwanted behavior, extensions should omit the origin label or provide their own unique value for it when creating such resources.

        If you need to deploy a non-DaemonSet resource, Gardener automatically ensures that it only runs on nodes that are allowed to host system components and extensions. For more information, see System Components Webhook.

        How to create kubeconfigs for the shoot cluster?

        Historically, Gardener extensions used to generate kubeconfigs with client certificates for components they deploy into the shoot control plane. For this, they reused the shoot cluster CA secret (ca) to issue new client certificates. With gardener/gardener#4661 we moved away from using client certificates in favor of short-lived, auto-rotated ServiceAccount tokens. These tokens are managed by gardener-resource-manager’s TokenRequestor. Extensions are supposed to reuse this mechanism for requesting tokens and a generic-token-kubeconfig for authenticating against shoot clusters.

        With GEP-18 (Shoot cluster CA rotation), a dedicated CA will be used for signing client certificates (gardener/gardener#5779) which will be rotated when triggered by the shoot owner. With this, extensions cannot reuse the ca secret anymore to issue client certificates. Hence, extensions must switch to short-lived ServiceAccount tokens in order to support the CA rotation feature.

        The generic-token-kubeconfig secret contains the CA bundle for establishing trust to shoot API servers. However, as the secret is immutable, its name changes with the rotation of the cluster CA. Extensions need to look up the generic-token-kubeconfig.secret.gardener.cloud/name annotation on the respective Cluster object in order to determine which secret contains the current CA bundle. The helper function extensionscontroller.GenericTokenKubeconfigSecretNameFromCluster can be used for this task.

        You can take a look at CA Rotation in Extensions for more details on the CA rotation feature in regard to extensions.

        How to create certificates for the shoot cluster?

        Gardener creates several certificate authorities (CA) that are used to create server certificates for various components. For example, the shoot’s etcd has its own CA, the kube-aggregator has its own CA as well, and both are different to the actual cluster’s CA.

        With GEP-18 (Shoot cluster CA rotation), extensions are required to do the same and generate dedicated CAs for their components (e.g. for signing a server certificate for cloud-controller-manager). They must not depend on the CA secrets managed by gardenlet.

        Please see CA Rotation in Extensions for the exact requirements that extensions need to fulfill in order to support the CA rotation feature.

        How to enforce a Pod Security Standard for extension namespaces?

        The pod-security.kubernetes.io/enforce namespace label enforces the Pod Security Standards.

        You can set the pod-security.kubernetes.io/enforce label for extension namespace by adding the security.gardener.cloud/pod-security-enforce annotation to your ControllerRegistration. The value of the annotation would be the value set for the pod-security.kubernetes.io/enforce label. It is advised to set the annotation with the most restrictive pod security standard that your extension pods comply with.

        If you are using the ./hack/generate-controller-registration.sh script to generate your ControllerRegistration you can use the -e, –pod-security-enforce option to set the security.gardener.cloud/pod-security-enforce annotation. If the option is not set, it defaults to baseline.

        3.14 - DNS Record

        Contract: DNSRecord Resources

        Every shoot cluster requires external DNS records that are publicly resolvable. The management of these DNS records requires provider-specific knowledge which is to be developed outside the Gardener’s core repository.

        Currently, Gardener uses DNSProvider and DNSEntry resources. However, this introduces undesired coupling of Gardener to a controller that does not adhere to the Gardener extension contracts. Because of this, we plan to stop using DNSProvider and DNSEntry resources for Gardener DNS records in the future and use the DNSRecord resources described here instead.

        What does Gardener create DNS records for?

        Internal Domain Name

        Every shoot cluster’s kube-apiserver running in the seed is exposed via a load balancer that has a public endpoint (IP or hostname). This endpoint is used by end-users and also by system components (that are running in another network, e.g., the kubelet or kube-proxy) to talk to the cluster. In order to be robust against changes of this endpoint (e.g., caused due to re-creation of the load balancer or move of the DNS record to another seed cluster), Gardener creates a so-called internal domain name for every shoot cluster. The internal domain name is a publicly resolvable DNS record that points to the load balancer of the kube-apiserver. Gardener uses this domain name in the kubeconfigs of all system components, instead of using directly the load balancer endpoint. This way Gardener does not need to recreate all kubeconfigs if the endpoint changes - it just needs to update the DNS record.

        External Domain Name

        The internal domain name is not configurable by end-users directly but configured by the Gardener administrator. However, end-users usually prefer to have another DNS name, maybe even using their own domain sometimes, to access their Kubernetes clusters. Gardener supports that by creating another DNS record, named external domain name, that actually points to the internal domain name. The kubeconfig handed out to end-users does contain this external domain name, i.e., users can access their clusters with the DNS name they like to.

        As not every end-user has an own domain, it is possible for Gardener administrators to configure so-called default domains. If configured, shoots that do not specify a domain explicitly get an external domain name based on a default domain (unless explicitly stated that this shoot should not get an external domain name (.spec.dns.provider=unmanaged).

        Ingress Domain Name (Deprecated)

        Gardener allows to deploy a nginx-ingress-controller into a shoot cluster (deprecated). This controller is exposed via a public load balancer (again, either IP or hostname). Gardener creates a wildcard DNS record pointing to this load balancer. Ingress resources can later use this wildcard DNS record to expose underlying applications.

        Seed Ingress

        If .spec.ingress is configured in the Seed, Gardener deploys the ingress controller mentioned in .spec.ingress.controller.kind to the seed cluster. Currently, the only supported kind is “nginx”. If the ingress field is set, then .spec.dns.provider must also be set. Gardener creates a wildcard DNS record pointing to the load balancer of the ingress controller. The Ingress resources of components like Plutono and Prometheus in the garden namespace and the shoot namespaces use this wildcard DNS record to expose their underlying applications.

        What needs to be implemented to support a new DNS provider?

        As part of the shoot flow, Gardener will create a number of DNSRecord resources in the seed cluster (one for each of the DNS records mentioned above) that need to be reconciled by an extension controller. These resources contain the following information:

        • The DNS provider type (e.g., aws-route53, google-clouddns, …)
        • A reference to a Secret object that contains the provider-specific credentials used to communicate with the provider’s API.
        • The fully qualified domain name (FQDN) of the DNS record, e.g. “api.<shoot domain>”.
        • The DNS record type, one of A, AAAA, CNAME, or TXT.
        • The DNS record values, that is a list of IP addresses for A records, a single hostname for CNAME records, or a list of texts for TXT records.

        Optionally, the DNSRecord resource may contain also the following information:

        • The region of the DNS record. If not specified, the region specified in the referenced Secret shall be used. If that is also not specified, the extension controller shall use a certain default region.
        • The DNS hosted zone of the DNS record. If not specified, it shall be determined automatically by the extension controller by getting all hosted zones of the account and searching for the longest zone name that is a suffix of the fully qualified domain name (FQDN) mentioned above.
        • The TTL of the DNS record in seconds. If not specified, it shall be set by the extension controller to 120.

        Example DNSRecord:

        ---
        apiVersion: v1
        kind: Secret
        metadata:
          name: dnsrecord-bar-external
          namespace: shoot--foo--bar
        type: Opaque
        data:
          # aws-route53 specific credentials here
        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: DNSRecord
        metadata:
          name: dnsrecord-external
          namespace: default
        spec:
          type: aws-route53
          secretRef:
            name: dnsrecord-bar-external
            namespace: shoot--foo--bar
        # region: eu-west-1
        # zone: ZFOO
          name: api.bar.foo.my-fancy-domain.com
          recordType: A
          values:
          - 1.2.3.4
        # ttl: 600
        

        In order to support a new DNS record provider, you need to write a controller that watches all DNSRecords with .spec.type=<my-provider-name>. You can take a look at the below referenced example implementation for the AWS route53 provider.

        Key Names in Secrets Containing Provider-Specific Credentials

        For compatibility with existing setups, extension controllers shall support two different namings of keys in secrets containing provider-specific credentials:

        • The naming used by the external-dns-management DNS controller. For example, on AWS the key names are AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION.
        • The naming used by other provider-specific extension controllers, e.g. for infrastructure. For example, on AWS the key names are accessKeyId, secretAccessKey, and region.

        Avoiding Reading the DNS Hosted Zones

        If the DNS hosted zone is not specified in the DNSRecord resource, during the first reconciliation the extension controller shall determine the correct DNS hosted zone for the specified FQDN and write it to the status of the resource:

        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: DNSRecord
        metadata:
          name: dnsrecord-external
          namespace: shoot--foo--bar
        spec:
          ...
        status:
          lastOperation: ...
          zone: ZFOO
        

        On subsequent reconciliations, the extension controller shall use the zone from the status and avoid reading the DNS hosted zones from the provider. If the DNSRecord resource specifies a zone in .spec.zone and the extension controller has written a value to .status.zone, the first one shall be considered with higher priority by the extension controller.

        Non-Provider Specific Information Required for DNS Record Creation

        Some providers might require further information that is not provider specific but already part of the shoot resource. As Gardener cannot know which information is required by providers, it simply mirrors the Shoot, Seed, and CloudProfile resources into the seed. They are part of the Cluster extension resource and can be used to extract information that is not part of the DNSRecord resource itself.

        Using DNSRecord Resources

        gardenlet manages DNSRecord resources for all three DNS records mentioned above (internal, external, and ingress). In order to successfully reconcile a shoot with the feature gate enabled, extension controllers for DNSRecord resources for types used in the default, internal, and custom domain secrets should be registered via ControllerRegistration resources.

        Note: For compatibility reasons, the spec.dns.providers section is still used to specify additional providers. Only the one marked as primary: true will be used for DNSRecord. All others are considered by the shoot-dns-service extension only (if deployed).

        Support for DNSRecord Resources in the Provider Extensions

        The following table contains information about the provider extension version that adds support for DNSRecord resources:

        ExtensionVersion
        provider-alicloudv1.26.0
        provider-awsv1.27.0
        provider-azurev1.21.0
        provider-gcpv1.18.0
        provider-openstackv1.21.0
        provider-vsphereN/A
        provider-equinix-metalN/A
        provider-kubevirtN/A
        provider-openshiftN/A

        Support for DNSRecord IPv6 recordType: AAAA in the Provider Extensions

        The following table contains information about the provider extension version that adds support for DNSRecord IPv6 recordType: AAAA:

        ExtensionVersion
        provider-alicloudN/A
        provider-awsN/A
        provider-azureN/A
        provider-gcpN/A
        provider-openstackN/A
        provider-vsphereN/A
        provider-equinix-metalN/A
        provider-kubevirtN/A
        provider-openshiftN/A
        provider-localv1.63.0

        References and Additional Resources

        3.15 - Extension

        Contract: Extension Resource

        Gardener defines common procedures which must be passed to create a functioning shoot cluster. Well known steps are represented by special resources like Infrastructure, OperatingSystemConfig or DNS. These resources are typically reconciled by dedicated controllers setting up the infrastructure on the hyperscaler or managing DNS entries, etc.

        But, some requirements don’t match with those special resources or don’t depend on being proceeded at a specific step in the creation / deletion flow of the shoot. They require a more generic hook. Therefore, Gardener offers the Extension resource.

        What is required to register and support an Extension type?

        Gardener creates one Extension resource per registered extension type in ControllerRegistration per shoot.

        apiVersion: core.gardener.cloud/v1beta1
        kind: ControllerRegistration
        metadata:
          name: extension-example
        spec:
          resources:
          - kind: Extension
            type: example
            globallyEnabled: true
            workerlessSupported: true
        

        If spec.resources[].globallyEnabled is true, then the Extension resources of the given type is created for every shoot cluster. Set to false, the Extension resource is only created if configured in the Shoot manifest. In case of workerless Shoot, a globally enabled Extension resource is created only if spec.resources[].workerlessSupported is also set to true. If an extension configured in the spec of a workerless Shoot is not supported yet, the admission request will be rejected.

        The Extension resources are created in the shoot namespace of the seed cluster.

        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: Extension
        metadata:
          name: example
          namespace: shoot--foo--bar
        spec:
          type: example
          providerConfig: {}
        

        Your controller needs to reconcile extensions.extensions.gardener.cloud. Since there can exist multiple Extension resources per shoot, each one holds a spec.type field to let controllers check their responsibility (similar to all other extension resources of Gardener).

        ProviderConfig

        It is possible to provide data in the Shoot resource which is copied to spec.providerConfig of the Extension resource.

        ---
        apiVersion: core.gardener.cloud/v1beta1
        kind: Shoot
        metadata:
          name: bar
          namespace: garden-foo
        spec:
          extensions:
          - type: example
            providerConfig:
              foo: bar
        ...
        

        results in

        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: Extension
        metadata:
          name: example
          namespace: shoot--foo--bar
        spec:
          type: example
          providerConfig:
            foo: bar
        

        Shoot Reconciliation Flow and Extension Status

        Gardener creates Extension resources as part of the Shoot reconciliation. Moreover, it is guaranteed that the Cluster resource exists before the Extension resource is created. Extensions can be reconciled at different stages during Shoot reconciliation depending on the defined extension lifecycle strategy in the respective ControllerRegistration resource. Please consult the Extension Lifecycle section for more information.

        For an Extension controller it is crucial to maintain the Extension’s status correctly. At the end Gardener checks the status of each Extension and only reports a successful shoot reconciliation if the state of the last operation is Succeeded.

        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: Extension
        metadata:
          generation: 1
          name: example
          namespace: shoot--foo--bar
        spec:
          type: example
        status:
          lastOperation:
            state: Succeeded
          observedGeneration: 1
        

        3.16 - Force Deletion

        Force Deletion

        From v1.81, Gardener supports Shoot Force Deletion. All extension controllers should also properly support it. This document outlines some important points that extension maintainers should keep in mind to support force deletion in their extensions.

        Overall Principles

        The following principles should always be upheld:

        • All resources pertaining to the extension and managed by it should be appropriately handled and cleaned up by the extension when force deletion is initiated.

        Implementation Details

        ForceDelete Actuator Methods

        Most extension controller implementations follow a common pattern where a generic Reconciler implementation delegates to an Actuator interface that contains the methods Reconcile, Delete, Migrate and Restore provided by the extension. A new method, ForceDelete has been added to all such Actuator interfaces; see the infrastructure Actuator interface as an example. The generic reconcilers call this method if the Shoot has annotation confirmation.gardener.cloud/force-deletion=true. Thus, it should be implemented by the extension controller to forcefully delete resources if not possible to delete them gracefully. If graceful deletion is possible, then in the ForceDelete, they can simply call the Delete method.

        Extension Controllers Based on Generic Actuators

        In practice, the implementation of many extension controllers (for example, the controlplane and worker controllers in most provider extensions) are based on a generic Actuator implementation that only delegates to extension methods for behavior that is truly provider-specific. In all such cases, the ForceDelete method has already been implemented with a method that should suit most of the extensions. If it doesn’t suit your extension, then the ForceDelete method needs to be overridden; see the Azure controlplane controller as an example.

        Extension Controllers Not Based on Generic Actuators

        The implementation of some extension controllers (for example, the infrastructure controllers in all provider extensions) are not based on a generic Actuator implementation. Such extension controllers must always provide a proper implementation of the ForceDelete method according to the above guidelines; see the AWS infrastructure controller as an example. In practice, this might result in code duplication between the different extensions, since the ForceDelete code is usually not OS-specific.

        Some General Implementation Examples

        • If the extension deploys only resources in the shoot cluster not backed by infrastructure in third-party systems, then performing the regular deletion code (actuator.Delete) will suffice in the majority of cases. (e.g - https://github.com/gardener/gardener-extension-shoot-networking-filter/blob/1d95a483d803874e8aa3b1de89431e221a7d574e/pkg/controller/lifecycle/actuator.go#L175-L178)
        • If the extension deploys resources which are backed by infrastructure in third-party systems:
          • If the resource is in the Seed cluster, the extension should remove the finalizers and delete the resource. This is needed especially if the resource is a custom resource since gardenlet will not be aware of this resource and cannot take action.
          • If the resource is in the Shoot and if it’s deployed by a ManagedResource, then gardenlet will take care to forcefully delete it in a later step of force-deletion. If the resource is not deployed via a ManagedResource, then it wouldn’t block the deletion flow anyway since it is in the Shoot cluster. In both cases, the extension controller can ignore the resource and return nil.

        3.17 - Healthcheck Library

        Health Check Library

        Goal

        Typically, an extension reconciles a specific resource (Custom Resource Definitions (CRDs)) and creates / modifies resources in the cluster (via helm, managed resources, kubectl, …). We call these API Objects ‘dependent objects’ - as they are bound to the lifecycle of the extension.

        The goal of this library is to enable extensions to setup health checks for their ‘dependent objects’ with minimal effort.

        Usage

        The library provides a generic controller with the ability to register any resource that satisfies the extension object interface. An example is the Worker CRD.

        Health check functions for commonly used dependent objects can be reused and registered with the controller, such as:

        • Deployment
        • DaemonSet
        • StatefulSet
        • ManagedResource (Gardener specific)

        See the below example taken from the provider-aws.

        health.DefaultRegisterExtensionForHealthCheck(
                       aws.Type,
                       extensionsv1alpha1.SchemeGroupVersion.WithKind(extensionsv1alpha1.WorkerResource),
                       func() runtime.Object { return &extensionsv1alpha1.Worker{} },
                       mgr, // controller runtime manager
                       opts, // options for the health check controller
                       nil, // custom predicates
                       map[extensionshealthcheckcontroller.HealthCheck]string{
                               general.CheckManagedResource(genericactuator.McmShootResourceName): string(gardencorev1beta1.ShootSystemComponentsHealthy),
                               general.CheckSeedDeployment(aws.MachineControllerManagerName):      string(gardencorev1beta1.ShootEveryNodeReady),
                               worker.SufficientNodesAvailable():                                  string(gardencorev1beta1.ShootEveryNodeReady),
                       })
        

        This creates a health check controller that reconciles the extensions.gardener.cloud/v1alpha1.Worker resource with the spec.type ‘aws’. Three health check functions are registered that are executed during reconciliation. Each health check is mapped to a single HealthConditionType that results in conditions with the same condition.type (see below). To contribute to the Shoot’s health, the following conditions can be used: SystemComponentsHealthy, EveryNodeReady, ControlPlaneHealthy, ObservabilityComponentsHealthy. In case of workerless Shoot the EveryNodeReady condition is not present, so it can’t be used.

        The Gardener/Gardenlet checks each extension for conditions matching these types. However, extensions are free to choose any HealthConditionType. For more information, see Contributing to Shoot Health Status Conditions.

        A health check has to satisfy the below interface. You can find implementation examples in the healtcheck folder.

        type HealthCheck interface {
            // Check is the function that executes the actual health check
            Check(context.Context, types.NamespacedName) (*SingleCheckResult, error)
            // InjectSeedClient injects the seed client
            InjectSeedClient(client.Client)
            // InjectShootClient injects the shoot client
            InjectShootClient(client.Client)
            // SetLoggerSuffix injects the logger
            SetLoggerSuffix(string, string)
            // DeepCopy clones the healthCheck
            DeepCopy() HealthCheck
        }
        

        The health check controller regularly (default: 30s) reconciles the extension resource and executes the registered health checks for the dependent objects. As a result, the controller writes condition(s) to the status of the extension containing the health check result. In our example, two checks are mapped to ShootEveryNodeReady and one to ShootSystemComponentsHealthy, leading to conditions with two distinct HealthConditionTypes (condition.type):

        status:
          conditions:
            - lastTransitionTime: "20XX-10-28T08:17:21Z"
              lastUpdateTime: "20XX-11-28T08:17:21Z"
              message: (1/1) Health checks successful
              reason: HealthCheckSuccessful
              status: "True"
              type: SystemComponentsHealthy
            - lastTransitionTime: "20XX-10-28T08:17:21Z"
              lastUpdateTime: "20XX-11-28T08:17:21Z"
              message: (2/2) Health checks successful
              reason: HealthCheckSuccessful
              status: "True"
              type: EveryNodeReady
        

        Please note that there are four statuses: True, False, Unknown, and Progressing.

        • True should be used for successful health checks.
        • False should be used for unsuccessful/failing health checks.
        • Unknown should be used when there was an error trying to determine the health status.
        • Progressing should be used to indicate that the health status did not succeed but for expected reasons (e.g., a cluster scale up/down could make the standard health check fail because something is wrong with the Machines, however, it’s actually an expected situation and known to be completed within a few minutes.)

        Health checks that report Progressing should also provide a timeout, after which this “progressing situation” is expected to be completed. The health check library will automatically transition the status to False if the timeout was exceeded.

        Additional Considerations

        It is up to the extension to decide how to conduct health checks, though it is recommended to make use of the build-in health check functionality of managed-resources for trivial checks. By deploying the depending resources via managed resources, the gardener resource manager conducts basic checks for different API objects out-of-the-box (e.g Deployments, DaemonSets, …) - and writes health conditions.

        By default, Gardener performs health checks for all the ManagedResources created in the shoot namespaces. Their status will be aggregated to the Shoot conditions according to the following rules:

        • Health checks of ManagedResource with .spec.class=nil are aggregated to the SystemComponentsHealthy condition
        • Health checks of ManagedResource with .spec.class!=nil are aggregated to the ControlPlaneHealthy condition unless the ManagedResource is labeled with care.gardener.cloud/condition-type=<other-condition-type>. In such case, it is aggregated to the <other-condition-type>.

        More sophisticated health checks should be implemented by the extension controller itself (implementing the HealthCheck interface).

        3.18 - Heartbeat

        Heartbeat Controller

        The heartbeat controller renews a dedicated Lease object named gardener-extension-heartbeat at regular 30 second intervals by default. This Lease is used for heartbeats similar to how gardenlet uses Lease objects for seed heartbeats (see gardenlet heartbeats).

        The gardener-extension-heartbeat Lease can be checked by other controllers to verify that the corresponding extension controller is still running. Currently, gardenlet checks this Lease when performing shoot health checks and expects to find the Lease inside the namespace where the extension controller is deployed by the corresponding ControllerInstallation. For each extension resource deployed in the Shoot control plane, gardenlet finds the corresponding gardener-extension-heartbeat Lease resource and checks whether the Lease’s .spec.renewTime is older than the allowed threshold for stale extension health checks - in this case, gardenlet considers the health check report for an extension resource as “outdated” and reflects this in the Shoot status.

        3.19 - Infrastructure

        Contract: Infrastructure Resource

        Every Kubernetes cluster requires some low-level infrastructure to be setup in order to work properly. Examples for that are networks, routing entries, security groups, IAM roles, etc. Before introducing the Infrastructure extension resource Gardener was using Terraform in order to create and manage these provider-specific resources (e.g., see here). Now, Gardener commissions an external, provider-specific controller to take over this task.

        Which infrastructure resources are required?

        Unfortunately, there is no general answer to this question as it is highly provider specific. Consider the above mentioned resources, i.e. VPC, subnets, route tables, security groups, IAM roles, SSH key pairs. Most of the resources are required in order to create VMs (the shoot cluster worker nodes), load balancers, and volumes.

        What needs to be implemented to support a new infrastructure provider?

        As part of the shoot flow Gardener will create a special CRD in the seed cluster that needs to be reconciled by an extension controller, for example:

        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: Infrastructure
        metadata:
          name: infrastructure
          namespace: shoot--foo--bar
        spec:
          type: azure
          region: eu-west-1
          secretRef:
            name: cloudprovider
            namespace: shoot--foo--bar
          providerConfig:
            apiVersion: azure.provider.extensions.gardener.cloud/v1alpha1
            kind: InfrastructureConfig
            resourceGroup:
              name: mygroup
            networks:
              vnet: # specify either 'name' or 'cidr'
              # name: my-vnet
                cidr: 10.250.0.0/16
              workers: 10.250.0.0/19
        

        The .spec.secretRef contains a reference to the provider secret pointing to the account that shall be used to create the needed resources. However, the most important section is the .spec.providerConfig. It contains an embedded declaration of the provider specific configuration for the infrastructure (that cannot be known by Gardener itself). You are responsible for designing how this configuration looks like. Gardener does not evaluate it but just copies this part from what has been provided by the end-user in the Shoot resource.

        After your controller has created the required resources in your provider’s infrastructure it needs to generate an output that can be used by other controllers in subsequent steps. An example for that is the Worker extension resource controller. It is responsible for creating virtual machines (shoot worker nodes) in this prepared infrastructure. Everything that it needs to know in order to do that (e.g. the network IDs, security group names, etc. (again: provider-specific)) needs to be provided as output in the Infrastructure resource:

        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: Infrastructure
        metadata:
          name: infrastructure
          namespace: shoot--foo--bar
        spec:
          ...
        status:
          lastOperation: ...
          providerStatus:
            apiVersion: azure.provider.extensions.gardener.cloud/v1alpha1
            kind: InfrastructureStatus
            resourceGroup:
              name: mygroup
            networks:
              vnet:
                name: my-vnet
              subnets:
              - purpose: nodes
                name: my-subnet
            availabilitySets:
            - purpose: nodes
              id: av-set-id
              name: av-set-name
            routeTables:
            - purpose: nodes
              name: route-table-name
            securityGroups:
            - purpose: nodes
              name: sec-group-name
        

        In order to support a new infrastructure provider you need to write a controller that watches all Infrastructures with .spec.type=<my-provider-name>. You can take a look at the below referenced example implementation for the Azure provider.

        Dynamic nodes network for shoot clusters

        Some environments do not allow end-users to statically define a CIDR for the network that shall be used for the shoot worker nodes. In these cases it is possible for the extension controllers to dynamically provision a network for the nodes (as part of their reconciliation loops), and to provide the CIDR in the status of the Infrastructure resource:

        ---
        apiVersion: extensions.gardener.cloud/v1alpha1
        kind: Infrastructure
        metadata:
          name: infrastructure
          namespace: shoot--foo--bar
        spec:
          ...
        status:
          lastOperation: ...
          providerStatus: ...
          nodesCIDR: 10.250.0.0/16
        

        Gardener will pick this nodesCIDR and use it to configure the VPN components to establish network connectivity between the control plane and the worker nodes. If the Shoot resource already specifies a nodes CIDR in .spec.networking.nodes and the extension controller provides also a value in .status.nodesCIDR in the Infrastructure resource then the latter one will always be considered with higher priority by Gardener.

        Non-provider specific information required for infrastructure creation

        Some providers might require further information that is not provider specific but already part of the shoot resource. One example for this is the GCP infrastructure controller which needs the pod and the service network of the cluster in order to prepare and configure the infrastructure correctly. As Gardener cannot know which information is required by providers it simply mirrors the Shoot, Seed, and CloudProfile resources into the seed. They are part of the Cluster extension resource and can be used to extract information that is not part of the Infrastructure resource itself.

        Implementation details

        Actuator interface

        Most existing infrastructure controller implementations follow a common pattern where a generic Reconciler delegates to an Actuator interface that contains the methods Reconcile, Delete, Migrate, and Restore. These methods are called by the generic Reconciler for the respective operations, and should be implemented by the extension according to the contract described here and the migration guidelines.

        ConfigValidator interface

        For infrastructure controllers, the generic Reconciler also delegates to a ConfigValidator interface that contains a single Validate method. This method is called by the generic Reconciler at the beginning of every reconciliation, and can be implemented by the extension to validate the .spec.providerConfig part of the Infrastructure resource with the respective cloud provider, typically the existence and validity of cloud provider resources such as AWS VPCs or GCP Cloud NAT IPs.

        The Validate method returns a list of errors. If this list is non-empty, the generic Reconciler will fail with an error. This error will have the error code ERR_CONFIGURATION_PROBLEM, unless there is at least one error in the list that has its ErrorType field set to field.ErrorTypeInternal.

        References and additional resources

        3.20 - Logging And Monitoring

        Logging and Monitoring for Extensions

        Gardener provides an integrated logging and monitoring stack for alerting, monitoring, and troubleshooting of its managed components by operators or end users. For further information how to make use of it in these roles, refer to the corresponding guides for exploring logs and for monitoring with Plutono.

        The components that constitute the logging and monitoring stack are managed by Gardener. By default, it deploys Prometheus and Alertmanager (managed via prometheus-operator, and Plutono into the garden namespace of all seed clusters. If the logging is enabled in the gardenlet configuration (logging.enabled), it will deploy fluent-operator and Vali in the garden namespace too.

        Each shoot namespace hosts managed logging and monitoring components. As part of the shoot reconciliation flow, Gardener deploys a shoot-specific Prometheus, blackbox-exporter, Plutono, and, if configured, an Alertmanager into the shoot namespace, next to the other control plane components. If the logging is enabled in the gardenlet configuration (logging.enabled) and the shoot purpose is not testing, it deploys a shoot-specific Vali in the shoot namespace too.

        The logging and monitoring stack is extensible by configuration. Gardener extensions can take advantage of that and contribute monitoring configurations encoded in ConfigMaps for their own, specific dashboards, alerts and other supported assets and integrate with it. As with other Gardener resources, they will be continuously reconciled. The extensions can also deploy directly fluent-operator custom resources which will be created in the seed cluster and plugged into the fluent-bit instance.

        This guide is about the roles and extensibility options of the logging and monitoring stack components, and how to integrate extensions with:

        Monitoring

        Seed Cluster

        Cache Prometheus

        The central Prometheus instance in the garden namespace (called “cache Prometheus”) fetches metrics and data from all seed cluster nodes and all seed cluster pods. It uses the federation concept to allow the shoot-specific instances to scrape only the metrics for the pods of the control plane they are responsible for. This mechanism allows to scrape the metrics for the nodes/pods once for the whole cluster, and to have them distributed afterwards. For more details, continue reading here.

        Typically, this is not necessary, but in case an extension wants to extend the configuration for this cache Prometheus, they can create the prometheus-operator’s custom resources and label them with prometheus=cache, for example:

        apiVersion: monitoring.coreos.com/v1
        kind: ServiceMonitor
        metadata:
          labels:
            prometheus: cache
          name: cache-my-component
          namespace: garden
        spec:
          selector:
            matchLabels:
              app: my-component
          endpoints:
          - metricRelabelings:
            - action: keep
              regex: ^(metric1|metric2|...)$
              sourceLabels:
              - __name__
            port: metrics
        

        Seed Prometheus

        Another Prometheus instance in the garden namespace (called “seed Prometheus”) fetches metrics and data from seed system components, kubelets, cAdvisors, and extensions. If you want your extension pods to be scraped then they must be annotated with prometheus.io/scrape=true and prometheus.io/port=<metrics-port>. For more details, continue reading here.

        Typically, this is not necessary, but in case an extension wants to extend the configuration for this seed Prometheus, they can create the prometheus-operator’s custom resources and label them with prometheus=seed, for example:

        apiVersion: monitoring.coreos.com/v1
        kind: ServiceMonitor
        metadata:
          labels:
            prometheus: seed
          name: seed-my-component
          namespace: garden
        spec:
          selector:
            matchLabels:
              app: my-component
          endpoints:
          - metricRelabelings:
            - action: keep
              regex: ^(metric1|metric2|...)$
              sourceLabels:
              - __name__
            port: metrics
        

        Aggregate Prometheus

        Another Prometheus instance in the garden namespace (called “aggregate Prometheus”) stores pre-aggregated data from the cache Prometheus and shoot Prometheus. An ingress exposes this Prometheus instance allowing it to be scraped from another cluster. For more details, continue reading here.

        Typically, this is not necessary, but in case an extension wants to extend the configuration for this aggregate Prometheus, they can create the prometheus-operator’s custom resources and label them with prometheus=aggregate, for example:

        apiVersion: monitoring.coreos.com/v1
        kind: ServiceMonitor
        metadata:
          labels:
            prometheus: aggregate
          name: aggregate-my-component
          namespace: garden
        spec:
          selector:
            matchLabels:
              app: my-component
          endpoints:
          - metricRelabelings:
            - action: keep
              regex: ^(metric1|metric2|...)$
              sourceLabels:
              - __name__
            port: metrics
        

        Plutono

        A Plutono instance is deployed by gardenlet into the seed cluster’s garden namespace for visualizing monitoring metrics and logs via dashboards. In order to provide custom dashboards, create a ConfigMap in the garden namespace labelled with dashboard.monitoring.gardener.cloud/seed=true that contains the respective JSON documents, for example:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          labels:
            dashboard.monitoring.gardener.cloud/seed: "true"
          name: extension-foo-my-custom-dashboard
          namespace: garden
        data:
          my-custom-dashboard.json: <dashboard-JSON-document>
        

        Shoot Cluster

        Shoot Prometheus

        The shoot-specific metrics are then made available to operators and users in the shoot Plutono, using the shoot Prometheus as data source.

        Extension controllers might deploy components as part of their reconciliation next to the shoot’s control plane. Examples for this would be a cloud-controller-manager or CSI controller deployments. Extensions that want to have their managed control plane components integrated with monitoring can contribute their per-shoot configuration for scraping Prometheus metrics, Alertmanager alerts or Plutono dashboards.

        Extensions Monitoring Integration

        In case an extension wants to extend the configuration for the shoot Prometheus, they can create the prometheus-operator’s custom resources and label them with prometheus=shoot.

        ServiceMonitor

        When the component runs in the seed cluster (e.g., as part of the shoot control plane), ServiceMonitor resources should be used:

        apiVersion: monitoring.coreos.com/v1
        kind: ServiceMonitor
        metadata:
          labels:
            prometheus: shoot
          name: shoot-my-controlplane-component
          namespace: shoot--foo--bar
        spec:
          selector:
            matchLabels:
              app: my-component
          endpoints:
          - metricRelabelings:
            - action: keep
              regex: ^(metric1|metric2|...)$
              sourceLabels:
              - __name__
            port: metrics
        

        In case HTTPS scheme is used, the CA certificate should be provided like this:

        spec:
          scheme: HTTPS
          tlsConfig:
            ca:
              secret:
                name: <name-of-ca-bundle-secret>
                key: bundle.crt
        

        In case the component requires credentials when contacting its metrics endpoint, provide them like this:

        spec:
          authorization:
            credentials:
              name: <name-of-secret-containing-credentials>
              key: <data-keyin-secret>
        

        If the component delegates authorization to the kube-apiserver of the shoot cluster, you can use the shoot-access-prometheus-shoot secret:

        spec:
          authorization:
            credentials:
              name: shoot-access-prometheus-shoot
              key: token
          # in case the component's server certificate is signed by the cluster CA: