Deployment

1 - Authentication Gardener Control Plane

Authentication of Gardener control plane components against the Garden cluster

Note: This document refers to Gardener’s API server, admission controller, controller manager and scheduler components. Any reference to the term Gardener control plane component can be replaced with any of the mentioned above.

There are several authentication possibilities depending on whether or not the concept of Virtual Garden is used.

Virtual Garden is not used, i.e., the runtime Garden cluster is also the target Garden cluster.

Automounted Service Account Token

The easiest way to deploy a Gardener control plane component will be to not provide kubeconfig at all. This way in-cluster configuration and an automounted service account token will be used. The drawback of this approach is that the automounted token will not be automatically rotated.

Service Account Token Volume Projection

Another solution will be to use Service Account Token Volume Projection combined with a kubeconfig referencing a token file (see example below).

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <CA-DATA>
    server: https://default.kubernetes.svc.cluster.local
  name: garden
contexts:
- context:
    cluster: garden
    user: garden
  name: garden
current-context: garden
users:
- name: garden
  user:
    tokenFile: /var/run/secrets/projected/serviceaccount/token

This will allow for automatic rotation of the service account token by the kubelet. The configuration can be achieved by setting both .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true and .Values.global.<GardenerControlPlaneComponent>.kubeconfig in the respective chart’s values.yaml file.

Virtual Garden is used, i.e., the runtime Garden cluster is different from the target Garden cluster.

Service Account

The easiest way to setup the authentication will be to create a service account and the respective roles will be bound to this service account in the target cluster. Then use the generated service account token and craft a kubeconfig which will be used by the workload in the runtime cluster. This approach does not provide a solution for the rotation of the service account token. However, this setup can be achieved by setting .Values.global.deployment.virtualGarden.enabled: true and following these steps:

1. Deploy the application part of the charts in the target cluster.
2. Get the service account token and craft the kubeconfig.
3. Set the crafted kubeconfig and deploy the runtime part of the charts in the runtime cluster.

Client Certificate

Another solution will be to bind the roles in the target cluster to a User subject instead of a service account and use a client certificate for authentication. This approach does not provide a solution for the client certificate rotation. However, this setup can be achieved by setting both .Values.global.deployment.virtualGarden.enabled: true and .Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name, then following these steps:

1. Generate a client certificate for the target cluster for the respective user.
2. Deploy the application part of the charts in the target cluster.
3. Craft a kubeconfig using the already generated client certificate.
4. Set the crafted kubeconfig and deploy the runtime part of the charts in the runtime cluster.

Projected Service Account Token

This approach requires an already deployed and configured oidc-webhook-authenticator for the target cluster. Also the runtime cluster should be registered as a trusted identity provider in the target cluster. Then projected service accounts tokens from the runtime cluster can be used to authenticate against the target cluster. The needed steps are as follows:

1. Deploy OWA and establish the needed trust.
2. Set .Values.global.deployment.virtualGarden.enabled: true and .Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name. Note: username value will depend on the trust configuration, e.g., <prefix>:system:serviceaccount:<namespace>:<serviceaccount>
3. Set .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true and .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.audience. Note: audience value will depend on the trust configuration, e.g., <cliend-id-from-trust-config>.
4. Craft a kubeconfig (see example below).
5. Deploy the application part of the charts in the target cluster.
6. Deploy the runtime part of the charts in the runtime cluster.

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <CA-DATA>
    server: https://virtual-garden.api
  name: virtual-garden
contexts:
- context:
    cluster: virtual-garden
    user: virtual-garden
  name: virtual-garden
current-context: virtual-garden
users:
- name: virtual-garden
  user:
    tokenFile: /var/run/secrets/projected/serviceaccount/token

2 - Configuring Logging

Configuring the Logging stack via Gardenlet configurations

Enable the Logging

In order to install the Gardener logging stack the logging.enabled configuration option has to be enabled in the Gardenlet configuration:

logging:
  enabled: true

From now on each Seed is going to have a logging stack which will collect logs from all pods and some systemd services. Logs related to Shoots with testing purpose are dropped in the fluent-bit output plugin. Shoots with a purpose different than testing have the same type of log aggregator (but different instance) as the Seed. The logs can be viewed in the Grafana in the garden namespace for the Seed components and in the respective shoot control plane namespaces.

Enable logs from the Shoot’s node systemd services.

The logs from the systemd services on each node can be retrieved by enabling the logging.shootNodeLogging option in the Gardenlet configuration:

logging:
  enabled: true
  shootNodeLogging:
    shootPurposes:
    - "evaluation"
    - "deployment"

Under the shootPurpose section just list all the shoot purposes for which the Shoot node logging feature will be enabled. Specifying the testing purpose has no effect because this purpose prevents the logging stack installation. Logs can be viewed in the operator Grafana! The dedicated labels are unit, syslog_identifier and nodename in the Explore menu.

Configuring the log processor

Under logging.fluentBit there is three optional sections.

• input: This overwrite the input configuration of the fluent-bit log processor.
• output: This overwrite the output configuration of the fluent-bit log processor.
• service: This overwrite the service configuration of the fluent-bit log processor.

logging:
  enabled: true
  fluentBit:
    output: |-
      [Output]
          ...      
    input: |-
      [Input]
          ...      
    service: |-
      [Service]
          ...      

additional egress IPBlock for allow-fluentbit NetworkPolicy

The optional setting under logging.fluentBit.networkPolicy.additionalEgressIPBlocks add additional egress IPBlock to allow-fluentbit NetworkPolicy to forward logs to a central system.

logging:
  enabled: true
  fluentBit:
    additionalEgressIpBlock:
      - 123.123.123.123/32

Configure central logging

For central logging, the output configuration of the fluent-bit log processor can be overwritten (logging.fluentBit.output) and the Loki instances deployments in Garden and Shoot namespace can be enabled/disabled (logging.loki.enabled), by default Loki is enabled.

logging:
  enabled: true
  fluentBit:
    output: |-
      [Output]
          ...      
  loki:
    enabled: false

Configuring central Loki storage capacity

By default, the central Loki has 100Gi of storage capacity. To overwrite the current central Loki storage capacity, the logging.loki.garden.storage setting in the gardenlet’s component configuration should be altered. If you need to increase it you can do so without losing the current data by specifying higher capacity. Doing so, the Loki’s PersistentVolume capacity will be increased instead of deleting the current PV. However, if you specify less capacity then the PersistentVolume will be deleted and with it the logs, too.

logging:
  enabled: true
  fluentBit:
    output: |-
      [Output]
          ...      
  loki:
    garden:
      storage: "200Gi"

3 - Deploy Gardenlet

Deploying Gardenlets

Gardenlets act as decentral “agents” to manage shoot clusters of a seed cluster.

To support scaleability in an automated way, gardenlets are deployed automatically. However, you can still deploy gardenlets manually to be more flexible, for example, when shoot clusters that need to be managed by Gardener are behind a firewall. The gardenlet only requires network connectivity from the gardenlet to the Garden cluster (not the other way round), so it can be used to register Kubernetes clusters with no public endpoint.

Procedure

1. First, an initial gardenlet needs to be deployed:

   • Deploy it manually if you have special requirements. More information: Deploy a Gardenlet Manually
   • Let the Gardener installer deploy it automatically otherwise. More information: Automatic Deployment of Gardenlets

2. To add additional seed clusters, it is recommended to use regular shoot clusters. You can do this by creating a ManagedSeed resource with a gardenlet section as described in Register Shoot as Seed.

4 - Deploy Gardenlet Automatically

Automatic Deployment of Gardenlets

The gardenlet can automatically deploy itself into shoot clusters, and register this cluster as a seed cluster. These clusters are called “managed seeds” (aka “shooted seeds”). This procedure is the preferred way to add additional seed clusters, because shoot clusters already come with production-grade qualities that are also demanded for seed clusters.

Prerequisites

The only prerequisite is to register an initial cluster as a seed cluster that has already a gardenlet deployed:

• This gardenlet was either deployed as part of a Gardener installation using a setup tool (for example, gardener/garden-setup) or
• the gardenlet was deployed manually
  • for a step-by-step manual installation Guide see: Deploy a Gardenlet Manually)

The initial cluster can be the garden cluster itself.

Self-Deployment of Gardenlets in Additional Managed Seed Clusters

For a better scalability, you usually need more seed clusters that you can create as follows:

1. Use the initial cluster as the seed cluster for other managed seed clusters. It hosts the control planes of the other seed clusters.
2. The gardenlet deployed in the initial cluster deploys itself automatically into the managed seed clusters.

The advantage of this approach is that there’s only one initial gardenlet installation required. Every other managed seed cluster has a gardenlet deployed automatically.

Related Links

Register Shoot as Seed

garden-setup

5 - Deploy Gardenlet Manually

Deploy a Gardenlet Manually

Manually deploying a gardenlet is required in the following cases:

• The Kubernetes cluster to be registered as a seed cluster has no public endpoint, because it is behind a firewall. The gardenlet must then be deployed into the cluster itself.

• The Kubernetes cluster to be registered as a seed cluster is managed externally (the Kubernetes cluster is not a shoot cluster, so Automatic Deployment of Gardenlets cannot be used).

• The gardenlet runs outside of the Kubernetes cluster that should be registered as a seed cluster. (The gardenlet is not restricted to run in the seed cluster or to be deployed into a Kubernetes cluster at all).

Once you’ve deployed a gardenlet manually, for example, behind a firewall, you can deploy new gardenlets automatically. The manually deployed gardenlet is then used as a template for the new gardenlets. More information: Automatic Deployment of Gardenlets.

Prerequisites

Kubernetes cluster that should be registered as a seed cluster

• Verify that the cluster has a supported Kubernetes version.

• Determine the nodes, pods, and services CIDR of the cluster. You need to configure this information in the Seed configuration. Gardener uses this information to check that the shoot cluster isn’t created with overlapping CIDR ranges.

• Every Seed cluster needs an Ingress controller which distributes external requests to internal components like grafana and prometheus. Gardener supports two approaches to achieve this:

a. Gardener managed Ingress controller and DNS records. For this configure the following lines in your Seed resource:

spec:
  dns:
    provider:
      type: aws-route53
      secretRef:
        name: ingress-secret
        namespace: garden
  ingress:
    domain: ingress.my-seed.example.com
    controller:
      kind: nginx
      providerConfig:
        <some-optional-provider-specific-config-for-the-ingressController>

⚠ Please note that if you set .spec.ingress then .spec.dns.ingressDomain must be nil.

b. Self-managed DNS record and Ingress controller:

⚠️ There should exist a DNS record *.ingress.<SEED-CLUSTER-DOMAIN> where <SEED-CLUSTER-DOMAIN> is the value of the .dns.ingressDomain field of a Seed cluster resource (or the respective Gardenlet configuration).

This is how it could be done for the Nginx ingress controller

Deploy nginx into the kube-system namespace in the Kubernetes cluster that should be registered as a Seed.

Nginx will on most cloud providers create the service with type LoadBalancer with an external ip.

NAME                        TYPE           CLUSTER-IP    EXTERNAL-IP
nginx-ingress-controller    LoadBalancer   10.0.15.46    34.200.30.30

Create a wildcard A record (e.g *.ingress.sweet-seed.. IN A 34.200.30.30) with your DNS provider and point it to the external ip of the ingress service. This ingress domain is later required to register the Seed cluster.

Please configure the ingress domain in the Seed specification as follows:

spec:
  dns:
    ingressDomain: ingress.sweet-seed.<my-domain>

⚠ Please note that if you set .spec.dns.ingressDomain then .spec.ingress must be nil.

kubeconfig for the Seed Cluster

The kubeconfig is required to deploy the gardenlet Helm chart to the seed cluster. The gardenlet requires certain privileges to be able to operate. These privileges are described in RBAC resources in the gardenlet Helm chart (see charts/gardener/gardenlet/charts/runtime/templates). The Helm chart contains a service account gardenlet that the gardenlet deployment uses by default to talk to the Seed API server.

If the gardenlet isn’t deployed in the seed cluster, the gardenlet can be configured to use a kubeconfig, which also requires the above-mentioned privileges, from a mounted directory. The kubeconfig is specified in section seedClientConnection.kubeconfig of the Gardenlet configuration. This configuration option isn’t used in the following, as this procedure only describes the recommended setup option where the gardenlet is running in the seed cluster itself.

Procedure Overview

1. Prepare the garden cluster:

   1. Create a bootstrap token secret in the kube-system namespace of the garden cluster
   2. Create RBAC roles for the gardenlet to allow bootstrapping in the garden cluster

2. Prepare the gardenlet Helm chart.

3. Automatically register shoot cluster as a seed cluster.

4. Deploy the gardenlet

5. Check that the gardenlet is successfully deployed

Create a bootstrap token secret in the kube-system namespace of the garden cluster

The gardenlet needs to talk to the Gardener API server residing in the garden cluster.

The gardenlet can be configured with an already existing garden cluster kubeconfig in one of the following ways:

• Either by specifying gardenClientConnection.kubeconfig in the Gardenlet configuration or

• by supplying the environment variable GARDEN_KUBECONFIG pointing to a mounted kubeconfig file).

The preferred way however, is to use the gardenlets ability to request a signed certificate for the garden cluster by leveraging Kubernetes Certificate Signing Requests. The gardenlet performs a TLS bootstrapping process that is similar to the Kubelet TLS Bootstrapping. Make sure that the API server of the garden cluster has bootstrap token authentication enabled.

The client credentials required for the gardenlets TLS bootstrapping process, need to be either token or certificate (OIDC isn’t supported) and have permissions to create a Certificate Signing Request (CSR). It’s recommended to use bootstrap tokens due to their desirable security properties (such as a limited token lifetime).

Therefore, first create a bootstrap token secret for the garden cluster:

apiVersion: v1
kind: Secret
metadata:
  # Name MUST be of form "bootstrap-token-<token id>"
  name: bootstrap-token-07401b
  namespace: kube-system

# Type MUST be 'bootstrap.kubernetes.io/token'
type: bootstrap.kubernetes.io/token
stringData:
  # Human readable description. Optional.
  description: "Token to be used by the gardenlet for Seed `sweet-seed`."

  # Token ID and secret. Required.
  token-id: 07401b # 6 characters
  token-secret: f395accd246ae52d # 16 characters

  # Expiration. Optional.
  # expiration: 2017-03-10T03:22:11Z

  # Allowed usages.
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"

When you later prepare the gardenlet Helm chart, a kubeconfig based on this token is shared with the gardenlet upon deployment.

Create RBAC roles for the gardenlet to allow bootstrapping in the garden cluster

This step is only required if the gardenlet you deploy is the first gardenlet in the Gardener installation. Additionally, when using the control plane chart, the following resources are already contained in the Helm chart, that is, if you use it you can skip these steps as the needed RBAC roles already exist.

The gardenlet uses the configured bootstrap kubeconfig in gardenClientConnection.bootstrapKubeconfig to request a signed certificate for the user gardener.cloud:system:seed:<seed-name> in the group gardener.cloud:system:seeds.

Create a ClusterRole and ClusterRoleBinding that grant full admin permissions to authenticated gardenlets.

Create the following resources in the garden cluster:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: gardener.cloud:system:seeds
rules:
  - apiGroups:
      - '*'
    resources:
      - '*'
    verbs:
      - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gardener.cloud:system:seeds
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gardener.cloud:system:seeds
subjects:
  - kind: Group
    name: gardener.cloud:system:seeds
    apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: gardener.cloud:system:seed-bootstrapper
rules:
  - apiGroups:
      - certificates.k8s.io
    resources:
      - certificatesigningrequests
    verbs:
      - create
      - get
  - apiGroups:
      - certificates.k8s.io
    resources:
      - certificatesigningrequests/seedclient
    verbs:
      - create
---
# A kubelet/gardenlet authenticating using bootstrap tokens is authenticated as a user in the group system:bootstrappers
# Allows the Gardenlet to create a CSR
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gardener.cloud:system:seed-bootstrapper
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gardener.cloud:system:seed-bootstrapper
subjects:
  - kind: Group
    name: system:bootstrappers
    apiGroup: rbac.authorization.k8s.io

ℹ️ After bootstrapping, the gardenlet has full administrative access to the garden cluster. You might be interested to harden this and limit its permissions to only resources related to the seed cluster it is responsible for. Please take a look into this document.

Prepare the gardenlet Helm chart

This section only describes the minimal configuration, using the global configuration values of the gardenlet Helm chart. For an overview over all values, see the configuration values. We refer to the global configuration values as gardenlet configuration in the remaining procedure.

1. Create a gardenlet configuration gardenlet-values.yaml based on this template.

2. Create a bootstrap kubeconfig based on the bootstrap token created in the garden cluster.

   Replace the <bootstrap-token> with token-id.token-secret (from our previous example: 07401b.f395accd246ae52d) from the bootstrap token secret.

   apiVersion: v1
   kind: Config
   current-context: gardenlet-bootstrap@default
   clusters:
   - cluster:
       certificate-authority-data: <ca-of-garden-cluster>
       server: https://<endpoint-of-garden-cluster>
     name: default
   contexts:
   - context:
       cluster: default
       user: gardenlet-bootstrap
     name: gardenlet-bootstrap@default
   users:
   - name: gardenlet-bootstrap
     user:
       token: <bootstrap-token>
   

3. In section gardenClientConnection.bootstrapKubeconfig of your gardenlet configuration, provide the bootstrap kubeconfig together with a name and namespace to the gardenlet Helm chart.

   gardenClientConnection:
     bootstrapKubeconfig:
       name: gardenlet-kubeconfig-bootstrap
       namespace: garden
       kubeconfig: |
               <bootstrap-kubeconfig>  # will be base64 encoded by helm
   

   The bootstrap kubeconfig is stored in the specified secret.

4. In section gardenClientConnection.kubeconfigSecret of your gardenlet configuration, define a name and a namespace where the gardenlet stores the real kubeconfig that it creates during the bootstrap process. If the secret doesn’t exist, the gardenlet creates it for you.

   gardenClientConnection:
     kubeconfigSecret:
       name: gardenlet-kubeconfig
       namespace: garden
   

Updating the garden cluster CA

The kubeconfig created by the gardenlet in step 4 will not be recreated as long as it exists, even if a new bootstrap kubeconfig is provided. To enable rotation of the garden cluster CA certificate, a new bundle can be provided via the gardenClientConnection.gardenClusterCACert field. If the provided bundle differs from the one currently in the gardenlet’s kubeconfig secret then it will be updated. To remove the CA completely (e.g. when switching to a publicly trusted endpoint) this field can be set to either none or null.

Automatically register shoot cluster as a seed cluster

A seed cluster can either be registered by manually creating the Seed resource or automatically by the gardenlet. This functionality is useful for managed seed clusters, as the gardenlet in the garden cluster deploys a copy of itself into the cluster with automatic registration of the Seed configured. However, it can also be used to have a streamlined seed cluster registration process when manually deploying the gardenlet.

This procedure doesn’t describe all the possible configurations for the Seed resource. For more information, see:

• Example Seed resource
• Configurable Seed settings.

Adjust the gardenlet component configuration

1. Supply the Seed resource in section seedConfig of your gardenlet configuration gardenlet-values.yaml.

2. Add the seedConfig to your gardenlet configuration gardenlet-values.yaml. The field seedConfig.spec.provider.type specifies the infrastructure provider type (for example, aws) of the seed cluster. For all supported infrastructure providers, see Known Extension Implementations.

   ....
   seedConfig:
     metadata:
       name: sweet-seed
     spec:
       dns:
         ingressDomain: ingress.sweet-seed.<my-domain> # see prerequisites
       networks: # see prerequisites
         nodes: 10.240.0.0/16
         pods: 100.244.0.0/16
         services: 100.32.0.0/13
         shootDefaults: # optional: non-overlapping default CIDRs for shoot clusters of that Seed
           pods: 100.96.0.0/11
           services: 100.64.0.0/13
       provider:
         region: eu-west-1
         type: <provider>
   

Optional: Enable backup and restore

The seed cluster can be set up with backup and restore for the main etcds of shoot clusters.

Gardener uses etcd-backup-restore that integrates with different storage providers to store the shoot cluster’s main etcd backups. Make sure to obtain client credentials that have sufficient permissions with the chosen storage provider.

Create a secret in the garden cluster with client credentials for the storage provider. The format of the secret is cloud provider specific and can be found in the repository of the respective Gardener extension. For example, the secret for AWS S3 can be found in the AWS provider extension (30-etcd-backup-secret.yaml).

apiVersion: v1
kind: Secret
metadata:
  name: sweet-seed-backup
  namespace: garden
type: Opaque
data:
  # client credentials format is provider specific

Configure the Seed resource in section seedConfig of your gardenlet configuration to use backup and restore:

...
seedConfig:
  metadata:
    name: sweet-seed
  spec:
    backup:
      provider: <provider>
      secretRef:
        name: sweet-seed-backup
        namespace: garden

Deploy the gardenlet

The gardenlet doesn’t have to run in the same Kubernetes cluster as the seed cluster it’s registering and reconciling, but it is in most cases advantageous to use in-cluster communication to talk to the Seed API server. Running a gardenlet outside of the cluster is mostly used for local development.

The gardenlet-values.yaml looks something like this (with automatic Seed registration and backup for shoot clusters enabled):

global:
  # Gardenlet configuration values
  gardenlet:
    enabled: true
    ...
    <default config>
    ...
    config:
      gardenClientConnection:
        ...
        bootstrapKubeconfig:
          name: gardenlet-bootstrap-kubeconfig
          namespace: garden
          kubeconfig: |
            apiVersion: v1
            clusters:
            - cluster:
                certificate-authority-data: <dummy>
                server: <my-garden-cluster-endpoint>
              name: my-kubernetes-cluster
            ....            

        kubeconfigSecret:
          name: gardenlet-kubeconfig
          namespace: garden
      ...
      <default config>
      ...
      seedConfig:
        metadata:
          name: sweet-seed
        spec:
          dns:
            ingressDomain: ingress.sweet-seed.<my-domain>
          networks:
            nodes: 10.240.0.0/16
            pods: 100.244.0.0/16
            services: 100.32.0.0/13
            shootDefaults:
              pods: 100.96.0.0/11
              services: 100.64.0.0/13
          provider:
            region: eu-west-1
            type: <provider>
          backup:
            provider: <provider>
            secretRef:
              name: sweet-seed-backup
              namespace: garden

Deploy the gardenlet Helm chart to the Kubernetes cluster.

helm install gardenlet charts/gardener/gardenlet \
  --namespace garden \
  -f gardenlet-values.yaml \
  --wait

This helm chart creates:

• A service account gardenlet that the gardenlet can use to talk to the Seed API server.
• RBAC roles for the service account (full admin rights at the moment).
• The secret (garden/gardenlet-bootstrap-kubeconfig) containing the bootstrap kubeconfig.
• The gardenlet deployment in the garden namespace.

Check that the gardenlet is successfully deployed

1. Check that the gardenlets certificate bootstrap was successful.

   Check if the secret gardenlet-kubeconfig in the namespace garden in the seed cluster is created and contains a kubeconfig with a valid certificate.

   1. Get the kubeconfig from the created secret.

      $ kubectl -n garden get secret gardenlet-kubeconfig -o json | jq -r .data.kubeconfig | base64 -d
      

   2. Test against the garden cluster and verify it’s working.

   3. Extract the client-certificate-data from the user gardenlet.

   4. View the certificate:

      $ openssl x509 -in ./gardenlet-cert -noout -text
      

      Check that the certificate is valid for a year (that is the lifetime of new certificates).

2. Check that the bootstrap secret gardenlet-bootstrap-kubeconfig has been deleted from the seed cluster in namespace garden.

3. Check that the seed cluster is registered and READY in the garden cluster.

   Check that the seed cluster sweet-seed exists and all conditions indicate that it’s available. If so, the Gardenlet is sending regular heartbeats and the seed bootstrapping was successful.

   Check that the conditions on the Seed resource look similar to the following:

   $ kubectl get seed sweet-seed -o json | jq .status.conditions
   [
     {
       "lastTransitionTime": "2020-07-17T09:17:29Z",
       "lastUpdateTime": "2020-07-17T09:17:29Z",
       "message": "Gardenlet is posting ready status.",
       "reason": "GardenletReady",
       "status": "True",
       "type": "GardenletReady"
     },
     {
       "lastTransitionTime": "2020-07-17T09:17:49Z",
       "lastUpdateTime": "2020-07-17T09:53:17Z",
       "message": "Seed cluster has been bootstrapped successfully.",
       "reason": "BootstrappingSucceeded",
       "status": "True",
       "type": "Bootstrapped"
     },
     {
       "lastTransitionTime": "2020-07-17T09:17:49Z",
       "lastUpdateTime": "2020-07-17T09:53:17Z",
       "message": "Backup Buckets are available.",
       "reason": "BackupBucketsAvailable",
       "status": "True",
       "type": "BackupBucketsReady"
     }
   ]
   

Related Links

Issue #1724: Harden Gardenlet RBAC privileges.

Backup and Restore.

6 - Feature Gates

Feature Gates in Gardener

This page contains an overview of the various feature gates an administrator can specify on different Gardener components.

Overview

Feature gates are a set of key=value pairs that describe Gardener features. You can turn these features on or off using the a component configuration file for a specific component.

Each Gardener component lets you enable or disable a set of feature gates that are relevant to that component. For example this is the configuration of the gardenlet component.

The following tables are a summary of the feature gates that you can set on different Gardener components.

• The “Since” column contains the Gardener release when a feature is introduced or its release stage is changed.
• The “Until” column, if not empty, contains the last Gardener release in which you can still use a feature gate.
• If a feature is in the Alpha or Beta state, you can find the feature listed in the Alpha/Beta feature gate table.
• If a feature is stable you can find all stages for that feature listed in the Graduated/Deprecated feature gate table.
• The Graduated/Deprecated feature gate table also lists deprecated and withdrawn features.

Feature gates for Alpha or Beta features

Feature gates for graduated or deprecated features

Using a feature

A feature can be in Alpha, Beta or GA stage. An Alpha feature means:

• Disabled by default.
• Might be buggy. Enabling the feature may expose bugs.
• Support for feature may be dropped at any time without notice.
• The API may change in incompatible ways in a later software release without notice.
• Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.

A Beta feature means:

• Enabled by default.
• The feature is well tested. Enabling the feature is considered safe.
• Support for the overall feature will not be dropped, though details may change.
• The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
• Recommended for only non-critical uses because of potential for incompatible changes in subsequent releases.

Please do try Beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.

A General Availability (GA) feature is also referred to as a stable feature. It means:

• The feature is always enabled; you cannot disable it.
• The corresponding feature gate is no longer needed.
• Stable versions of features will appear in released software for many subsequent versions.

List of Feature Gates

7 - Gardenlet API Access

Scoped API Access for Gardenlets

By default, gardenlets have administrative access in the garden cluster. They are able to execute any API request on any object independent of whether the object is related to the seed cluster the gardenlet is responsible for. As RBAC is not powerful enough for fine-grained checks and for the sake of security, Gardener provides two optional but recommended configurations for your environments that scope the API access for gardenlets.

Similar to the Node authorization mode in Kubernetes, Gardener features a SeedAuthorizer plugin. It is a special-purpose authorization plugin that specifically authorizes API requests made by the gardenlets.

Likewise, similar to the NodeRestriction admission plugin in Kubernetes, Gardener features a SeedRestriction plugin. It is a special-purpose admission plugin that specifically limits the Kubernetes objects gardenlets can modify.

📚 You might be interested to look into the design proposal for scoped Kubelet API access from the Kubernetes community. It can be translated to Gardener and Gardenlets with their Seed and Shoot resources.

Flow Diagram

The following diagram shows how the two plugins are included in the request flow of a gardenlet. When they are not enabled then the kube-apiserver is internally authorizing the request via RBAC before forwarding the request directly to the gardener-apiserver, i.e., the gardener-admission-controller would not be consulted (this is not entirely correct because it also serves other admission webhook handlers, but for simplicity reasons this document focuses on the API access scope only).

When enabling the plugins, there is one additional step for each before the gardener-apiserver responds to the request.

Please note that the example shows a request to an object (Shoot) residing in one of the API groups served by gardener-apiserver. However, the gardenlet is also interacting with objects in API groups served by the kube-apiserver (e.g., Secret,ConfigMap, etc.). In this case, the consultation of the SeedRestriction admission plugin is performed by the kube-apiserver itself before it forwards the request to the gardener-apiserver.

Today, the following rules are implemented:

[1] If you use ManagedSeed resources then the gardenlet reconciling them (“parent gardenlet”) may be allowed to submit certain requests for the Seed resources resulting out of such ManagedSeed reconciliations (even if the “parent gardenlet” is not responsible for them):

• ℹ️ It is allowed to delete the Seed resources if the corresponding ManagedSeed objects already have a deletionTimestamp (this is secure as gardenlets themselves don’t have permissions for deleting ManagedSeeds).
• ⚠ It is allowed to create or update Seed resources if the corresponding ManagedSeed objects use a seed template, i.e., .spec.seedTemplate != nil. In this case, there is at least one gardenlet in your system which is responsible for two or more Seeds. Please keep in mind that this use case is not recommended for production scenarios (you should only have one dedicated gardenlet per seed cluster), hence, the security improvements discussed in this document might be limited.

SeedAuthorizer Authorization Webhook Enablement

The SeedAuthorizer is implemented as Kubernetes authorization webhook and part of the gardener-admission-controller component running in the garden cluster.

🎛 In order to activate it, you have to follow these steps:

1. Set the following flags for the kube-apiserver of the garden cluster (i.e., the kube-apiserver whose API is extended by Gardener):

   • --authorization-mode=RBAC,Node,Webhook (please note that Webhook should appear after RBAC in the list [1]; Node might not be needed if you use a virtual garden cluster)
   • --authorization-webhook-config-file=<path-to-the-webhook-config-file>
   • --authorization-webhook-cache-authorized-ttl=0
   • --authorization-webhook-cache-unauthorized-ttl=0

2. The webhook config file (stored at <path-to-the-webhook-config-file>) should look as follows:

   apiVersion: v1
   kind: Config
   clusters:
   - name: garden
     cluster:
       certificate-authority-data: base64(CA-CERT-OF-GARDENER-ADMISSION-CONTROLLER)
       server: https://gardener-admission-controller.garden/webhooks/auth/seed
   users:
   - name: kube-apiserver
     user: {}
   contexts:
   - name: auth-webhook
     context:
       cluster: garden
       user: kube-apiserver
   current-context: auth-webhook
   

3. When deploying the Gardener controlplane Helm chart, set .global.rbac.seedAuthorizer.enabled=true. This will prevent that the RBAC resources granting global access for all gardenlets will be deployed.

4. Delete the existing RBAC resources granting global access for all gardenlets by running:

   kubectl delete \
     clusterrole.rbac.authorization.k8s.io/gardener.cloud:system:seeds \
     clusterrolebinding.rbac.authorization.k8s.io/gardener.cloud:system:seeds \
     --ignore-not-found
   

Please note that you should activate the SeedRestriction admission handler as well.

[1] The reason for the fact that Webhook authorization plugin should appear after RBAC is that the kube-apiserver will be depending on the gardener-admission-controller (serving the webhook). However, the gardener-admission-controller can only start when gardener-apiserver runs, but gardener-apiserver itself can only start when kube-apiserver runs. If Webhook is before RBAC then gardener-apiserver might not be able to start, leading to a deadlock.

Authorizer Decisions

As mentioned earlier, it’s the authorizer’s job to evaluate API requests and return one of the following decisions:

• DecisionAllow: The request is allowed, further configured authorizers won’t be consulted.
• DecisionDeny: The request is denied, further configured authorizers won’t be consulted.
• DecisionNoOpinion: A decision cannot be made, further configured authorizers will be consulted.

For backwards compatibility, no requests are denied at the moment, so that they are still deferred to a subsequent authorizer like RBAC. Though, this might change in the future.

First, the SeedAuthorizer extracts the Seed name from the API request. This requires a proper TLS certificate the gardenlet uses to contact the API server and is automatically given if TLS bootstrapping is used. Concretely, the authorizer checks the certificate for name gardener.cloud:system:seed:<seed-name> and group gardener.cloud:system:seeds. In cases where this information is missing e.g., when a custom Kubeconfig is used, the authorizer cannot make any decision. Thus, RBAC is still a considerable option to restrict the gardenlet’s access permission if the above explained preconditions are not given.

With the Seed name at hand, the authorizer checks for an existing path from the resource that a request is being made for to the Seed belonging to the gardenlet. Take a look at the Implementation Details section for more information.

Implementation Details

Internally, the SeedAuthorizer uses a directed, acyclic graph data structure in order to efficiently respond to authorization requests for gardenlets:

• A vertex in this graph represents a Kubernetes resource with its kind, namespace, and name (e.g., Shoot:garden-my-project/my-shoot).
• An edge from vertex u to vertex v in this graph exists when
  • (1) v is referred by u and v is a Seed, or when
  • (2) u is referred by v, or when
  • (3) u is strictly associated with v.

For example, a Shoot refers to a Seed, a CloudProfile, a SecretBinding, etc., so it has an outgoing edge to the Seed (1) and incoming edges from the CloudProfile and SecretBinding vertices (2). However, there might also be a ShootState or a BackupEntry resource strictly associated with this Shoot, hence, it has incoming edges from these vertices (3).

In above picture the resources that are actively watched have are shaded. Gardener resources are green while Kubernetes resources are blue. It shows the dependencies between the resources and how the graph is built based on above rules.

ℹ️ Above picture shows all resources that may be accessed by gardenlets, except for the Quota resource which is only included for completeness.

Now, when a gardenlet wants to access certain resources then the SeedAuthorizer uses a Depth-First traversal starting from the vertex representing the resource in question, e.g., from a Project vertex. If there is a path from the Project vertex to the vertex representing the Seed the gardenlet is responsible for then it allows the request.

Metrics

The SeedAuthorizer registers the following metrics related to the mentioned graph implementation:

Debug Handler

When the .server.enableDebugHandlers field in the gardener-admission-controller’s component configuration is set to true then it serves a handler that can be used for debugging the resource dependency graph under /debug/resource-dependency-graph.

🚨 Only use this setting for development purposes as it enables unauthenticated users to view all data if they have access to the gardener-admission-controller component.

The handler renders an HTML page displaying the current graph with a list of vertices and its associated incoming and outgoing edges to other vertices. Depending on the size of the Gardener landscape (and consequently, the size of the graph), it might not be possible to render it in its entirety. If there are more than 2000 vertices then the default filtering will selected for kind=Seed to prevent overloading the output.

Example output:

-------------------------------------------------------------------------------
|
| # Seed:my-seed
|   <- (11)
|     BackupBucket:73972fe2-3d7e-4f61-a406-b8f9e670e6b7
|     BackupEntry:garden-my-project/shoot--dev--my-shoot--4656a460-1a69-4f00-9372-7452cbd38ee3
|     ControllerInstallation:dns-external-mxt8m
|     ControllerInstallation:extension-shoot-cert-service-4qw5j
|     ControllerInstallation:networking-calico-bgrb2
|     ControllerInstallation:os-gardenlinux-qvb5z
|     ControllerInstallation:provider-gcp-w4mvf
|     Secret:garden/backup
|     Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------
|
| # Shoot:garden-my-project/my-shoot
|   <- (5)
|     CloudProfile:gcp
|     Namespace:garden-my-project
|     Secret:garden-my-project/my-dns-secret
|     SecretBinding:garden-my-project/my-credentials
|     ShootState:garden-my-project/my-shoot
|   -> (1)
|     Seed:my-seed
|
-------------------------------------------------------------------------------
|
| # ShootState:garden-my-project/my-shoot
|   -> (1)
|     Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------

... (etc., similarly for the other resources)

There are anchor links to easily jump from one resource to another, and the page provides means for filtering the results based on the kind, namespace, and/or name.

Pitfalls

When there is a relevant update to an existing resource, i.e., when a reference to another resource is changed, then the corresponding vertex (along with all associated edges) is first deleted from the graph before it gets added again with the up-to-date edges. However, this does only work for vertices belonging to resources that are only created in exactly one “watch handler”. For example, the vertex for a SecretBinding can either be created in the SecretBinding handler itself or in the Shoot handler. In such cases, deleting the vertex before (re-)computing the edges might lead to race conditions and potentially renders the graph invalid. Consequently, instead of deleting the vertex, only the edges the respective handler is responsible for are deleted. If the vertex ends up with no remaining edges then it also gets deleted automatically. Afterwards, the vertex can either be added again or the updated edges can be created.

SeedRestriction Admission Webhook Enablement

The SeedRestriction is implemented as Kubernetes admission webhook and part of the gardener-admission-controller component running in the garden cluster.

🎛 In order to activate it, you have to set .global.admission.seedRestriction.enabled=true when using the Gardener controlplane Helm chart. This will add an additional webhook in the existing ValidatingWebhookConfiguration of the gardener-admission-controller which contains the configuration for the SeedRestriction handler. Please note that it should only be activated when the SeedAuthorizer is active as well.

Admission Decisions

The admission’s purpose is to perform extended validation on requests which require the body of the object in question. Additionally, it handles CREATE requests of gardenlets (above discussed resource dependency graph cannot be used in such cases because there won’t be any vertex/edge for non-existing resources).

Gardenlets are restricted to only create new resources which are somehow related to the seed clusters they are responsible for.

8 - Getting Started Locally

Deploying Gardener locally

This document will walk you through deploying Gardener on your local machine. If you encounter difficulties, please open an issue so that we can make this process easier.

Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as both garden and seed cluster (please refer to the architecture overview) for simplicity.

Based on Skaffold, the container images for all required components will be built and deployed into the cluster (via their Helm charts).

Prerequisites

• Make sure that you have followed the Local Setup guide up until the Get the sources step.
• Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least 8 CPUs and 8Gi memory; see here how to configure the resources for Docker for Mac).

  Please note that 8 CPU / 8Gi memory might not be enough for more than two Shoot clusters, i.e., you might need to increase these values if you want to run additional Shoots. If you plan on following the optional steps to create a second seed cluster, the required resources will be more - at least 10 CPUs and 18Gi memory. Additionally, please configure at least 120Gi of disk size for the Docker daemon. Tip: With docker system df and docker system prune -a you can cleanup unused data.

Setting up the KinD cluster (garden and seed)

make kind-up

This command sets up a new KinD cluster named gardener-local and stores the kubeconfig in the ./example/gardener-local/kind/kubeconfig file.

It might be helpful to copy this file to $HOME/.kube/config since you will need to target this KinD cluster multiple times. Alternatively, make sure to set your KUBECONFIG environment variable to ./example/gardener-local/kind/kubeconfig for all future steps via export KUBECONFIG=example/gardener-local/kind/kubeconfig.

All following steps assume that you are using this kubeconfig.

Additionally, this command also deploys a local container registry to the cluster as well as a few registry mirrors, that are set up as a pull-through cache for all upstream registries Gardener uses by default. This is done to speed up image pulls across local clusters. The local registry can be accessed as localhost:5001 for pushing and pulling. The storage directories of the registries are mounted to the host machine under dev/local-registry. With this, mirrored images don’t have to be pulled again after recreating the cluster.

The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy support (the default kindnet CNI doesn’t provide NetworkPolicy support). Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.

Setting up Gardener

make gardener-up

This will first build the images based (which might take a bit if you do it for the first time). Afterwards, the Gardener resources will be deployed into the cluster.

Creating a Shoot cluster

You can wait for the Seed to be ready by running

kubectl wait --for=condition=gardenletready --for=condition=extensionsready --for=condition=bootstrapped seed local --timeout=5m

Alternatively, you can run kubectl get seed local and wait for the STATUS to indicate readiness:

NAME    STATUS   PROVIDER   REGION   AGE     VERSION       K8S VERSION
local   Ready    local      local    4m42s   vX.Y.Z-dev    v1.21.1

In order to create a first shoot cluster, just run

kubectl apply -f example/provider-local/shoot.yaml

You can wait for the Shoot to be ready by running

kubectl wait --for=condition=apiserveravailable --for=condition=controlplanehealthy --for=condition=everynodeready --for=condition=systemcomponentshealthy shoot local -n garden-local --timeout=10m

Alternatively, you can run kubectl -n garden-local get shoot local and wait for the LAST OPERATION to reach 100%:

NAME    CLOUDPROFILE   PROVIDER   REGION   K8S VERSION   HIBERNATION   LAST OPERATION            STATUS    AGE
local   local          local      local    1.21.0        Awake         Create Processing (43%)   healthy   94s

(Optional): You could also execute a simple e2e test (creating and deleting a shoot) by running

make test-e2e-local-simple KUBECONFIG="$PWD/example/gardener-local/kind/kubeconfig"

Accessing the Shoot cluster

⚠️ Please note that in this setup shoot clusters are not accessible by default when you download the kubeconfig and try to communicate with them. The reason is that your host most probably cannot resolve the DNS names of the clusters since provider-local extension runs inside the KinD cluster (see this for more details). Hence, if you want to access the shoot cluster, you have to run the following command which will extend your /etc/hosts file with the required information to make the DNS names resolvable:

cat <<EOF | sudo tee -a /etc/hosts

# Manually created to access local Gardener shoot clusters.
# TODO: Remove this again when the shoot cluster access is no longer required.
127.0.0.1 api.local.local.external.local.gardener.cloud
127.0.0.1 api.local.local.internal.local.gardener.cloud

127.0.0.1 api.e2e-managedseed.garden.external.local.gardener.cloud
127.0.0.1 api.e2e-managedseed.garden.internal.local.gardener.cloud
127.0.0.1 api.e2e-hibernated.local.external.local.gardener.cloud
127.0.0.1 api.e2e-hibernated.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-unpriv.local.external.local.gardener.cloud
127.0.0.1 api.e2e-unpriv.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-wake-up.local.external.local.gardener.cloud
127.0.0.1 api.e2e-wake-up.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-migrate.local.external.local.gardener.cloud
127.0.0.1 api.e2e-migrate.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-rotate.local.external.local.gardener.cloud
127.0.0.1 api.e2e-rotate.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-default.local.external.local.gardener.cloud
127.0.0.1 api.e2e-default.local.internal.local.gardener.cloud
EOF

Now you can access it by running

kubectl -n garden-local get secret local.kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > /tmp/kubeconfig-shoot-local.yaml
kubectl --kubeconfig=/tmp/kubeconfig-shoot-local.yaml get nodes

(Optional): Setting up a second seed cluster

There are cases where you would want to create a second cluster seed in your local setup. For example, if you want to test the control plane migration feature. The following steps describe how to do that.

make kind2-up

This command sets up a new KinD cluster named gardener-local2 and stores its kubeconfig in the ./example/gardener-local/kind2/kubeconfig file.

In order to deploy required resources in the KinD cluster that you just created, run:

make gardenlet-kind2-up

The following steps assume that your are using the kubeconfig that points to the gardener-local cluster (first KinD cluster): export KUBECONFIG=example/gardener-local/kind/kubeconfig.

You can wait for the local2 Seed to be ready by running:

kubectl wait --for=condition=gardenletready --for=condition=extensionsready --for=condition=bootstrapped seed local2 --timeout=5m

Alternatively, you can run kubectl get seed local2 and wait for the STATUS to indicate readiness:

NAME    STATUS   PROVIDER   REGION   AGE     VERSION       K8S VERSION
local2  Ready    local      local    4m42s   vX.Y.Z-dev    v1.21.1

If you want to perform control plane migration you can follow the steps outlined here to migrate the shoot cluster to the second seed you just created.

Deleting the Shoot cluster

./hack/usage/delete shoot local garden-local

(Optional): Tear down the second seed cluster

make kind2-down

Tear down the Gardener environment

make kind-down

Further reading

This setup makes use of the local provider extension. You can read more about it in this document.

9 - Image Vector

Image Vector

The Gardenlet is deploying several different container images into the seed and the shoot clusters. The image repositories and tags are defined in a central image vector file. Obviously, the image versions defined there must fit together with the deployment manifests (e.g., some command-line flags do only exist in certain versions).

Example

images:
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: gcr.io/google_containers/pause-amd64
  tag: "3.0"
  version: 1.17.x
  architectures:
  - amd64
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: gcr.io/google_containers/pause-amd64
  tag: "3.1"
  version: ">= 1.18"
  architectures:
  - amd64
...

That means that the Gardenlet will use the pause-container in with tag 3.0 for all seed/shoot clusters with Kubernetes version 1.17.x, and tag 3.1 for all clusters with Kubernetes >= 1.18.

Image Vector Architecture

images:
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: registry.k8s.io/pause
  tag: "3.0"
  architectures:
  - amd64
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: registry.k8s.io/pause
  tag: "3.0"
  architectures:
  - arm64
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: registry.k8s.io/pause
  tag: "3.0"
  architectures:
  - amd64
  - arm64
...

Architectures is an optional field of image. It is a list of strings specifying CPU architecture of machines on which this image can be used. The valid options for architectures field are as follow:

• amd64 : This sepecifies image can run only on machines having CPU architecture amd64.
• arm64 : This sepecifies image can run only on machines having CPU architecture arm64.

If image doesn’t specifies any architectures then by default it is considered to support both amd64 and arm64 architectures.

Overwrite image vector

In some environment it is not possible to use these “pre-defined” images that come with a Gardener release. A prominent example for that is Alicloud in China which does not allow access to Google’s GCR. In these cases you might want to overwrite certain images, e.g., point the pause-container to a different registry.

⚠️ If you specify an image that does not fit to the resource manifest then the seed/shoot reconciliation might fail.

In order to overwrite the images you must provide a similar file to Gardenlet:

images:
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: my-custom-image-registry/pause-amd64
  tag: "3.0"
  version: 1.17.x
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: my-custom-image-registry/pause-amd64
  tag: "3.1"
  version: ">= 1.18"
...

During deployment of the gardenlet create a ConfigMap containing the above content and mount it as a volume into the gardenlet pod. Next, specify the environment variable IMAGEVECTOR_OVERWRITE whose value must be the path to the file you just mounted:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gardenlet-images-overwrite
  namespace: garden
data:
  images_overwrite.yaml: |
    images:
    - ...    
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gardenlet
  namespace: garden
spec:
  template:
    ...
    spec:
      containers:
      - name: gardenlet
        env:
        - name: IMAGEVECTOR_OVERWRITE
          value: /charts-overwrite/images_overwrite.yaml
        volumeMounts:
        - name: gardenlet-images-overwrite
          mountPath: /charts-overwrite
        ...
      volumes:
      - name: gardenlet-images-overwrite
        configMap:
          name: gardenlet-images-overwrite
  ...

Image vectors for dependent components

The gardenlet is deploying a lot of different components that might deploy other images themselves. These components might use an image vector as well. Operators might want to customize the image locations for these transitive images as well, hence, they might need to specify an image vector overwrite for the components directly deployed by Gardener.

It is possible to specify the IMAGEVECTOR_OVERWRITE_COMPONENTS environment variable to the gardenlet that points to a file with the following content:

components:
- name: etcd-druid
  imageVectorOverwrite: |
    images:
    - name: etcd
      tag: v1.2.3
      repository: etcd/etcd    
...

The gardenlet will, if supported by the directly deployed component (etcd-druid in this example), inject the given imageVectorOverwrite into the Deployment manifest. The respective component is responsible for using the overwritten images instead of its defaults.

10 - Migration V0 To V1

Migration from Gardener v0 to v1

Please refer to the document for older Gardener versions.

11 - Secret Binding Provider Controller

SecretBinding Provider Controller

This page describes the process on how to enable the SecretBinding provider controller.

Overview

With Gardener v1.38.0 the SecretBinding resource does now contain a new optional field .provider.type (details about the motivation can be found in https://github.com/gardener/gardener/issues/4888). To make the process of setting the new field automated and afterwards to enforce validation on the new field in backwards compatible manner, Gardener features the SecretBinding provider controller and a feature gate - SecretBindingProviderValidation.

Process

A Gardener landscape operator can follow the following steps:

1. Enable the SecretBinding provider controller of Gardener Controller Manager.

   The SecretBinding provider controller is responsible to populate the .provider.type field of a SecretBinding based on its current usage by Shoot resources. For example if a Shoot crazy-botany with .provider.type=aws is using a SecretBinding my-secret-binding, then the SecretBinding provider controller will take care to set the .provider.type field of the SecretBinding to the same provider type (aws). To enable the SecretBinding provider controller, in the ControllerManagerConfiguration set the controller.secretBindingProvider.concurentSyncs field (e.g set it to 5). Although that it is not recommended, the API allows Shoots from different provider types to reference the same SecretBinding (assuming that backing Secret contains data for both of the provider types). To preserve the backwards compatibility for such SecretBindings, the provider controller will maintain the multiple provider types in the field (it will join them with separator , - for example aws,gcp).

2. Disable the SecretBinding provider controller and enable SecretBindingProviderValidation feature gate of Gardener API server.

   The SecretBindingProviderValidation feature gate of Gardener API server enables set of validations for the SecretBinding provider field. It forbids creating a Shoot that has a different provider type from the referenced SecretBinding’s one. It also enforces immutability on the field. After making sure that SecretBinding provider controller is enabled and it populated the .provider.type field of a majority of the SecretBindings on a Gardener landscape (the SecretBindings that are unused will have their provider type unset), a Gardener landscape operator has to disable the SecretBinding provider controller and to enable the SecretBindingProviderValidation feature gate of Gardener API server. To disable the SecretBinding provider controller, in the ControllerManagerConfiguration set the controller.secretBindingProvider.concurentSyncs field to 0.

Implementation History

• Gardener v1.38: The SecretBinding resource has a new optional field .provider.type. The SecretBinding provider controller is disabled by default. The SecretBindingProviderValidation feature gate of Gardener API server is disabled by default.
• Gardener v1.42: The SecretBinding provider controller is enabled by default.
• Gardener v1.51: The SecretBindingProviderValidation feature gate of Gardener API server is enabled by default and the SecretBinding provider controller is disabled by default.
• Gardener v1.53: The SecretBindingProviderValidation feature gate of Gardener API server is unconditionally enabled (can no longer be disabled).
• Gardener v1.55: The SecretBindingProviderValidation feature gate of Gardener API server and the SecretBinding provider controller are removed.

12 - Setup Gardener

Deploying the Gardener into a Kubernetes cluster

Similar to Kubernetes, Gardener consists out of control plane components (Gardener API server, Gardener controller manager, Gardener scheduler), and an agent component (Gardenlet). The control plane is deployed in the so-called garden cluster while the agent is installed into every seed cluster. Please note that it is possible to use the garden cluster as seed cluster by simply deploying the Gardenlet into it.

We are providing Helm charts in order to manage the various resources of the components. Please always make sure that you use the Helm chart version that matches the Gardener version you want to deploy.

Deploying the Gardener control plane (API server, admission controller, controller manager, scheduler)

The configuration values depict the various options to configure the different components. Please consult this document for component specific configurations and this document for authentication related specifics.

Also note that all resources and deployments need to be created in the garden namespace (not overrideable). If you enable the Gardener admission controller as part of you setup, please make sure the garden namespace is labelled with app: gardener. Otherwise, the backing service account for the admission controller Pod might not be created successfully. No action is necessary, if you deploy the garden namespace with the Gardener control plane Helm chart.

After preparing your values in a separate controlplane-values.yaml file (values.yaml can be used as starting point), you can run the following command against your garden cluster:

helm install charts/gardener/controlplane \
  --namespace garden \
  --name gardener-controlplane \
  -f controlplane-values.yaml \
  --wait

Deploying Gardener extensions

Gardener is an extensible system that does not contain the logic for provider-specific things like DNS management, cloud infrastructures, network plugins, operating system configs, and many more.

You have to install extension controllers for these parts. Please consult the documentation regarding extensions to get more information.

Deploying the Gardener agent (Gardenlet)

Please refer to this document on how to deploy a Gardenlet.

13 - Version Skew Policy

Version Skew Policy

This document describes the maximum version skew supported between various Gardener components.

Supported Gardener Versions

Gardener versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version, following Semantic Versioning terminology.

The Gardener project maintains release branches for the most recent three minor releases.

Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility. Patch releases are cut from those branches at a regular cadence, plus additional urgent releases when required.

For more information, see this document.

Supported Version Skew

Technically, we follow the same policy as the Kubernetes project. However, given that our release cadence is much more frequent compared to Kubernetes (every 14d vs. every 120d), in many cases it is possible to skip a version. Still, to be on the safe side, it is highly recommended to follow the described policy.

gardener-apiserver

In multi-instance setups of Gardener, the newest and oldest gardener-apiserver instances must be within one minor version.

Example:

• newest gardener-apiserver is at 1.37
• other gardener-apiserver instances are supported at 1.37 and v1.36

gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet

gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet must not be newer than the gardener-apiserver instances they communicate with. They are expected to match the gardener-apiserver minor version, but may be up to one minor version older (to allow live upgrades).

Example:

• gardener-apiserver is at v1.37
• gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet are supported at 1.37 and v1.36

Supported Component Upgrade Order

The supported version skew between components has implications on the order in which components must be upgraded. This section describes the order in which components must be upgraded to transition an existing Gardener installation from version 1.37 to version 1.38.

gardener-apiserver

Pre-requisites:

• In a single-instance setup, the existing gardener-apiserver instance is 1.37
• In a multi-instance setup, all gardener-apiserver instances are at 1.37 or 1.38 (this ensures maximum skew of 1 minor version between the oldest and newest gardener-apiserver instance)
• The gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet instances that communicate with this gardener-apiserver are at version 1.37 (this ensures they are not newer than the existing API server version and are within 1 minor version of the new API server version)

Action:

• Upgrade gardener-apiserver to 1.38

gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet

Pre-requisites:

• The gardener-apiserver instances these components communicate with are at 1.38 (in multi-instance setups in which these components can communicate with any gardener-apiserver instance in the cluster, all gardener-apiserver instances must be upgraded before upgrading these components)

Action:

• Upgrade gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet to 1.38

Supported Kubernetes Versions

Please refer to this document.