Deployment

1 - Authentication Gardener Control Plane

Authentication of Gardener Control Plane Components Against the Garden Cluster

Note: This document refers to Gardener’s API server, admission controller, controller manager and scheduler components. Any reference to the term Gardener control plane component can be replaced with any of the mentioned above.

There are several authentication possibilities depending on whether or not the concept of Virtual Garden is used.

Virtual Garden is not used, i.e., the runtime Garden cluster is also the target Garden cluster.

Automounted Service Account Token

The easiest way to deploy a Gardener control plane component is to not provide a kubeconfig at all. This way in-cluster configuration and an automounted service account token will be used. The drawback of this approach is that the automounted token will not be automatically rotated.

Service Account Token Volume Projection

Another solution is to use Service Account Token Volume Projection combined with a kubeconfig referencing a token file (see the example below).

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <CA-DATA>
    server: https://default.kubernetes.svc.cluster.local
  name: garden
contexts:
- context:
    cluster: garden
    user: garden
  name: garden
current-context: garden
users:
- name: garden
  user:
    tokenFile: /var/run/secrets/projected/serviceaccount/token

This will allow for automatic rotation of the service account token by the kubelet. The configuration can be achieved by setting both .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true and .Values.global.<GardenerControlPlaneComponent>.kubeconfig in the respective chart’s values.yaml file.

Virtual Garden is used, i.e., the runtime Garden cluster is different from the target Garden cluster.

Service Account

The easiest way to setup the authentication is to create a service account and the respective roles will be bound to this service account in the target cluster. Then use the generated service account token and craft a kubeconfig, which will be used by the workload in the runtime cluster. This approach does not provide a solution for the rotation of the service account token. However, this setup can be achieved by setting .Values.global.deployment.virtualGarden.enabled: true and following these steps:

1. Deploy the application part of the charts in the target cluster.
2. Get the service account token and craft the kubeconfig.
3. Set the crafted kubeconfig and deploy the runtime part of the charts in the runtime cluster.

Client Certificate

Another solution is to bind the roles in the target cluster to a User subject instead of a service account and use a client certificate for authentication. This approach does not provide a solution for the client certificate rotation. However, this setup can be achieved by setting both .Values.global.deployment.virtualGarden.enabled: true and .Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name, then following these steps:

1. Generate a client certificate for the target cluster for the respective user.
2. Deploy the application part of the charts in the target cluster.
3. Craft a kubeconfig using the already generated client certificate.
4. Set the crafted kubeconfig and deploy the runtime part of the charts in the runtime cluster.

Projected Service Account Token

This approach requires an already deployed and configured oidc-webhook-authenticator for the target cluster. Also, the runtime cluster should be registered as a trusted identity provider in the target cluster. Then, projected service accounts tokens from the runtime cluster can be used to authenticate against the target cluster. The needed steps are as follows:

1. Deploy OWA and establish the needed trust.
2. Set .Values.global.deployment.virtualGarden.enabled: true and .Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name.

   Note: username value will depend on the trust configuration, e.g., <prefix>:system:serviceaccount:<namespace>:<serviceaccount>

3. Set .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true and .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.audience.

   Note: audience value will depend on the trust configuration, e.g., <cliend-id-from-trust-config>.

4. Craft a kubeconfig (see the example below).
5. Deploy the application part of the charts in the target cluster.
6. Deploy the runtime part of the charts in the runtime cluster.

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <CA-DATA>
    server: https://virtual-garden.api
  name: virtual-garden
contexts:
- context:
    cluster: virtual-garden
    user: virtual-garden
  name: virtual-garden
current-context: virtual-garden
users:
- name: virtual-garden
  user:
    tokenFile: /var/run/secrets/projected/serviceaccount/token

2 - Configuring Logging

Configuring the Logging Stack via gardenlet Configurations

Enable the Logging

In order to install the Gardener logging stack, the logging.enabled configuration option has to be enabled in the Gardenlet configuration:

logging:
  enabled: true

From now on, each Seed is going to have a logging stack which will collect logs from all pods and some systemd services. Logs related to Shoots with testing purpose are dropped in the fluent-bit output plugin. Shoots with a purpose different than testing have the same type of log aggregator (but different instance) as the Seed. The logs can be viewed in the Grafana in the garden namespace for the Seed components and in the respective shoot control plane namespaces.

Enable Logs from the Shoot’s Node systemd Services

The logs from the systemd services on each node can be retrieved by enabling the logging.shootNodeLogging option in the gardenlet configuration:

logging:
  enabled: true
  shootNodeLogging:
    shootPurposes:
    - "evaluation"
    - "deployment"

Under the shootPurpose section, just list all the shoot purposes for which the Shoot node logging feature will be enabled. Specifying the testing purpose has no effect because this purpose prevents the logging stack installation. Logs can be viewed in the operator Grafana! The dedicated labels are unit, syslog_identifier, and nodename in the Explore menu.

Configuring the Log Processor

Under logging.fluentBit there are three optional sections:

• input: This overwrites the input configuration of the fluent-bit log processor.
• output: This overwrites the output configuration of the fluent-bit log processor.
• service: This overwrites the service configuration of the fluent-bit log processor.

logging:
  enabled: true
  fluentBit:
    output: |-
      [Output]
          ...      
    input: |-
      [Input]
          ...      
    service: |-
      [Service]
          ...      

Additional egress IPBlock for allow-fluentbit NetworkPolicy

The optional setting under logging.fluentBit.networkPolicy.additionalEgressIPBlocks adds an additional egress IPBlock to allow-fluentbit NetworkPolicy to forward logs to a central system.

logging:
  enabled: true
  fluentBit:
    additionalEgressIpBlock:
      - 123.123.123.123/32

Configure Central Logging

For central logging, the output configuration of the fluent-bit log processor can be overwritten (logging.fluentBit.output) and the Loki instances deployments in the Garden and Shoot namespace can be enabled/disabled (logging.loki.enabled), by default Loki is enabled.

logging:
  enabled: true
  fluentBit:
    output: |-
      [Output]
          ...      
  loki:
    enabled: false

Configuring Central Loki Storage Capacity

By default, the central Loki has 100Gi of storage capacity. To overwrite the current central Loki storage capacity, the logging.loki.garden.storage setting in the gardenlet’s component configuration should be altered. If you need to increase it, you can do so without losing the current data by specifying a higher capacity. By doing so, the Loki’s PersistentVolume capacity will be increased instead of deleting the current PV. However, if you specify less capacity, then the PersistentVolume will be deleted and with it the logs, too.

logging:
  enabled: true
  fluentBit:
    output: |-
      [Output]
          ...      
  loki:
    garden:
      storage: "200Gi"

3 - Deploy Gardenlet

Deploying Gardenlets

Gardenlets act as decentral “agents” to manage the shoot clusters of a seed cluster.

To support scaleability in an automated way, gardenlets are deployed automatically. However, you can still deploy gardenlets manually to be more flexible, for example, when the shoot clusters that need to be managed by Gardener are behind a firewall. The gardenlet only requires network connectivity from the gardenlet to the Garden cluster (not the other way round), so it can be used to register Kubernetes clusters with no public endpoint.

Procedure

1. First, an initial gardenlet needs to be deployed:

   • Deploy it manually if you have special requirements. For more information, see Deploy a Gardenlet Manually.
   • Let the Gardener installer deploy it automatically otherwise. For more information, see Automatic Deployment of Gardenlets.

2. To add additional seed clusters, it is recommended to use regular shoot clusters. You can do this by creating a ManagedSeed resource with a gardenlet section as described in Register Shoot as Seed.

4 - Deploy Gardenlet Automatically

Automatic Deployment of gardenlets

The gardenlet can automatically deploy itself into shoot clusters, and register a cluster as a seed cluster. These clusters are called “managed seeds” (aka “shooted seeds”). This procedure is the preferred way to add additional seed clusters, because shoot clusters already come with production-grade qualities that are also demanded for seed clusters.

Prerequisites

The only prerequisite is to register an initial cluster as a seed cluster that already has a gardenlet deployed in one of the following ways:

• The gardenlet was deployed as part of a Gardener installation using a setup tool (for example, gardener/garden-setup).
• The gardenlet was deployed manually (for a step-by-step manual installation guide, see Deploy a Gardenlet Manually).

The initial cluster can be the garden cluster itself.

Self-Deployment of gardenlets in Additional Managed Seed Clusters

For a better scalability, you usually need more seed clusters that you can create, as follows:

1. Use the initial cluster as the seed cluster for other managed seed clusters. It hosts the control planes of the other seed clusters.
2. The gardenlet deployed in the initial cluster deploys itself automatically into the managed seed clusters.

The advantage of this approach is that there’s only one initial gardenlet installation required. Every other managed seed cluster has a gardenlet deployed automatically.

Related Links

• Register Shoot as Seed
• garden-setup

5 - Deploy Gardenlet Manually

Deploy a gardenlet Manually

Manually deploying a gardenlet is required in the following cases:

• The Kubernetes cluster to be registered as a seed cluster has no public endpoint, because it is behind a firewall. The gardenlet must then be deployed into the cluster itself.

• The Kubernetes cluster to be registered as a seed cluster is managed externally (the Kubernetes cluster is not a shoot cluster, so Automatic Deployment of Gardenlets cannot be used).

• The gardenlet runs outside of the Kubernetes cluster that should be registered as a seed cluster. (The gardenlet is not restricted to run in the seed cluster or to be deployed into a Kubernetes cluster at all).

Once you’ve deployed a gardenlet manually, for example, behind a firewall, you can deploy new gardenlets automatically. The manually deployed gardenlet is then used as a template for the new gardenlets. For more information, see Automatic Deployment of Gardenlets.

Prerequisites

Kubernetes Cluster that Should Be Registered as a Seed Cluster

• Verify that the cluster has a supported Kubernetes version.

• Determine the nodes, pods, and services CIDR of the cluster. You need to configure this information in the Seed configuration. Gardener uses this information to check that the shoot cluster isn’t created with overlapping CIDR ranges.

• Every seed cluster needs an Ingress controller which distributes external requests to internal components like Grafana and Prometheus. For this, configure the following lines in your Seed resource:

spec:
  dns:
    provider:
      type: aws-route53
      secretRef:
        name: ingress-secret
        namespace: garden
  ingress:
    domain: ingress.my-seed.example.com
    controller:
      kind: nginx
      providerConfig:
        <some-optional-provider-specific-config-for-the-ingressController>

kubeconfig for the Seed Cluster

The kubeconfig is required to deploy the gardenlet Helm chart to the seed cluster. The gardenlet requires certain privileges to be able to operate. These privileges are described in RBAC resources in the gardenlet Helm chart (see charts/gardener/gardenlet/templates). The Helm chart contains a service account gardenlet that the gardenlet deployment uses by default to talk to the Seed API server.

If the gardenlet isn’t deployed in the seed cluster, the gardenlet can be configured to use a kubeconfig, which also requires the above-mentioned privileges, from a mounted directory. The kubeconfig is specified in the seedClientConnection.kubeconfig section of the Gardenlet configuration. This configuration option isn’t used in the following, as this procedure only describes the recommended setup option where the gardenlet is running in the seed cluster itself.

Procedure Overview

1. Prepare the garden cluster:

   1. Create a bootstrap token secret in the kube-system namespace of the garden cluster
   2. Create RBAC roles for the gardenlet to allow bootstrapping in the garden cluster

2. Prepare the gardenlet Helm chart.

3. Automatically register shoot cluster as a seed cluster.

4. Deploy the gardenlet

5. Check that the gardenlet is successfully deployed

Create a Bootstrap Token Secret in the kube-system Namespace of the Garden Cluster

The gardenlet needs to talk to the Gardener API server residing in the garden cluster.

The gardenlet can be configured with an already existing garden cluster kubeconfig in one of the following ways:

• By specifying gardenClientConnection.kubeconfig in the Gardenlet configuration.
• By supplying the environment variable GARDEN_KUBECONFIG pointing to a mounted kubeconfig file).

The preferred way, however, is to use the gardenlet’s ability to request a signed certificate for the garden cluster by leveraging Kubernetes Certificate Signing Requests. The gardenlet performs a TLS bootstrapping process that is similar to the Kubelet TLS Bootstrapping. Make sure that the API server of the garden cluster has bootstrap token authentication enabled.

The client credentials required for the gardenlet’s TLS bootstrapping process need to be either token or certificate (OIDC isn’t supported) and have permissions to create a Certificate Signing Request (CSR). It’s recommended to use bootstrap tokens due to their desirable security properties (such as a limited token lifetime).

Therefore, first create a bootstrap token secret for the garden cluster:

apiVersion: v1
kind: Secret
metadata:
  # Name MUST be of form "bootstrap-token-<token id>"
  name: bootstrap-token-07401b
  namespace: kube-system

# Type MUST be 'bootstrap.kubernetes.io/token'
type: bootstrap.kubernetes.io/token
stringData:
  # Human readable description. Optional.
  description: "Token to be used by the gardenlet for Seed `sweet-seed`."

  # Token ID and secret. Required.
  token-id: 07401b # 6 characters
  token-secret: f395accd246ae52d # 16 characters

  # Expiration. Optional.
  # expiration: 2017-03-10T03:22:11Z

  # Allowed usages.
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"

When you later prepare the gardenlet Helm chart, a kubeconfig based on this token is shared with the gardenlet upon deployment.

Create RBAC Roles for the gardenlet to Allow Bootstrapping in the Garden Cluster

This step is only required if the gardenlet you deploy is the first gardenlet in the Gardener installation. Additionally, when using the control plane chart, the following resources are already contained in the Helm chart, that is, if you use it you can skip these steps as the needed RBAC roles already exist.

The gardenlet uses the configured bootstrap kubeconfig in gardenClientConnection.bootstrapKubeconfig to request a signed certificate for the user gardener.cloud:system:seed:<seed-name> in the group gardener.cloud:system:seeds.

Create a ClusterRole and ClusterRoleBinding that grant full admin permissions to authenticated gardenlets.

Create the following resources in the garden cluster:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: gardener.cloud:system:seeds
rules:
  - apiGroups:
      - '*'
    resources:
      - '*'
    verbs:
      - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gardener.cloud:system:seeds
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gardener.cloud:system:seeds
subjects:
  - kind: Group
    name: gardener.cloud:system:seeds
    apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: gardener.cloud:system:seed-bootstrapper
rules:
  - apiGroups:
      - certificates.k8s.io
    resources:
      - certificatesigningrequests
    verbs:
      - create
      - get
  - apiGroups:
      - certificates.k8s.io
    resources:
      - certificatesigningrequests/seedclient
    verbs:
      - create
---
# A kubelet/gardenlet authenticating using bootstrap tokens is authenticated as a user in the group system:bootstrappers
# Allows the Gardenlet to create a CSR
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gardener.cloud:system:seed-bootstrapper
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gardener.cloud:system:seed-bootstrapper
subjects:
  - kind: Group
    name: system:bootstrappers
    apiGroup: rbac.authorization.k8s.io

ℹ️ After bootstrapping, the gardenlet has full administrative access to the garden cluster. You might be interested to harden this and limit its permissions to only resources related to the seed cluster it is responsible for. Please take a look at Scoped API Access for Gardenlets.

Prepare the gardenlet Helm Chart

This section only describes the minimal configuration, using the global configuration values of the gardenlet Helm chart. For an overview over all values, see the configuration values. We refer to the global configuration values as gardenlet configuration in the following procedure.

1. Create a gardenlet configuration gardenlet-values.yaml based on this template.

2. Create a bootstrap kubeconfig based on the bootstrap token created in the garden cluster.

   Replace the <bootstrap-token> with token-id.token-secret (from our previous example: 07401b.f395accd246ae52d) from the bootstrap token secret.

   apiVersion: v1
   kind: Config
   current-context: gardenlet-bootstrap@default
   clusters:
   - cluster:
       certificate-authority-data: <ca-of-garden-cluster>
       server: https://<endpoint-of-garden-cluster>
     name: default
   contexts:
   - context:
       cluster: default
       user: gardenlet-bootstrap
     name: gardenlet-bootstrap@default
   users:
   - name: gardenlet-bootstrap
     user:
       token: <bootstrap-token>
   

3. In the gardenClientConnection.bootstrapKubeconfig section of your gardenlet configuration, provide the bootstrap kubeconfig together with a name and namespace to the gardenlet Helm chart.

   gardenClientConnection:
     bootstrapKubeconfig:
       name: gardenlet-kubeconfig-bootstrap
       namespace: garden
       kubeconfig: |
               <bootstrap-kubeconfig>  # will be base64 encoded by helm
   

   The bootstrap kubeconfig is stored in the specified secret.

4. In the gardenClientConnection.kubeconfigSecret section of your gardenlet configuration, define a name and a namespace where the gardenlet stores the real kubeconfig that it creates during the bootstrap process. If the secret doesn’t exist, the gardenlet creates it for you.

   gardenClientConnection:
     kubeconfigSecret:
       name: gardenlet-kubeconfig
       namespace: garden
   

Updating the Garden Cluster CA

The kubeconfig created by the gardenlet in step 4 will not be recreated as long as it exists, even if a new bootstrap kubeconfig is provided. To enable rotation of the garden cluster CA certificate, a new bundle can be provided via the gardenClientConnection.gardenClusterCACert field. If the provided bundle differs from the one currently in the gardenlet’s kubeconfig secret then it will be updated. To remove the CA completely (e.g. when switching to a publicly trusted endpoint), this field can be set to either none or null.

Automatically Register a Shoot Cluster as a Seed Cluster

A seed cluster can either be registered by manually creating the Seed resource or automatically by the gardenlet. This functionality is useful for managed seed clusters, as the gardenlet in the garden cluster deploys a copy of itself into the cluster with automatic registration of the Seed configured. However, it can also be used to have a streamlined seed cluster registration process when manually deploying the gardenlet.

This procedure doesn’t describe all the possible configurations for the Seed resource. For more information, see:

• Example Seed resource
• Configurable Seed settings

Adjust the gardenlet Component Configuration

1. Supply the Seed resource in the seedConfig section of your gardenlet configuration gardenlet-values.yaml.

2. Add the seedConfig to your gardenlet configuration gardenlet-values.yaml. The field seedConfig.spec.provider.type specifies the infrastructure provider type (for example, aws) of the seed cluster. For all supported infrastructure providers, see Known Extension Implementations.

   ....
   seedConfig:
     metadata:
       name: sweet-seed
     spec:
       dns:
         provider:
           type: <provider>
           secretRef:
             name: ingress-secret
             namespace: garden
       ingress: # see prerequisites
         domain: ingress.dev.my-seed.example.com
         controller:
           kind: nginx
       networks: # see prerequisites
         nodes: 10.240.0.0/16
         pods: 100.244.0.0/16
         services: 100.32.0.0/13
         shootDefaults: # optional: non-overlapping default CIDRs for shoot clusters of that Seed
           pods: 100.96.0.0/11
           services: 100.64.0.0/13
       provider:
         region: eu-west-1
         type: <provider>
   

Optional: Enable HA Mode

You may consider running gardenlet with multiple replicas, especially if the seed cluster is configured to host HA shoot control planes. Therefore, the following Helm chart values define the degree of high availability you want to achieve for the gardenlet deployment.

replicaCount: 2 # or more if a higher failure tolerance is required.
failureToleranceType: zone # One of `zone` or `node` - defines how replicas are spread.

Optional: Enable Backup and Restore

The seed cluster can be set up with backup and restore for the main etcds of shoot clusters.

Gardener uses etcd-backup-restore that integrates with different storage providers to store the shoot cluster’s main etcd backups. Make sure to obtain client credentials that have sufficient permissions with the chosen storage provider.

Create a secret in the garden cluster with client credentials for the storage provider. The format of the secret is cloud provider specific and can be found in the repository of the respective Gardener extension. For example, the secret for AWS S3 can be found in the AWS provider extension (30-etcd-backup-secret.yaml).

apiVersion: v1
kind: Secret
metadata:
  name: sweet-seed-backup
  namespace: garden
type: Opaque
data:
  # client credentials format is provider specific

Configure the Seed resource in the seedConfig section of your gardenlet configuration to use backup and restore:

...
seedConfig:
  metadata:
    name: sweet-seed
  spec:
    backup:
      provider: <provider>
      secretRef:
        name: sweet-seed-backup
        namespace: garden

Deploy the gardenlet

The gardenlet doesn’t have to run in the same Kubernetes cluster as the seed cluster it’s registering and reconciling, but it is in most cases advantageous to use in-cluster communication to talk to the Seed API server. Running a gardenlet outside of the cluster is mostly used for local development.

The gardenlet-values.yaml looks something like this (with automatic Seed registration and backup for shoot clusters enabled):

<default config>
...
config:
  gardenClientConnection:
    ...
    bootstrapKubeconfig:
      name: gardenlet-bootstrap-kubeconfig
      namespace: garden
      kubeconfig: |
        apiVersion: v1
        clusters:
        - cluster:
            certificate-authority-data: <dummy>
            server: <my-garden-cluster-endpoint>
          name: my-kubernetes-cluster
        ....        

    kubeconfigSecret:
      name: gardenlet-kubeconfig
      namespace: garden
  ...
  <default config>
  ...
  seedConfig:
    metadata:
      name: sweet-seed
    spec:
      dns:
        provider:
          type: <provider>
          secretRef:
            name: ingress-secret
            namespace: garden
      ingress: # see prerequisites
        domain: ingress.dev.my-seed.example.com
        controller:
          kind: nginx
      networks:
        nodes: 10.240.0.0/16
        pods: 100.244.0.0/16
        services: 100.32.0.0/13
        shootDefaults:
          pods: 100.96.0.0/11
          services: 100.64.0.0/13
      provider:
        region: eu-west-1
        type: <provider>
      backup:
        provider: <provider>
        secretRef:
          name: sweet-seed-backup
          namespace: garden

Deploy the gardenlet Helm chart to the Kubernetes cluster:

helm install gardenlet charts/gardener/gardenlet \
  --namespace garden \
  -f gardenlet-values.yaml \
  --wait

This helm chart creates:

• A service account gardenlet that the gardenlet can use to talk to the Seed API server.
• RBAC roles for the service account (full admin rights at the moment).
• The secret (garden/gardenlet-bootstrap-kubeconfig) containing the bootstrap kubeconfig.
• The gardenlet deployment in the garden namespace.

Check that the gardenlet Is Successfully Deployed

1. Check that the gardenlets certificate bootstrap was successful.

   Check if the secret gardenlet-kubeconfig in the namespace garden in the seed cluster is created and contains a kubeconfig with a valid certificate.

   1. Get the kubeconfig from the created secret.

      $ kubectl -n garden get secret gardenlet-kubeconfig -o json | jq -r .data.kubeconfig | base64 -d
      

   2. Test against the garden cluster and verify it’s working.

   3. Extract the client-certificate-data from the user gardenlet.

   4. View the certificate:

      $ openssl x509 -in ./gardenlet-cert -noout -text
      

      Check that the certificate is valid for a year (that is the lifetime of new certificates).

2. Check that the bootstrap secret gardenlet-bootstrap-kubeconfig has been deleted from the seed cluster in namespace garden.

3. Check that the seed cluster is registered and READY in the garden cluster.

   Check that the seed cluster sweet-seed exists and all conditions indicate that it’s available. If so, the Gardenlet is sending regular heartbeats and the seed bootstrapping was successful.

   Check that the conditions on the Seed resource look similar to the following:

   $ kubectl get seed sweet-seed -o json | jq .status.conditions
   [
     {
       "lastTransitionTime": "2020-07-17T09:17:29Z",
       "lastUpdateTime": "2020-07-17T09:17:29Z",
       "message": "Gardenlet is posting ready status.",
       "reason": "GardenletReady",
       "status": "True",
       "type": "GardenletReady"
     },
     {
       "lastTransitionTime": "2020-07-17T09:17:49Z",
       "lastUpdateTime": "2020-07-17T09:53:17Z",
       "message": "Seed cluster has been bootstrapped successfully.",
       "reason": "BootstrappingSucceeded",
       "status": "True",
       "type": "Bootstrapped"
     },
     {
       "lastTransitionTime": "2020-07-17T09:17:49Z",
       "lastUpdateTime": "2020-07-17T09:53:17Z",
       "message": "Backup Buckets are available.",
       "reason": "BackupBucketsAvailable",
       "status": "True",
       "type": "BackupBucketsReady"
     }
   ]
   

Related Links

• Issue #1724: Harden Gardenlet RBAC privileges.
• Backup and Restore.

6 - Feature Gates

Feature Gates in Gardener

This page contains an overview of the various feature gates an administrator can specify on different Gardener components.

Overview

Feature gates are a set of key=value pairs that describe Gardener features. You can turn these features on or off using the component configuration file for a specific component.

Each Gardener component lets you enable or disable a set of feature gates that are relevant to that component. For example, this is the configuration of the gardenlet component.

The following tables are a summary of the feature gates that you can set on different Gardener components.

• The “Since” column contains the Gardener release when a feature is introduced or its release stage is changed.
• The “Until” column, if not empty, contains the last Gardener release in which you can still use a feature gate.
• If a feature is in the Alpha or Beta state, you can find the feature listed in the Alpha/Beta feature gate table.
• If a feature is stable you can find all stages for that feature listed in the Graduated/Deprecated feature gate table.
• The Graduated/Deprecated feature gate table also lists deprecated and withdrawn features.

Feature Gates for Alpha or Beta Features

Feature Gates for Graduated or Deprecated Features

Using a Feature

A feature can be in Alpha, Beta or GA stage. An Alpha feature means:

• Disabled by default.
• Might be buggy. Enabling the feature may expose bugs.
• Support for feature may be dropped at any time without notice.
• The API may change in incompatible ways in a later software release without notice.
• Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.

A Beta feature means:

• Enabled by default.
• The feature is well tested. Enabling the feature is considered safe.
• Support for the overall feature will not be dropped, though details may change.
• The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
• Recommended for only non-critical uses because of potential for incompatible changes in subsequent releases.

Please do try Beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.

A General Availability (GA) feature is also referred to as a stable feature. It means:

• The feature is always enabled; you cannot disable it.
• The corresponding feature gate is no longer needed.
• Stable versions of features will appear in released software for many subsequent versions.

List of Feature Gates

7 - gardenlet API Access

Scoped API Access for gardenlets

By default, gardenlets have administrative access in the garden cluster. They are able to execute any API request on any object independent of whether the object is related to the seed cluster the gardenlet is responsible for. As RBAC is not powerful enough for fine-grained checks and for the sake of security, Gardener provides two optional but recommended configurations for your environments that scope the API access for gardenlets.

Similar to the Node authorization mode in Kubernetes, Gardener features a SeedAuthorizer plugin. It is a special-purpose authorization plugin that specifically authorizes API requests made by the gardenlets.

Likewise, similar to the NodeRestriction admission plugin in Kubernetes, Gardener features a SeedRestriction plugin. It is a special-purpose admission plugin that specifically limits the Kubernetes objects gardenlets can modify.

📚 You might be interested to look into the design proposal for scoped Kubelet API access from the Kubernetes community. It can be translated to Gardener and Gardenlets with their Seed and Shoot resources.

Flow Diagram

The following diagram shows how the two plugins are included in the request flow of a gardenlet. When they are not enabled, then the kube-apiserver is internally authorizing the request via RBAC before forwarding the request directly to the gardener-apiserver, i.e., the gardener-admission-controller would not be consulted (this is not entirely correct because it also serves other admission webhook handlers, but for simplicity reasons this document focuses on the API access scope only).

When enabling the plugins, there is one additional step for each before the gardener-apiserver responds to the request.

Please note that the example shows a request to an object (Shoot) residing in one of the API groups served by gardener-apiserver. However, the gardenlet is also interacting with objects in API groups served by the kube-apiserver (e.g., Secret,ConfigMap). In this case, the consultation of the SeedRestriction admission plugin is performed by the kube-apiserver itself before it forwards the request to the gardener-apiserver.

Today, the following rules are implemented:

[1] If you use ManagedSeed resources then the gardenlet reconciling them (“parent gardenlet”) may be allowed to submit certain requests for the Seed resources resulting out of such ManagedSeed reconciliations (even if the “parent gardenlet” is not responsible for them):

ℹ️ It is allowed to delete the Seed resources if the corresponding ManagedSeed objects already have a deletionTimestamp (this is secure as gardenlets themselves don’t have permissions for deleting ManagedSeeds).

SeedAuthorizer Authorization Webhook Enablement

The SeedAuthorizer is implemented as a Kubernetes authorization webhook and part of the gardener-admission-controller component running in the garden cluster.

🎛 In order to activate it, you have to follow these steps:

1. Set the following flags for the kube-apiserver of the garden cluster (i.e., the kube-apiserver whose API is extended by Gardener):

   • --authorization-mode=RBAC,Node,Webhook (please note that Webhook should appear after RBAC in the list [1]; Node might not be needed if you use a virtual garden cluster)
   • --authorization-webhook-config-file=<path-to-the-webhook-config-file>
   • --authorization-webhook-cache-authorized-ttl=0
   • --authorization-webhook-cache-unauthorized-ttl=0

2. The webhook config file (stored at <path-to-the-webhook-config-file>) should look as follows:

   apiVersion: v1
   kind: Config
   clusters:
   - name: garden
     cluster:
       certificate-authority-data: base64(CA-CERT-OF-GARDENER-ADMISSION-CONTROLLER)
       server: https://gardener-admission-controller.garden/webhooks/auth/seed
   users:
   - name: kube-apiserver
     user: {}
   contexts:
   - name: auth-webhook
     context:
       cluster: garden
       user: kube-apiserver
   current-context: auth-webhook
   

3. When deploying the Gardener controlplane Helm chart, set .global.rbac.seedAuthorizer.enabled=true. This will ensure that the RBAC resources granting global access for all gardenlets will be deployed.

4. Delete the existing RBAC resources granting global access for all gardenlets by running:

   kubectl delete \
     clusterrole.rbac.authorization.k8s.io/gardener.cloud:system:seeds \
     clusterrolebinding.rbac.authorization.k8s.io/gardener.cloud:system:seeds \
     --ignore-not-found
   

Please note that you should activate the SeedRestriction admission handler as well.

[1] The reason for the fact that Webhook authorization plugin should appear after RBAC is that the kube-apiserver will be depending on the gardener-admission-controller (serving the webhook). However, the gardener-admission-controller can only start when gardener-apiserver runs, but gardener-apiserver itself can only start when kube-apiserver runs. If Webhook is before RBAC, then gardener-apiserver might not be able to start, leading to a deadlock.

Authorizer Decisions

As mentioned earlier, it’s the authorizer’s job to evaluate API requests and return one of the following decisions:

• DecisionAllow: The request is allowed, further configured authorizers won’t be consulted.
• DecisionDeny: The request is denied, further configured authorizers won’t be consulted.
• DecisionNoOpinion: A decision cannot be made, further configured authorizers will be consulted.

For backwards compatibility, no requests are denied at the moment, so that they are still deferred to a subsequent authorizer like RBAC. Though, this might change in the future.

First, the SeedAuthorizer extracts the Seed name from the API request. This requires a proper TLS certificate that the gardenlet uses to contact the API server and is automatically given if TLS bootstrapping is used. Concretely, the authorizer checks the certificate for name gardener.cloud:system:seed:<seed-name> and group gardener.cloud:system:seeds. In cases where this information is missing e.g., when a custom Kubeconfig is used, the authorizer cannot make any decision. Thus, RBAC is still a considerable option to restrict the gardenlet’s access permission if the above explained preconditions are not given.

With the Seed name at hand, the authorizer checks for an existing path from the resource that a request is being made for to the Seed belonging to the gardenlet. Take a look at the Implementation Details section for more information.

Implementation Details

Internally, the SeedAuthorizer uses a directed, acyclic graph data structure in order to efficiently respond to authorization requests for gardenlets:

• A vertex in this graph represents a Kubernetes resource with its kind, namespace, and name (e.g., Shoot:garden-my-project/my-shoot).
• An edge from vertex u to vertex v in this graph exists when
  • (1) v is referred by u and v is a Seed, or when
  • (2) u is referred by v, or when
  • (3) u is strictly associated with v.

For example, a Shoot refers to a Seed, a CloudProfile, a SecretBinding, etc., so it has an outgoing edge to the Seed (1) and incoming edges from the CloudProfile and SecretBinding vertices (2). However, there might also be a ShootState or a BackupEntry resource strictly associated with this Shoot, hence, it has incoming edges from these vertices (3).

In the above picture, the resources that are actively watched are shaded. Gardener resources are green, while Kubernetes resources are blue. It shows the dependencies between the resources and how the graph is built based on the above rules.

ℹ️ The above picture shows all resources that may be accessed by gardenlets, except for the Quota resource which is only included for completeness.

Now, when a gardenlet wants to access certain resources, then the SeedAuthorizer uses a Depth-First traversal starting from the vertex representing the resource in question, e.g., from a Project vertex. If there is a path from the Project vertex to the vertex representing the Seed the gardenlet is responsible for. then it allows the request.

Metrics

The SeedAuthorizer registers the following metrics related to the mentioned graph implementation:

Debug Handler

When the .server.enableDebugHandlers field in the gardener-admission-controller’s component configuration is set to true, then it serves a handler that can be used for debugging the resource dependency graph under /debug/resource-dependency-graph.

🚨 Only use this setting for development purposes, as it enables unauthenticated users to view all data if they have access to the gardener-admission-controller component.

The handler renders an HTML page displaying the current graph with a list of vertices and its associated incoming and outgoing edges to other vertices. Depending on the size of the Gardener landscape (and consequently, the size of the graph), it might not be possible to render it in its entirety. If there are more than 2000 vertices, then the default filtering will selected for kind=Seed to prevent overloading the output.

Example output:

-------------------------------------------------------------------------------
|
| # Seed:my-seed
|   <- (11)
|     BackupBucket:73972fe2-3d7e-4f61-a406-b8f9e670e6b7
|     BackupEntry:garden-my-project/shoot--dev--my-shoot--4656a460-1a69-4f00-9372-7452cbd38ee3
|     ControllerInstallation:dns-external-mxt8m
|     ControllerInstallation:extension-shoot-cert-service-4qw5j
|     ControllerInstallation:networking-calico-bgrb2
|     ControllerInstallation:os-gardenlinux-qvb5z
|     ControllerInstallation:provider-gcp-w4mvf
|     Secret:garden/backup
|     Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------
|
| # Shoot:garden-my-project/my-shoot
|   <- (5)
|     CloudProfile:gcp
|     Namespace:garden-my-project
|     Secret:garden-my-project/my-dns-secret
|     SecretBinding:garden-my-project/my-credentials
|     ShootState:garden-my-project/my-shoot
|   -> (1)
|     Seed:my-seed
|
-------------------------------------------------------------------------------
|
| # ShootState:garden-my-project/my-shoot
|   -> (1)
|     Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------

... (etc., similarly for the other resources)

There are anchor links to easily jump from one resource to another, and the page provides means for filtering the results based on the kind, namespace, and/or name.

Pitfalls

When there is a relevant update to an existing resource, i.e., when a reference to another resource is changed, then the corresponding vertex (along with all associated edges) is first deleted from the graph before it gets added again with the up-to-date edges. However, this does only work for vertices belonging to resources that are only created in exactly one “watch handler”. For example, the vertex for a SecretBinding can either be created in the SecretBinding handler itself or in the Shoot handler. In such cases, deleting the vertex before (re-)computing the edges might lead to race conditions and potentially renders the graph invalid. Consequently, instead of deleting the vertex, only the edges the respective handler is responsible for are deleted. If the vertex ends up with no remaining edges, then it also gets deleted automatically. Afterwards, the vertex can either be added again or the updated edges can be created.

SeedRestriction Admission Webhook Enablement

The SeedRestriction is implemented as Kubernetes admission webhook and part of the gardener-admission-controller component running in the garden cluster.

🎛 In order to activate it, you have to set .global.admission.seedRestriction.enabled=true when using the Gardener controlplane Helm chart. This will add an additional webhook in the existing ValidatingWebhookConfiguration of the gardener-admission-controller which contains the configuration for the SeedRestriction handler. Please note that it should only be activated when the SeedAuthorizer is active as well.

Admission Decisions

The admission’s purpose is to perform extended validation on requests which require the body of the object in question. Additionally, it handles CREATE requests of gardenlets (the above discussed resource dependency graph cannot be used in such cases because there won’t be any vertex/edge for non-existing resources).

Gardenlets are restricted to only create new resources which are somehow related to the seed clusters they are responsible for.

8 - Getting Started Locally

Deploying Gardener Locally

This document will walk you through deploying Gardener on your local machine. If you encounter difficulties, please open an issue so that we can make this process easier.

Overview

Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as both garden and seed cluster (please refer to the architecture overview) for simplicity.

Based on Skaffold, the container images for all required components will be built and deployed into the cluster (via their Helm charts).

Alternatives

When deploying Gardener on your local machine you might face several limitations:

• Your machine doesn’t have enough compute resources (see prerequisites) for hosting a second seed cluster or multiple shoot clusters.
• Testing Gardener’s IPv6 features requires a Linux machine and native IPv6 connectivity to the internet, but you’re on macOS or don’t have IPv6 connectivity in your office environment or via your home ISP.

In these cases, you might want to check out one of the following options that run the setup described in this guide elsewhere for circumventing these limitations:

• remote local setup: deploy on a remote pod for more compute resources
• dev box on Google Cloud: deploy on a Google Cloud machine for more compute resource and/or simple IPv4/IPv6 dual-stack networking

Prerequisites

• Make sure that you have followed the Local Setup guide up until the Get the sources step.
• Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least 8 CPUs and 8Gi memory; see here how to configure the resources for Docker for Mac).

  Please note that 8 CPU / 8Gi memory might not be enough for more than two Shoot clusters, i.e., you might need to increase these values if you want to run additional Shoots. If you plan on following the optional steps to create a second seed cluster, the required resources will be more - at least 10 CPUs and 18Gi memory. Additionally, please configure at least 120Gi of disk size for the Docker daemon. Tip: You can clean up unused data with docker system df and docker system prune -a.

• Make sure the kind docker network is using the CIDR 172.18.0.0/16.
  • If the network does not exist, it can be created with docker network create kind --subnet 172.18.0.0/16
  • If the network already exists, the CIDR can be checked with docker network inspect kind | jq '.[].IPAM.Config[].Subnet'. If it is not 172.18.0.0/16, delete the network with docker network rm kind and create it with the command above.

Setting Up the KinD Cluster (Garden and Seed)

make kind-up

If you want to setup an IPv6 KinD cluster, use make kind-up IPFAMILY=ipv6 instead.

This command sets up a new KinD cluster named gardener-local and stores the kubeconfig in the ./example/gardener-local/kind/local/kubeconfig file.

It might be helpful to copy this file to $HOME/.kube/config, since you will need to target this KinD cluster multiple times. Alternatively, make sure to set your KUBECONFIG environment variable to ./example/gardener-local/kind/local/kubeconfig for all future steps via export KUBECONFIG=example/gardener-local/kind/local/kubeconfig.

All of the following steps assume that you are using this kubeconfig.

Additionally, this command also deploys a local container registry to the cluster, as well as a few registry mirrors, that are set up as a pull-through cache for all upstream registries Gardener uses by default. This is done to speed up image pulls across local clusters. The local registry can be accessed as localhost:5001 for pushing and pulling. The storage directories of the registries are mounted to the host machine under dev/local-registry. With this, mirrored images don’t have to be pulled again after recreating the cluster.

The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy support (the default kindnet CNI doesn’t provide NetworkPolicy support). Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.

Outgoing IPv6 Single-Stack Networking (optional)

If you want to test IPv6-related features, we need to configure NAT for outgoing traffic from the kind network to the internet. After make kind-up IPFAMILY=ipv6, check the network created by kind:

$ docker network inspect kind | jq '.[].IPAM.Config[].Subnet'
"172.18.0.0/16"
"fc00:f853:ccd:e793::/64"

Determine which device is used for outgoing internet traffic by looking at the default route:

$ ip route show default
default via 192.168.195.1 dev enp3s0 proto dhcp src 192.168.195.34 metric 100

Configure NAT for traffic from the kind cluster to the internet using the IPv6 range and the network device from the previous two steps:

ip6tables -t nat -A POSTROUTING -o enp3s0 -s fc00:f853:ccd:e793::/64 -j MASQUERADE

Setting Up Gardener

make gardener-up

This will first build the base image (which might take a bit if you do it for the first time). Afterwards, the Gardener resources will be deployed into the cluster.

Developing Gardener

make gardener-dev

This is similar to make gardener-up but additionally starts a skaffold dev loop. After the initial deployment, skaffold starts watching source files. Once it has detected changes, press any key to trigger a new build and deployment of the changed components.

Tip: you can set the SKAFFOLD_MODULE environment variable to select specific modules of the skaffold configuration (see skaffold.yaml) that skaffold should watch, build, and deploy. This significantly reduces turnaround times during development.

For example, if you want to develop changes to gardenlet:

# initial deployment of all components
make gardener-up
# start iterating on gardenlet without deploying other components
make gardener-dev SKAFFOLD_MODULE=gardenlet

Creating a Shoot Cluster

You can wait for the Seed to be ready by running:

./hack/usage/wait-for.sh seed local GardenletReady Bootstrapped SeedSystemComponentsHealthy ExtensionsReady

Alternatively, you can run kubectl get seed local and wait for the STATUS to indicate readiness:

NAME    STATUS   PROVIDER   REGION   AGE     VERSION       K8S VERSION
local   Ready    local      local    4m42s   vX.Y.Z-dev    v1.21.1

In order to create a first shoot cluster, just run:

kubectl apply -f example/provider-local/shoot.yaml

You can wait for the Shoot to be ready by running:

NAMESPACE=garden-local ./hack/usage/wait-for.sh shoot APIServerAvailable ControlPlaneHealthy ObservabilityComponentsHealthy EveryNodeReady SystemComponentsHealthy

Alternatively, you can run kubectl -n garden-local get shoot local and wait for the LAST OPERATION to reach 100%:

NAME    CLOUDPROFILE   PROVIDER   REGION   K8S VERSION   HIBERNATION   LAST OPERATION            STATUS    AGE
local   local          local      local    1.21.0        Awake         Create Processing (43%)   healthy   94s

(Optional): You could also execute a simple e2e test (creating and deleting a shoot) by running:

make test-e2e-local-simple KUBECONFIG="$PWD/example/gardener-local/kind/local/kubeconfig"

Accessing the Shoot Cluster

⚠️ Please note that in this setup, shoot clusters are not accessible by default when you download the kubeconfig and try to communicate with them. The reason is that your host most probably cannot resolve the DNS names of the clusters since provider-local extension runs inside the KinD cluster (for more details, see DNSRecord). Hence, if you want to access the shoot cluster, you have to run the following command which will extend your /etc/hosts file with the required information to make the DNS names resolvable:

cat <<EOF | sudo tee -a /etc/hosts

# Manually created to access local Gardener shoot clusters.
# TODO: Remove this again when the shoot cluster access is no longer required.
127.0.0.1 api.local.local.external.local.gardener.cloud
127.0.0.1 api.local.local.internal.local.gardener.cloud

127.0.0.1 api.e2e-managedseed.garden.external.local.gardener.cloud
127.0.0.1 api.e2e-managedseed.garden.internal.local.gardener.cloud
127.0.0.1 api.e2e-hibernated.local.external.local.gardener.cloud
127.0.0.1 api.e2e-hibernated.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-unpriv.local.external.local.gardener.cloud
127.0.0.1 api.e2e-unpriv.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-wake-up.local.external.local.gardener.cloud
127.0.0.1 api.e2e-wake-up.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-migrate.local.external.local.gardener.cloud
127.0.0.1 api.e2e-migrate.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-rotate.local.external.local.gardener.cloud
127.0.0.1 api.e2e-rotate.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-default.local.external.local.gardener.cloud
127.0.0.1 api.e2e-default.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-update-node.local.external.local.gardener.cloud
127.0.0.1 api.e2e-update-node.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-update-zone.local.external.local.gardener.cloud
127.0.0.1 api.e2e-update-zone.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-upgrade.local.external.local.gardener.cloud
127.0.0.1 api.e2e-upgrade.local.internal.local.gardener.cloud
EOF

To access the Shoot, you can acquire a kubeconfig by using the shoots/adminkubeconfig subresource.

(Optional): Setting Up a Second Seed Cluster

There are cases where you would want to create a second seed cluster in your local setup. For example, if you want to test the control plane migration feature. The following steps describe how to do that.

If you are on macOS, add a new IP address on your loopback device which will be necessary for the new KinD cluster that you will create. On macOS, the default loopback device is lo0.

sudo ip addr add 127.0.0.2 dev lo0                                     # adding 127.0.0.2 ip to the loopback interface

Next, setup the second KinD cluster:

make kind2-up

This command sets up a new KinD cluster named gardener-local2 and stores its kubeconfig in the ./example/gardener-local/kind/local2/kubeconfig file.

In order to deploy required resources in the KinD cluster that you just created, run:

make gardenlet-kind2-up

The following steps assume that you are using the kubeconfig that points to the gardener-local cluster (first KinD cluster): export KUBECONFIG=example/gardener-local/kind/local/kubeconfig.

You can wait for the local2 Seed to be ready by running:

./hack/usage/wait-for.sh seed local2 GardenletReady Bootstrapped ExtensionsReady

Alternatively, you can run kubectl get seed local2 and wait for the STATUS to indicate readiness:

NAME    STATUS   PROVIDER   REGION   AGE     VERSION       K8S VERSION
local2  Ready    local      local    4m42s   vX.Y.Z-dev    v1.21.1

If you want to perform control plane migration, you can follow the steps outlined in Control Plane Migration to migrate the shoot cluster to the second seed you just created.

Deleting the Shoot Cluster

./hack/usage/delete shoot local garden-local

(Optional): Tear Down the Second Seed Cluster

make kind2-down

Tear Down the Gardener Environment

make kind-down

Remote Local Setup

Just like Prow is executing the KinD based integration tests in a K8s pod, it is possible to interactively run this KinD based Gardener development environment, aka “local setup”, in a “remote” K8s pod.

k apply -f docs/deployment/content/remote-local-setup.yaml
k exec -it deployment/remote-local-setup -- sh

tmux -u a

Caveats

Please refer to the TMUX documentation for working effectively inside the remote-local-setup pod.

To access Grafana, Prometheus or other components in a browser, two port forwards are needed:

The port forward from the laptop to the pod:

k port-forward deployment/remote-local-setup 3000

The port forward in the remote-local-setup pod to the respective component:

k port-forward -n shoot--local--local deployment/grafana 3000

Related Links

• Local Provider Extension

9 - Getting Started Locally With Extensions

Deploying Gardener Locally and Enabling Provider-Extensions

This document will walk you through deploying Gardener on your local machine and bootstrapping your own seed clusters on an existing Kubernetes cluster. It is supposed to run your local Gardener developments on a real infrastructure. For running Gardener only entirely local, please check the getting started locally documentation. If you encounter difficulties, please open an issue so that we can make this process easier.

Overview

Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as garden cluster. Any Kubernetes cluster could be used as seed clusters in order to support provider extensions (please refer to the architecture overview). This guide is tested for using Kubernetes clusters provided by Gardener, AWS, Azure, and GCP as seed so far.

Based on Skaffold, the container images for all required components will be built and deployed into the clusters (via their Helm charts).

Prerequisites

• Make sure that you have followed the Local Setup guide up until the Get the sources step.
• Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least 8 CPUs and 8Gi memory; see the Docker documentation for how to configure the resources for Docker for Mac).

  Additionally, please configure at least 120Gi of disk size for the Docker daemon. Tip: You can clean up unused data with docker system df and docker system prune -a.

• Make sure that you have access to a Kubernetes cluster you can use as a seed cluster in this setup.
  • The seed cluster requires at least 16 CPUs in total to run one shoot cluster
  • You could use any Kubernetes cluster for your seed cluster. However, using a Gardener shoot cluster for your seed simplifies some configuration steps.
  • When bootstrapping gardenlet to the cluster, your new seed will have the same provider type as the shoot cluster you use - an AWS shoot will become an AWS seed, a GCP shoot will become a GCP seed, etc. (only relevant when using a Gardener shoot as seed).

Provide Infrastructure Credentials and Configuration

As this setup is running on a real infrastructure, you have to provide credentials for DNS, the infrastructure, and the kubeconfig for the Gardener cluster you want to use as seed.

There are .gitignore entries for all files and directories which include credentials. Nevertheless, please double check and make sure that credentials are not commited.

DNS

Gardener control plane requires DNS for default and internal domains. Thus, you have to configure a valid DNS provider for your setup.

Please maintain your DNS provider configuration and credentials at ./example/provider-extensions/garden/controlplane/values.yaml.

You can find a template for the file at ./example/provider-extensions/garden/controlplane/values.yaml.tmpl.

Infrastructure

Infrastructure secrets and the corresponding secret bindings should be maintained at:

• ./example/provider-extensions/garden/project/credentials/infrastructure-secrets.yaml
• ./example/provider-extensions/garden/project/credentials/secretbindings.yaml

There are templates with .tmpl suffixes for the files in the same folder.

Seed Cluster Preparation

The kubeconfig of your Kubernetes cluster you would like to use as seed should be placed at ./example/provider-extensions/seed/kubeconfig. Additionally, please maintain the configuration of your seed in ./example/provider-extensions/gardenlet/values.yaml. It is automatically copied from values.yaml.tmpl in the same directory when you run make gardener-extensions-up for the first time. It also includes explanations of the properties you should set.

Using a Gardener cluster as seed simplifies the process, because some configuration options can be taken from shoot-info and creating DNS entries and TLS certificates is automated.

However, you can use different Kubernetes clusters for your seed too and configure these things manually. Please configure the options of ./example/provider-extensions/gardenlet/values.yaml upfront. For configuring DNS and TLS certificates, make gardener-extensions-up, which is explained later, will pause and tell you what to do.

External Controllers

You might plan to deploy and register external controllers for networking, operating system, providers, etc. Please put ControllerDeployments and ControllerRegistrations into the ./example/provider-extensions/garden/controllerregistrations directory. The whole content of this folder will be applied to your KinD cluster.

CloudProfiles

There are no demo CloudProfiles yet. Thus, please copy CloudProfiles from another landscape to the ./example/provider-extensions/garden/cloudprofiles directory or create your own CloudProfiles based on the gardener examples. Please check the GitHub repository of your desired provider-extension. Most of them include example CloudProfiles. All files you place in this folder will be applied to your KinD cluster.

Setting Up the KinD Cluster

make kind-extensions-up

This command sets up a new KinD cluster named gardener-local and stores the kubeconfig in the ./example/gardener-local/kind/extensions/kubeconfig file.

It might be helpful to copy this file to $HOME/.kube/config, since you will need to target this KinD cluster multiple times. Alternatively, make sure to set your KUBECONFIG environment variable to ./example/gardener-local/kind/extensions/kubeconfig for all future steps via export KUBECONFIG=$PWD/example/gardener-local/kind/extensions/kubeconfig.

All of the following steps assume that you are using this kubeconfig.

Additionally, this command deploys a local container registry to the cluster as well as a few registry mirrors that are set up as a pull-through cache for all upstream registries Gardener uses by default. This is done to speed up image pulls across local clusters. The local registry can be accessed as localhost:5001 for pushing and pulling. The storage directories of the registries are mounted to your machine under dev/local-registry. With this, mirrored images don’t have to be pulled again after recreating the cluster.

The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy support (the default kindnet CNI doesn’t provide NetworkPolicy support). Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.

Setting Up Gardener (Garden on KinD, Seed on Gardener Cluster)

make gardener-extensions-up

This will first prepare the basic configuration of your KinD and Gardener clusters. Afterwards, the images for the Garden cluster are built and deployed into the KinD cluster. Finally, the images for the Seed cluster are built, pushed to a container registry on the Seed, and the gardenlet is started.

Adding Additional Seeds

Additional seed(s) can be added by running

make gardener-extensions-up SEED_NAME=<seed-name>

The seed cluster preparations are similar to the first seed:

The kubeconfig of your Kubernetes cluster you would like to use as seed should be placed at ./example/provider-extensions/seed/kubeconfig-<seed-name>. Additionally, please maintain the configuration of your seed in ./example/provider-extensions/gardenlet/values-<seed-name>.yaml. It is automatically copied from values.yaml.tmpl in the same directory when you run make gardener-extensions-up SEED_NAME=<seed-name> for the first time. It also includes explanations of the properties you should set.

Removing a Seed

If you have multiple seeds and want to remove one, just use

make gardener-extensions-down SEED_NAME=<seed-name>

If it is not the last seed, this command will only remove the seed, but leave the local Gardener cluster and the other seeds untouched. To remove all seeds and to cleanup the local Gardener cluster, you have to run the command for each seed.

Pause and Unpause the KinD Cluster

The KinD cluster can be paused by stopping and keeping its docker container. This can be done by running:

make kind-extensions-down

When you run make kind-extensions-up again, you will start the docker container with your previous Gardener configuration again.

This provides the option to switch off your local KinD cluster fast without leaving orphaned infrastructure elements behind.

Creating a Shoot Cluster

You can wait for the Seed to be ready by running:

kubectl wait --for=condition=gardenletready seed provider-extensions --timeout=5m

make kind-extensions-up already includes such a check. However, it might be useful when you wake up your Seed from hibernation or unpause you KinD cluster.

Alternatively, you can run kubectl get seed provider-extensions and wait for the STATUS to indicate readiness:

NAME                  STATUS   PROVIDER   REGION         AGE    VERSION      K8S VERSION
provider-extensions   Ready    gcp        europe-west1   111m   v1.61.0-dev   v1.24.7

In order to create a first shoot cluster, please create your own Shoot definition and apply it to your KinD cluster. gardener-scheduler includes candidateDeterminationStrategy: MinimalDistance configuration so you are able to run schedule Shoots of different providers on your Seed.

You can wait for your Shoots to be ready by running kubectl -n garden-local get shoots and wait for the LAST OPERATION to reach 100%. The output depends on your Shoot definition. This is an example output:

NAME        CLOUDPROFILE   PROVIDER   REGION         K8S VERSION   HIBERNATION   LAST OPERATION               STATUS    AGE
aws         aws            aws        eu-west-1      1.24.3        Awake         Create Processing (43%)      healthy   84s
aws-arm64   aws            aws        eu-west-1      1.24.3        Awake         Create Processing (43%)      healthy   65s
azure       az             azure      westeurope     1.24.2        Awake         Create Processing (43%)      healthy   57s
gcp         gcp            gcp        europe-west1   1.24.3        Awake         Create Processing (43%)      healthy   94s

Accessing the Shoot Cluster

Your shoot clusters will have a public DNS entries for their API servers, so that they could be reached via the Internet via kubectl after you have created their kubeconfig.

We encourage you to use the adminkubeconfig subresource for accessing your shoot cluster. You can find an example how to use it in Accessing Shoot Clusters.

Deleting the Shoot Clusters

Before tearing down your environment, you have to delete your shoot clusters. This is highly recommended because otherwise you would leave orphaned items on your infrastructure accounts.

./hack/usage/delete shoot <your-shoot> garden-local

Tear Down the Gardener Environment

Before you delete your local KinD cluster, you should shut down your Shoots and Seed in a clean way to avoid orphaned infrastructure elements in your projects.

Please ensure that your KinD and Seed clusters are online (not paused or hibernated) and run:

make gardener-extensions-down

This will delete all Shoots first (this could take a couple of minutes), then uninstall gardenlet from the Seed and the gardener components from the KinD. Finally, the additional components like container registry, etc., are deleted from both clusters.

When this is done, you can securely delete your local KinD cluster by running:

make kind-extensions-clean

10 - Image Vector

Image Vector

The Gardenlet is deploying several different container images into the seed and the shoot clusters. The image repositories and tags are defined in a central image vector file. Obviously, the image versions defined there must fit together with the deployment manifests (e.g., some command-line flags do only exist in certain versions).

Example

images:
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: registry.k8s.io/pause
  tag: "3.4"
  version: "1.20.x"
  architectures:
  - amd64
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: registry.k8s.io/pause
  tag: "3.5"
  version: ">= 1.21"
  architectures:
  - amd64
...

That means that the Gardenlet will use the pause-container with tag 3.4 for all seed/shoot clusters with Kubernetes version 1.20.x, and tag 3.5 for all clusters with Kubernetes >= 1.21.

Image Vector Architecture

images:
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: registry.k8s.io/pause
  tag: "3.5"
  architectures:
  - amd64
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: registry.k8s.io/pause
  tag: "3.5"
  architectures:
  - arm64
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: registry.k8s.io/pause
  tag: "3.5"
  architectures:
  - amd64
  - arm64
...

architectures is an optional field of image. It is a list of strings specifying CPU architecture of machines on which this image can be used. The valid options for the architectures field are as follows:

• amd64 : This specifies that the image can run only on machines having CPU architecture amd64.
• arm64 : This specifies that the image can run only on machines having CPU architecture arm64.

If an image doesn’t specify any architectures, then by default it is considered to support both amd64 and arm64 architectures.

Overwrite Image Vector

In some environments it is not possible to use these “pre-defined” images that come with a Gardener release. A prominent example for that is Alicloud in China, which does not allow access to Google’s GCR. In these cases, you might want to overwrite certain images, e.g., point the pause-container to a different registry.

⚠️ If you specify an image that does not fit to the resource manifest, then the seed/shoot reconciliation might fail.

In order to overwrite the images, you must provide a similar file to gardenlet:

images:
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: my-custom-image-registry/pause
  tag: "3.4"
  version: "1.20.x"
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: my-custom-image-registry/pause
  tag: "3.5"
  version: ">= 1.21"
...

During deployment of the gardenlet, create a ConfigMap containing the above content and mount it as a volume into the gardenlet pod. Next, specify the environment variable IMAGEVECTOR_OVERWRITE, whose value must be the path to the file you just mounted:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gardenlet-images-overwrite
  namespace: garden
data:
  images_overwrite.yaml: |
    images:
    - ...    
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gardenlet
  namespace: garden
spec:
  template:
    ...
    spec:
      containers:
      - name: gardenlet
        env:
        - name: IMAGEVECTOR_OVERWRITE
          value: /charts-overwrite/images_overwrite.yaml
        volumeMounts:
        - name: gardenlet-images-overwrite
          mountPath: /charts-overwrite
        ...
      volumes:
      - name: gardenlet-images-overwrite
        configMap:
          name: gardenlet-images-overwrite
  ...

Image Vectors for Dependent Components

The gardenlet is deploying a lot of different components that might deploy other images themselves. These components might use an image vector as well. Operators might want to customize the image locations for these transitive images as well, hence, they might need to specify an image vector overwrite for the components directly deployed by Gardener.

It is possible to specify the IMAGEVECTOR_OVERWRITE_COMPONENTS environment variable to the gardenlet that points to a file with the following content:

components:
- name: etcd-druid
  imageVectorOverwrite: |
    images:
    - name: etcd
      tag: v1.2.3
      repository: etcd/etcd    
...

The gardenlet will, if supported by the directly deployed component (etcd-druid in this example), inject the given imageVectorOverwrite into the Deployment manifest. The respective component is responsible for using the overwritten images instead of its defaults.

11 - Migration V0 To V1

Migration from Gardener v0 to v1

Please refer to the document for older Gardener versions.

12 - Secret Binding Provider Controller

SecretBinding Provider Controller

This page describes the process on how to enable the SecretBinding provider controller.

Overview

With Gardener v1.38.0, the SecretBinding resource now contains a new optional field .provider.type (details about the motivation can be found in https://github.com/gardener/gardener/issues/4888). To make the process of setting the new field automated and afterwards to enforce validation on the new field in backwards compatible manner, Gardener features the SecretBinding provider controller and a feature gate - SecretBindingProviderValidation.

Process

A Gardener landscape operator can follow the following steps:

1. Enable the SecretBinding provider controller of Gardener Controller Manager.

   The SecretBinding provider controller is responsible for populating the .provider.type field of a SecretBinding based on its current usage by Shoot resources. For example, if a Shoot crazy-botany with .provider.type=aws is using a SecretBinding my-secret-binding, then the SecretBinding provider controller will take care to set the .provider.type field of the SecretBinding to the same provider type (aws). To enable the SecretBinding provider controller, set the controller.secretBindingProvider.concurentSyncs field in the ControllerManagerConfiguration (e.g set it to 5). Although that it is not recommended, the API allows Shoots from different provider types to reference the same SecretBinding (assuming that the backing Secret contains data for both of the provider types). To preserve the backwards compatibility for such SecretBindings, the provider controller will maintain the multiple provider types in the field (it will join them with the separator , - for example aws,gcp).

2. Disable the SecretBinding provider controller and enable the SecretBindingProviderValidation feature gate of Gardener API server.

   The SecretBindingProviderValidation feature gate of Gardener API server enables a set of validations for the SecretBinding provider field. It forbids creating a Shoot that has a different provider type from the referenced SecretBinding’s one. It also enforces immutability on the field. After making sure that SecretBinding provider controller is enabled and it populated the .provider.type field of a majority of the SecretBindings on a Gardener landscape (the SecretBindings that are unused will have their provider type unset), a Gardener landscape operator has to disable the SecretBinding provider controller and to enable the SecretBindingProviderValidation feature gate of Gardener API server. To disable the SecretBinding provider controller, set the controller.secretBindingProvider.concurentSyncs field in the ControllerManagerConfiguration to 0.

Implementation History

• Gardener v1.38: The SecretBinding resource has a new optional field .provider.type. The SecretBinding provider controller is disabled by default. The SecretBindingProviderValidation feature gate of Gardener API server is disabled by default.
• Gardener v1.42: The SecretBinding provider controller is enabled by default.
• Gardener v1.51: The SecretBindingProviderValidation feature gate of Gardener API server is enabled by default and the SecretBinding provider controller is disabled by default.
• Gardener v1.53: The SecretBindingProviderValidation feature gate of Gardener API server is unconditionally enabled (can no longer be disabled).
• Gardener v1.55: The SecretBindingProviderValidation feature gate of Gardener API server and the SecretBinding provider controller are removed.

13 - Setup Gardener

Deploying Gardener into a Kubernetes Cluster

Similar to Kubernetes, Gardener consists out of control plane components (Gardener API server, Gardener controller manager, Gardener scheduler), and an agent component (gardenlet). The control plane is deployed in the so-called garden cluster, while the agent is installed into every seed cluster. Please note that it is possible to use the garden cluster as seed cluster by simply deploying the gardenlet into it.

We are providing Helm charts in order to manage the various resources of the components. Please always make sure that you use the Helm chart version that matches the Gardener version you want to deploy.

Deploying the Gardener Control Plane (API Server, Admission Controller, Controller Manager, Scheduler)

The configuration values depict the various options to configure the different components. Please consult Gardener Configuration and Usage for component specific configurations and Authentication of Gardener Control Plane Components Against the Garden Cluster for authentication related specifics.

Also, note that all resources and deployments need to be created in the garden namespace (not overrideable). If you enable the Gardener admission controller as part of you setup, please make sure the garden namespace is labelled with app: gardener. Otherwise, the backing service account for the admission controller Pod might not be created successfully. No action is necessary if you deploy the garden namespace with the Gardener control plane Helm chart.

After preparing your values in a separate controlplane-values.yaml file (values.yaml can be used as starting point), you can run the following command against your garden cluster:

helm install charts/gardener/controlplane \
  --namespace garden \
  --name gardener-controlplane \
  -f controlplane-values.yaml \
  --wait

Deploying Gardener Extensions

Gardener is an extensible system that does not contain the logic for provider-specific things like DNS management, cloud infrastructures, network plugins, operating system configs, and many more.

You have to install extension controllers for these parts. Please consult the documentation regarding extensions to get more information.

Deploying the Gardener Agent (gardenlet)

Please refer to Deploying Gardenlets on how to deploy a gardenlet.

14 - Version Skew Policy

Version Skew Policy

This document describes the maximum version skew supported between various Gardener components.

Supported Gardener Versions

Gardener versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version, following Semantic Versioning terminology.

The Gardener project maintains release branches for the three most recent minor releases.

Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility. Patch releases are cut from those branches at a regular cadence, plus additional urgent releases when required.

For more information, see the Releases document.

Supported Version Skew

Technically, we follow the same policy as the Kubernetes project. However, given that our release cadence is much more frequent compared to Kubernetes (every 14d vs. every 120d), in many cases it is possible to skip a version. Still, to be on the safe side, it is highly recommended to follow the described policy.

gardener-apiserver

In multi-instance setups of Gardener, the newest and oldest gardener-apiserver instances must be within one minor version.

Example:

• newest gardener-apiserver is at 1.37
• other gardener-apiserver instances are supported at 1.37 and v1.36

gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet

gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet must not be newer than the gardener-apiserver instances they communicate with. They are expected to match the gardener-apiserver minor version, but may be up to one minor version older (to allow live upgrades).

Example:

• gardener-apiserver is at v1.37
• gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet are supported at 1.37 and v1.36

Supported Component Upgrade Order

The supported version skew between components has implications on the order in which components must be upgraded. This section describes the order in which components must be upgraded to transition an existing Gardener installation from version 1.37 to version 1.38.

gardener-apiserver

Prerequisites:

• In a single-instance setup, the existing gardener-apiserver instance is 1.37.
• In a multi-instance setup, all gardener-apiserver instances are at 1.37 or 1.38 (this ensures maximum skew of 1 minor version between the oldest and newest gardener-apiserver instance).
• The gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet instances that communicate with this gardener-apiserver are at version 1.37 (this ensures they are not newer than the existing API server version and are within 1 minor version of the new API server version).

Actions:

• Upgrade gardener-apiserver to 1.38.

gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet

Prerequisites:

• The gardener-apiserver instances these components communicate with are at 1.38 (in multi-instance setups in which these components can communicate with any gardener-apiserver instance in the cluster, all gardener-apiserver instances must be upgraded before upgrading these components)

Actions:

• Upgrade gardener-controller-manager, gardener-scheduler, gardener-admission-controller, and gardenlet to 1.38

Supported Kubernetes Versions

Please refer to Supported Kubernetes Versions.