This is the multi-page printable view of this section. Click here to print.
Deployment
1 - Authentication Gardener Control Plane
Authentication of Gardener Control Plane Components Against the Garden Cluster
Note: This document refers to Gardener’s API server, admission controller, controller manager and scheduler components. Any reference to the term Gardener control plane component can be replaced with any of the mentioned above.
There are several authentication possibilities depending on whether or not the concept of Virtual Garden is used.
Virtual Garden is not used, i.e., the runtime
Garden cluster is also the target
Garden cluster.
Automounted Service Account Token
The easiest way to deploy a Gardener control plane component is to not provide a kubeconfig
at all. This way in-cluster configuration and an automounted service account token will be used. The drawback of this approach is that the automounted token will not be automatically rotated.
Service Account Token Volume Projection
Another solution is to use Service Account Token Volume Projection combined with a kubeconfig
referencing a token file (see the example below).
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: <CA-DATA>
server: https://default.kubernetes.svc.cluster.local
name: garden
contexts:
- context:
cluster: garden
user: garden
name: garden
current-context: garden
users:
- name: garden
user:
tokenFile: /var/run/secrets/projected/serviceaccount/token
This will allow for automatic rotation of the service account token by the kubelet
. The configuration can be achieved by setting both .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true
and .Values.global.<GardenerControlPlaneComponent>.kubeconfig
in the respective chart’s values.yaml
file.
Virtual Garden is used, i.e., the runtime
Garden cluster is different from the target
Garden cluster.
Service Account
The easiest way to setup the authentication is to create a service account and the respective roles will be bound to this service account in the target
cluster. Then use the generated service account token and craft a kubeconfig
, which will be used by the workload in the runtime
cluster. This approach does not provide a solution for the rotation of the service account token. However, this setup can be achieved by setting .Values.global.deployment.virtualGarden.enabled: true
and following these steps:
- Deploy the
application
part of the charts in thetarget
cluster. - Get the service account token and craft the
kubeconfig
. - Set the crafted
kubeconfig
and deploy theruntime
part of the charts in theruntime
cluster.
Client Certificate
Another solution is to bind the roles in the target
cluster to a User
subject instead of a service account and use a client certificate for authentication. This approach does not provide a solution for the client certificate rotation. However, this setup can be achieved by setting both .Values.global.deployment.virtualGarden.enabled: true
and .Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name
, then following these steps:
- Generate a client certificate for the
target
cluster for the respective user. - Deploy the
application
part of the charts in thetarget
cluster. - Craft a
kubeconfig
using the already generated client certificate. - Set the crafted
kubeconfig
and deploy theruntime
part of the charts in theruntime
cluster.
Projected Service Account Token
This approach requires an already deployed and configured oidc-webhook-authenticator for the target
cluster. Also, the runtime
cluster should be registered as a trusted identity provider in the target
cluster. Then, projected service accounts tokens from the runtime
cluster can be used to authenticate against the target
cluster. The needed steps are as follows:
- Deploy OWA and establish the needed trust.
- Set
.Values.global.deployment.virtualGarden.enabled: true
and.Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name
.Note: username value will depend on the trust configuration, e.g.,
<prefix>:system:serviceaccount:<namespace>:<serviceaccount>
- Set
.Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true
and.Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.audience
.Note: audience value will depend on the trust configuration, e.g.,
<client-id-from-trust-config>
. - Craft a kubeconfig (see the example below).
- Deploy the
application
part of the charts in thetarget
cluster. - Deploy the
runtime
part of the charts in theruntime
cluster.
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: <CA-DATA>
server: https://virtual-garden.api
name: virtual-garden
contexts:
- context:
cluster: virtual-garden
user: virtual-garden
name: virtual-garden
current-context: virtual-garden
users:
- name: virtual-garden
user:
tokenFile: /var/run/secrets/projected/serviceaccount/token
2 - Configuring Logging
Configuring the Logging Stack via gardenlet Configurations
Enable the Logging
In order to install the Gardener logging stack, the logging.enabled
configuration option has to be enabled in the Gardenlet configuration:
logging:
enabled: true
From now on, each Seed is going to have a logging stack which will collect logs from all pods and some systemd services. Logs related to Shoots with testing
purpose are dropped in the fluent-bit
output plugin. Shoots with a purpose different than testing
have the same type of log aggregator (but different instance) as the Seed. The logs can be viewed in the Plutono in the garden
namespace for the Seed components and in the respective shoot control plane namespaces.
Enable Logs from the Shoot’s Node systemd Services
The logs from the systemd services on each node can be retrieved by enabling the logging.shootNodeLogging
option in the gardenlet configuration:
logging:
enabled: true
shootNodeLogging:
shootPurposes:
- "evaluation"
- "deployment"
Under the shootPurpose
section, just list all the shoot purposes for which the Shoot node logging feature will be enabled. Specifying the testing
purpose has no effect because this purpose prevents the logging stack installation.
Logs can be viewed in the operator Plutono!
The dedicated labels are unit
, syslog_identifier
, and nodename
in the Explore
menu.
Configuring Central Vali Storage Capacity
By default, the central Vali has 100Gi
of storage capacity.
To overwrite the current central Vali storage capacity, the logging.vali.garden.storage
setting in the gardenlet’s component configuration should be altered.
If you need to increase it, you can do so without losing the current data by specifying a higher capacity. By doing so, the Vali’s PersistentVolume
capacity will be increased instead of deleting the current PV.
However, if you specify less capacity, then the PersistentVolume
will be deleted and with it the logs, too.
logging:
enabled: true
vali:
garden:
storage: "200Gi"
3 - Deploy Gardenlet
Deploying Gardenlets
Gardenlets act as decentralized agents to manage the shoot clusters of a seed cluster.
Procedure
After you have deployed the Gardener control plane, you need one or more seed clusters in order to be able to create shoot clusters.
You can either register an existing cluster as “seed” (this could also be the cluster in which the control plane runs), or you can create new clusters (typically shoots, i.e., this approach registers at least one first initial seed) and then register them as “seeds”.
The following sections describe the scenarios.
Register A First Seed Cluster
If you have not registered a seed cluster yet (thus, you need to deploy a first, so-called “unmanaged seed”), your approach depends on how you deployed the Gardener control plane.
Gardener Control Plane Deployed Via gardener/controlplane
Helm chart
You can follow Deploy a gardenlet Manually.
Gardener Control Plane Deployed Via gardener-operator
- If you want to register the same cluster in which
gardener-operator
runs, or if you want to register another cluster that is reachable (network-wise) forgardener-operator
, you can follow Deploy gardenlet viagardener-operator
. - If you want to register a cluster that is not reachable (network-wise) (e.g., because it runs behind a firewall), you can follow Deploy a gardenlet Manually.
Register Further Seed Clusters
If you already have a seed cluster, and you want to deploy further seed clusters (so-called “managed seeds”), you can follow Deploy a gardenlet Automatically.
4 - Deploy Gardenlet Automatically
Deploy a gardenlet Automatically
The gardenlet can automatically deploy itself into shoot clusters, and register them as seed clusters. These clusters are called “managed seeds” (aka “shooted seeds”). This procedure is the preferred way to add additional seed clusters, because shoot clusters already come with production-grade qualities that are also demanded for seed clusters.
Prerequisites
The only prerequisite is to register an initial cluster as a seed cluster that already has a deployed gardenlet (for available options see Deploying Gardenlets).
💡 Tip
The initial seed cluster can be the garden cluster itself, but for better separation of concerns, it is recommended to only register other clusters as seeds.
Auto-Deployment of Gardenlets into Shoot Clusters
For a better scalability of your Gardener landscape (e.g., when the total number of Shoot
s grows), you usually need more seed clusters that you can create, as follows:
- Use the initial seed cluster (“unmanaged seed”) to create shoot clusters that you later register as seed clusters.
- The gardenlet deployed in the initial cluster can deploy itself into the shoot clusters (which eventually makes them getting registered as seeds) if
ManagedSeed
resources are created.
The advantage of this approach is that there’s only one initial gardenlet installation required. Every other managed seed cluster gets an automatically deployed gardenlet.
Related Links
5 - Deploy Gardenlet Manually
Deploy a gardenlet Manually
Manually deploying a gardenlet is usually only required if the Kubernetes cluster to be registered as a seed cluster is managed via third-party tooling (i.e., the Kubernetes cluster is not a shoot cluster, so Deploy a gardenlet Automatically cannot be used).
In this case, gardenlet
needs to be deployed manually, meaning that its Helm chart must be installed.
💡 Tip
Once you’ve deployed a gardenlet manually, you can deploy new gardenlets automatically. The manually deployed gardenlet is then used as a template for the new gardenlets. For more information, see Deploy a gardenlet Automatically.
Prerequisites
Kubernetes Cluster that Should Be Registered as a Seed Cluster
Verify that the cluster has a supported Kubernetes version.
Determine the nodes, pods, and services CIDR of the cluster. You need to configure this information in the
Seed
configuration. Gardener uses this information to check that the shoot cluster isn’t created with overlapping CIDR ranges.Every seed cluster needs an Ingress controller which distributes external requests to internal components like Plutono and Prometheus. For this, configure the following lines in your Seed resource:
spec: dns: provider: type: aws-route53 secretRef: name: ingress-secret namespace: garden ingress: domain: ingress.my-seed.example.com controller: kind: nginx providerConfig: <some-optional-provider-specific-config-for-the-ingressController>
Procedure Overview
- Prepare the garden cluster:
- Prepare the gardenlet Helm chart.
- Automatically register shoot cluster as a seed cluster.
- Deploy the gardenlet
- Check that the gardenlet is successfully deployed
Create a Bootstrap Token Secret in the kube-system
Namespace of the Garden Cluster
The gardenlet needs to talk to the Gardener API server residing in the garden cluster.
Use gardenlet’s ability to request a signed certificate for the garden cluster by leveraging Kubernetes Certificate Signing Requests. The gardenlet performs a TLS bootstrapping process that is similar to the Kubelet TLS Bootstrapping. Make sure that the API server of the garden cluster has bootstrap token authentication enabled.
The client credentials required for the gardenlet’s TLS bootstrapping process need to be either token
or certificate
(OIDC isn’t supported) and have permissions to create a Certificate Signing Request (CSR).
It’s recommended to use bootstrap tokens due to their desirable security properties (such as a limited token lifetime).
Therefore, first create a bootstrap token secret for the garden cluster:
apiVersion: v1
kind: Secret
metadata:
# Name MUST be of form "bootstrap-token-<token id>"
name: bootstrap-token-07401b
namespace: kube-system
# Type MUST be 'bootstrap.kubernetes.io/token'
type: bootstrap.kubernetes.io/token
stringData:
# Human readable description. Optional.
description: "Token to be used by the gardenlet for Seed `sweet-seed`."
# Token ID and secret. Required.
token-id: 07401b # 6 characters
token-secret: f395accd246ae52d # 16 characters
# Expiration. Optional.
# expiration: 2017-03-10T03:22:11Z
# Allowed usages.
usage-bootstrap-authentication: "true"
usage-bootstrap-signing: "true"
When you later prepare the gardenlet Helm chart, a kubeconfig
based on this token is shared with the gardenlet upon deployment.
Prepare the gardenlet Helm Chart
This section only describes the minimal configuration, using the global configuration values of the gardenlet Helm chart. For an overview over all values, see the configuration values. We refer to the global configuration values as gardenlet configuration in the following procedure.
Create a gardenlet configuration
gardenlet-values.yaml
based on this template.Create a bootstrap
kubeconfig
based on the bootstrap token created in the garden cluster.Replace the
<bootstrap-token>
withtoken-id.token-secret
(from our previous example:07401b.f395accd246ae52d
) from the bootstrap token secret.apiVersion: v1 kind: Config current-context: gardenlet-bootstrap@default clusters: - cluster: certificate-authority-data: <ca-of-garden-cluster> server: https://<endpoint-of-garden-cluster> name: default contexts: - context: cluster: default user: gardenlet-bootstrap name: gardenlet-bootstrap@default users: - name: gardenlet-bootstrap user: token: <bootstrap-token>
In the
gardenClientConnection.bootstrapKubeconfig
section of your gardenlet configuration, provide the bootstrapkubeconfig
together with a name and namespace to the gardenlet Helm chart.gardenClientConnection: bootstrapKubeconfig: name: gardenlet-kubeconfig-bootstrap namespace: garden kubeconfig: | <bootstrap-kubeconfig> # will be base64 encoded by helm
The bootstrap
kubeconfig
is stored in the specified secret.In the
gardenClientConnection.kubeconfigSecret
section of your gardenlet configuration, define a name and a namespace where the gardenlet stores the realkubeconfig
that it creates during the bootstrap process. If the secret doesn’t exist, the gardenlet creates it for you.gardenClientConnection: kubeconfigSecret: name: gardenlet-kubeconfig namespace: garden
Updating the Garden Cluster CA
The kubeconfig created by the gardenlet in step 4 will not be recreated as long as it exists, even if a new bootstrap kubeconfig is provided.
To enable rotation of the garden cluster CA certificate, a new bundle can be provided via the gardenClientConnection.gardenClusterCACert
field.
If the provided bundle differs from the one currently in the gardenlet’s kubeconfig secret then it will be updated.
To remove the CA completely (e.g. when switching to a publicly trusted endpoint), this field can be set to either none
or null
.
Prepare Seed Specification
When gardenlet starts, it tries to register a Seed
resource in the garden cluster based on the specification provided in seedConfig
in its configuration.
This procedure doesn’t describe all the possible configurations for the
Seed
resource. For more information, see:
Supply the
Seed
resource in theseedConfig
section of your gardenlet configurationgardenlet-values.yaml
.Add the
seedConfig
to your gardenlet configurationgardenlet-values.yaml
. The fieldseedConfig.spec.provider.type
specifies the infrastructure provider type (for example,aws
) of the seed cluster. For all supported infrastructure providers, see Known Extension Implementations.# ... seedConfig: metadata: name: sweet-seed labels: environment: evaluation annotations: custom.gardener.cloud/option: special spec: dns: provider: type: <provider> secretRef: name: ingress-secret namespace: garden ingress: # see prerequisites domain: ingress.dev.my-seed.example.com controller: kind: nginx networks: # see prerequisites nodes: 10.240.0.0/16 pods: 100.244.0.0/16 services: 100.32.0.0/13 shootDefaults: # optional: non-overlapping default CIDRs for shoot clusters of that Seed pods: 100.96.0.0/11 services: 100.64.0.0/13 provider: region: eu-west-1 type: <provider>
Apart from the seed’s name, seedConfig.metadata
can optionally contain labels
and annotations
.
gardenlet will set the labels of the registered Seed
object to the labels given in the seedConfig
plus gardener.cloud/role=seed
.
Any custom labels on the Seed
object will be removed on the next restart of gardenlet.
If a label is removed from the seedConfig
it is removed from the Seed
object as well.
In contrast to labels, annotations in the seedConfig
are added to existing annotations on the Seed
object.
Thus, custom annotations that are added to the Seed
object during runtime are not removed by gardenlet on restarts.
Furthermore, if an annotation is removed from the seedConfig
, gardenlet does not remove it from the Seed
object.
Optional: Enable HA Mode
You may consider running gardenlet
with multiple replicas, especially if the seed cluster is configured to host HA shoot control planes.
Therefore, the following Helm chart values define the degree of high availability you want to achieve for the gardenlet
deployment.
replicaCount: 2 # or more if a higher failure tolerance is required.
failureToleranceType: zone # One of `zone` or `node` - defines how replicas are spread.
Optional: Enable Backup and Restore
The seed cluster can be set up with backup and restore for the main etcds
of shoot clusters.
Gardener uses etcd-backup-restore that integrates with different storage providers to store the shoot cluster’s main etcd
backups.
Make sure to obtain client credentials that have sufficient permissions with the chosen storage provider.
Create a secret in the garden cluster with client credentials for the storage provider. The format of the secret is cloud provider specific and can be found in the repository of the respective Gardener extension. For example, the secret for AWS S3 can be found in the AWS provider extension (30-etcd-backup-secret.yaml).
apiVersion: v1
kind: Secret
metadata:
name: sweet-seed-backup
namespace: garden
type: Opaque
data:
# client credentials format is provider specific
Configure the Seed
resource in the seedConfig
section of your gardenlet configuration to use backup and restore:
# ...
seedConfig:
metadata:
name: sweet-seed
spec:
backup:
provider: <provider>
secretRef:
name: sweet-seed-backup
namespace: garden
Optional: Enable Self-Upgrades
In order to take off the continuous task of deploying gardenlet’s Helm chart in case you want to upgrade its version, it supports self-upgrades.
The way this works is that it pulls information (its configuration and deployment values) from a seedmanagement.gardener.cloud/v1alpha1.Gardenlet
resource in the garden cluster.
This resource must be in the garden
namespace and must have the same name as the Seed
the gardenlet is responsible for.
For more information, see this section.
In order to make gardenlet automatically create a corresponding seedmanagement.gardener.cloud/v1alpha1.Gardenlet
resource, you must provide
selfUpgrade:
deployment:
helm:
ociRepository:
ref: <url-to-oci-repository-containing-gardenlet-helm-chart>
in your gardenlet-values.yaml
file.
Please replace the ref
placeholder with the URL to the OCI repository containing the gardenlet Helm chart you are installing.
ℹ️
If you don’t configure this
selfUpgrade
section in the initial deployment, you can also do it later, or you directly create the correspondingseedmanagement.gardener.cloud/v1alpha1.Gardenlet
resource in the garden cluster.
Deploy the gardenlet
The gardenlet-values.yaml
looks something like this (with backup for shoot clusters enabled):
# <default config>
# ...
config:
gardenClientConnection:
# ...
bootstrapKubeconfig:
name: gardenlet-bootstrap-kubeconfig
namespace: garden
kubeconfig: |
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <dummy>
server: <my-garden-cluster-endpoint>
name: my-kubernetes-cluster
# ...
kubeconfigSecret:
name: gardenlet-kubeconfig
namespace: garden
# ...
# <default config>
# ...
seedConfig:
metadata:
name: sweet-seed
spec:
dns:
provider:
type: <provider>
secretRef:
name: ingress-secret
namespace: garden
ingress: # see prerequisites
domain: ingress.dev.my-seed.example.com
controller:
kind: nginx
networks:
nodes: 10.240.0.0/16
pods: 100.244.0.0/16
services: 100.32.0.0/13
shootDefaults:
pods: 100.96.0.0/11
services: 100.64.0.0/13
provider:
region: eu-west-1
type: <provider>
backup:
provider: <provider>
secretRef:
name: sweet-seed-backup
namespace: garden
Deploy the gardenlet Helm chart to the Kubernetes cluster:
helm install gardenlet charts/gardener/gardenlet \
--namespace garden \
-f gardenlet-values.yaml \
--wait
This Helm chart creates:
- A service account
gardenlet
that the gardenlet can use to talk to the Seed API server. - RBAC roles for the service account (full admin rights at the moment).
- The secret (
garden
/gardenlet-bootstrap-kubeconfig
) containing the bootstrapkubeconfig
. - The gardenlet deployment in the
garden
namespace.
Check that the gardenlet Is Successfully Deployed
Check that the gardenlets certificate bootstrap was successful.
Check if the secret
gardenlet-kubeconfig
in the namespacegarden
in the seed cluster is created and contains akubeconfig
with a valid certificate.Get the
kubeconfig
from the created secret.$ kubectl -n garden get secret gardenlet-kubeconfig -o json | jq -r .data.kubeconfig | base64 -d
Test against the garden cluster and verify it’s working.
Extract the
client-certificate-data
from the usergardenlet
.View the certificate:
$ openssl x509 -in ./gardenlet-cert -noout -text
Check that the bootstrap secret
gardenlet-bootstrap-kubeconfig
has been deleted from the seed cluster in namespacegarden
.Check that the seed cluster is registered and
READY
in the garden cluster.Check that the seed cluster
sweet-seed
exists and all conditions indicate that it’s available. If so, the Gardenlet is sending regular heartbeats and the seed bootstrapping was successful.Check that the conditions on the
Seed
resource look similar to the following:$ kubectl get seed sweet-seed -o json | jq .status.conditions [ { "lastTransitionTime": "2020-07-17T09:17:29Z", "lastUpdateTime": "2020-07-17T09:17:29Z", "message": "Gardenlet is posting ready status.", "reason": "GardenletReady", "status": "True", "type": "GardenletReady" }, { "lastTransitionTime": "2020-07-17T09:17:49Z", "lastUpdateTime": "2020-07-17T09:53:17Z", "message": "Backup Buckets are available.", "reason": "BackupBucketsAvailable", "status": "True", "type": "BackupBucketsReady" } ]
Self Upgrades
In order to keep your gardenlets in such “unmanaged seeds” up-to-date (i.e., in seeds which are no shoot clusters), its Helm chart must be regularly deployed. This requires network connectivity to such clusters which can be challenging if they reside behind a firewall or in restricted environments. It is much simpler if gardenlet could keep itself up-to-date, based on configuration read from the garden cluster. This approach greatly reduces operational complexity.
gardenlet runs a controller which watches for seedmanagement.gardener.cloud/v1alpha1.Gardenlet
resources in the garden cluster in the garden
namespace having the same name as the Seed
the gardenlet is responsible for.
Such resources contain its component configuration and deployment values.
Most notably, a URL to an OCI repository containing gardenlet’s Helm chart is included.
An example Gardenlet
resource looks like this:
apiVersion: seedmanagement.gardener.cloud/v1alpha1
kind: Gardenlet
metadata:
name: local
namespace: garden
spec:
deployment:
replicaCount: 1
revisionHistoryLimit: 2
helm:
ociRepository:
ref: <url-to-gardenlet-chart-repository>:v1.97.0
config:
apiVersion: gardenlet.config.gardener.cloud/v1alpha1
kind: GardenletConfiguration
gardenClientConnection:
kubeconfigSecret:
name: gardenlet-kubeconfig
namespace: garden
controllers:
shoot:
reconcileInMaintenanceOnly: true
respectSyncPeriodOverwrite: true
shootState:
concurrentSyncs: 0
featureGates:
DefaultSeccompProfile: true
ShootManagedIssuer: true
etcdConfig:
featureGates:
UseEtcdWrapper: true
logging:
enabled: true
vali:
enabled: true
shootNodeLogging:
shootPurposes:
- infrastructure
- production
- development
- evaluation
seedConfig:
apiVersion: core.gardener.cloud/v1beta1
kind: Seed
metadata:
labels:
base: kind
spec:
backup:
provider: local
region: local
secretRef:
name: backup-local
namespace: garden
dns:
provider:
secretRef:
name: internal-domain-internal-local-gardener-cloud
namespace: garden
type: local
ingress:
controller:
kind: nginx
domain: ingress.local.seed.local.gardener.cloud
networks:
nodes: 172.18.0.0/16
pods: 10.1.0.0/16
services: 10.2.0.0/16
shootDefaults:
pods: 10.3.0.0/16
services: 10.4.0.0/16
provider:
region: local
type: local
zones:
- "0"
settings:
excessCapacityReservation:
enabled: false
scheduling:
visible: true
verticalPodAutoscaler:
enabled: true
On reconciliation, gardenlet downloads the Helm chart, renders it with the provided values, and then applies it to its own cluster. Hence, in order to keep a gardenlet up-to-date, it is enough to update the tag/digest of the OCI repository ref for the Helm chart:
spec:
deployment:
helm:
ociRepository:
ref: <url-to-gardenlet-chart-repository>:v1.97.0
This way, network connectivity to the cluster in which gardenlet runs is not required at all (at least for deployment purposes).
When you delete this resource, nothing happens: gardenlet remains running with the configuration as before.
However, self-upgrades are obviously not possible anymore.
In order to upgrade it, you have to either recreate the Gardenlet
object, or redeploy the Helm chart.
Related Links
6 - Deploy Gardenlet Via Operator
Deploy a gardenlet Via gardener-operator
The gardenlet can automatically be deployed by gardener-operator
into existing Kubernetes clusters in order to register them as seeds.
Prerequisites
Using this method only works when gardener-operator
is managing the garden cluster.
If you have used the gardener/controlplane
Helm chart for the deployment of the Gardener control plane, please refer to this document.
💡 Tip
The initial seed cluster can be the garden cluster itself, but for better separation of concerns, it is recommended to only register other clusters as seeds.
Deployment of gardenlets
Using this method, gardener-operator
is only taking care of the very first deployment of gardenlet.
Once running, the gardenlet leverages the self upgrade strategy in order to keep itself up-to-date.
Concretely, gardener-operator
only acts when there is no respective Seed
resource yet.
In order to request a gardenlet deployment, create following resource in the (virtual) garden cluster:
apiVersion: seedmanagement.gardener.cloud/v1alpha1
kind: Gardenlet
metadata:
name: local
namespace: garden
spec:
deployment:
replicaCount: 1
revisionHistoryLimit: 2
helm:
ociRepository:
ref: <url-to-gardenlet-chart-repository>:v1.97.0
config:
apiVersion: gardenlet.config.gardener.cloud/v1alpha1
kind: GardenletConfiguration
controllers:
shoot:
reconcileInMaintenanceOnly: true
respectSyncPeriodOverwrite: true
shootState:
concurrentSyncs: 0
featureGates:
ShootManagedIssuer: true
etcdConfig:
featureGates:
UseEtcdWrapper: true
logging:
enabled: true
vali:
enabled: true
shootNodeLogging:
shootPurposes:
- infrastructure
- production
- development
- evaluation
seedConfig:
apiVersion: core.gardener.cloud/v1beta1
kind: Seed
metadata:
labels:
base: kind
spec:
backup:
provider: local
region: local
secretRef:
name: backup-local
namespace: garden
dns:
provider:
secretRef:
name: internal-domain-internal-local-gardener-cloud
namespace: garden
type: local
ingress:
controller:
kind: nginx
domain: ingress.local.seed.local.gardener.cloud
networks:
nodes: 172.18.0.0/16
pods: 10.1.0.0/16
services: 10.2.0.0/16
shootDefaults:
pods: 10.3.0.0/16
services: 10.4.0.0/16
provider:
region: local
type: local
zones:
- "0"
settings:
excessCapacityReservation:
enabled: false
scheduling:
visible: true
verticalPodAutoscaler:
enabled: true
This causes gardener-operator
to deploy gardenlet to the same cluster where it is running.
Once it comes up, gardenlet will create a Seed
resource with the same name and uses the Gardenlet
resource for self-upgrades (see this document).
Remote Clusters
If you want gardener-operator
to deploy gardenlet into some other cluster, create a kubeconfig Secret
and reference it in the Gardenlet
resource:
apiVersion: v1
kind: Secret
metadata:
name: remote-cluster-kubeconfig
namespace: garden
type: Opaque
data:
kubeconfig: base64(kubeconfig-to-remote-cluster)
---
apiVersion: seedmanagement.gardener.cloud/v1alpha1
kind: Gardenlet
metadata:
name: local
namespace: garden
spec:
kubeconfigSecretRef:
name: remote-cluster-kubeconfig
# ...
After successful deployment of gardenlet, gardener-operator
will delete the remote-cluster-kubeconfig
Secret
and set .spec.kubeconfigSecretRef
to nil
.
This is because the kubeconfig will never ever be needed anymore (gardener-operator
is only responsible for initial deployment, and gardenlet updates itself with an in-cluster kubeconfig).
7 - Feature Gates
Feature Gates in Gardener
This page contains an overview of the various feature gates an administrator can specify on different Gardener components.
Overview
Feature gates are a set of key=value pairs that describe Gardener features. You can turn these features on or off using the component configuration file for a specific component.
Each Gardener component lets you enable or disable a set of feature gates that are relevant to that component. For example, this is the configuration of the gardenlet component.
The following tables are a summary of the feature gates that you can set on different Gardener components.
- The “Since” column contains the Gardener release when a feature is introduced or its release stage is changed.
- The “Until” column, if not empty, contains the last Gardener release in which you can still use a feature gate.
- If a feature is in the Alpha or Beta state, you can find the feature listed in the Alpha/Beta feature gate table.
- If a feature is stable you can find all stages for that feature listed in the Graduated/Deprecated feature gate table.
- The Graduated/Deprecated feature gate table also lists deprecated and withdrawn features.
Feature Gates for Alpha or Beta Features
Feature | Default | Stage | Since | Until |
---|---|---|---|---|
DefaultSeccompProfile | false | Alpha | 1.54 | |
ShootForceDeletion | false | Alpha | 1.81 | 1.90 |
ShootForceDeletion | true | Beta | 1.91 | |
UseNamespacedCloudProfile | false | Alpha | 1.92 | |
ShootManagedIssuer | false | Alpha | 1.93 | |
ShootCredentialsBinding | false | Alpha | 1.98 | 1.106 |
ShootCredentialsBinding | true | Beta | 1.107 | |
NewWorkerPoolHash | false | Alpha | 1.98 | |
NewVPN | false | Alpha | 1.104 |
Feature Gates for Graduated or Deprecated Features
Feature | Default | Stage | Since | Until |
---|---|---|---|---|
NodeLocalDNS | false | Alpha | 1.7 | 1.25 |
NodeLocalDNS | Removed | 1.26 | ||
KonnectivityTunnel | false | Alpha | 1.6 | 1.26 |
KonnectivityTunnel | Removed | 1.27 | ||
MountHostCADirectories | false | Alpha | 1.11 | 1.25 |
MountHostCADirectories | true | Beta | 1.26 | 1.27 |
MountHostCADirectories | true | GA | 1.27 | |
MountHostCADirectories | Removed | 1.30 | ||
DisallowKubeconfigRotationForShootInDeletion | false | Alpha | 1.28 | 1.31 |
DisallowKubeconfigRotationForShootInDeletion | true | Beta | 1.32 | 1.35 |
DisallowKubeconfigRotationForShootInDeletion | true | GA | 1.36 | 1.37 |
DisallowKubeconfigRotationForShootInDeletion | Removed | 1.38 | ||
Logging | false | Alpha | 0.13 | 1.40 |
Logging | Removed | 1.41 | ||
AdminKubeconfigRequest | false | Alpha | 1.24 | 1.38 |
AdminKubeconfigRequest | true | Beta | 1.39 | 1.41 |
AdminKubeconfigRequest | true | GA | 1.42 | 1.49 |
AdminKubeconfigRequest | Removed | 1.50 | ||
UseDNSRecords | false | Alpha | 1.27 | 1.38 |
UseDNSRecords | true | Beta | 1.39 | 1.43 |
UseDNSRecords | true | GA | 1.44 | 1.49 |
UseDNSRecords | Removed | 1.50 | ||
CachedRuntimeClients | false | Alpha | 1.7 | 1.33 |
CachedRuntimeClients | true | Beta | 1.34 | 1.44 |
CachedRuntimeClients | true | GA | 1.45 | 1.49 |
CachedRuntimeClients | Removed | 1.50 | ||
DenyInvalidExtensionResources | false | Alpha | 1.31 | 1.41 |
DenyInvalidExtensionResources | true | Beta | 1.42 | 1.44 |
DenyInvalidExtensionResources | true | GA | 1.45 | 1.49 |
DenyInvalidExtensionResources | Removed | 1.50 | ||
RotateSSHKeypairOnMaintenance | false | Alpha | 1.28 | 1.44 |
RotateSSHKeypairOnMaintenance | true | Beta | 1.45 | 1.47 |
RotateSSHKeypairOnMaintenance (deprecated) | false | Beta | 1.48 | 1.50 |
RotateSSHKeypairOnMaintenance (deprecated) | Removed | 1.51 | ||
ShootMaxTokenExpirationOverwrite | false | Alpha | 1.43 | 1.44 |
ShootMaxTokenExpirationOverwrite | true | Beta | 1.45 | 1.47 |
ShootMaxTokenExpirationOverwrite | true | GA | 1.48 | 1.50 |
ShootMaxTokenExpirationOverwrite | Removed | 1.51 | ||
ShootMaxTokenExpirationValidation | false | Alpha | 1.43 | 1.45 |
ShootMaxTokenExpirationValidation | true | Beta | 1.46 | 1.47 |
ShootMaxTokenExpirationValidation | true | GA | 1.48 | 1.50 |
ShootMaxTokenExpirationValidation | Removed | 1.51 | ||
WorkerPoolKubernetesVersion | false | Alpha | 1.35 | 1.45 |
WorkerPoolKubernetesVersion | true | Beta | 1.46 | 1.49 |
WorkerPoolKubernetesVersion | true | GA | 1.50 | 1.51 |
WorkerPoolKubernetesVersion | Removed | 1.52 | ||
DisableDNSProviderManagement | false | Alpha | 1.41 | 1.49 |
DisableDNSProviderManagement | true | Beta | 1.50 | 1.51 |
DisableDNSProviderManagement | true | GA | 1.52 | 1.59 |
DisableDNSProviderManagement | Removed | 1.60 | ||
SecretBindingProviderValidation | false | Alpha | 1.38 | 1.50 |
SecretBindingProviderValidation | true | Beta | 1.51 | 1.52 |
SecretBindingProviderValidation | true | GA | 1.53 | 1.54 |
SecretBindingProviderValidation | Removed | 1.55 | ||
SeedKubeScheduler | false | Alpha | 1.15 | 1.54 |
SeedKubeScheduler | false | Deprecated | 1.55 | 1.60 |
SeedKubeScheduler | Removed | 1.61 | ||
ShootCARotation | false | Alpha | 1.42 | 1.50 |
ShootCARotation | true | Beta | 1.51 | 1.56 |
ShootCARotation | true | GA | 1.57 | 1.59 |
ShootCARotation | Removed | 1.60 | ||
ShootSARotation | false | Alpha | 1.48 | 1.50 |
ShootSARotation | true | Beta | 1.51 | 1.56 |
ShootSARotation | true | GA | 1.57 | 1.59 |
ShootSARotation | Removed | 1.60 | ||
ReversedVPN | false | Alpha | 1.22 | 1.41 |
ReversedVPN | true | Beta | 1.42 | 1.62 |
ReversedVPN | true | GA | 1.63 | 1.69 |
ReversedVPN | Removed | 1.70 | ||
ForceRestore | Removed | 1.66 | ||
SeedChange | false | Alpha | 1.12 | 1.52 |
SeedChange | true | Beta | 1.53 | 1.68 |
SeedChange | true | GA | 1.69 | 1.72 |
SeedChange | Removed | 1.73 | ||
CopyEtcdBackupsDuringControlPlaneMigration | false | Alpha | 1.37 | 1.52 |
CopyEtcdBackupsDuringControlPlaneMigration | true | Beta | 1.53 | 1.68 |
CopyEtcdBackupsDuringControlPlaneMigration | true | GA | 1.69 | 1.72 |
CopyEtcdBackupsDuringControlPlaneMigration | Removed | 1.73 | ||
ManagedIstio | false | Alpha | 1.5 | 1.18 |
ManagedIstio | true | Beta | 1.19 | |
ManagedIstio | true | Deprecated | 1.48 | 1.69 |
ManagedIstio | Removed | 1.70 | ||
APIServerSNI | false | Alpha | 1.7 | 1.18 |
APIServerSNI | true | Beta | 1.19 | |
APIServerSNI | true | Deprecated | 1.48 | 1.72 |
APIServerSNI | Removed | 1.73 | ||
HAControlPlanes | false | Alpha | 1.49 | 1.70 |
HAControlPlanes | true | Beta | 1.71 | 1.72 |
HAControlPlanes | true | GA | 1.73 | 1.73 |
HAControlPlanes | Removed | 1.74 | ||
FullNetworkPoliciesInRuntimeCluster | false | Alpha | 1.66 | 1.70 |
FullNetworkPoliciesInRuntimeCluster | true | Beta | 1.71 | 1.72 |
FullNetworkPoliciesInRuntimeCluster | true | GA | 1.73 | 1.73 |
FullNetworkPoliciesInRuntimeCluster | Removed | 1.74 | ||
DisableScalingClassesForShoots | false | Alpha | 1.73 | 1.78 |
DisableScalingClassesForShoots | true | Beta | 1.79 | 1.80 |
DisableScalingClassesForShoots | true | GA | 1.81 | 1.81 |
DisableScalingClassesForShoots | Removed | 1.82 | ||
ContainerdRegistryHostsDir | false | Alpha | 1.77 | 1.85 |
ContainerdRegistryHostsDir | true | Beta | 1.86 | 1.86 |
ContainerdRegistryHostsDir | true | GA | 1.87 | 1.87 |
ContainerdRegistryHostsDir | Removed | 1.88 | ||
WorkerlessShoots | false | Alpha | 1.70 | 1.78 |
WorkerlessShoots | true | Beta | 1.79 | 1.85 |
WorkerlessShoots | true | GA | 1.86 | 1.87 |
WorkerlessShoots | Removed | 1.88 | ||
MachineControllerManagerDeployment | false | Alpha | 1.73 | |
MachineControllerManagerDeployment | true | Beta | 1.81 | 1.81 |
MachineControllerManagerDeployment | true | GA | 1.82 | 1.91 |
MachineControllerManagerDeployment | Removed | 1.92 | ||
APIServerFastRollout | true | Beta | 1.82 | 1.89 |
APIServerFastRollout | true | GA | 1.90 | 1.91 |
APIServerFastRollout | Removed | 1.92 | ||
UseGardenerNodeAgent | false | Alpha | 1.82 | 1.88 |
UseGardenerNodeAgent | true | Beta | 1.89 | 1.89 |
UseGardenerNodeAgent | true | GA | 1.90 | 1.91 |
UseGardenerNodeAgent | Removed | 1.92 | ||
CoreDNSQueryRewriting | false | Alpha | 1.55 | 1.95 |
CoreDNSQueryRewriting | true | Beta | 1.96 | 1.96 |
CoreDNSQueryRewriting | true | GA | 1.97 | 1.100 |
CoreDNSQueryRewriting | Removed | 1.101 | ||
MutableShootSpecNetworkingNodes | false | Alpha | 1.64 | 1.95 |
MutableShootSpecNetworkingNodes | true | Beta | 1.96 | 1.96 |
MutableShootSpecNetworkingNodes | true | GA | 1.97 | 1.100 |
MutableShootSpecNetworkingNodes | Removed | 1.101 | ||
VPAForETCD | false | Alpha | 1.94 | 1.96 |
VPAForETCD | true | Beta | 1.97 | 1.104 |
VPAForETCD | true | GA | 1.105 | |
VPAAndHPAForAPIServer | false | Alpha | 1.95 | 1.100 |
VPAAndHPAForAPIServer | true | Beta | 1.101 | 1.104 |
VPAAndHPAForAPIServer | true | GA | 1.105 | |
HVPA | false | Alpha | 0.31 | 1.105 |
HVPA | false | Deprecated | 1.106 | |
HVPAForShootedSeed | false | Alpha | 0.32 | 1.105 |
HVPAForShootedSeed | false | Deprecated | 1.106 | |
IPv6SingleStack | false | Alpha | 1.63 | |
IPv6SingleStack | Removed | 1.107 |
Using a Feature
A feature can be in Alpha, Beta or GA stage. An Alpha feature means:
- Disabled by default.
- Might be buggy. Enabling the feature may expose bugs.
- Support for feature may be dropped at any time without notice.
- The API may change in incompatible ways in a later software release without notice.
- Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.
A Beta feature means:
- Enabled by default.
- The feature is well tested. Enabling the feature is considered safe.
- Support for the overall feature will not be dropped, though details may change.
- The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
- Recommended for only non-critical uses because of potential for incompatible changes in subsequent releases.
Please do try Beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.
A General Availability (GA) feature is also referred to as a stable feature. It means:
- The feature is always enabled; you cannot disable it.
- The corresponding feature gate is no longer needed.
- Stable versions of features will appear in released software for many subsequent versions.
List of Feature Gates
Feature | Relevant Components | Description |
---|---|---|
HVPA | gardenlet , gardener-operator | Enables simultaneous horizontal and vertical scaling in garden or seed clusters. |
HVPAForShootedSeed | gardenlet | Enables simultaneous horizontal and vertical scaling in managed seed (aka “shooted seed”) clusters. |
DefaultSeccompProfile | gardenlet , gardener-operator | Enables the defaulting of the seccomp profile for Gardener managed workload in the garden or seed to RuntimeDefault . |
ShootForceDeletion | gardener-apiserver | Allows forceful deletion of Shoots by annotating them with the confirmation.gardener.cloud/force-deletion annotation. |
UseNamespacedCloudProfile | gardener-apiserver | Enables usage of NamespacedCloudProfile s in Shoot s. |
ShootManagedIssuer | gardenlet | Enables the shoot managed issuer functionality described in GEP 24. |
VPAForETCD | gardenlet , gardener-operator | Enables VPA for etcd-main and etcd-events , regardless of HVPA enablement. |
VPAAndHPAForAPIServer | gardenlet , gardener-operator | Enables an autoscaling mechanism for kube-apiserver of shoot or virtual garden clusters, and the gardener-apiserver . They are scaled simultaneously by VPA and HPA on the same metric (CPU and memory usage). The pod-trashing cycle between VPA and HPA scaling on the same metric is avoided by configuring the HPA to scale on average usage (not on average utilization) and by picking the target average utilization values in sync with VPA’s allowed maximums. The feature gate takes precedence over the HVPA feature gate when they are both enabled. |
ShootCredentialsBinding | gardener-apiserver | Enables usage of CredentialsBindingName in Shoot s. |
NewWorkerPoolHash | gardenlet | Enables usage of the new worker pool hash calculation. The new calculation supports rolling worker pools if kubeReserved , systemReserved , evicitonHard or cpuManagerPolicy in the kubelet configuration are changed. All provider extensions must be upgraded to support this feature first. Existing worker pools are not immediately migrated to the new hash variant, since this would trigger the replacement of all nodes. The migration happens when a rolling update is triggered according to the old or new hash version calculation. |
NewVPN | gardenlet | Enables usage of the new implementation of the VPN (go rewrite) using an IPv6 transfer network. |
8 - Getting Started Locally
Deploying Gardener Locally
This document will walk you through deploying Gardener on your local machine. If you encounter difficulties, please open an issue so that we can make this process easier.
Overview
Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as both garden and seed cluster (please refer to the architecture overview) for simplicity.
Based on Skaffold, the container images for all required components will be built and deployed into the cluster (via their Helm charts).
Alternatives
When deploying Gardener on your local machine you might face several limitations:
- Your machine doesn’t have enough compute resources (see prerequisites) for hosting a second seed cluster or multiple shoot clusters.
- Testing Gardener’s IPv6 features requires a Linux machine and native IPv6 connectivity to the internet, but you’re on macOS or don’t have IPv6 connectivity in your office environment or via your home ISP.
In these cases, you might want to check out one of the following options that run the setup described in this guide elsewhere for circumventing these limitations:
- remote local setup: deploy on a remote pod for more compute resources
- dev box on Google Cloud: deploy on a Google Cloud machine for more compute resource and/or simple IPv4/IPv6 dual-stack networking
Prerequisites
- Make sure that you have followed the Local Setup guide up until the Get the sources step.
- Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least
8
CPUs and8Gi
memory; see here how to configure the resources for Docker for Mac).Please note that 8 CPU / 8Gi memory might not be enough for more than two
Shoot
clusters, i.e., you might need to increase these values if you want to run additionalShoot
s. If you plan on following the optional steps to create a second seed cluster, the required resources will be more - at least10
CPUs and18Gi
memory. Additionally, please configure at least120Gi
of disk size for the Docker daemon. Tip: You can clean up unused data withdocker system df
anddocker system prune -a
.
Setting Up the KinD Cluster (Garden and Seed)
make kind-up
If you want to setup an IPv6 KinD cluster, use
make kind-up IPFAMILY=ipv6
instead.
This command sets up a new KinD cluster named gardener-local
and stores the kubeconfig in the ./example/gardener-local/kind/local/kubeconfig
file.
It might be helpful to copy this file to
$HOME/.kube/config
, since you will need to target this KinD cluster multiple times. Alternatively, make sure to set yourKUBECONFIG
environment variable to./example/gardener-local/kind/local/kubeconfig
for all future steps viaexport KUBECONFIG=$PWD/example/gardener-local/kind/local/kubeconfig
.
All of the following steps assume that you are using this kubeconfig.
Additionally, this command also deploys a local container registry to the cluster, as well as a few registry mirrors, that are set up as a pull-through cache for all upstream registries Gardener uses by default. This is done to speed up image pulls across local clusters.
You will need to add
127.0.0.1 garden.local.gardener.cloud
to your /etc/hosts.
The local registry can now be accessed either via localhost:5001
or garden.local.gardener.cloud:5001
for pushing and pulling.
The storage directories of the registries are mounted to the host machine under dev/local-registry
.
With this, mirrored images don’t have to be pulled again after recreating the cluster.
The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy
support (the default kindnet
CNI doesn’t provide NetworkPolicy
support).
Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.
Setting Up IPv6 Single-Stack Networking (optional)
First, ensure that your /etc/hosts
file contains an entry resolving garden.local.gardener.cloud
to the IPv6 loopback address:
::1 garden.local.gardener.cloud
Typically, only ip6-localhost
is mapped to ::1
on linux machines.
However, we need garden.local.gardener.cloud
to resolve to both 127.0.0.1
and ::1
so that we can talk to our registry via a single address (garden.local.gardener.cloud:5001
).
Next, we need to configure NAT for outgoing traffic from the kind network to the internet.
After executing make kind-up IPFAMILY=ipv6
, execute the following command to set up the corresponding iptables rules:
ip6tables -t nat -A POSTROUTING -o $(ip route show default | awk '{print $5}') -s fd00:10::/64 -j MASQUERADE
Setting Up Gardener
make gardener-up
If you want to setup an IPv6 ready Gardener, use
make gardener-up IPFAMILY=ipv6
instead.
This will first build the base images (which might take a bit if you do it for the first time). Afterwards, the Gardener resources will be deployed into the cluster.
Developing Gardener
make gardener-dev
This is similar to make gardener-up
but additionally starts a skaffold dev loop.
After the initial deployment, skaffold starts watching source files.
Once it has detected changes, press any key to trigger a new build and deployment of the changed components.
Tip: you can set the SKAFFOLD_MODULE
environment variable to select specific modules of the skaffold configuration (see skaffold.yaml
) that skaffold should watch, build, and deploy.
This significantly reduces turnaround times during development.
For example, if you want to develop changes to gardenlet:
# initial deployment of all components
make gardener-up
# start iterating on gardenlet without deploying other components
make gardener-dev SKAFFOLD_MODULE=gardenlet
Debugging Gardener
make gardener-debug
This is using skaffold debugging features. In the Gardener case, Go debugging using Delve is the most relevant use case. Please see the skaffold debugging documentation how to set up your IDE accordingly or check the examples below (GoLand, VS Code).
SKAFFOLD_MODULE
environment variable is working the same way as described for Developing Gardener. However, skaffold is not watching for changes when debugging,
because it would like to avoid interrupting your debugging session.
For example, if you want to debug gardenlet:
# initial deployment of all components
make gardener-up
# start debugging gardenlet without deploying other components
make gardener-debug SKAFFOLD_MODULE=gardenlet
In debugging flow, skaffold builds your container images, reconfigures your pods and creates port forwardings for the Delve
debugging ports to your localhost.
The default port is 56268
. If you debug multiple pods at the same time, the port of the second pod will be forwarded to 56269
and so on.
Please check your console output for the concrete port-forwarding on your machine.
Note: Resuming or stopping only a single goroutine (Go Issue 25578, 31132) is currently not supported, so the action will cause all the goroutines to get activated or paused. (vscode-go wiki)
This means that when a goroutine of gardenlet (or any other gardener-core component you try to debug) is paused on a breakpoint, all the other goroutines are paused. Hence, when the whole gardenlet process is paused, it can not renew its lease and can not respond to the liveness and readiness probes. Skaffold automatically increases timeoutSeconds
of liveness and readiness probes to 600. Anyway, we were facing problems when debugging that pods have been killed after a while.
Thus, leader election, health and readiness checks for gardener-admission-controller
, gardener-apiserver
, gardener-controller-manager
, gardener-scheduler
,gardenlet
and operator
are disabled when debugging.
If you have similar problems with other components which are not deployed by skaffold, you could temporarily turn off the leader election and disable liveness and readiness probes there too.
Debugging in GoLand
- Edit your Run/Debug Configurations.
- Add a new Go Remote configuration.
- Set the port to
56268
(or any increment of it when debugging multiple components). - Recommended: Change the behavior of On disconnect to Leave it running.
Debugging in VS Code
- Create or edit your
.vscode/launch.json
configuration. - Add the following configuration:
{
"name": "go remote",
"type": "go",
"request": "attach",
"mode": "remote",
"port": 56268, // or any increment of it when debugging multiple components
"host": "127.0.0.1"
}
Since the ko builder is used in Skaffold to build the images, it’s not necessary to specify the cwd
and remotePath
options as they match the workspace folder (ref).
Creating a Shoot
Cluster
You can wait for the Seed
to be ready by running:
./hack/usage/wait-for.sh seed local GardenletReady SeedSystemComponentsHealthy ExtensionsReady
Alternatively, you can run kubectl get seed local
and wait for the STATUS
to indicate readiness:
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
local Ready local local 4m42s vX.Y.Z-dev v1.25.1
In order to create a first shoot cluster, just run:
kubectl apply -f example/provider-local/shoot.yaml
You can wait for the Shoot
to be ready by running:
NAMESPACE=garden-local ./hack/usage/wait-for.sh shoot local APIServerAvailable ControlPlaneHealthy ObservabilityComponentsHealthy EveryNodeReady SystemComponentsHealthy
Alternatively, you can run kubectl -n garden-local get shoot local
and wait for the LAST OPERATION
to reach 100%
:
NAME CLOUDPROFILE PROVIDER REGION K8S VERSION HIBERNATION LAST OPERATION STATUS AGE
local local local local 1.25.1 Awake Create Processing (43%) healthy 94s
If you don’t need any worker pools, you can create a workerless Shoot
by running:
kubectl apply -f example/provider-local/shoot-workerless.yaml
(Optional): You could also execute a simple e2e test (creating and deleting a shoot) by running:
make test-e2e-local-simple KUBECONFIG="$PWD/example/gardener-local/kind/local/kubeconfig"
Accessing the Shoot
Cluster
⚠️ Please note that in this setup, shoot clusters are not accessible by default when you download the kubeconfig and try to communicate with them.
The reason is that your host most probably cannot resolve the DNS names of the clusters since provider-local
extension runs inside the KinD cluster (for more details, see DNSRecord).
Hence, if you want to access the shoot cluster, you have to run the following command which will extend your /etc/hosts
file with the required information to make the DNS names resolvable:
cat <<EOF | sudo tee -a /etc/hosts
# Begin of Gardener local setup section
# Shoot API server domains
172.18.255.1 api.local.local.external.local.gardener.cloud
172.18.255.1 api.local.local.internal.local.gardener.cloud
# Ingress
172.18.255.1 p-seed.ingress.local.seed.local.gardener.cloud
172.18.255.1 g-seed.ingress.local.seed.local.gardener.cloud
172.18.255.1 gu-local--local.ingress.local.seed.local.gardener.cloud
172.18.255.1 p-local--local.ingress.local.seed.local.gardener.cloud
172.18.255.1 v-local--local.ingress.local.seed.local.gardener.cloud
# E2e tests
172.18.255.1 api.e2e-managedseed.garden.external.local.gardener.cloud
172.18.255.1 api.e2e-managedseed.garden.internal.local.gardener.cloud
172.18.255.1 api.e2e-hib.local.external.local.gardener.cloud
172.18.255.1 api.e2e-hib.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-hib-wl.local.external.local.gardener.cloud
172.18.255.1 api.e2e-hib-wl.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-unpriv.local.external.local.gardener.cloud
172.18.255.1 api.e2e-unpriv.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-wake-up.local.external.local.gardener.cloud
172.18.255.1 api.e2e-wake-up.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-wake-up-wl.local.external.local.gardener.cloud
172.18.255.1 api.e2e-wake-up-wl.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-wake-up-ncp.local.external.local.gardener.cloud
172.18.255.1 api.e2e-wake-up-ncp.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-migrate.local.external.local.gardener.cloud
172.18.255.1 api.e2e-migrate.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-migrate-wl.local.external.local.gardener.cloud
172.18.255.1 api.e2e-migrate-wl.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-mgr-hib.local.external.local.gardener.cloud
172.18.255.1 api.e2e-mgr-hib.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-rotate.local.external.local.gardener.cloud
172.18.255.1 api.e2e-rotate.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-rotate-wl.local.external.local.gardener.cloud
172.18.255.1 api.e2e-rotate-wl.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-default.local.external.local.gardener.cloud
172.18.255.1 api.e2e-default.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-default-wl.local.external.local.gardener.cloud
172.18.255.1 api.e2e-default-wl.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-force-delete.local.external.local.gardener.cloud
172.18.255.1 api.e2e-force-delete.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-fd-hib.local.external.local.gardener.cloud
172.18.255.1 api.e2e-fd-hib.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-upd-node.local.external.local.gardener.cloud
172.18.255.1 api.e2e-upd-node.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-upd-node-wl.local.external.local.gardener.cloud
172.18.255.1 api.e2e-upd-node-wl.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-upgrade.local.external.local.gardener.cloud
172.18.255.1 api.e2e-upgrade.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-upgrade-wl.local.external.local.gardener.cloud
172.18.255.1 api.e2e-upgrade-wl.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-upg-hib.local.external.local.gardener.cloud
172.18.255.1 api.e2e-upg-hib.local.internal.local.gardener.cloud
172.18.255.1 api.e2e-upg-hib-wl.local.external.local.gardener.cloud
172.18.255.1 api.e2e-upg-hib-wl.local.internal.local.gardener.cloud
172.18.255.1 gu-local--e2e-rotate.ingress.local.seed.local.gardener.cloud
172.18.255.1 gu-local--e2e-rotate-wl.ingress.local.seed.local.gardener.cloud
# End of Gardener local setup section
EOF
To access the Shoot
, you can acquire a kubeconfig
by using the shoots/adminkubeconfig
subresource.
For convenience a helper script is provided in the hack
directory. By default the script will generate a kubeconfig for a Shoot
named “local” in the garden-local
namespace valid for one hour.
./hack/usage/generate-admin-kubeconf.sh > admin-kubeconf.yaml
If you want to change the default namespace or shoot name, you can do so by passing different values as arguments.
./hack/usage/generate-admin-kubeconf.sh --namespace <namespace> --shoot-name <shootname> > admin-kubeconf.yaml
To access an Ingress resource from the Seed
, use the Ingress host with port 8448
(https://<ingress-host>:8448
, for example https://gu-local--local.ingress.local.seed.local.gardener.cloud:8448
).
(Optional): Setting Up a Second Seed Cluster
There are cases where you would want to create a second seed cluster in your local setup. For example, if you want to test the control plane migration feature. The following steps describe how to do that.
If you are on macOS, add a new IP address on your loopback device which will be necessary for the new KinD cluster that you will create. On macOS, the default loopback device is lo0
.
sudo ip addr add 172.18.255.2 dev lo0 # adding 172.18.255.2 ip to the loopback interface
Next, setup the second KinD cluster:
make kind2-up
This command sets up a new KinD cluster named gardener-local2
and stores its kubeconfig in the ./example/gardener-local/kind/local2/kubeconfig
file.
In order to deploy required resources in the KinD cluster that you just created, run:
make gardenlet-kind2-up
The following steps assume that you are using the kubeconfig that points to the gardener-local
cluster (first KinD cluster): export KUBECONFIG=$PWD/example/gardener-local/kind/local/kubeconfig
.
You can wait for the local2
Seed
to be ready by running:
./hack/usage/wait-for.sh seed local2 GardenletReady SeedSystemComponentsHealthy ExtensionsReady
Alternatively, you can run kubectl get seed local2
and wait for the STATUS
to indicate readiness:
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
local2 Ready local local 4m42s vX.Y.Z-dev v1.25.1
If you want to perform control plane migration, you can follow the steps outlined in Control Plane Migration to migrate the shoot cluster to the second seed you just created.
Deleting the Shoot
Cluster
./hack/usage/delete shoot local garden-local
(Optional): Tear Down the Second Seed Cluster
make kind2-down
Tear Down the Gardener Environment
make kind-down
Alternative Way to Set Up Garden and Seed Leveraging gardener-operator
Instead of starting Garden and Seed via make kind-up gardener-up
, you can also use gardener-operator
to create your local dev landscape.
In this setup, the virtual garden cluster has its own load balancer, so you have to create an own DNS entry in your /etc/hosts
:
cat <<EOF | sudo tee -a /etc/hosts
# Manually created to access local Gardener virtual garden cluster.
# TODO: Remove this again when the virtual garden cluster access is no longer required.
172.18.255.3 api.virtual-garden.local.gardener.cloud
EOF
You can bring up gardener-operator
with this command:
make kind-operator-up operator-up
Afterwards, you can create your local Garden
and install gardenlet
into the KinD cluster with this command:
make operator-seed-up
You find the kubeconfig for the KinD cluster at ./example/gardener-local/kind/operator/kubeconfig
.
The one for the virtual garden is accessible at ./example/operator/virtual-garden/kubeconfig
.
❗ Important
When you create non-HA shoot clusters (i.e.,
Shoot
s with.spec.controlPlane.highAvailability.failureTolerance != zone
), then they are not exposed via172.18.255.1
(ref). Instead, you need to find out under which Istio instance they got exposed, and put the corresponding IP address into your/etc/hosts
file:# replace <shoot-namespace> with your shoot namespace (e.g., `shoot--foo--bar`): kubectl -n "$(kubectl -n <shoot-namespace> get gateway kube-apiserver -o jsonpath={.spec.selector.istio} | sed 's/.*--/istio-ingress--/')" get svc istio-ingressgateway -o jsonpath={.status.loadBalancer.ingress..ip}
When the shoot cluster is HA (i.e.,
.spec.controlPlane.highAvailability.failureTolerance == zone
), then you can access it via172.18.255.1
.
Similar as in the section Developing Gardener it’s possible to run a Skaffold development loop as well using:
make operator-seed-dev
ℹ️ Please note that in this setup Skaffold is only watching for changes in the following components:
gardenlet
gardenlet/chart
gardener-resource-manager
gardener-node-agent
Finally, please use this command to tear down your environment:
make kind-operator-down
This setup supports creating shoots and managed seeds the same way as explained in the previous chapters. However, the development loop has limitations and the debugging setup is not working yet.
Remote Local Setup
Just like Prow is executing the KinD based integration tests in a K8s pod, it is possible to interactively run this KinD based Gardener development environment, aka “local setup”, in a “remote” K8s pod.
k apply -f docs/deployment/content/remote-local-setup.yaml
k exec -it remote-local-setup-0 -- sh
tmux a
Caveats
Please refer to the TMUX documentation for working effectively inside the remote-local-setup pod.
To access Plutono, Prometheus or other components in a browser, two port forwards are needed:
The port forward from the laptop to the pod:
k port-forward remote-local-setup-0 3000
The port forward in the remote-local-setup pod to the respective component:
k port-forward -n shoot--local--local deployment/plutono 3000
Related Links
9 - Getting Started Locally With Extensions
Deploying Gardener Locally and Enabling Provider-Extensions
This document will walk you through deploying Gardener on your local machine and bootstrapping your own seed clusters on an existing Kubernetes cluster. It is supposed to run your local Gardener developments on a real infrastructure. For running Gardener only entirely local, please check the getting started locally documentation. If you encounter difficulties, please open an issue so that we can make this process easier.
Overview
Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as garden cluster. Any Kubernetes cluster could be used as seed clusters in order to support provider extensions (please refer to the architecture overview). This guide is tested for using Kubernetes clusters provided by Gardener, AWS, Azure, and GCP as seed so far.
Based on Skaffold, the container images for all required components will be built and deployed into the clusters (via their Helm charts).
Prerequisites
- Make sure that you have followed the Local Setup guide up until the Get the sources step.
- Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least
8
CPUs and8Gi
memory; see the Docker documentation for how to configure the resources for Docker for Mac).Additionally, please configure at least
120Gi
of disk size for the Docker daemon. Tip: You can clean up unused data withdocker system df
anddocker system prune -a
. - Make sure that you have access to a Kubernetes cluster you can use as a seed cluster in this setup.
- The seed cluster requires at least 16 CPUs in total to run one shoot cluster
- You could use any Kubernetes cluster for your seed cluster. However, using a Gardener shoot cluster for your seed simplifies some configuration steps.
- When bootstrapping
gardenlet
to the cluster, your new seed will have the same provider type as the shoot cluster you use - an AWS shoot will become an AWS seed, a GCP shoot will become a GCP seed, etc. (only relevant when using a Gardener shoot as seed).
Provide Infrastructure Credentials and Configuration
As this setup is running on a real infrastructure, you have to provide credentials for DNS, the infrastructure, and the kubeconfig for the Kubernetes cluster you want to use as seed.
There are
.gitignore
entries for all files and directories which include credentials. Nevertheless, please double check and make sure that credentials are not committed to the version control system.
DNS
Gardener control plane requires DNS for default and internal domains. Thus, you have to configure a valid DNS provider for your setup.
Please maintain your DNS provider configuration and credentials at ./example/provider-extensions/garden/controlplane/domain-secrets.yaml
.
You can find a template for the file at ./example/provider-extensions/garden/controlplane/domain-secrets.yaml.tmpl
.
Infrastructure
Infrastructure secrets and the corresponding secret bindings should be maintained at:
./example/provider-extensions/garden/project/credentials/infrastructure-secrets.yaml
./example/provider-extensions/garden/project/credentials/secretbindings.yaml
There are templates with .tmpl
suffixes for the files in the same folder.
Projects
The projects and the namespaces associated with them should be maintained at ./example/provider-extensions/garden/project/project.yaml
.
You can find a template for the file at ./example/provider-extensions/garden/project/project.yaml.tmpl
.
Seed Cluster Preparation
The kubeconfig
of your Kubernetes cluster you would like to use as seed should be placed at ./example/provider-extensions/seed/kubeconfig
.
Additionally, please maintain the configuration of your seed in ./example/provider-extensions/gardenlet/values.yaml
. It is automatically copied from values.yaml.tmpl
in the same directory when you run make gardener-extensions-up
for the first time. It also includes explanations of the properties you should set.
Using a Gardener Shoot cluster as seed simplifies the process, because some configuration options can be taken from shoot-info
and creating DNS entries and TLS certificates is automated.
However, you can use different Kubernetes clusters for your seed too and configure these things manually. Please configure the options of ./example/provider-extensions/gardenlet/values.yaml
upfront. For configuring DNS and TLS certificates, make gardener-extensions-up
, which is explained later, will pause and tell you what to do.
External Controllers
You might plan to deploy and register external controllers for networking, operating system, providers, etc. Please put ControllerDeployment
s and ControllerRegistration
s into the ./example/provider-extensions/garden/controllerregistrations
directory. The whole content of this folder will be applied to your KinD cluster.
CloudProfile
s
There are no demo CloudProfiles
yet. Thus, please copy CloudProfiles
from another landscape to the ./example/provider-extensions/garden/cloudprofiles
directory or create your own CloudProfiles
based on the gardener examples. Please check the GitHub repository of your desired provider-extension. Most of them include example CloudProfile
s. All files you place in this folder will be applied to your KinD cluster.
Setting Up the KinD Cluster
make kind-extensions-up
This command sets up a new KinD cluster named gardener-extensions
and stores the kubeconfig in the ./example/gardener-local/kind/extensions/kubeconfig
file.
It might be helpful to copy this file to
$HOME/.kube/config
, since you will need to target this KinD cluster multiple times. Alternatively, make sure to set yourKUBECONFIG
environment variable to./example/gardener-local/kind/extensions/kubeconfig
for all future steps viaexport KUBECONFIG=$PWD/example/gardener-local/kind/extensions/kubeconfig
.
All of the following steps assume that you are using this kubeconfig.
Additionally, this command deploys a local container registry to the cluster as well as a few registry mirrors that are set up as a pull-through cache for all upstream registries Gardener uses by default. This is done to speed up image pulls across local clusters.
You will need to add
127.0.0.1 garden.local.gardener.cloud
to your /etc/hosts.
The local registry can now be accessed either via localhost:5001
or garden.local.gardener.cloud:5001
for pushing and pulling.
The storage directories of the registries are mounted to your machine under dev/local-registry
.
With this, mirrored images don’t have to be pulled again after recreating the cluster.
The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy
support (the default kindnet
CNI doesn’t provide NetworkPolicy
support).
Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.
Setting Up Gardener (Garden on KinD, Seed on Gardener Cluster)
make gardener-extensions-up
This will first prepare the basic configuration of your KinD and Gardener clusters.
Afterwards, the images for the Garden cluster are built and deployed into the KinD cluster.
Finally, the images for the Seed cluster are built, pushed to a container registry on the Seed, and the gardenlet
is started.
If support for workload identity is required you can invoke the top command with DEV_SETUP_WITH_WORKLOAD_IDENTITY_SUPPORT
variable set to true
.
This will cause the Gardener Discovery Server to be deployed and exposed through the seed cluster.
External systems can be then configured to trust the workload identity issuer of the local Garden cluster.
DEV_SETUP_WITH_WORKLOAD_IDENTITY_SUPPORT=true make gardener-extensions-up
❗ Important
The Gardener Discovery Server is started with a token which is valid for 48 hours. Rerun
DEV_SETUP_WITH_WORKLOAD_IDENTITY_SUPPORT=true make gardener-extensions-up
in order to renew the token.When working with multiple seed clusters you need to only pass
DEV_SETUP_WITH_WORKLOAD_IDENTITY_SUPPORT=true
for the one seed cluster that will be used to expose the workload identity documents. A single Garden cluster needs only one Gardener Discovery Server.
Adding Additional Seeds
Additional seed(s) can be added by running
make gardener-extensions-up SEED_NAME=<seed-name>
The seed cluster preparations are similar to the first seed:
The kubeconfig
of your Kubernetes cluster you would like to use as seed should be placed at ./example/provider-extensions/seed/kubeconfig-<seed-name>
.
Additionally, please maintain the configuration of your seed in ./example/provider-extensions/gardenlet/values-<seed-name>.yaml
. It is automatically copied from values.yaml.tmpl
in the same directory when you run make gardener-extensions-up SEED_NAME=<seed-name>
for the first time. It also includes explanations of the properties you should set.
Removing a Seed
If you have multiple seeds and want to remove one, just use
make gardener-extensions-down SEED_NAME=<seed-name>
If it is not the last seed, this command will only remove the seed, but leave the local Gardener cluster and the other seeds untouched. To remove all seeds and to cleanup the local Gardener cluster, you have to run the command for each seed.
💡 Tip
If using development setup that supports workload identity pass
DEV_SETUP_WITH_WORKLOAD_IDENTITY_SUPPORT=true
when removing the seed that was used to host the Gardener Discovery Server.DEV_SETUP_WITH_WORKLOAD_IDENTITY_SUPPORT=true make gardener-extensions-down SEED_NAME=<seed-name>
Rotate credentials of container image registry in a Seed
There is a container image registry in each Seed cluster where Gardener images required for the Seed and the Shoot nodes are pushed to. This registry is password protected.
The password is generated when the Seed is deployed via make gardener-extensions-up
. Afterward, it is not rotated automatically.
Otherwise, this could break the update of gardener-node-agent
, because it might not be able to pull its own new image anymore
This is no general issue of gardener-node-agent
, but a limitation provider-extensions
setup. Gardener does not support protected container images out of the box. The function was added for this scenario only.
However, if you want to rotate the credentials for any reason, there are two options for it.
- run
make gardener-extensions-up
(to ensure that your images are up-to-date) reconcile
all shoots on the seed where you want to rotate the registry password- run
kubectl delete secrets -n registry registry-password
on your seed cluster - run
make gardener-extensions-up
reconcile
the shoots again
or
reconcile
all shoots on the seed where you want to rotate the registry password- run
kubectl delete secrets -n registry registry-password
on your seed cluster - run
./example/provider-extensions/registry-seed/deploy-registry.sh <path to seed kubeconfig> <seed registry hostname>
reconcile
the shoots again
Pause and Unpause the KinD Cluster
The KinD cluster can be paused by stopping and keeping its docker container. This can be done by running:
make kind-extensions-down
When you run make kind-extensions-up
again, you will start the docker container with your previous Gardener configuration again.
This provides the option to switch off your local KinD cluster fast without leaving orphaned infrastructure elements behind.
Creating a Shoot
Cluster
You can wait for the Seed
to be ready by running:
kubectl wait --for=condition=gardenletready seed provider-extensions --timeout=5m
make kind-extensions-up
already includes such a check. However, it might be useful when you wake up your Seed
from hibernation or unpause you KinD cluster.
Alternatively, you can run kubectl get seed provider-extensions
and wait for the STATUS
to indicate readiness:
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
provider-extensions Ready gcp europe-west1 111m v1.61.0-dev v1.24.7
In order to create a first shoot cluster, please create your own Shoot
definition and apply it to your KinD cluster. gardener-scheduler
includes candidateDeterminationStrategy: MinimalDistance
configuration so you are able to run schedule Shoot
s of different providers on your Seed
.
You can wait for your Shoot
s to be ready by running kubectl -n garden-local get shoots
and wait for the LAST OPERATION
to reach 100%
. The output depends on your Shoot
definition. This is an example output:
NAME CLOUDPROFILE PROVIDER REGION K8S VERSION HIBERNATION LAST OPERATION STATUS AGE
aws aws aws eu-west-1 1.24.3 Awake Create Processing (43%) healthy 84s
aws-arm64 aws aws eu-west-1 1.24.3 Awake Create Processing (43%) healthy 65s
azure az azure westeurope 1.24.2 Awake Create Processing (43%) healthy 57s
gcp gcp gcp europe-west1 1.24.3 Awake Create Processing (43%) healthy 94s
Accessing the Shoot
Cluster
Your shoot clusters will have a public DNS entries for their API servers, so that they could be reached via the Internet via kubectl
after you have created their kubeconfig
.
We encourage you to use the adminkubeconfig subresource for accessing your shoot cluster. You can find an example how to use it in Accessing Shoot Clusters.
Deleting the Shoot
Clusters
Before tearing down your environment, you have to delete your shoot clusters. This is highly recommended because otherwise you would leave orphaned items on your infrastructure accounts.
./hack/usage/delete shoot <your-shoot> garden-local
Tear Down the Gardener Environment
Before you delete your local KinD cluster, you should shut down your Shoots
and Seed
in a clean way to avoid orphaned infrastructure elements in your projects.
Please ensure that your KinD and Seed clusters are online (not paused or hibernated) and run:
make gardener-extensions-down
This will delete all Shoots
first (this could take a couple of minutes), then uninstall gardenlet
from the Seed and the gardener components from the KinD. Finally, the additional components like container registry, etc., are deleted from both clusters.
When this is done, you can securely delete your local KinD cluster by running:
make kind-extensions-clean
10 - Image Vector
Image Vector
The Gardener components are deploying several different container images into the garden, seed, and the shoot clusters. The image repositories and tags are defined in a central image vector file. Obviously, the image versions defined there must fit together with the deployment manifests (e.g., some command-line flags do only exist in certain versions).
Example
images:
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.4"
targetVersion: "1.20.x"
architectures:
- amd64
- arm64
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
ref: registry.k8s.io/pause:3.5
targetVersion: ">= 1.21"
architectures:
- amd64
- arm64
That means that Gardener will use the pause-container
with tag 3.4
for all clusters with Kubernetes version 1.20.x
, and the image with ref registry.k8s.io/pause:3.5
for all clusters with Kubernetes >= 1.21
.
ℹ️ Note
As you can see, it is possible to provide the full image reference via the
ref
field. Another option is to use therepository
andtag
fields.tag
may also be a digest only (starting withsha256:...
), or it can contain both tag and digest (v1.2.3@sha256:...
).
Architectures
images:
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.5"
architectures:
- amd64
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
ref: registry.k8s.io/pause:3.5
architectures:
- arm64
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
ref: registry.k8s.io/pause:3.5
architectures:
- amd64
- arm64
architectures
is an optional field of image. It is a list of strings specifying CPU architecture of machines on which this image can be used. The valid options for the architectures field are as follows:
amd64
: This specifies that the image can run only on machines having CPU architectureamd64
.arm64
: This specifies that the image can run only on machines having CPU architecturearm64
.
If an image doesn’t specify any architectures, then by default it is considered to support both amd64
and arm64
architectures.
Overwriting Image Vector
In some environments it is not possible to use these “pre-defined” images that come with a Gardener release.
A prominent example for that is Alicloud in China, which does not allow access to Google’s GCR.
In these cases, you might want to overwrite certain images, e.g., point the pause-container
to a different registry.
⚠️ If you specify an image that does not fit to the resource manifest, then the reconciliations might fail.
In order to overwrite the images, you must provide a similar file to the Gardener component:
images:
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: my-custom-image-registry/pause
tag: "3.4"
version: "1.20.x"
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
ref: my-custom-image-registry/pause:3.5
version: ">= 1.21"
❗ Important
When the overwriting file contains
ref
for an image but the source file doesn’t, then this invalidates bothrepository
andtag
of the source. When it containsrepository
for an image but the source file usesref
, then this invalidatesref
of the source.
For gardenlet
, you can create a ConfigMap
containing the above content and mount it as a volume into the gardenlet
pod.
Next, specify the environment variable IMAGEVECTOR_OVERWRITE
, whose value must be the path to the file you just mounted.
The approach works similarly for gardener-operator
.
apiVersion: v1
kind: ConfigMap
metadata:
name: gardenlet-images-overwrite
namespace: garden
data:
images_overwrite.yaml: |
images:
- ...
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: gardenlet
namespace: garden
spec:
template:
spec:
containers:
- name: gardenlet
env:
- name: IMAGEVECTOR_OVERWRITE
value: /imagevector-overwrite/images_overwrite.yaml
volumeMounts:
- name: gardenlet-images-overwrite
mountPath: /imagevector-overwrite
volumes:
- name: gardenlet-images-overwrite
configMap:
name: gardenlet-images-overwrite
Image Vectors for Dependent Components
Gardener is deploying a lot of different components that might deploy other images themselves. These components might use an image vector as well. Operators might want to customize the image locations for these transitive images as well, hence, they might need to specify an image vector overwrite for the components directly deployed by Gardener.
It is possible to specify the IMAGEVECTOR_OVERWRITE_COMPONENTS
environment variable to Gardener that points to a file with the following content:
components:
- name: etcd-druid
imageVectorOverwrite: |
images:
- name: etcd
tag: v1.2.3
repository: etcd/etcd
Gardener will, if supported by the directly deployed component (etcd-druid
in this example), inject the given imageVectorOverwrite
into the Deployment
manifest.
The respective component is responsible for using the overwritten images instead of its defaults.
Helm Chart Image Vector
Some Gardener components might also deploy packaged Helm charts which are pulled from an OCI repository.
The concepts are the very same as for the container images.
The only difference is that the environment variable for overwriting this chart image vector is called IMAGEVECTOR_OVERWRITE_CHARTS
.
11 - Migration V0 To V1
Migration from Gardener v0
to v1
Please refer to the document for older Gardener versions.
12 - Scoped API Access for gardenlets and Extensions
Scoped API Access for gardenlets and Extensions
By default, gardenlet
s have administrative access in the garden cluster.
They are able to execute any API request on any object independent of whether the object is related to the seed cluster the gardenlet
is responsible for.
As RBAC is not powerful enough for fine-grained checks and for the sake of security, Gardener provides two optional but recommended configurations for your environments that scope the API access for gardenlet
s.
Similar to the Node
authorization mode in Kubernetes, Gardener features a SeedAuthorizer
plugin.
It is a special-purpose authorization plugin that specifically authorizes API requests made by the gardenlet
s.
Likewise, similar to the NodeRestriction
admission plugin in Kubernetes, Gardener features a SeedRestriction
plugin.
It is a special-purpose admission plugin that specifically limits the Kubernetes objects gardenlet
s can modify.
📚 You might be interested to look into the design proposal for scoped Kubelet API access from the Kubernetes community.
It can be translated to Gardener and Gardenlets with their Seed
and Shoot
resources.
Historically, gardenlet
has been the only component running in the seed cluster that has access to both the seed cluster and the garden cluster.
Starting from Gardener v1.74.0
, extensions running on seed clusters can also get access to the garden cluster using a token for a dedicated ServiceAccount.
Extensions using this mechanism only get permission to read global resources like CloudProfiles
(this is granted to all authenticated users) unless the plugins described in this document are enabled.
Generally, the plugins handle extension clients exactly like gardenlet clients with some minor exceptions.
Extension clients in the sense of the plugins are clients authenticated as a ServiceAccount
with the extension-
name prefix in a seed-
namespace of the garden cluster.
Other ServiceAccounts
are not considered as seed clients, not handled by the plugins, and only get the described read access to global resources.
Flow Diagram
The following diagram shows how the two plugins are included in the request flow of a gardenlet
.
When they are not enabled, then the kube-apiserver
is internally authorizing the request via RBAC before forwarding the request directly to the gardener-apiserver
, i.e., the gardener-admission-controller
would not be consulted (this is not entirely correct because it also serves other admission webhook handlers, but for simplicity reasons this document focuses on the API access scope only).
When enabling the plugins, there is one additional step for each before the gardener-apiserver
responds to the request.
Please note that the example shows a request to an object (Shoot
) residing in one of the API groups served by gardener-apiserver
.
However, the gardenlet
is also interacting with objects in API groups served by the kube-apiserver
(e.g., Secret
,ConfigMap
).
In this case, the consultation of the SeedRestriction
admission plugin is performed by the kube-apiserver
itself before it forwards the request to the gardener-apiserver
.
Implemented Rules
Today, the following rules are implemented:
Resource | Verbs | Path(s) | Description |
---|---|---|---|
BackupBucket | get , list , watch , create , update , patch , delete | BackupBucket -> Seed | Allow get , list , watch requests for all BackupBucket s. Allow only create , update , patch , delete requests for BackupBucket s assigned to the gardenlet ’s Seed . |
BackupEntry | get , list , watch , create , update , patch | BackupEntry -> Seed | Allow get , list , watch requests for all BackupEntry s. Allow only create , update , patch requests for BackupEntry s assigned to the gardenlet ’s Seed and referencing BackupBucket s assigned to the gardenlet ’s Seed . |
Bastion | get , list , watch , create , update , patch | Bastion -> Seed | Allow get , list , watch requests for all Bastion s. Allow only create , update , patch requests for Bastion s assigned to the gardenlet ’s Seed . |
CertificateSigningRequest | get , create | CertificateSigningRequest -> Seed | Allow only get , create requests for CertificateSigningRequest s related to the gardenlet ’s Seed . |
CloudProfile | get | CloudProfile -> Shoot -> Seed | Allow only get requests for CloudProfile s referenced by Shoot s that are assigned to the gardenlet ’s Seed . |
ClusterRoleBinding | create , get , update , patch , delete | ClusterRoleBinding -> ManagedSeed -> Shoot -> Seed | Allow create , get , update , patch requests for ManagedSeed s in the bootstrapping phase assigned to the gardenlet’s Seed s. Allow delete requests from gardenlets bootstrapped via ManagedSeed s. |
ConfigMap | get | ConfigMap -> Shoot -> Seed | Allow only get requests for ConfigMap s referenced by Shoot s that are assigned to the gardenlet ’s Seed . Allows reading the kube-system/cluster-identity ConfigMap . |
ControllerRegistration | get , list , watch | ControllerRegistration -> ControllerInstallation -> Seed | Allow get , list , watch requests for all ControllerRegistration s. |
ControllerDeployment | get | ControllerDeployment -> ControllerInstallation -> Seed | Allow get requests for ControllerDeployments s referenced by ControllerInstallation s assigned to the gardenlet ’s Seed . |
ControllerInstallation | get , list , watch , update , patch | ControllerInstallation -> Seed | Allow get , list , watch requests for all ControllerInstallation s. Allow only update , patch requests for ControllerInstallation s assigned to the gardenlet ’s Seed . |
CredentialsBinding | get | CredentialsBinding -> Shoot -> Seed | Allow only get requests for CredentialsBinding s referenced by Shoot s that are assigned to the gardenlet ’s Seed . |
Event | create , patch | none | Allow to create or patch all kinds of Event s. |
ExposureClass | get | ExposureClass -> Shoot -> Seed | Allow get requests for ExposureClass es referenced by Shoot s that are assigned to the gardenlet ’s Seed . Deny get requests to other ExposureClass es. |
Gardenlet | get , list , watch , update , patch , create | Gardenlet -> Seed | Allow get , list , watch requests for all Gardenlet s. Allow only create , update , and patch requests for Gardenlet s belonging to the gardenlet ’s Seed . |
Lease | create , get , watch , update | Lease -> Seed | Allow create , get , update , and delete requests for Lease s of the gardenlet ’s Seed . |
ManagedSeed | get , list , watch , update , patch | ManagedSeed -> Shoot -> Seed | Allow get , list , watch requests for all ManagedSeed s. Allow only update , patch requests for ManagedSeed s referencing a Shoot assigned to the gardenlet ’s Seed . |
Namespace | get | Namespace -> Shoot -> Seed | Allow get requests for Namespace s of Shoot s that are assigned to the gardenlet ’s Seed . Always allow get requests for the garden Namespace . |
NamespacedCloudProfile | get | NamespacedCloudProfile -> Shoot -> Seed | Allow only get requests for NamespacedCloudProfile s referenced by Shoot s that are assigned to the gardenlet ’s Seed . |
Project | get | Project -> Namespace -> Shoot -> Seed | Allow get requests for Project s referenced by the Namespace of Shoot s that are assigned to the gardenlet ’s Seed . |
SecretBinding | get | SecretBinding -> Shoot -> Seed | Allow only get requests for SecretBinding s referenced by Shoot s that are assigned to the gardenlet ’s Seed . |
Secret | create , get , update , patch , delete (, list , watch ) | Secret -> Seed , Secret -> Shoot -> Seed , Secret -> SecretBinding -> Shoot -> Seed , Secret -> CredentialsBinding -> Shoot -> Seed , BackupBucket -> Seed | Allow get , list , watch requests for all Secret s in the seed-<name> namespace. Allow only create , get , update , patch , delete requests for the Secret s related to resources assigned to the gardenlet ’s Seed s. |
Seed | get , list , watch , create , update , patch , delete | Seed | Allow get , list , watch requests for all Seed s. Allow only create , update , patch , delete requests for the gardenlet ’s Seed s. [1] |
ServiceAccount | create , get , update , patch , delete | ServiceAccount -> ManagedSeed -> Shoot -> Seed , ServiceAccount -> Namespace -> Seed | Allow create , get , update , patch requests for ManagedSeed s in the bootstrapping phase assigned to the gardenlet ’s Seed s. Allow delete requests from gardenlets bootstrapped via ManagedSeed s. Allow all verbs on ServiceAccount s in seed-specific namespace. |
Shoot | get , list , watch , update , patch | Shoot -> Seed | Allow get , list , watch requests for all Shoot s. Allow only update , patch requests for Shoot s assigned to the gardenlet ’s Seed . |
ShootState | get , create , update , patch | ShootState -> Shoot -> Seed | Allow only get , create , update , patch requests for ShootState s belonging by Shoot s that are assigned to the gardenlet ’s Seed . |
WorkloadIdentity | get | WorkloadIdentity -> CredentialsBinding -> Shoot -> Seed | Allow only get requests for WorkloadIdentities referenced by CredentialsBinding s referenced by Shoot s that are assigned to the gardenlet ’s Seed . |
[1] If you use
ManagedSeed
resources then thegardenlet
reconciling them (“parentgardenlet
”) may be allowed to submit certain requests for theSeed
resources resulting out of suchManagedSeed
reconciliations (even if the “parentgardenlet
” is not responsible for them):
ℹ️ It is allowed to delete the Seed
resources if the corresponding ManagedSeed
objects already have a deletionTimestamp
(this is secure as gardenlet
s themselves don’t have permissions for deleting ManagedSeed
s).
Rule Exceptions for Extension Clients
Extension clients are allowed to perform the same operations as gardenlet clients with the following exceptions:
- Extension clients are granted the read-only subset of verbs for
CertificateSigningRequests
,ClusterRoleBindings
, andServiceAccounts
(to prevent privilege escalation). - Extension clients are granted full access to
Lease
objects but only in the seed-specific namespace.
When the need arises, more exceptions might be added to the access rules for resources that are already handled by the plugins.
E.g., if an extension needs to populate additional shoot-specific InternalSecrets
, according handling can be introduced.
Permissions for resources that are not handled by the plugins can be granted using additional RBAC rules (independent of the plugins).
SeedAuthorizer
Authorization Webhook Enablement
The SeedAuthorizer
is implemented as a Kubernetes authorization webhook and part of the gardener-admission-controller
component running in the garden cluster.
🎛 In order to activate it, you have to follow these steps:
Set the following flags for the
kube-apiserver
of the garden cluster (i.e., thekube-apiserver
whose API is extended by Gardener):--authorization-mode=RBAC,Node,Webhook
(please note thatWebhook
should appear afterRBAC
in the list [1];Node
might not be needed if you use a virtual garden cluster)--authorization-webhook-config-file=<path-to-the-webhook-config-file>
--authorization-webhook-cache-authorized-ttl=0
--authorization-webhook-cache-unauthorized-ttl=0
The webhook config file (stored at
<path-to-the-webhook-config-file>
) should look as follows:apiVersion: v1 kind: Config clusters: - name: garden cluster: certificate-authority-data: base64(CA-CERT-OF-GARDENER-ADMISSION-CONTROLLER) server: https://gardener-admission-controller.garden/webhooks/auth/seed users: - name: kube-apiserver user: {} contexts: - name: auth-webhook context: cluster: garden user: kube-apiserver current-context: auth-webhook
When deploying the Gardener
controlplane
Helm chart, set.global.rbac.seedAuthorizer.enabled=true
. This will ensure that the RBAC resources granting global access for allgardenlet
s will be deployed.Delete the existing RBAC resources granting global access for all
gardenlet
s by running:kubectl delete \ clusterrole.rbac.authorization.k8s.io/gardener.cloud:system:seeds \ clusterrolebinding.rbac.authorization.k8s.io/gardener.cloud:system:seeds \ --ignore-not-found
Please note that you should activate the SeedRestriction
admission handler as well.
[1] The reason for the fact that
Webhook
authorization plugin should appear afterRBAC
is that thekube-apiserver
will be depending on thegardener-admission-controller
(serving the webhook). However, thegardener-admission-controller
can only start whengardener-apiserver
runs, butgardener-apiserver
itself can only start whenkube-apiserver
runs. IfWebhook
is beforeRBAC
, thengardener-apiserver
might not be able to start, leading to a deadlock.
Authorizer Decisions
As mentioned earlier, it’s the authorizer’s job to evaluate API requests and return one of the following decisions:
DecisionAllow
: The request is allowed, further configured authorizers won’t be consulted.DecisionDeny
: The request is denied, further configured authorizers won’t be consulted.DecisionNoOpinion
: A decision cannot be made, further configured authorizers will be consulted.
For backwards compatibility, no requests are denied at the moment, so that they are still deferred to a subsequent authorizer like RBAC. Though, this might change in the future.
First, the SeedAuthorizer
extracts the Seed
name from the API request.
This step considers the following two cases:
- If the authenticated user belongs to the
gardener.cloud:system:seeds
group, it is considered a gardenlet client.- This requires a proper TLS certificate that the
gardenlet
uses to contact the API server and is automatically given if TLS bootstrapping is used. - The authorizer extracts the seed name from the username by stripping the
gardener.cloud:system:seed:
prefix. - In cases where this information is missing e.g., when a custom Kubeconfig is used, the authorizer cannot make any decision. Thus, RBAC is still a considerable option to restrict the
gardenlet
’s access permission if the above explained preconditions are not given.
- This requires a proper TLS certificate that the
- If the authenticated user belongs to the
system:serviceaccounts
group, it is considered an extension client under the following conditions:- The
ServiceAccount
must be located in aseed-
namespace. I.e., the user has to belong to a group with thesystem:serviceaccounts:seed-
prefix. The seed name is extracted from this group by stripping the prefix. - The
ServiceAccount
must have theextension-
prefix. I.e., the username must have thesystem:serviceaccount:seed-<seed-name>:extension-
prefix.
- The
With the Seed
name at hand, the authorizer checks for an existing path from the resource that a request is being made for to the Seed
belonging to the gardenlet
/extension.
Take a look at the Implementation Details section for more information.
Implementation Details
Internally, the SeedAuthorizer
uses a directed, acyclic graph data structure in order to efficiently respond to authorization requests for gardenlet
s/extensions:
- A vertex in this graph represents a Kubernetes resource with its kind, namespace, and name (e.g.,
Shoot:garden-my-project/my-shoot
). - An edge from vertex
u
to vertexv
in this graph exists when- (1)
v
is referred byu
andv
is aSeed
, or when - (2)
u
is referred byv
, or when - (3)
u
is strictly associated withv
.
- (1)
For example, a Shoot
refers to a Seed
, a CloudProfile
, a SecretBinding
, etc., so it has an outgoing edge to the Seed
(1) and incoming edges from the CloudProfile
and SecretBinding
vertices (2).
However, there might also be a ShootState
or a BackupEntry
resource strictly associated with this Shoot
, hence, it has incoming edges from these vertices (3).
In the above picture, the resources that are actively watched are shaded. Gardener resources are green, while Kubernetes resources are blue. It shows the dependencies between the resources and how the graph is built based on the above rules.
ℹ️ The above picture shows all resources that may be accessed by gardenlet
s/extensions, except for the Quota
resource which is only included for completeness.
Now, when a gardenlet
/extension wants to access certain resources, then the SeedAuthorizer
uses a Depth-First traversal starting from the vertex representing the resource in question, e.g., from a Project
vertex.
If there is a path from the Project
vertex to the vertex representing the Seed
the gardenlet
/extension is responsible for. then it allows the request.
Metrics
The SeedAuthorizer
registers the following metrics related to the mentioned graph implementation:
Metric | Description |
---|---|
gardener_admission_controller_seed_authorizer_graph_update_duration_seconds | Histogram of duration of resource dependency graph updates in seed authorizer, i.e., how long does it take to update the graph’s vertices/edges when a resource is created, changed, or deleted. |
gardener_admission_controller_seed_authorizer_graph_path_check_duration_seconds | Histogram of duration of checks whether a path exists in the resource dependency graph in seed authorizer. |
Debug Handler
When the .server.enableDebugHandlers
field in the gardener-admission-controller
’s component configuration is set to true
, then it serves a handler that can be used for debugging the resource dependency graph under /debug/resource-dependency-graph
.
🚨 Only use this setting for development purposes, as it enables unauthenticated users to view all data if they have access to the gardener-admission-controller
component.
The handler renders an HTML page displaying the current graph with a list of vertices and its associated incoming and outgoing edges to other vertices.
Depending on the size of the Gardener landscape (and consequently, the size of the graph), it might not be possible to render it in its entirety.
If there are more than 2000 vertices, then the default filtering will selected for kind=Seed
to prevent overloading the output.
Example output:
-------------------------------------------------------------------------------
|
| # Seed:my-seed
| <- (11)
| BackupBucket:73972fe2-3d7e-4f61-a406-b8f9e670e6b7
| BackupEntry:garden-my-project/shoot--dev--my-shoot--4656a460-1a69-4f00-9372-7452cbd38ee3
| ControllerInstallation:dns-external-mxt8m
| ControllerInstallation:extension-shoot-cert-service-4qw5j
| ControllerInstallation:networking-calico-bgrb2
| ControllerInstallation:os-gardenlinux-qvb5z
| ControllerInstallation:provider-gcp-w4mvf
| Secret:garden/backup
| Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------
|
| # Shoot:garden-my-project/my-shoot
| <- (5)
| CloudProfile:gcp
| Namespace:garden-my-project
| Secret:garden-my-project/my-dns-secret
| SecretBinding:garden-my-project/my-credentials
| ShootState:garden-my-project/my-shoot
| -> (1)
| Seed:my-seed
|
-------------------------------------------------------------------------------
|
| # ShootState:garden-my-project/my-shoot
| -> (1)
| Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------
... (etc., similarly for the other resources)
There are anchor links to easily jump from one resource to another, and the page provides means for filtering the results based on the kind
, namespace
, and/or name
.
Pitfalls
When there is a relevant update to an existing resource, i.e., when a reference to another resource is changed, then the corresponding vertex (along with all associated edges) is first deleted from the graph before it gets added again with the up-to-date edges.
However, this does only work for vertices belonging to resources that are only created in exactly one “watch handler”.
For example, the vertex for a SecretBinding
can either be created in the SecretBinding
handler itself or in the Shoot
handler.
In such cases, deleting the vertex before (re-)computing the edges might lead to race conditions and potentially renders the graph invalid.
Consequently, instead of deleting the vertex, only the edges the respective handler is responsible for are deleted.
If the vertex ends up with no remaining edges, then it also gets deleted automatically.
Afterwards, the vertex can either be added again or the updated edges can be created.
SeedRestriction
Admission Webhook Enablement
The SeedRestriction
is implemented as Kubernetes admission webhook and part of the gardener-admission-controller
component running in the garden cluster.
🎛 In order to activate it, you have to set .global.admission.seedRestriction.enabled=true
when using the Gardener controlplane
Helm chart.
This will add an additional webhook in the existing ValidatingWebhookConfiguration
of the gardener-admission-controller
which contains the configuration for the SeedRestriction
handler.
Please note that it should only be activated when the SeedAuthorizer
is active as well.
Admission Decisions
The admission’s purpose is to perform extended validation on requests which require the body of the object in question.
Additionally, it handles CREATE
requests of gardenlet
s/extensions (the above discussed resource dependency graph cannot be used in such cases because there won’t be any vertex/edge for non-existing resources).
Gardenlets/extensions are restricted to only create new resources which are somehow related to the seed clusters they are responsible for.
13 - Secret Binding Provider Controller
SecretBinding Provider Controller
This page describes the process on how to enable the SecretBinding provider controller.
Overview
With Gardener v1.38.0, the SecretBinding resource now contains a new optional field .provider.type
(details about the motivation can be found in https://github.com/gardener/gardener/issues/4888). To make the process of setting the new field automated and afterwards to enforce validation on the new field in backwards compatible manner, Gardener features the SecretBinding provider controller and a feature gate - SecretBindingProviderValidation
.
Process
A Gardener landscape operator can follow the following steps:
Enable the SecretBinding provider controller of Gardener Controller Manager.
The SecretBinding provider controller is responsible for populating the
.provider.type
field of a SecretBinding based on its current usage by Shoot resources. For example, if a Shootcrazy-botany
with.provider.type=aws
is using a SecretBindingmy-secret-binding
, then the SecretBinding provider controller will take care to set the.provider.type
field of the SecretBinding to the same provider type (aws
). To enable the SecretBinding provider controller, set thecontroller.secretBindingProvider.concurrentSyncs
field in the ControllerManagerConfiguration (e.g set it to5
). Although that it is not recommended, the API allows Shoots from different provider types to reference the same SecretBinding (assuming that the backing Secret contains data for both of the provider types). To preserve the backwards compatibility for such SecretBindings, the provider controller will maintain the multiple provider types in the field (it will join them with the separator,
- for exampleaws,gcp
).Disable the SecretBinding provider controller and enable the
SecretBindingProviderValidation
feature gate of Gardener API server.The
SecretBindingProviderValidation
feature gate of Gardener API server enables a set of validations for the SecretBinding provider field. It forbids creating a Shoot that has a different provider type from the referenced SecretBinding’s one. It also enforces immutability on the field. After making sure that SecretBinding provider controller is enabled and it populated the.provider.type
field of a majority of the SecretBindings on a Gardener landscape (the SecretBindings that are unused will have their provider type unset), a Gardener landscape operator has to disable the SecretBinding provider controller and to enable theSecretBindingProviderValidation
feature gate of Gardener API server. To disable the SecretBinding provider controller, set thecontroller.secretBindingProvider.concurrentSyncs
field in the ControllerManagerConfiguration to0
.
Implementation History
- Gardener v1.38: The SecretBinding resource has a new optional field
.provider.type
. The SecretBinding provider controller is disabled by default. TheSecretBindingProviderValidation
feature gate of Gardener API server is disabled by default. - Gardener v1.42: The SecretBinding provider controller is enabled by default.
- Gardener v1.51: The
SecretBindingProviderValidation
feature gate of Gardener API server is enabled by default and the SecretBinding provider controller is disabled by default. - Gardener v1.53: The
SecretBindingProviderValidation
feature gate of Gardener API server is unconditionally enabled (can no longer be disabled). - Gardener v1.55: The
SecretBindingProviderValidation
feature gate of Gardener API server and the SecretBinding provider controller are removed.
14 - Setup Gardener
Deploying Gardener into a Kubernetes Cluster
Similar to Kubernetes, Gardener consists out of control plane components (Gardener API server, Gardener controller manager, Gardener scheduler), and an agent component (gardenlet). The control plane is deployed in the so-called garden cluster, while the agent is installed into every seed cluster. Please note that it is possible to use the garden cluster as seed cluster by simply deploying the gardenlet into it.
We are providing Helm charts in order to manage the various resources of the components. Please always make sure that you use the Helm chart version that matches the Gardener version you want to deploy.
Deploying the Gardener Control Plane (API Server, Admission Controller, Controller Manager, Scheduler)
In order to deploy the control plane components, please first deploy gardener-operator
and create a Garden
resource.
🚨 Caution
Below approach is deprecated and will be removed after v1.135 of Gardener has been released (around beginning of 2026).
The configuration values depict the various options to configure the different components. Please consult Gardener Configuration and Usage for component specific configurations and Authentication of Gardener Control Plane Components Against the Garden Cluster for authentication related specifics.
Also, note that all resources and deployments need to be created in the garden
namespace (not overrideable).
If you enable the Gardener admission controller as part of you setup, please make sure the garden
namespace is labelled with app: gardener
.
Otherwise, the backing service account for the admission controller Pod might not be created successfully.
No action is necessary if you deploy the garden
namespace with the Gardener control plane Helm chart.
After preparing your values in a separate controlplane-values.yaml
file (values.yaml can be used as starting point), you can run the following command against your garden cluster:
helm install charts/gardener/controlplane \
--namespace garden \
--name gardener-controlplane \
-f controlplane-values.yaml \
--wait
Deploying Gardener Extensions
Gardener is an extensible system that does not contain the logic for provider-specific things like DNS management, cloud infrastructures, network plugins, operating system configs, and many more.
You have to install extension controllers for these parts. Please consult the documentation regarding extensions to get more information.
Deploying the Gardener Agent (gardenlet)
Please refer to Deploying Gardenlets on how to deploy a gardenlet.
15 - Version Skew Policy
Version Skew Policy
This document describes the maximum version skew supported between various Gardener components.
Supported Gardener Versions
Gardener versions are expressed as x.y.z
, where x
is the major version, y
is the minor version, and z
is the patch version, following Semantic Versioning terminology.
The Gardener project maintains release branches for the three most recent minor releases.
Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility. Patch releases are cut from those branches at a regular cadence, plus additional urgent releases when required.
For more information, see the Releases document.
Supported Version Skew
Technically, we follow the same policy as the Kubernetes project.
However, given that our release cadence is much more frequent compared to Kubernetes (every 14d
vs. every 120d
), in many cases it might be possible to skip versions, though we do not test these upgrade paths.
Consequently, in general it might not work, and to be on the safe side, it is highly recommended to follow the described policy.
🚨 Note that downgrading Gardener versions is generally not tested during development and should be considered unsupported.
gardener-apiserver
In multi-instance setups of Gardener, the newest and oldest gardener-apiserver
instances must be within one minor version.
Example:
- newest
gardener-apiserver
is at 1.37 - other
gardener-apiserver
instances are supported at 1.37 and 1.36
gardener-controller-manager, gardener-scheduler, gardener-admission-controller
gardener-controller-manager
, gardener-scheduler
, and gardener-admission-controller
must not be newer than the gardener-apiserver
instances they communicate with.
They are expected to match the gardener-apiserver
minor version, but may be up to one minor version older (to allow live upgrades).
Example:
gardener-apiserver
is at 1.37gardener-controller-manager
,gardener-scheduler
, andgardener-admission-controller
are supported at 1.37 and 1.36
gardenlet
gardenlet
must not be newer thangardener-apiserver
gardenlet
may be up to two minor versions older thangardener-apiserver
Example:
gardener-apiserver
is at 1.37gardenlet
is supported at 1.37, 1.36, and 1.35
gardener-operator
Since gardener-operator
manages the Gardener control plane components (gardener-apiserver
, gardener-controller-manager
, gardener-scheduler
, gardener-admission-controller
), it follows the same policy as for gardener-apiserver
.
It implements additional start-up checks to ensure adherence to this policy.
Concretely, gardener-operator
will crash when
- its gets downgraded.
- its version gets upgraded and skips at least one minor version.
Supported Component Upgrade Order
The supported version skew between components has implications on the order in which components must be upgraded. This section describes the order in which components must be upgraded to transition an existing Gardener installation from version 1.37 to version 1.38.
gardener-apiserver
Prerequisites:
- In a single-instance setup, the existing
gardener-apiserver
instance is 1.37. - In a multi-instance setup, all
gardener-apiserver
instances are at 1.37 or 1.38 (this ensures maximum skew of 1 minor version between the oldest and newestgardener-apiserver
instance). - The
gardener-controller-manager
,gardener-scheduler
, andgardener-admission-controller
instances that communicate with thisgardener-apiserver
are at version 1.37 (this ensures they are not newer than the existing API server version and are within 1 minor version of the new API server version). gardenlet
instances on all seeds are at version 1.37 or 1.36 (this ensures they are not newer than the existing API server version and are within 2 minor versions of the new API server version).
Actions:
- Upgrade
gardener-apiserver
to 1.38.
gardener-controller-manager, gardener-scheduler, gardener-admission-controller
Prerequisites:
- The
gardener-apiserver
instances these components communicate with are at 1.38 (in multi-instance setups in which these components can communicate with anygardener-apiserver
instance in the cluster, allgardener-apiserver
instances must be upgraded before upgrading these components).
Actions:
- Upgrade
gardener-controller-manager
,gardener-scheduler
, andgardener-admission-controller
to 1.38
gardenlet
Prerequisites:
- The
gardener-apiserver
instances thegardenlet
communicates with are at 1.38.
Actions:
- Optionally upgrade
gardenlet
instances to 1.38 (or they can be left at 1.37 or 1.36).
⚠️ Warning
Running a landscape with
gardenlet
instances that are persistently two minor versions behindgardener-apiserver
means they must be upgraded before the Gardener control plane can be upgraded.
gardener-operator
Prerequisites:
- All
gardener-operator
instances are at 1.37.
Actions:
- Upgrade
gardener-operator
to 1.38.
Supported Gardener Extension Versions
Extensions are maintained and released separately and independently of the gardener/gardener
repository.
Consequently, providing version constraints is not possible in this document.
Sometimes, the documentation of extensions contains compatibility information (e.g., “this extension version is only compatible with Gardener versions higher than 1.80”, see this example).
However, since all extensions typically make use of the extensions library (example), a general constraint is that no extension must depend on a version of the extensions library higher than the version of gardenlet
.
Example 1:
gardener-apiserver
and other Gardener control plane components are at 1.37.- All
gardenlet
s are at 1.37. - Only extensions are supported which depend on 1.37 or lower of the extensions library.
Example 2:
gardener-apiserver
and other Gardener control plane components are at 1.37.- Some
gardenlet
s are at 1.37, others are at 1.36. - Only extensions are supported which depend on 1.36 or lower of the extensions library.
Supported Kubernetes Versions
Please refer to Supported Kubernetes Versions.