This is the multi-page printable view of this section. Click here to print.
Deployment
1 - Authentication Gardener Control Plane
Authentication of Gardener Control Plane Components Against the Garden Cluster
Note: This document refers to Gardener’s API server, admission controller, controller manager and scheduler components. Any reference to the term Gardener control plane component can be replaced with any of the mentioned above.
There are several authentication possibilities depending on whether or not the concept of Virtual Garden is used.
Virtual Garden is not used, i.e., the runtime
Garden cluster is also the target
Garden cluster.
Automounted Service Account Token
The easiest way to deploy a Gardener control plane component is to not provide a kubeconfig
at all. This way in-cluster configuration and an automounted service account token will be used. The drawback of this approach is that the automounted token will not be automatically rotated.
Service Account Token Volume Projection
Another solution is to use Service Account Token Volume Projection combined with a kubeconfig
referencing a token file (see the example below).
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: <CA-DATA>
server: https://default.kubernetes.svc.cluster.local
name: garden
contexts:
- context:
cluster: garden
user: garden
name: garden
current-context: garden
users:
- name: garden
user:
tokenFile: /var/run/secrets/projected/serviceaccount/token
This will allow for automatic rotation of the service account token by the kubelet
. The configuration can be achieved by setting both .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true
and .Values.global.<GardenerControlPlaneComponent>.kubeconfig
in the respective chart’s values.yaml
file.
Virtual Garden is used, i.e., the runtime
Garden cluster is different from the target
Garden cluster.
Service Account
The easiest way to setup the authentication is to create a service account and the respective roles will be bound to this service account in the target
cluster. Then use the generated service account token and craft a kubeconfig
, which will be used by the workload in the runtime
cluster. This approach does not provide a solution for the rotation of the service account token. However, this setup can be achieved by setting .Values.global.deployment.virtualGarden.enabled: true
and following these steps:
- Deploy the
application
part of the charts in thetarget
cluster. - Get the service account token and craft the
kubeconfig
. - Set the crafted
kubeconfig
and deploy theruntime
part of the charts in theruntime
cluster.
Client Certificate
Another solution is to bind the roles in the target
cluster to a User
subject instead of a service account and use a client certificate for authentication. This approach does not provide a solution for the client certificate rotation. However, this setup can be achieved by setting both .Values.global.deployment.virtualGarden.enabled: true
and .Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name
, then following these steps:
- Generate a client certificate for the
target
cluster for the respective user. - Deploy the
application
part of the charts in thetarget
cluster. - Craft a
kubeconfig
using the already generated client certificate. - Set the crafted
kubeconfig
and deploy theruntime
part of the charts in theruntime
cluster.
Projected Service Account Token
This approach requires an already deployed and configured oidc-webhook-authenticator for the target
cluster. Also, the runtime
cluster should be registered as a trusted identity provider in the target
cluster. Then, projected service accounts tokens from the runtime
cluster can be used to authenticate against the target
cluster. The needed steps are as follows:
- Deploy OWA and establish the needed trust.
- Set
.Values.global.deployment.virtualGarden.enabled: true
and.Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name
.Note: username value will depend on the trust configuration, e.g.,
<prefix>:system:serviceaccount:<namespace>:<serviceaccount>
- Set
.Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true
and.Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.audience
.Note: audience value will depend on the trust configuration, e.g.,
<cliend-id-from-trust-config>
. - Craft a kubeconfig (see the example below).
- Deploy the
application
part of the charts in thetarget
cluster. - Deploy the
runtime
part of the charts in theruntime
cluster.
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: <CA-DATA>
server: https://virtual-garden.api
name: virtual-garden
contexts:
- context:
cluster: virtual-garden
user: virtual-garden
name: virtual-garden
current-context: virtual-garden
users:
- name: virtual-garden
user:
tokenFile: /var/run/secrets/projected/serviceaccount/token
2 - Configuring Logging
Configuring the Logging Stack via gardenlet Configurations
Enable the Logging
In order to install the Gardener logging stack, the logging.enabled
configuration option has to be enabled in the Gardenlet configuration:
logging:
enabled: true
From now on, each Seed is going to have a logging stack which will collect logs from all pods and some systemd services. Logs related to Shoots with testing
purpose are dropped in the fluent-bit
output plugin. Shoots with a purpose different than testing
have the same type of log aggregator (but different instance) as the Seed. The logs can be viewed in the Grafana in the garden
namespace for the Seed components and in the respective shoot control plane namespaces.
Enable Logs from the Shoot’s Node systemd Services
The logs from the systemd services on each node can be retrieved by enabling the logging.shootNodeLogging
option in the gardenlet configuration:
logging:
enabled: true
shootNodeLogging:
shootPurposes:
- "evaluation"
- "deployment"
Under the shootPurpose
section, just list all the shoot purposes for which the Shoot node logging feature will be enabled. Specifying the testing
purpose has no effect because this purpose prevents the logging stack installation.
Logs can be viewed in the operator Grafana!
The dedicated labels are unit
, syslog_identifier
, and nodename
in the Explore
menu.
Configuring the Log Processor
Under logging.fluentBit
there are three optional sections:
input
: This overwrites the input configuration of the fluent-bit log processor.output
: This overwrites the output configuration of the fluent-bit log processor.service
: This overwrites the service configuration of the fluent-bit log processor.
logging:
enabled: true
fluentBit:
output: |-
[Output]
...
input: |-
[Input]
...
service: |-
[Service]
...
Additional egress IPBlock for allow-fluentbit NetworkPolicy
The optional setting under logging.fluentBit.networkPolicy.additionalEgressIPBlocks
adds an additional egress IPBlock to allow-fluentbit
NetworkPolicy to forward logs to a central system.
logging:
enabled: true
fluentBit:
additionalEgressIpBlock:
- 123.123.123.123/32
Configure Central Logging
For central logging, the output configuration of the fluent-bit log processor can be overwritten (logging.fluentBit.output
) and the Loki instances deployments in the Garden and Shoot namespace can be enabled/disabled (logging.loki.enabled
), by default Loki is enabled.
logging:
enabled: true
fluentBit:
output: |-
[Output]
...
loki:
enabled: false
Configuring Central Loki Storage Capacity
By default, the central Loki has 100Gi
of storage capacity.
To overwrite the current central Loki storage capacity, the logging.loki.garden.storage
setting in the gardenlet’s component configuration should be altered.
If you need to increase it, you can do so without losing the current data by specifying a higher capacity. By doing so, the Loki’s PersistentVolume
capacity will be increased instead of deleting the current PV.
However, if you specify less capacity, then the PersistentVolume
will be deleted and with it the logs, too.
logging:
enabled: true
fluentBit:
output: |-
[Output]
...
loki:
garden:
storage: "200Gi"
3 - Deploy Gardenlet
Deploying Gardenlets
Gardenlets act as decentral “agents” to manage the shoot clusters of a seed cluster.
To support scaleability in an automated way, gardenlets are deployed automatically. However, you can still deploy gardenlets manually to be more flexible, for example, when the shoot clusters that need to be managed by Gardener are behind a firewall. The gardenlet only requires network connectivity from the gardenlet to the Garden cluster (not the other way round), so it can be used to register Kubernetes clusters with no public endpoint.
Procedure
First, an initial gardenlet needs to be deployed:
- Deploy it manually if you have special requirements. For more information, see Deploy a Gardenlet Manually.
- Let the Gardener installer deploy it automatically otherwise. For more information, see Automatic Deployment of Gardenlets.
To add additional seed clusters, it is recommended to use regular shoot clusters. You can do this by creating a
ManagedSeed
resource with agardenlet
section as described in Register Shoot as Seed.
4 - Deploy Gardenlet Automatically
Automatic Deployment of gardenlets
The gardenlet can automatically deploy itself into shoot clusters, and register a cluster as a seed cluster. These clusters are called “managed seeds” (aka “shooted seeds”). This procedure is the preferred way to add additional seed clusters, because shoot clusters already come with production-grade qualities that are also demanded for seed clusters.
Prerequisites
The only prerequisite is to register an initial cluster as a seed cluster that already has a gardenlet deployed in one of the following ways:
- The gardenlet was deployed as part of a Gardener installation using a setup tool (for example,
gardener/garden-setup
). - The gardenlet was deployed manually (for a step-by-step manual installation guide, see Deploy a Gardenlet Manually).
The initial cluster can be the garden cluster itself.
Self-Deployment of gardenlets in Additional Managed Seed Clusters
For a better scalability, you usually need more seed clusters that you can create, as follows:
- Use the initial cluster as the seed cluster for other managed seed clusters. It hosts the control planes of the other seed clusters.
- The gardenlet deployed in the initial cluster deploys itself automatically into the managed seed clusters.
The advantage of this approach is that there’s only one initial gardenlet installation required. Every other managed seed cluster has a gardenlet deployed automatically.
Related Links
5 - Deploy Gardenlet Manually
Deploy a gardenlet Manually
Manually deploying a gardenlet is required in the following cases:
The Kubernetes cluster to be registered as a seed cluster has no public endpoint, because it is behind a firewall. The gardenlet must then be deployed into the cluster itself.
The Kubernetes cluster to be registered as a seed cluster is managed externally (the Kubernetes cluster is not a shoot cluster, so Automatic Deployment of Gardenlets cannot be used).
The gardenlet runs outside of the Kubernetes cluster that should be registered as a seed cluster. (The gardenlet is not restricted to run in the seed cluster or to be deployed into a Kubernetes cluster at all).
Once you’ve deployed a gardenlet manually, for example, behind a firewall, you can deploy new gardenlets automatically. The manually deployed gardenlet is then used as a template for the new gardenlets. For more information, see Automatic Deployment of Gardenlets.
Prerequisites
Kubernetes Cluster that Should Be Registered as a Seed Cluster
Verify that the cluster has a supported Kubernetes version.
Determine the nodes, pods, and services CIDR of the cluster. You need to configure this information in the
Seed
configuration. Gardener uses this information to check that the shoot cluster isn’t created with overlapping CIDR ranges.Every seed cluster needs an Ingress controller which distributes external requests to internal components like Grafana and Prometheus. For this, configure the following lines in your Seed resource:
spec:
dns:
provider:
type: aws-route53
secretRef:
name: ingress-secret
namespace: garden
ingress:
domain: ingress.my-seed.example.com
controller:
kind: nginx
providerConfig:
<some-optional-provider-specific-config-for-the-ingressController>
kubeconfig
for the Seed Cluster
The kubeconfig
is required to deploy the gardenlet Helm chart to the seed cluster.
The gardenlet requires certain privileges to be able to operate.
These privileges are described in RBAC resources in the gardenlet Helm chart (see charts/gardener/gardenlet/templates).
The Helm chart contains a service account gardenlet
that the gardenlet deployment uses by default to talk to the Seed API server.
If the gardenlet isn’t deployed in the seed cluster, the gardenlet can be configured to use a
kubeconfig
, which also requires the above-mentioned privileges, from a mounted directory. Thekubeconfig
is specified in theseedClientConnection.kubeconfig
section of the Gardenlet configuration. This configuration option isn’t used in the following, as this procedure only describes the recommended setup option where the gardenlet is running in the seed cluster itself.
Procedure Overview
Prepare the garden cluster:
Create a Bootstrap Token Secret in the kube-system
Namespace of the Garden Cluster
The gardenlet needs to talk to the Gardener API server residing in the garden cluster.
The gardenlet can be configured with an already existing garden cluster kubeconfig
in one of the following ways:
- By specifying
gardenClientConnection.kubeconfig
in the Gardenlet configuration. - By supplying the environment variable
GARDEN_KUBECONFIG
pointing to a mountedkubeconfig
file).
The preferred way, however, is to use the gardenlet’s ability to request a signed certificate for the garden cluster by leveraging Kubernetes Certificate Signing Requests. The gardenlet performs a TLS bootstrapping process that is similar to the Kubelet TLS Bootstrapping. Make sure that the API server of the garden cluster has bootstrap token authentication enabled.
The client credentials required for the gardenlet’s TLS bootstrapping process
need to be either token
or certificate
(OIDC isn’t supported) and have permissions
to create a Certificate Signing Request (CSR).
It’s recommended to use bootstrap tokens
due to their desirable security properties (such as a limited token lifetime).
Therefore, first create a bootstrap token secret for the garden cluster:
apiVersion: v1
kind: Secret
metadata:
# Name MUST be of form "bootstrap-token-<token id>"
name: bootstrap-token-07401b
namespace: kube-system
# Type MUST be 'bootstrap.kubernetes.io/token'
type: bootstrap.kubernetes.io/token
stringData:
# Human readable description. Optional.
description: "Token to be used by the gardenlet for Seed `sweet-seed`."
# Token ID and secret. Required.
token-id: 07401b # 6 characters
token-secret: f395accd246ae52d # 16 characters
# Expiration. Optional.
# expiration: 2017-03-10T03:22:11Z
# Allowed usages.
usage-bootstrap-authentication: "true"
usage-bootstrap-signing: "true"
When you later prepare the gardenlet Helm chart,
a kubeconfig
based on this token is shared with the gardenlet upon deployment.
Create RBAC Roles for the gardenlet to Allow Bootstrapping in the Garden Cluster
This step is only required if the gardenlet you deploy is the first gardenlet in the Gardener installation. Additionally, when using the control plane chart, the following resources are already contained in the Helm chart, that is, if you use it you can skip these steps as the needed RBAC roles already exist.
The gardenlet uses the configured bootstrap kubeconfig
in gardenClientConnection.bootstrapKubeconfig
to request a signed certificate for the user gardener.cloud:system:seed:<seed-name>
in the group gardener.cloud:system:seeds
.
Create a ClusterRole
and ClusterRoleBinding
that grant full admin permissions to authenticated gardenlets.
Create the following resources in the garden cluster:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: gardener.cloud:system:seeds
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: gardener.cloud:system:seeds
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: gardener.cloud:system:seeds
subjects:
- kind: Group
name: gardener.cloud:system:seeds
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: gardener.cloud:system:seed-bootstrapper
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests
verbs:
- create
- get
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/seedclient
verbs:
- create
---
# A kubelet/gardenlet authenticating using bootstrap tokens is authenticated as a user in the group system:bootstrappers
# Allows the Gardenlet to create a CSR
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: gardener.cloud:system:seed-bootstrapper
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: gardener.cloud:system:seed-bootstrapper
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
ℹ️ After bootstrapping, the gardenlet has full administrative access to the garden cluster. You might be interested to harden this and limit its permissions to only resources related to the seed cluster it is responsible for. Please take a look at Scoped API Access for Gardenlets.
Prepare the gardenlet Helm Chart
This section only describes the minimal configuration, using the global configuration values of the gardenlet Helm chart. For an overview over all values, see the configuration values. We refer to the global configuration values as gardenlet configuration in the following procedure.
Create a gardenlet configuration
gardenlet-values.yaml
based on this template.Create a bootstrap
kubeconfig
based on the bootstrap token created in the garden cluster.Replace the
<bootstrap-token>
withtoken-id.token-secret
(from our previous example:07401b.f395accd246ae52d
) from the bootstrap token secret.apiVersion: v1 kind: Config current-context: gardenlet-bootstrap@default clusters: - cluster: certificate-authority-data: <ca-of-garden-cluster> server: https://<endpoint-of-garden-cluster> name: default contexts: - context: cluster: default user: gardenlet-bootstrap name: gardenlet-bootstrap@default users: - name: gardenlet-bootstrap user: token: <bootstrap-token>
In the
gardenClientConnection.bootstrapKubeconfig
section of your gardenlet configuration, provide the bootstrapkubeconfig
together with a name and namespace to the gardenlet Helm chart.gardenClientConnection: bootstrapKubeconfig: name: gardenlet-kubeconfig-bootstrap namespace: garden kubeconfig: | <bootstrap-kubeconfig> # will be base64 encoded by helm
The bootstrap
kubeconfig
is stored in the specified secret.In the
gardenClientConnection.kubeconfigSecret
section of your gardenlet configuration, define a name and a namespace where the gardenlet stores the realkubeconfig
that it creates during the bootstrap process. If the secret doesn’t exist, the gardenlet creates it for you.gardenClientConnection: kubeconfigSecret: name: gardenlet-kubeconfig namespace: garden
Updating the Garden Cluster CA
The kubeconfig created by the gardenlet in step 4 will not be recreated as long as it exists, even if a new bootstrap kubeconfig is provided. To enable rotation of the garden cluster CA certificate, a new bundle can be provided via the gardenClientConnection.gardenClusterCACert
field. If the provided bundle differs from the one currently in the gardenlet’s kubeconfig secret then it will be updated. To remove the CA completely (e.g. when switching to a publicly trusted endpoint), this field can be set to either none
or null
.
Automatically Register a Shoot Cluster as a Seed Cluster
A seed cluster can either be registered by manually creating
the Seed
resource
or automatically by the gardenlet.
This functionality is useful for managed seed clusters,
as the gardenlet in the garden cluster deploys a copy of itself
into the cluster with automatic registration of the Seed
configured.
However, it can also be used to have a streamlined seed cluster registration process when manually deploying the gardenlet.
This procedure doesn’t describe all the possible configurations for the
Seed
resource. For more information, see:
Adjust the gardenlet Component Configuration
Supply the
Seed
resource in theseedConfig
section of your gardenlet configurationgardenlet-values.yaml
.Add the
seedConfig
to your gardenlet configurationgardenlet-values.yaml
. The fieldseedConfig.spec.provider.type
specifies the infrastructure provider type (for example,aws
) of the seed cluster. For all supported infrastructure providers, see Known Extension Implementations..... seedConfig: metadata: name: sweet-seed spec: dns: provider: type: <provider> secretRef: name: ingress-secret namespace: garden ingress: # see prerequisites domain: ingress.dev.my-seed.example.com controller: kind: nginx networks: # see prerequisites nodes: 10.240.0.0/16 pods: 100.244.0.0/16 services: 100.32.0.0/13 shootDefaults: # optional: non-overlapping default CIDRs for shoot clusters of that Seed pods: 100.96.0.0/11 services: 100.64.0.0/13 provider: region: eu-west-1 type: <provider>
Optional: Enable HA Mode
You may consider running gardenlet
with multiple replicas, especially if the seed cluster is configured to host HA shoot control planes.
Therefore, the following Helm chart values define the degree of high availability you want to achieve for the gardenlet
deployment.
replicaCount: 2 # or more if a higher failure tolerance is required.
failureToleranceType: zone # One of `zone` or `node` - defines how replicas are spread.
Optional: Enable Backup and Restore
The seed cluster can be set up with backup and restore
for the main etcds
of shoot clusters.
Gardener uses etcd-backup-restore
that integrates with different storage providers
to store the shoot cluster’s main etcd
backups.
Make sure to obtain client credentials that have sufficient permissions with the chosen storage provider.
Create a secret in the garden cluster with client credentials for the storage provider. The format of the secret is cloud provider specific and can be found in the repository of the respective Gardener extension. For example, the secret for AWS S3 can be found in the AWS provider extension (30-etcd-backup-secret.yaml).
apiVersion: v1
kind: Secret
metadata:
name: sweet-seed-backup
namespace: garden
type: Opaque
data:
# client credentials format is provider specific
Configure the Seed
resource in the seedConfig
section of your gardenlet configuration to use backup and restore:
...
seedConfig:
metadata:
name: sweet-seed
spec:
backup:
provider: <provider>
secretRef:
name: sweet-seed-backup
namespace: garden
Deploy the gardenlet
The gardenlet doesn’t have to run in the same Kubernetes cluster as the seed cluster it’s registering and reconciling, but it is in most cases advantageous to use in-cluster communication to talk to the Seed API server. Running a gardenlet outside of the cluster is mostly used for local development.
The gardenlet-values.yaml
looks something like this
(with automatic Seed registration and backup for shoot clusters enabled):
<default config>
...
config:
gardenClientConnection:
...
bootstrapKubeconfig:
name: gardenlet-bootstrap-kubeconfig
namespace: garden
kubeconfig: |
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <dummy>
server: <my-garden-cluster-endpoint>
name: my-kubernetes-cluster
....
kubeconfigSecret:
name: gardenlet-kubeconfig
namespace: garden
...
<default config>
...
seedConfig:
metadata:
name: sweet-seed
spec:
dns:
provider:
type: <provider>
secretRef:
name: ingress-secret
namespace: garden
ingress: # see prerequisites
domain: ingress.dev.my-seed.example.com
controller:
kind: nginx
networks:
nodes: 10.240.0.0/16
pods: 100.244.0.0/16
services: 100.32.0.0/13
shootDefaults:
pods: 100.96.0.0/11
services: 100.64.0.0/13
provider:
region: eu-west-1
type: <provider>
backup:
provider: <provider>
secretRef:
name: sweet-seed-backup
namespace: garden
Deploy the gardenlet Helm chart to the Kubernetes cluster:
helm install gardenlet charts/gardener/gardenlet \
--namespace garden \
-f gardenlet-values.yaml \
--wait
This helm chart creates:
- A service account
gardenlet
that the gardenlet can use to talk to the Seed API server. - RBAC roles for the service account (full admin rights at the moment).
- The secret (
garden
/gardenlet-bootstrap-kubeconfig
) containing the bootstrapkubeconfig
. - The gardenlet deployment in the
garden
namespace.
Check that the gardenlet Is Successfully Deployed
Check that the gardenlets certificate bootstrap was successful.
Check if the secret
gardenlet-kubeconfig
in the namespacegarden
in the seed cluster is created and contains akubeconfig
with a valid certificate.Get the
kubeconfig
from the created secret.$ kubectl -n garden get secret gardenlet-kubeconfig -o json | jq -r .data.kubeconfig | base64 -d
Test against the garden cluster and verify it’s working.
Extract the
client-certificate-data
from the usergardenlet
.View the certificate:
$ openssl x509 -in ./gardenlet-cert -noout -text
Check that the certificate is valid for a year (that is the lifetime of new certificates).
Check that the bootstrap secret
gardenlet-bootstrap-kubeconfig
has been deleted from the seed cluster in namespacegarden
.Check that the seed cluster is registered and
READY
in the garden cluster.Check that the seed cluster
sweet-seed
exists and all conditions indicate that it’s available. If so, the Gardenlet is sending regular heartbeats and the seed bootstrapping was successful.Check that the conditions on the
Seed
resource look similar to the following:$ kubectl get seed sweet-seed -o json | jq .status.conditions [ { "lastTransitionTime": "2020-07-17T09:17:29Z", "lastUpdateTime": "2020-07-17T09:17:29Z", "message": "Gardenlet is posting ready status.", "reason": "GardenletReady", "status": "True", "type": "GardenletReady" }, { "lastTransitionTime": "2020-07-17T09:17:49Z", "lastUpdateTime": "2020-07-17T09:53:17Z", "message": "Seed cluster has been bootstrapped successfully.", "reason": "BootstrappingSucceeded", "status": "True", "type": "Bootstrapped" }, { "lastTransitionTime": "2020-07-17T09:17:49Z", "lastUpdateTime": "2020-07-17T09:53:17Z", "message": "Backup Buckets are available.", "reason": "BackupBucketsAvailable", "status": "True", "type": "BackupBucketsReady" } ]
Related Links
6 - Feature Gates
Feature Gates in Gardener
This page contains an overview of the various feature gates an administrator can specify on different Gardener components.
Overview
Feature gates are a set of key=value pairs that describe Gardener features. You can turn these features on or off using the component configuration file for a specific component.
Each Gardener component lets you enable or disable a set of feature gates that are relevant to that component. For example, this is the configuration of the gardenlet component.
The following tables are a summary of the feature gates that you can set on different Gardener components.
- The “Since” column contains the Gardener release when a feature is introduced or its release stage is changed.
- The “Until” column, if not empty, contains the last Gardener release in which you can still use a feature gate.
- If a feature is in the Alpha or Beta state, you can find the feature listed in the Alpha/Beta feature gate table.
- If a feature is stable you can find all stages for that feature listed in the Graduated/Deprecated feature gate table.
- The Graduated/Deprecated feature gate table also lists deprecated and withdrawn features.
Feature Gates for Alpha or Beta Features
Feature | Default | Stage | Since | Until |
---|---|---|---|---|
HVPA | false | Alpha | 0.31 | |
HVPAForShootedSeed | false | Alpha | 0.32 | |
ManagedIstio | false | Alpha | 1.5 | 1.18 |
ManagedIstio | true | Beta | 1.19 | |
ManagedIstio (deprecated) | true | Beta | 1.48 | |
APIServerSNI | false | Alpha | 1.7 | 1.18 |
APIServerSNI | true | Beta | 1.19 | |
APIServerSNI (deprecated) | true | Beta | 1.48 | |
SeedChange | false | Alpha | 1.12 | 1.52 |
SeedChange | true | Beta | 1.53 | |
CopyEtcdBackupsDuringControlPlaneMigration | false | Alpha | 1.37 | 1.52 |
CopyEtcdBackupsDuringControlPlaneMigration | true | Beta | 1.53 | |
HAControlPlanes | false | Alpha | 1.49 | |
DefaultSeccompProfile | false | Alpha | 1.54 | |
CoreDNSQueryRewriting | false | Alpha | 1.55 | |
IPv6SingleStack | false | Alpha | 1.63 | |
MutableShootSpecNetworkingNodes | false | Alpha | 1.64 | |
FullNetworkPoliciesInRuntimeCluster | false | Alpha | 1.66 |
Feature Gates for Graduated or Deprecated Features
Feature | Default | Stage | Since | Until |
---|---|---|---|---|
NodeLocalDNS | false | Alpha | 1.7 | |
NodeLocalDNS | Removed | 1.26 | ||
KonnectivityTunnel | false | Alpha | 1.6 | |
KonnectivityTunnel | Removed | 1.27 | ||
MountHostCADirectories | false | Alpha | 1.11 | 1.25 |
MountHostCADirectories | true | Beta | 1.26 | 1.27 |
MountHostCADirectories | true | GA | 1.27 | |
MountHostCADirectories | Removed | 1.30 | ||
DisallowKubeconfigRotationForShootInDeletion | false | Alpha | 1.28 | 1.31 |
DisallowKubeconfigRotationForShootInDeletion | true | Beta | 1.32 | 1.35 |
DisallowKubeconfigRotationForShootInDeletion | true | GA | 1.36 | |
DisallowKubeconfigRotationForShootInDeletion | Removed | 1.38 | ||
Logging | false | Alpha | 0.13 | 1.40 |
Logging | Removed | 1.41 | ||
AdminKubeconfigRequest | false | Alpha | 1.24 | 1.38 |
AdminKubeconfigRequest | true | Beta | 1.39 | 1.41 |
AdminKubeconfigRequest | true | GA | 1.42 | 1.49 |
AdminKubeconfigRequest | Removed | 1.50 | ||
UseDNSRecords | false | Alpha | 1.27 | 1.38 |
UseDNSRecords | true | Beta | 1.39 | 1.43 |
UseDNSRecords | true | GA | 1.44 | 1.49 |
UseDNSRecords | Removed | 1.50 | ||
CachedRuntimeClients | false | Alpha | 1.7 | 1.33 |
CachedRuntimeClients | true | Beta | 1.34 | 1.44 |
CachedRuntimeClients | true | GA | 1.45 | 1.49 |
CachedRuntimeClients | Removed | 1.50 | ||
DenyInvalidExtensionResources | false | Alpha | 1.31 | 1.41 |
DenyInvalidExtensionResources | true | Beta | 1.42 | 1.44 |
DenyInvalidExtensionResources | true | GA | 1.45 | 1.49 |
DenyInvalidExtensionResources | Removed | 1.50 | ||
RotateSSHKeypairOnMaintenance | false | Alpha | 1.28 | 1.44 |
RotateSSHKeypairOnMaintenance | true | Beta | 1.45 | 1.47 |
RotateSSHKeypairOnMaintenance (deprecated) | false | Beta | 1.48 | 1.50 |
RotateSSHKeypairOnMaintenance (deprecated) | Removed | 1.51 | ||
ShootMaxTokenExpirationOverwrite | false | Alpha | 1.43 | 1.44 |
ShootMaxTokenExpirationOverwrite | true | Beta | 1.45 | 1.47 |
ShootMaxTokenExpirationOverwrite | true | GA | 1.48 | 1.50 |
ShootMaxTokenExpirationOverwrite | Removed | 1.51 | ||
ShootMaxTokenExpirationValidation | false | Alpha | 1.43 | 1.45 |
ShootMaxTokenExpirationValidation | true | Beta | 1.46 | 1.47 |
ShootMaxTokenExpirationValidation | true | GA | 1.48 | 1.50 |
ShootMaxTokenExpirationValidation | Removed | 1.51 | ||
WorkerPoolKubernetesVersion | false | Alpha | 1.35 | 1.45 |
WorkerPoolKubernetesVersion | true | Beta | 1.46 | 1.49 |
WorkerPoolKubernetesVersion | true | GA | 1.50 | 1.51 |
WorkerPoolKubernetesVersion | Removed | 1.52 | ||
DisableDNSProviderManagement | false | Alpha | 1.41 | 1.49 |
DisableDNSProviderManagement | true | Beta | 1.50 | 1.51 |
DisableDNSProviderManagement | true | GA | 1.52 | 1.59 |
DisableDNSProviderManagement | Removed | 1.60 | ||
SecretBindingProviderValidation | false | Alpha | 1.38 | 1.50 |
SecretBindingProviderValidation | true | Beta | 1.51 | 1.52 |
SecretBindingProviderValidation | true | GA | 1.53 | 1.54 |
SecretBindingProviderValidation | Removed | 1.55 | ||
SeedKubeScheduler | false | Alpha | 1.15 | 1.54 |
SeedKubeScheduler | false | Deprecated | 1.55 | 1.60 |
SeedKubeScheduler | Removed | 1.61 | ||
ShootCARotation | false | Alpha | 1.42 | 1.50 |
ShootCARotation | true | Beta | 1.51 | 1.56 |
ShootCARotation | true | GA | 1.57 | 1.59 |
ShootCARotation | Removed | 1.60 | ||
ShootSARotation | false | Alpha | 1.48 | 1.50 |
ShootSARotation | true | Beta | 1.51 | 1.56 |
ShootSARotation | true | GA | 1.57 | 1.59 |
ShootSARotation | Removed | 1.60 | ||
ReversedVPN | false | Alpha | 1.22 | 1.41 |
ReversedVPN | true | Beta | 1.42 | 1.62 |
ReversedVPN | true | GA | 1.63 | |
ForceRestore | false | Removed | 1.66 |
Using a Feature
A feature can be in Alpha, Beta or GA stage. An Alpha feature means:
- Disabled by default.
- Might be buggy. Enabling the feature may expose bugs.
- Support for feature may be dropped at any time without notice.
- The API may change in incompatible ways in a later software release without notice.
- Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.
A Beta feature means:
- Enabled by default.
- The feature is well tested. Enabling the feature is considered safe.
- Support for the overall feature will not be dropped, though details may change.
- The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
- Recommended for only non-critical uses because of potential for incompatible changes in subsequent releases.
Please do try Beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.
A General Availability (GA) feature is also referred to as a stable feature. It means:
- The feature is always enabled; you cannot disable it.
- The corresponding feature gate is no longer needed.
- Stable versions of features will appear in released software for many subsequent versions.
List of Feature Gates
Feature | Relevant Components | Description |
---|---|---|
HVPA | gardenlet , gardener-operator | Enables simultaneous horizontal and vertical scaling in garden or seed clusters. |
HVPAForShootedSeed | gardenlet | Enables simultaneous horizontal and vertical scaling in managed seed (aka “shooted seed”) clusters. |
ManagedIstio (deprecated) | gardenlet | Enables a Gardener-tailored Istio in each Seed cluster. Disable this feature if Istio is already installed in the cluster. Istio is not automatically removed if this feature is disabled. See the detailed documentation for more information. |
APIServerSNI (deprecated) | gardenlet | Enables only one LoadBalancer to be used for every Shoot cluster API server in a Seed. Enable this feature when ManagedIstio is enabled or Istio is manually deployed in the Seed cluster. See GEP-8 for more details. |
SeedChange | gardener-apiserver | Enables updating the spec.seedName field during shoot validation from a non-empty value in order to trigger shoot control plane migration. |
ReversedVPN | gardenlet | Reverses the connection setup of the VPN tunnel between the Seed and the Shoot cluster(s). It allows Seed and Shoot clusters to be in different networks with only direct access in one direction (Shoot -> Seed). In addition to that, it reduces the amount of load balancers required, i.e. no load balancers are required for the VPN tunnel anymore. It requires APIServerSNI and kubernetes version 1.18 or higher to work. Details can be found in GEP-14. |
CopyEtcdBackupsDuringControlPlaneMigration | gardenlet | Enables the copy of etcd backups from the object store of the source seed to the object store of the destination seed during control plane migration. |
SecretBindingProviderValidation | gardener-apiserver | Enables validations on Gardener API server that: - requires the provider type of a SecretBinding to be set (on SecretBinding creation) - requires the SecretBinding provider type to match the Shoot provider type (on Shoot creation) - enforces immutability on the provider type of a SecretBinding |
HAControlPlanes | gardener-apiserver | HAControlPlanes allows shoot control planes to be run in high availability mode. |
DefaultSeccompProfile | gardenlet , gardener-operator | Enables the defaulting of the seccomp profile for Gardener managed workload in the garden or seed to RuntimeDefault . |
CoreDNSQueryRewriting | gardenlet | Enables automatic DNS query rewriting in shoot cluster’s CoreDNS to shortcut name resolution of fully qualified in-cluster and out-of-cluster names, which follow a user-defined pattern. Details can be found in DNS Search Path Optimization. |
IPv6SingleStack | gardener-apiserver | Allows creating shoot clusters with IPv6 single-stack networking (GEP-21). |
MutableShootSpecNetworkingNodes | gardener-apiserver | Allows updating the field spec.networking.nodes . The validity of the values has to be checked in the provider extensions. Only enable this feature gate when your system runs provider extensions which have implemented the validation. |
FullNetworkPoliciesInRuntimeCluster | gardenlet | Enables gardenlet’s NetworkPolicy controller to place ‘deny-all’ network policies in all relevant namespaces in the runtime cluster. |
7 - gardenlet API Access
Scoped API Access for gardenlets
By default, gardenlet
s have administrative access in the garden cluster.
They are able to execute any API request on any object independent of whether the object is related to the seed cluster the gardenlet
is responsible for.
As RBAC is not powerful enough for fine-grained checks and for the sake of security, Gardener provides two optional but recommended configurations for your environments that scope the API access for gardenlet
s.
Similar to the Node
authorization mode in Kubernetes, Gardener features a SeedAuthorizer
plugin.
It is a special-purpose authorization plugin that specifically authorizes API requests made by the gardenlet
s.
Likewise, similar to the NodeRestriction
admission plugin in Kubernetes, Gardener features a SeedRestriction
plugin.
It is a special-purpose admission plugin that specifically limits the Kubernetes objects gardenlet
s can modify.
📚 You might be interested to look into the design proposal for scoped Kubelet API access from the Kubernetes community.
It can be translated to Gardener and Gardenlets with their Seed
and Shoot
resources.
Flow Diagram
The following diagram shows how the two plugins are included in the request flow of a gardenlet
.
When they are not enabled, then the kube-apiserver
is internally authorizing the request via RBAC before forwarding the request directly to the gardener-apiserver
, i.e., the gardener-admission-controller
would not be consulted (this is not entirely correct because it also serves other admission webhook handlers, but for simplicity reasons this document focuses on the API access scope only).
When enabling the plugins, there is one additional step for each before the gardener-apiserver
responds to the request.
Please note that the example shows a request to an object (Shoot
) residing in one of the API groups served by gardener-apiserver
.
However, the gardenlet
is also interacting with objects in API groups served by the kube-apiserver
(e.g., Secret
,ConfigMap
).
In this case, the consultation of the SeedRestriction
admission plugin is performed by the kube-apiserver
itself before it forwards the request to the gardener-apiserver
.
Today, the following rules are implemented:
Resource | Verbs | Path(s) | Description |
---|---|---|---|
BackupBucket | get , list , watch , create , update , patch , delete | BackupBucket -> Seed | Allow get , list , watch requests for all BackupBucket s. Allow only create , update , patch , delete requests for BackupBucket s assigned to the gardenlet ’s Seed . |
BackupEntry | get , list , watch , create , update , patch | BackupEntry -> Seed | Allow get , list , watch requests for all BackupEntry s. Allow only create , update , patch requests for BackupEntry s assigned to the gardenlet ’s Seed and referencing BackupBucket s assigned to the gardenlet ’s Seed . |
Bastion | get , list , watch , create , update , patch | Bastion -> Seed | Allow get , list , watch requests for all Bastion s. Allow only create , update , patch requests for Bastion s assigned to the gardenlet ’s Seed . |
CertificateSigningRequest | get , create | CertificateSigningRequest -> Seed | Allow only get , create requests for CertificateSigningRequest s related to the gardenlet ’s Seed . |
CloudProfile | get | CloudProfile -> Shoot -> Seed | Allow only get requests for CloudProfile s referenced by Shoot s that are assigned to the gardenlet ’s Seed . |
ClusterRoleBinding | create , get , update , patch , delete | ClusterRoleBinding -> ManagedSeed -> Shoot -> Seed | Allow create , get , update , patch requests for ManagedSeed s in the bootstrapping phase assigned to the gardenlet’s Seed s. Allow delete requests from gardenlets bootstrapped via ManagedSeed s. |
ConfigMap | get | ConfigMap -> Shoot -> Seed | Allow only get requests for ConfigMap s referenced by Shoot s that are assigned to the gardenlet ’s Seed . Allows reading the kube-system/cluster-identity ConfigMap . |
ControllerRegistration | get , list , watch | ControllerRegistration -> ControllerInstallation -> Seed | Allow get , list , watch requests for all ControllerRegistration s. |
ControllerDeployment | get | ControllerDeployment -> ControllerInstallation -> Seed | Allow get requests for ControllerDeployments s referenced by ControllerInstallation s assigned to the gardenlet ’s Seed . |
ControllerInstallation | get , list , watch , update , patch | ControllerInstallation -> Seed | Allow get , list , watch requests for all ControllerInstallation s. Allow only update , patch requests for ControllerInstallation s assigned to the gardenlet ’s Seed . |
Event | create , patch | none | Allow to create or patch all kinds of Event s. |
ExposureClass | get | ExposureClass -> Shoot -> Seed | Allow get requests for ExposureClass es referenced by Shoot s that are assigned to the gardenlet ’s Seed . Deny get requests to other ExposureClass es. |
Lease | create , get , watch , update | Lease -> Seed | Allow create , get , update , and delete requests for Lease s of the gardenlet ’s Seed . |
ManagedSeed | get , list , watch , update , patch | ManagedSeed -> Shoot -> Seed | Allow get , list , watch requests for all ManagedSeed s. Allow only update , patch requests for ManagedSeed s referencing a Shoot assigned to the gardenlet ’s Seed . |
Namespace | get | Namespace -> Shoot -> Seed | Allow get requests for Namespace s of Shoot s that are assigned to the gardenlet ’s Seed . Always allow get requests for the garden Namespace . |
Project | get | Project -> Namespace -> Shoot -> Seed | Allow get requests for Project s referenced by the Namespace of Shoot s that are assigned to the gardenlet ’s Seed . |
SecretBinding | get | SecretBinding -> Shoot -> Seed | Allow only get requests for SecretBinding s referenced by Shoot s that are assigned to the gardenlet ’s Seed . |
Secret | create , get , update , patch , delete (, list , watch ) | Secret -> Seed , Secret -> Shoot -> Seed , Secret -> SecretBinding -> Shoot -> Seed , BackupBucket -> Seed | Allow get , list , watch requests for all Secret s in the seed-<name> namespace. Allow only create , get , update , patch , delete requests for the Secret s related to resources assigned to the gardenlet ’s Seed s. |
Seed | get , list , watch , create , update , patch , delete | Seed | Allow get , list , watch requests for all Seed s. Allow only create , update , patch , delete requests for the gardenlet ’s Seed s. [1] |
ServiceAccount | create , get , update , patch , delete | ServiceAccount -> ManagedSeed -> Shoot -> Seed | Allow create , get , update , patch requests for ManagedSeed s in the bootstrapping phase assigned to the gardenlet ’s Seed s. Allow delete requests from gardenlets bootstrapped via ManagedSeed s. |
Shoot | get , list , watch , update , patch | Shoot -> Seed | Allow get , list , watch requests for all Shoot s. Allow only update , patch requests for Shoot s assigned to the gardenlet ’s Seed . |
ShootState | get , create , update , patch | ShootState -> Shoot -> Seed | Allow only get , create , update , patch requests for ShootState s belonging by Shoot s that are assigned to the gardenlet ’s Seed . |
[1] If you use
ManagedSeed
resources then thegardenlet
reconciling them (“parentgardenlet
”) may be allowed to submit certain requests for theSeed
resources resulting out of suchManagedSeed
reconciliations (even if the “parentgardenlet
” is not responsible for them):
ℹ️ It is allowed to delete the Seed
resources if the corresponding ManagedSeed
objects already have a deletionTimestamp
(this is secure as gardenlet
s themselves don’t have permissions for deleting ManagedSeed
s).
SeedAuthorizer
Authorization Webhook Enablement
The SeedAuthorizer
is implemented as a Kubernetes authorization webhook and part of the gardener-admission-controller
component running in the garden cluster.
🎛 In order to activate it, you have to follow these steps:
Set the following flags for the
kube-apiserver
of the garden cluster (i.e., thekube-apiserver
whose API is extended by Gardener):--authorization-mode=RBAC,Node,Webhook
(please note thatWebhook
should appear afterRBAC
in the list [1];Node
might not be needed if you use a virtual garden cluster)--authorization-webhook-config-file=<path-to-the-webhook-config-file>
--authorization-webhook-cache-authorized-ttl=0
--authorization-webhook-cache-unauthorized-ttl=0
The webhook config file (stored at
<path-to-the-webhook-config-file>
) should look as follows:apiVersion: v1 kind: Config clusters: - name: garden cluster: certificate-authority-data: base64(CA-CERT-OF-GARDENER-ADMISSION-CONTROLLER) server: https://gardener-admission-controller.garden/webhooks/auth/seed users: - name: kube-apiserver user: {} contexts: - name: auth-webhook context: cluster: garden user: kube-apiserver current-context: auth-webhook
When deploying the Gardener
controlplane
Helm chart, set.global.rbac.seedAuthorizer.enabled=true
. This will ensure that the RBAC resources granting global access for allgardenlet
s will be deployed.Delete the existing RBAC resources granting global access for all
gardenlet
s by running:kubectl delete \ clusterrole.rbac.authorization.k8s.io/gardener.cloud:system:seeds \ clusterrolebinding.rbac.authorization.k8s.io/gardener.cloud:system:seeds \ --ignore-not-found
Please note that you should activate the SeedRestriction
admission handler as well.
[1] The reason for the fact that
Webhook
authorization plugin should appear afterRBAC
is that thekube-apiserver
will be depending on thegardener-admission-controller
(serving the webhook). However, thegardener-admission-controller
can only start whengardener-apiserver
runs, butgardener-apiserver
itself can only start whenkube-apiserver
runs. IfWebhook
is beforeRBAC
, thengardener-apiserver
might not be able to start, leading to a deadlock.
Authorizer Decisions
As mentioned earlier, it’s the authorizer’s job to evaluate API requests and return one of the following decisions:
DecisionAllow
: The request is allowed, further configured authorizers won’t be consulted.DecisionDeny
: The request is denied, further configured authorizers won’t be consulted.DecisionNoOpinion
: A decision cannot be made, further configured authorizers will be consulted.
For backwards compatibility, no requests are denied at the moment, so that they are still deferred to a subsequent authorizer like RBAC. Though, this might change in the future.
First, the SeedAuthorizer
extracts the Seed
name from the API request. This requires a proper TLS certificate that the gardenlet
uses to contact the API server and is automatically given if TLS bootstrapping is used.
Concretely, the authorizer checks the certificate for name gardener.cloud:system:seed:<seed-name>
and group gardener.cloud:system:seeds
.
In cases where this information is missing e.g., when a custom Kubeconfig is used, the authorizer cannot make any decision. Thus, RBAC is still a considerable option to restrict the gardenlet
’s access permission if the above explained preconditions are not given.
With the Seed
name at hand, the authorizer checks for an existing path from the resource that a request is being made for to the Seed
belonging to the gardenlet
. Take a look at the Implementation Details section for more information.
Implementation Details
Internally, the SeedAuthorizer
uses a directed, acyclic graph data structure in order to efficiently respond to authorization requests for gardenlet
s:
- A vertex in this graph represents a Kubernetes resource with its kind, namespace, and name (e.g.,
Shoot:garden-my-project/my-shoot
). - An edge from vertex
u
to vertexv
in this graph exists when- (1)
v
is referred byu
andv
is aSeed
, or when - (2)
u
is referred byv
, or when - (3)
u
is strictly associated withv
.
- (1)
For example, a Shoot
refers to a Seed
, a CloudProfile
, a SecretBinding
, etc., so it has an outgoing edge to the Seed
(1) and incoming edges from the CloudProfile
and SecretBinding
vertices (2).
However, there might also be a ShootState
or a BackupEntry
resource strictly associated with this Shoot
, hence, it has incoming edges from these vertices (3).
In the above picture, the resources that are actively watched are shaded. Gardener resources are green, while Kubernetes resources are blue. It shows the dependencies between the resources and how the graph is built based on the above rules.
ℹ️ The above picture shows all resources that may be accessed by gardenlet
s, except for the Quota
resource which is only included for completeness.
Now, when a gardenlet
wants to access certain resources, then the SeedAuthorizer
uses a Depth-First traversal starting from the vertex representing the resource in question, e.g., from a Project
vertex.
If there is a path from the Project
vertex to the vertex representing the Seed
the gardenlet
is responsible for. then it allows the request.
Metrics
The SeedAuthorizer
registers the following metrics related to the mentioned graph implementation:
Metric | Description |
---|---|
gardener_admission_controller_seed_authorizer_graph_update_duration_seconds | Histogram of duration of resource dependency graph updates in seed authorizer, i.e., how long does it take to update the graph’s vertices/edges when a resource is created, changed, or deleted. |
gardener_admission_controller_seed_authorizer_graph_path_check_duration_seconds | Histogram of duration of checks whether a path exists in the resource dependency graph in seed authorizer. |
Debug Handler
When the .server.enableDebugHandlers
field in the gardener-admission-controller
’s component configuration is set to true
, then it serves a handler that can be used for debugging the resource dependency graph under /debug/resource-dependency-graph
.
🚨 Only use this setting for development purposes, as it enables unauthenticated users to view all data if they have access to the gardener-admission-controller
component.
The handler renders an HTML page displaying the current graph with a list of vertices and its associated incoming and outgoing edges to other vertices.
Depending on the size of the Gardener landscape (and consequently, the size of the graph), it might not be possible to render it in its entirety.
If there are more than 2000 vertices, then the default filtering will selected for kind=Seed
to prevent overloading the output.
Example output:
-------------------------------------------------------------------------------
|
| # Seed:my-seed
| <- (11)
| BackupBucket:73972fe2-3d7e-4f61-a406-b8f9e670e6b7
| BackupEntry:garden-my-project/shoot--dev--my-shoot--4656a460-1a69-4f00-9372-7452cbd38ee3
| ControllerInstallation:dns-external-mxt8m
| ControllerInstallation:extension-shoot-cert-service-4qw5j
| ControllerInstallation:networking-calico-bgrb2
| ControllerInstallation:os-gardenlinux-qvb5z
| ControllerInstallation:provider-gcp-w4mvf
| Secret:garden/backup
| Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------
|
| # Shoot:garden-my-project/my-shoot
| <- (5)
| CloudProfile:gcp
| Namespace:garden-my-project
| Secret:garden-my-project/my-dns-secret
| SecretBinding:garden-my-project/my-credentials
| ShootState:garden-my-project/my-shoot
| -> (1)
| Seed:my-seed
|
-------------------------------------------------------------------------------
|
| # ShootState:garden-my-project/my-shoot
| -> (1)
| Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------
... (etc., similarly for the other resources)
There are anchor links to easily jump from one resource to another, and the page provides means for filtering the results based on the kind
, namespace
, and/or name
.
Pitfalls
When there is a relevant update to an existing resource, i.e., when a reference to another resource is changed, then the corresponding vertex (along with all associated edges) is first deleted from the graph before it gets added again with the up-to-date edges.
However, this does only work for vertices belonging to resources that are only created in exactly one “watch handler”.
For example, the vertex for a SecretBinding
can either be created in the SecretBinding
handler itself or in the Shoot
handler.
In such cases, deleting the vertex before (re-)computing the edges might lead to race conditions and potentially renders the graph invalid.
Consequently, instead of deleting the vertex, only the edges the respective handler is responsible for are deleted.
If the vertex ends up with no remaining edges, then it also gets deleted automatically.
Afterwards, the vertex can either be added again or the updated edges can be created.
SeedRestriction
Admission Webhook Enablement
The SeedRestriction
is implemented as Kubernetes admission webhook and part of the gardener-admission-controller
component running in the garden cluster.
🎛 In order to activate it, you have to set .global.admission.seedRestriction.enabled=true
when using the Gardener controlplane
Helm chart.
This will add an additional webhook in the existing ValidatingWebhookConfiguration
of the gardener-admission-controller
which contains the configuration for the SeedRestriction
handler.
Please note that it should only be activated when the SeedAuthorizer
is active as well.
Admission Decisions
The admission’s purpose is to perform extended validation on requests which require the body of the object in question.
Additionally, it handles CREATE
requests of gardenlet
s (the above discussed resource dependency graph cannot be used in such cases because there won’t be any vertex/edge for non-existing resources).
Gardenlets are restricted to only create new resources which are somehow related to the seed clusters they are responsible for.
8 - Getting Started Locally
Deploying Gardener Locally
This document will walk you through deploying Gardener on your local machine. If you encounter difficulties, please open an issue so that we can make this process easier.
Overview
Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as both garden and seed cluster (please refer to the architecture overview) for simplicity.
Based on Skaffold, the container images for all required components will be built and deployed into the cluster (via their Helm charts).
Alternatives
When deploying Gardener on your local machine you might face several limitations:
- Your machine doesn’t have enough compute resources (see prerequisites) for hosting a second seed cluster or multiple shoot clusters.
- Testing Gardener’s IPv6 features requires a Linux machine and native IPv6 connectivity to the internet, but you’re on macOS or don’t have IPv6 connectivity in your office environment or via your home ISP.
In these cases, you might want to check out one of the following options that run the setup described in this guide elsewhere for circumventing these limitations:
- remote local setup: deploy on a remote pod for more compute resources
- dev box on Google Cloud: deploy on a Google Cloud machine for more compute resource and/or simple IPv4/IPv6 dual-stack networking
Prerequisites
- Make sure that you have followed the Local Setup guide up until the Get the sources step.
- Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least
8
CPUs and8Gi
memory; see here how to configure the resources for Docker for Mac).Please note that 8 CPU / 8Gi memory might not be enough for more than two
Shoot
clusters, i.e., you might need to increase these values if you want to run additionalShoot
s. If you plan on following the optional steps to create a second seed cluster, the required resources will be more - at least10
CPUs and18Gi
memory. Additionally, please configure at least120Gi
of disk size for the Docker daemon. Tip: You can clean up unused data withdocker system df
anddocker system prune -a
. - Make sure the
kind
docker network is using the CIDR172.18.0.0/16
.- If the network does not exist, it can be created with
docker network create kind --subnet 172.18.0.0/16
- If the network already exists, the CIDR can be checked with
docker network inspect kind | jq '.[].IPAM.Config[].Subnet'
. If it is not172.18.0.0/16
, delete the network withdocker network rm kind
and create it with the command above.
- If the network does not exist, it can be created with
Setting Up the KinD Cluster (Garden and Seed)
make kind-up
If you want to setup an IPv6 KinD cluster, use
make kind-up IPFAMILY=ipv6
instead.
This command sets up a new KinD cluster named gardener-local
and stores the kubeconfig in the ./example/gardener-local/kind/local/kubeconfig
file.
It might be helpful to copy this file to
$HOME/.kube/config
, since you will need to target this KinD cluster multiple times. Alternatively, make sure to set yourKUBECONFIG
environment variable to./example/gardener-local/kind/local/kubeconfig
for all future steps viaexport KUBECONFIG=example/gardener-local/kind/local/kubeconfig
.
All of the following steps assume that you are using this kubeconfig.
Additionally, this command also deploys a local container registry to the cluster, as well as a few registry mirrors, that are set up as a pull-through cache for all upstream registries Gardener uses by default.
This is done to speed up image pulls across local clusters.
The local registry can be accessed as localhost:5001
for pushing and pulling.
The storage directories of the registries are mounted to the host machine under dev/local-registry
.
With this, mirrored images don’t have to be pulled again after recreating the cluster.
The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy
support (the default kindnet
CNI doesn’t provide NetworkPolicy
support).
Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.
Outgoing IPv6 Single-Stack Networking (optional)
If you want to test IPv6-related features, we need to configure NAT for outgoing traffic from the kind network to the internet.
After make kind-up IPFAMILY=ipv6
, check the network created by kind:
$ docker network inspect kind | jq '.[].IPAM.Config[].Subnet'
"172.18.0.0/16"
"fc00:f853:ccd:e793::/64"
Determine which device is used for outgoing internet traffic by looking at the default route:
$ ip route show default
default via 192.168.195.1 dev enp3s0 proto dhcp src 192.168.195.34 metric 100
Configure NAT for traffic from the kind cluster to the internet using the IPv6 range and the network device from the previous two steps:
ip6tables -t nat -A POSTROUTING -o enp3s0 -s fc00:f853:ccd:e793::/64 -j MASQUERADE
Setting Up Gardener
make gardener-up
This will first build the base image (which might take a bit if you do it for the first time). Afterwards, the Gardener resources will be deployed into the cluster.
Developing Gardener
make gardener-dev
This is similar to make gardener-up
but additionally starts a skaffold dev loop.
After the initial deployment, skaffold starts watching source files.
Once it has detected changes, press any key to trigger a new build and deployment of the changed components.
Tip: you can set the SKAFFOLD_MODULE
environment variable to select specific modules of the skaffold configuration (see skaffold.yaml
) that skaffold should watch, build, and deploy.
This significantly reduces turnaround times during development.
For example, if you want to develop changes to gardenlet:
# initial deployment of all components
make gardener-up
# start iterating on gardenlet without deploying other components
make gardener-dev SKAFFOLD_MODULE=gardenlet
Creating a Shoot
Cluster
You can wait for the Seed
to be ready by running:
./hack/usage/wait-for.sh seed local GardenletReady Bootstrapped SeedSystemComponentsHealthy ExtensionsReady
Alternatively, you can run kubectl get seed local
and wait for the STATUS
to indicate readiness:
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
local Ready local local 4m42s vX.Y.Z-dev v1.21.1
In order to create a first shoot cluster, just run:
kubectl apply -f example/provider-local/shoot.yaml
You can wait for the Shoot
to be ready by running:
NAMESPACE=garden-local ./hack/usage/wait-for.sh shoot APIServerAvailable ControlPlaneHealthy ObservabilityComponentsHealthy EveryNodeReady SystemComponentsHealthy
Alternatively, you can run kubectl -n garden-local get shoot local
and wait for the LAST OPERATION
to reach 100%
:
NAME CLOUDPROFILE PROVIDER REGION K8S VERSION HIBERNATION LAST OPERATION STATUS AGE
local local local local 1.21.0 Awake Create Processing (43%) healthy 94s
(Optional): You could also execute a simple e2e test (creating and deleting a shoot) by running:
make test-e2e-local-simple KUBECONFIG="$PWD/example/gardener-local/kind/local/kubeconfig"
Accessing the Shoot
Cluster
⚠️ Please note that in this setup, shoot clusters are not accessible by default when you download the kubeconfig and try to communicate with them.
The reason is that your host most probably cannot resolve the DNS names of the clusters since provider-local
extension runs inside the KinD cluster (for more details, see DNSRecord).
Hence, if you want to access the shoot cluster, you have to run the following command which will extend your /etc/hosts
file with the required information to make the DNS names resolvable:
cat <<EOF | sudo tee -a /etc/hosts
# Manually created to access local Gardener shoot clusters.
# TODO: Remove this again when the shoot cluster access is no longer required.
127.0.0.1 api.local.local.external.local.gardener.cloud
127.0.0.1 api.local.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-managedseed.garden.external.local.gardener.cloud
127.0.0.1 api.e2e-managedseed.garden.internal.local.gardener.cloud
127.0.0.1 api.e2e-hibernated.local.external.local.gardener.cloud
127.0.0.1 api.e2e-hibernated.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-unpriv.local.external.local.gardener.cloud
127.0.0.1 api.e2e-unpriv.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-wake-up.local.external.local.gardener.cloud
127.0.0.1 api.e2e-wake-up.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-migrate.local.external.local.gardener.cloud
127.0.0.1 api.e2e-migrate.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-rotate.local.external.local.gardener.cloud
127.0.0.1 api.e2e-rotate.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-default.local.external.local.gardener.cloud
127.0.0.1 api.e2e-default.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-update-node.local.external.local.gardener.cloud
127.0.0.1 api.e2e-update-node.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-update-zone.local.external.local.gardener.cloud
127.0.0.1 api.e2e-update-zone.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-upgrade.local.external.local.gardener.cloud
127.0.0.1 api.e2e-upgrade.local.internal.local.gardener.cloud
EOF
To access the Shoot
, you can acquire a kubeconfig
by using the shoots/adminkubeconfig
subresource.
(Optional): Setting Up a Second Seed Cluster
There are cases where you would want to create a second seed cluster in your local setup. For example, if you want to test the control plane migration feature. The following steps describe how to do that.
If you are on macOS, add a new IP address on your loopback device which will be necessary for the new KinD cluster that you will create. On macOS, the default loopback device is lo0
.
sudo ip addr add 127.0.0.2 dev lo0 # adding 127.0.0.2 ip to the loopback interface
Next, setup the second KinD cluster:
make kind2-up
This command sets up a new KinD cluster named gardener-local2
and stores its kubeconfig in the ./example/gardener-local/kind/local2/kubeconfig
file.
In order to deploy required resources in the KinD cluster that you just created, run:
make gardenlet-kind2-up
The following steps assume that you are using the kubeconfig that points to the gardener-local
cluster (first KinD cluster): export KUBECONFIG=example/gardener-local/kind/local/kubeconfig
.
You can wait for the local2
Seed
to be ready by running:
./hack/usage/wait-for.sh seed local2 GardenletReady Bootstrapped ExtensionsReady
Alternatively, you can run kubectl get seed local2
and wait for the STATUS
to indicate readiness:
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
local2 Ready local local 4m42s vX.Y.Z-dev v1.21.1
If you want to perform control plane migration, you can follow the steps outlined in Control Plane Migration to migrate the shoot cluster to the second seed you just created.
Deleting the Shoot
Cluster
./hack/usage/delete shoot local garden-local
(Optional): Tear Down the Second Seed Cluster
make kind2-down
Tear Down the Gardener Environment
make kind-down
Remote Local Setup
Just like Prow is executing the KinD based integration tests in a K8s pod, it is possible to interactively run this KinD based Gardener development environment, aka “local setup”, in a “remote” K8s pod.
k apply -f docs/deployment/content/remote-local-setup.yaml
k exec -it deployment/remote-local-setup -- sh
tmux -u a
Caveats
Please refer to the TMUX documentation for working effectively inside the remote-local-setup pod.
To access Grafana, Prometheus or other components in a browser, two port forwards are needed:
The port forward from the laptop to the pod:
k port-forward deployment/remote-local-setup 3000
The port forward in the remote-local-setup pod to the respective component:
k port-forward -n shoot--local--local deployment/grafana 3000
Related Links
9 - Getting Started Locally With Extensions
Deploying Gardener Locally and Enabling Provider-Extensions
This document will walk you through deploying Gardener on your local machine and bootstrapping your own seed clusters on an existing Kubernetes cluster. It is supposed to run your local Gardener developments on a real infrastructure. For running Gardener only entirely local, please check the getting started locally documentation. If you encounter difficulties, please open an issue so that we can make this process easier.
Overview
Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as garden cluster. Any Kubernetes cluster could be used as seed clusters in order to support provider extensions (please refer to the architecture overview). This guide is tested for using Kubernetes clusters provided by Gardener, AWS, Azure, and GCP as seed so far.
Based on Skaffold, the container images for all required components will be built and deployed into the clusters (via their Helm charts).
Prerequisites
- Make sure that you have followed the Local Setup guide up until the Get the sources step.
- Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least
8
CPUs and8Gi
memory; see the Docker documentation for how to configure the resources for Docker for Mac).Additionally, please configure at least
120Gi
of disk size for the Docker daemon. Tip: You can clean up unused data withdocker system df
anddocker system prune -a
. - Make sure that you have access to a Kubernetes cluster you can use as a seed cluster in this setup.
- The seed cluster requires at least 16 CPUs in total to run one shoot cluster
- You could use any Kubernetes cluster for your seed cluster. However, using a Gardener shoot cluster for your seed simplifies some configuration steps.
- When bootstrapping
gardenlet
to the cluster, your new seed will have the same provider type as the shoot cluster you use - an AWS shoot will become an AWS seed, a GCP shoot will become a GCP seed, etc. (only relevant when using a Gardener shoot as seed).
Provide Infrastructure Credentials and Configuration
As this setup is running on a real infrastructure, you have to provide credentials for DNS, the infrastructure, and the kubeconfig for the Gardener cluster you want to use as seed.
There are
.gitignore
entries for all files and directories which include credentials. Nevertheless, please double check and make sure that credentials are not commited.
DNS
Gardener control plane requires DNS for default and internal domains. Thus, you have to configure a valid DNS provider for your setup.
Please maintain your DNS provider configuration and credentials at ./example/provider-extensions/garden/controlplane/values.yaml
.
You can find a template for the file at ./example/provider-extensions/garden/controlplane/values.yaml.tmpl
.
Infrastructure
Infrastructure secrets and the corresponding secret bindings should be maintained at:
./example/provider-extensions/garden/project/credentials/infrastructure-secrets.yaml
./example/provider-extensions/garden/project/credentials/secretbindings.yaml
There are templates with .tmpl
suffixes for the files in the same folder.
Seed Cluster Preparation
The kubeconfig
of your Kubernetes cluster you would like to use as seed should be placed at ./example/provider-extensions/seed/kubeconfig
.
Additionally, please maintain the configuration of your seed in ./example/provider-extensions/gardenlet/values.yaml
. It is automatically copied from values.yaml.tmpl
in the same directory when you run make gardener-extensions-up
for the first time. It also includes explanations of the properties you should set.
Using a Gardener cluster as seed simplifies the process, because some configuration options can be taken from shoot-info
and creating DNS entries and TLS certificates is automated.
However, you can use different Kubernetes clusters for your seed too and configure these things manually. Please configure the options of ./example/provider-extensions/gardenlet/values.yaml
upfront. For configuring DNS and TLS certificates, make gardener-extensions-up
, which is explained later, will pause and tell you what to do.
External Controllers
You might plan to deploy and register external controllers for networking, operating system, providers, etc. Please put ControllerDeployment
s and ControllerRegistration
s into the ./example/provider-extensions/garden/controllerregistrations
directory. The whole content of this folder will be applied to your KinD cluster.
CloudProfile
s
There are no demo CloudProfiles
yet. Thus, please copy CloudProfiles
from another landscape to the ./example/provider-extensions/garden/cloudprofiles
directory or create your own CloudProfiles
based on the gardener examples. Please check the GitHub repository of your desired provider-extension. Most of them include example CloudProfile
s. All files you place in this folder will be applied to your KinD cluster.
Setting Up the KinD Cluster
make kind-extensions-up
This command sets up a new KinD cluster named gardener-local
and stores the kubeconfig in the ./example/gardener-local/kind/extensions/kubeconfig
file.
It might be helpful to copy this file to
$HOME/.kube/config
, since you will need to target this KinD cluster multiple times. Alternatively, make sure to set yourKUBECONFIG
environment variable to./example/gardener-local/kind/extensions/kubeconfig
for all future steps viaexport KUBECONFIG=$PWD/example/gardener-local/kind/extensions/kubeconfig
.
All of the following steps assume that you are using this kubeconfig.
Additionally, this command deploys a local container registry to the cluster as well as a few registry mirrors that are set up as a pull-through cache for all upstream registries Gardener uses by default.
This is done to speed up image pulls across local clusters.
The local registry can be accessed as localhost:5001
for pushing and pulling.
The storage directories of the registries are mounted to your machine under dev/local-registry
.
With this, mirrored images don’t have to be pulled again after recreating the cluster.
The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy
support (the default kindnet
CNI doesn’t provide NetworkPolicy
support).
Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.
Setting Up Gardener (Garden on KinD, Seed on Gardener Cluster)
make gardener-extensions-up
This will first prepare the basic configuration of your KinD and Gardener clusters.
Afterwards, the images for the Garden cluster are built and deployed into the KinD cluster.
Finally, the images for the Seed cluster are built, pushed to a container registry on the Seed, and the gardenlet
is started.
Adding Additional Seeds
Additional seed(s) can be added by running
make gardener-extensions-up SEED_NAME=<seed-name>
The seed cluster preparations are similar to the first seed:
The kubeconfig
of your Kubernetes cluster you would like to use as seed should be placed at ./example/provider-extensions/seed/kubeconfig-<seed-name>
.
Additionally, please maintain the configuration of your seed in ./example/provider-extensions/gardenlet/values-<seed-name>.yaml
. It is automatically copied from values.yaml.tmpl
in the same directory when you run make gardener-extensions-up SEED_NAME=<seed-name>
for the first time. It also includes explanations of the properties you should set.
Removing a Seed
If you have multiple seeds and want to remove one, just use
make gardener-extensions-down SEED_NAME=<seed-name>
If it is not the last seed, this command will only remove the seed, but leave the local Gardener cluster and the other seeds untouched. To remove all seeds and to cleanup the local Gardener cluster, you have to run the command for each seed.
Pause and Unpause the KinD Cluster
The KinD cluster can be paused by stopping and keeping its docker container. This can be done by running:
make kind-extensions-down
When you run make kind-extensions-up
again, you will start the docker container with your previous Gardener configuration again.
This provides the option to switch off your local KinD cluster fast without leaving orphaned infrastructure elements behind.
Creating a Shoot
Cluster
You can wait for the Seed
to be ready by running:
kubectl wait --for=condition=gardenletready seed provider-extensions --timeout=5m
make kind-extensions-up
already includes such a check. However, it might be useful when you wake up your Seed
from hibernation or unpause you KinD cluster.
Alternatively, you can run kubectl get seed provider-extensions
and wait for the STATUS
to indicate readiness:
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
provider-extensions Ready gcp europe-west1 111m v1.61.0-dev v1.24.7
In order to create a first shoot cluster, please create your own Shoot
definition and apply it to your KinD cluster. gardener-scheduler
includes candidateDeterminationStrategy: MinimalDistance
configuration so you are able to run schedule Shoot
s of different providers on your Seed
.
You can wait for your Shoot
s to be ready by running kubectl -n garden-local get shoots
and wait for the LAST OPERATION
to reach 100%
. The output depends on your Shoot
definition. This is an example output:
NAME CLOUDPROFILE PROVIDER REGION K8S VERSION HIBERNATION LAST OPERATION STATUS AGE
aws aws aws eu-west-1 1.24.3 Awake Create Processing (43%) healthy 84s
aws-arm64 aws aws eu-west-1 1.24.3 Awake Create Processing (43%) healthy 65s
azure az azure westeurope 1.24.2 Awake Create Processing (43%) healthy 57s
gcp gcp gcp europe-west1 1.24.3 Awake Create Processing (43%) healthy 94s
Accessing the Shoot
Cluster
Your shoot clusters will have a public DNS entries for their API servers, so that they could be reached via the Internet via kubectl
after you have created their kubeconfig
.
We encourage you to use the adminkubeconfig subresource for accessing your shoot cluster. You can find an example how to use it in Accessing Shoot Clusters.
Deleting the Shoot
Clusters
Before tearing down your environment, you have to delete your shoot clusters. This is highly recommended because otherwise you would leave orphaned items on your infrastructure accounts.
./hack/usage/delete shoot <your-shoot> garden-local
Tear Down the Gardener Environment
Before you delete your local KinD cluster, you should shut down your Shoots
and Seed
in a clean way to avoid orphaned infrastructure elements in your projects.
Please ensure that your KinD and Seed clusters are online (not paused or hibernated) and run:
make gardener-extensions-down
This will delete all Shoots
first (this could take a couple of minutes), then uninstall gardenlet
from the Seed and the gardener components from the KinD. Finally, the additional components like container registry, etc., are deleted from both clusters.
When this is done, you can securely delete your local KinD cluster by running:
make kind-extensions-clean
10 - Image Vector
Image Vector
The Gardenlet is deploying several different container images into the seed and the shoot clusters. The image repositories and tags are defined in a central image vector file. Obviously, the image versions defined there must fit together with the deployment manifests (e.g., some command-line flags do only exist in certain versions).
Example
images:
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.4"
version: "1.20.x"
architectures:
- amd64
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.5"
version: ">= 1.21"
architectures:
- amd64
...
That means that the Gardenlet will use the pause-container
with tag 3.4
for all seed/shoot clusters with Kubernetes version 1.20.x
, and tag 3.5
for all clusters with Kubernetes >= 1.21
.
Image Vector Architecture
images:
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.5"
architectures:
- amd64
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.5"
architectures:
- arm64
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.5"
architectures:
- amd64
- arm64
...
architectures
is an optional field of image. It is a list of strings specifying CPU architecture of machines on which this image can be used. The valid options for the architectures field are as follows:
amd64
: This specifies that the image can run only on machines having CPU architectureamd64
.arm64
: This specifies that the image can run only on machines having CPU architecturearm64
.
If an image doesn’t specify any architectures, then by default it is considered to support both amd64
and arm64
architectures.
Overwrite Image Vector
In some environments it is not possible to use these “pre-defined” images that come with a Gardener release.
A prominent example for that is Alicloud in China, which does not allow access to Google’s GCR.
In these cases, you might want to overwrite certain images, e.g., point the pause-container
to a different registry.
⚠️ If you specify an image that does not fit to the resource manifest, then the seed/shoot reconciliation might fail.
In order to overwrite the images, you must provide a similar file to gardenlet:
images:
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: my-custom-image-registry/pause
tag: "3.4"
version: "1.20.x"
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: my-custom-image-registry/pause
tag: "3.5"
version: ">= 1.21"
...
During deployment of the gardenlet, create a ConfigMap
containing the above content and mount it as a volume into the gardenlet pod.
Next, specify the environment variable IMAGEVECTOR_OVERWRITE
, whose value must be the path to the file you just mounted:
apiVersion: v1
kind: ConfigMap
metadata:
name: gardenlet-images-overwrite
namespace: garden
data:
images_overwrite.yaml: |
images:
- ...
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: gardenlet
namespace: garden
spec:
template:
...
spec:
containers:
- name: gardenlet
env:
- name: IMAGEVECTOR_OVERWRITE
value: /charts-overwrite/images_overwrite.yaml
volumeMounts:
- name: gardenlet-images-overwrite
mountPath: /charts-overwrite
...
volumes:
- name: gardenlet-images-overwrite
configMap:
name: gardenlet-images-overwrite
...
Image Vectors for Dependent Components
The gardenlet is deploying a lot of different components that might deploy other images themselves. These components might use an image vector as well. Operators might want to customize the image locations for these transitive images as well, hence, they might need to specify an image vector overwrite for the components directly deployed by Gardener.
It is possible to specify the IMAGEVECTOR_OVERWRITE_COMPONENTS
environment variable to the gardenlet that points to a file with the following content:
components:
- name: etcd-druid
imageVectorOverwrite: |
images:
- name: etcd
tag: v1.2.3
repository: etcd/etcd
...
The gardenlet will, if supported by the directly deployed component (etcd-druid
in this example), inject the given imageVectorOverwrite
into the Deployment
manifest.
The respective component is responsible for using the overwritten images instead of its defaults.
11 - Migration V0 To V1
Migration from Gardener v0
to v1
Please refer to the document for older Gardener versions.
12 - Secret Binding Provider Controller
SecretBinding Provider Controller
This page describes the process on how to enable the SecretBinding provider controller.
Overview
With Gardener v1.38.0, the SecretBinding resource now contains a new optional field .provider.type
(details about the motivation can be found in https://github.com/gardener/gardener/issues/4888). To make the process of setting the new field automated and afterwards to enforce validation on the new field in backwards compatible manner, Gardener features the SecretBinding provider controller and a feature gate - SecretBindingProviderValidation
.
Process
A Gardener landscape operator can follow the following steps:
Enable the SecretBinding provider controller of Gardener Controller Manager.
The SecretBinding provider controller is responsible for populating the
.provider.type
field of a SecretBinding based on its current usage by Shoot resources. For example, if a Shootcrazy-botany
with.provider.type=aws
is using a SecretBindingmy-secret-binding
, then the SecretBinding provider controller will take care to set the.provider.type
field of the SecretBinding to the same provider type (aws
). To enable the SecretBinding provider controller, set thecontroller.secretBindingProvider.concurentSyncs
field in the ControllerManagerConfiguration (e.g set it to5
). Although that it is not recommended, the API allows Shoots from different provider types to reference the same SecretBinding (assuming that the backing Secret contains data for both of the provider types). To preserve the backwards compatibility for such SecretBindings, the provider controller will maintain the multiple provider types in the field (it will join them with the separator,
- for exampleaws,gcp
).Disable the SecretBinding provider controller and enable the
SecretBindingProviderValidation
feature gate of Gardener API server.The
SecretBindingProviderValidation
feature gate of Gardener API server enables a set of validations for the SecretBinding provider field. It forbids creating a Shoot that has a different provider type from the referenced SecretBinding’s one. It also enforces immutability on the field. After making sure that SecretBinding provider controller is enabled and it populated the.provider.type
field of a majority of the SecretBindings on a Gardener landscape (the SecretBindings that are unused will have their provider type unset), a Gardener landscape operator has to disable the SecretBinding provider controller and to enable theSecretBindingProviderValidation
feature gate of Gardener API server. To disable the SecretBinding provider controller, set thecontroller.secretBindingProvider.concurentSyncs
field in the ControllerManagerConfiguration to0
.
Implementation History
- Gardener v1.38: The SecretBinding resource has a new optional field
.provider.type
. The SecretBinding provider controller is disabled by default. TheSecretBindingProviderValidation
feature gate of Gardener API server is disabled by default. - Gardener v1.42: The SecretBinding provider controller is enabled by default.
- Gardener v1.51: The
SecretBindingProviderValidation
feature gate of Gardener API server is enabled by default and the SecretBinding provider controller is disabled by default. - Gardener v1.53: The
SecretBindingProviderValidation
feature gate of Gardener API server is unconditionally enabled (can no longer be disabled). - Gardener v1.55: The
SecretBindingProviderValidation
feature gate of Gardener API server and the SecretBinding provider controller are removed.
13 - Setup Gardener
Deploying Gardener into a Kubernetes Cluster
Similar to Kubernetes, Gardener consists out of control plane components (Gardener API server, Gardener controller manager, Gardener scheduler), and an agent component (gardenlet). The control plane is deployed in the so-called garden cluster, while the agent is installed into every seed cluster. Please note that it is possible to use the garden cluster as seed cluster by simply deploying the gardenlet into it.
We are providing Helm charts in order to manage the various resources of the components. Please always make sure that you use the Helm chart version that matches the Gardener version you want to deploy.
Deploying the Gardener Control Plane (API Server, Admission Controller, Controller Manager, Scheduler)
The configuration values depict the various options to configure the different components. Please consult Gardener Configuration and Usage for component specific configurations and Authentication of Gardener Control Plane Components Against the Garden Cluster for authentication related specifics.
Also, note that all resources and deployments need to be created in the garden
namespace (not overrideable).
If you enable the Gardener admission controller as part of you setup, please make sure the garden
namespace is labelled with app: gardener
.
Otherwise, the backing service account for the admission controller Pod might not be created successfully.
No action is necessary if you deploy the garden
namespace with the Gardener control plane Helm chart.
After preparing your values in a separate controlplane-values.yaml
file (values.yaml can be used as starting point), you can run the following command against your garden cluster:
helm install charts/gardener/controlplane \
--namespace garden \
--name gardener-controlplane \
-f controlplane-values.yaml \
--wait
Deploying Gardener Extensions
Gardener is an extensible system that does not contain the logic for provider-specific things like DNS management, cloud infrastructures, network plugins, operating system configs, and many more.
You have to install extension controllers for these parts. Please consult the documentation regarding extensions to get more information.
Deploying the Gardener Agent (gardenlet)
Please refer to Deploying Gardenlets on how to deploy a gardenlet.
14 - Version Skew Policy
Version Skew Policy
This document describes the maximum version skew supported between various Gardener components.
Supported Gardener Versions
Gardener versions are expressed as x.y.z
, where x
is the major version, y
is the minor version, and z
is the patch version, following Semantic Versioning terminology.
The Gardener project maintains release branches for the three most recent minor releases.
Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility. Patch releases are cut from those branches at a regular cadence, plus additional urgent releases when required.
For more information, see the Releases document.
Supported Version Skew
Technically, we follow the same policy as the Kubernetes project.
However, given that our release cadence is much more frequent compared to Kubernetes (every 14d
vs. every 120d
), in many cases it is possible to skip a version.
Still, to be on the safe side, it is highly recommended to follow the described policy.
gardener-apiserver
In multi-instance setups of Gardener, the newest and oldest gardener-apiserver
instances must be within one minor version.
Example:
- newest
gardener-apiserver
is at 1.37 - other
gardener-apiserver
instances are supported at 1.37 and v1.36
gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet
gardener-controller-manager
, gardener-scheduler
, gardener-admission-controller
, and gardenlet
must not be newer than the gardener-apiserver
instances they communicate with.
They are expected to match the gardener-apiserver
minor version, but may be up to one minor version older (to allow live upgrades).
Example:
gardener-apiserver
is at v1.37gardener-controller-manager
,gardener-scheduler
,gardener-admission-controller
, andgardenlet
are supported at 1.37 and v1.36
Supported Component Upgrade Order
The supported version skew between components has implications on the order in which components must be upgraded. This section describes the order in which components must be upgraded to transition an existing Gardener installation from version 1.37 to version 1.38.
gardener-apiserver
Prerequisites:
- In a single-instance setup, the existing
gardener-apiserver
instance is 1.37. - In a multi-instance setup, all
gardener-apiserver
instances are at 1.37 or 1.38 (this ensures maximum skew of 1 minor version between the oldest and newestgardener-apiserver
instance). - The
gardener-controller-manager
,gardener-scheduler
,gardener-admission-controller
, andgardenlet
instances that communicate with thisgardener-apiserver
are at version 1.37 (this ensures they are not newer than the existing API server version and are within 1 minor version of the new API server version).
Actions:
- Upgrade
gardener-apiserver
to 1.38.
gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet
Prerequisites:
- The
gardener-apiserver
instances these components communicate with are at 1.38 (in multi-instance setups in which these components can communicate with anygardener-apiserver
instance in the cluster, allgardener-apiserver
instances must be upgraded before upgrading these components)
Actions:
- Upgrade
gardener-controller-manager
,gardener-scheduler
,gardener-admission-controller
, andgardenlet
to 1.38
Supported Kubernetes Versions
Please refer to Supported Kubernetes Versions.