This is the multi-page printable view of this section. Click here to print.
Deployment
1 - Authentication Gardener Control Plane
Authentication of Gardener control plane components against the Garden cluster
Note: This document refers to Gardener’s API server, admission controller, controller manager and scheduler components. Any reference to the term Gardener control plane component can be replaced with any of the mentioned above.
There are several authentication possibilities depending on whether or not the concept of Virtual Garden is used.
Virtual Garden is not used, i.e., the runtime
Garden cluster is also the target
Garden cluster.
Automounted Service Account Token
The easiest way to deploy a Gardener control plane component will be to not provide kubeconfig
at all. This way in-cluster configuration and an automounted service account token will be used. The drawback of this approach is that the automounted token will not be automatically rotated.
Service Account Token Volume Projection
Another solution will be to use Service Account Token Volume Projection combined with a kubeconfig
referencing a token file (see example below).
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: <CA-DATA>
server: https://default.kubernetes.svc.cluster.local
name: garden
contexts:
- context:
cluster: garden
user: garden
name: garden
current-context: garden
users:
- name: garden
user:
tokenFile: /var/run/secrets/projected/serviceaccount/token
This will allow for automatic rotation of the service account token by the kubelet
. The configuration can be achieved by setting both .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true
and .Values.global.<GardenerControlPlaneComponent>.kubeconfig
in the respective chart’s values.yaml
file.
Virtual Garden is used, i.e., the runtime
Garden cluster is different from the target
Garden cluster.
Service Account
The easiest way to setup the authentication will be to create a service account and the respective roles will be bound to this service account in the target
cluster. Then use the generated service account token and craft a kubeconfig
which will be used by the workload in the runtime
cluster. This approach does not provide a solution for the rotation of the service account token. However, this setup can be achieved by setting .Values.global.deployment.virtualGarden.enabled: true
and following these steps:
- Deploy the
application
part of the charts in thetarget
cluster. - Get the service account token and craft the
kubeconfig
. - Set the crafted
kubeconfig
and deploy theruntime
part of the charts in theruntime
cluster.
Client Certificate
Another solution will be to bind the roles in the target
cluster to a User
subject instead of a service account and use a client certificate for authentication. This approach does not provide a solution for the client certificate rotation. However, this setup can be achieved by setting both .Values.global.deployment.virtualGarden.enabled: true
and .Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name
, then following these steps:
- Generate a client certificate for the
target
cluster for the respective user. - Deploy the
application
part of the charts in thetarget
cluster. - Craft a
kubeconfig
using the already generated client certificate. - Set the crafted
kubeconfig
and deploy theruntime
part of the charts in theruntime
cluster.
Projected Service Account Token
This approach requires an already deployed and configured oidc-webhook-authenticator for the target
cluster. Also the runtime
cluster should be registered as a trusted identity provider in the target
cluster. Then projected service accounts tokens from the runtime
cluster can be used to authenticate against the target
cluster. The needed steps are as follows:
- Deploy OWA and establish the needed trust.
- Set
.Values.global.deployment.virtualGarden.enabled: true
and.Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name
. Note: username value will depend on the trust configuration, e.g.,<prefix>:system:serviceaccount:<namespace>:<serviceaccount>
- Set
.Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true
and.Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.audience
. Note: audience value will depend on the trust configuration, e.g.,<cliend-id-from-trust-config>
. - Craft a kubeconfig (see example below).
- Deploy the
application
part of the charts in thetarget
cluster. - Deploy the
runtime
part of the charts in theruntime
cluster.
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: <CA-DATA>
server: https://virtual-garden.api
name: virtual-garden
contexts:
- context:
cluster: virtual-garden
user: virtual-garden
name: virtual-garden
current-context: virtual-garden
users:
- name: virtual-garden
user:
tokenFile: /var/run/secrets/projected/serviceaccount/token
2 - Configuring Logging
Configuring the Logging stack via Gardenlet configurations
Enable the Logging
In order to install the Gardener logging stack the logging.enabled
configuration option has to be enabled in the Gardenlet configuration:
logging:
enabled: true
From now on each Seed is going to have a logging stack which will collect logs from all pods and some systemd services. Logs related to Shoots with testing
purpose are dropped in the fluent-bit
output plugin. Shoots with a purpose different than testing
have the same type of log aggregator (but different instance) as the Seed. The logs can be viewed in the Grafana in the garden
namespace for the Seed components and in the respective shoot control plane namespaces.
Enable logs from the Shoot’s node systemd services.
The logs from the systemd services on each node can be retrieved by enabling the logging.shootNodeLogging
option in the Gardenlet configuration:
logging:
enabled: true
shootNodeLogging:
shootPurposes:
- "evaluation"
- "deployment"
Under the shootPurpose
section just list all the shoot purposes for which the Shoot node logging feature will be enabled. Specifying the testing
purpose has no effect because this purpose prevents the logging stack installation.
Logs can be viewed in the operator Grafana!
The dedicated labels are unit
, syslog_identifier
and nodename
in the Explore
menu.
Configuring the log processor
Under logging.fluentBit
there is three optional sections.
input
: This overwrite the input configuration of the fluent-bit log processor.output
: This overwrite the output configuration of the fluent-bit log processor.service
: This overwrite the service configuration of the fluent-bit log processor.
logging:
enabled: true
fluentBit:
output: |-
[Output]
...
input: |-
[Input]
...
service: |-
[Service]
...
Configuring the Loki PriorityClass
The central Loki, which is in the garden
namespace, contains all the logs from the most important seed components. When the central Loki PriorityClass
is with low value then its pods can be preempted and often moved from one node to another while Kubernetes tries to free space for more important pods. The persistent volume will be detached/attached again as well. Based on the performance of the underlying infrastructure, this leads to great central Loki downtime. To give greater priority of the seed Loki you can use the logging.loki.garden.priority
option.
logging:
enabled: true
loki:
garden:
priority: 100
Configure central logging
For central logging, the output configuration of the fluent-bit log processor can be overwritten (logging.fluentBit.output
) and the Loki instances deployments in Garden and Shoot namespace can be enabled/disabled (logging.loki.enabled
), by default Loki is enabled.
logging:
enabled: true
fluentBit:
output: |-
[Output]
...
loki:
enabled: false
Configuring central Loki storage capacity
By default, the central Loki has 100Gi
of storage capacity.
To overwrite the current central Loki storage capacity, the logging.loki.garden.storage
setting in the gardenlet’s component configuration should be altered.
If you need to increase it you can do so without losing the current data by specifying higher capacity. Doing so, the Loki’s PersistentVolume
capacity will be increased instead of deleting the current PV.
However, if you specify less capacity then the PersistentVolume
will be deleted and with it the logs, too.
logging:
enabled: true
fluentBit:
output: |-
[Output]
...
loki:
garden:
storage: "200Gi"
3 - Deploy Gardenlet
Deploying Gardenlets
Gardenlets act as decentral “agents” to manage shoot clusters of a seed cluster.
To support scaleability in an automated way, gardenlets are deployed automatically. However, you can still deploy gardenlets manually to be more flexible, for example, when shoot clusters that need to be managed by Gardener are behind a firewall. The gardenlet only requires network connectivity from the gardenlet to the Garden cluster (not the other way round), so it can be used to register Kubernetes clusters with no public endpoint.
Procedure
First, an initial gardenlet needs to be deployed:
- Deploy it manually if you have special requirements. More information: Deploy a Gardenlet Manually
- Let the Gardener installer deploy it automatically otherwise. More information: Automatic Deployment of Gardenlets
To add additional seed clusters, it is recommended to use regular shoot clusters. You can do this by creating a
ManagedSeed
resource with agardenlet
section as described in Register Shoot as Seed.
4 - Deploy Gardenlet Automatically
Automatic Deployment of Gardenlets
The gardenlet can automatically deploy itself into shoot clusters, and register this cluster as a seed cluster. These clusters are called “managed seeds” (aka “shooted seeds”). This procedure is the preferred way to add additional seed clusters, because shoot clusters already come with production-grade qualities that are also demanded for seed clusters.
Prerequisites
The only prerequisite is to register an initial cluster as a seed cluster that has already a gardenlet deployed:
- This gardenlet was either deployed as part of a Gardener installation using a setup tool (for example,
gardener/garden-setup
) or - the gardenlet was deployed manually
- for a step-by-step manual installation Guide see: Deploy a Gardenlet Manually)
The initial cluster can be the garden cluster itself.
Self-Deployment of Gardenlets in Additional Managed Seed Clusters
For a better scalability, you usually need more seed clusters that you can create as follows:
- Use the initial cluster as the seed cluster for other managed seed clusters. It hosts the control planes of the other seed clusters.
- The gardenlet deployed in the initial cluster deploys itself automatically into the managed seed clusters.
The advantage of this approach is that there’s only one initial gardenlet installation required. Every other managed seed cluster has a gardenlet deployed automatically.
Related Links
5 - Deploy Gardenlet Manually
Deploy a Gardenlet Manually
Manually deploying a gardenlet is required in the following cases:
The Kubernetes cluster to be registered as a seed cluster has no public endpoint, because it is behind a firewall. The gardenlet must then be deployed into the cluster itself.
The Kubernetes cluster to be registered as a seed cluster is managed externally (the Kubernetes cluster is not a shoot cluster, so Automatic Deployment of Gardenlets cannot be used).
The gardenlet runs outside of the Kubernetes cluster that should be registered as a seed cluster. (The gardenlet is not restricted to run in the seed cluster or to be deployed into a Kubernetes cluster at all).
Once you’ve deployed a gardenlet manually, for example, behind a firewall, you can deploy new gardenlets automatically. The manually deployed gardenlet is then used as a template for the new gardenlets. More information: Automatic Deployment of Gardenlets.
Prerequisites
Kubernetes cluster that should be registered as a seed cluster
Verify that the cluster has a supported Kubernetes version.
Determine the nodes, pods, and services CIDR of the cluster. You need to configure this information in the
Seed
configuration. Gardener uses this information to check that the shoot cluster isn’t created with overlapping CIDR ranges.Every Seed cluster needs an Ingress controller which distributes external requests to internal components like grafana and prometheus. Gardener supports two approaches to achieve this:
a. Gardener managed Ingress controller and DNS records. For this configure the following lines in your Seed resource:
spec:
dns:
provider:
type: aws-route53
secretRef:
name: ingress-secret
namespace: garden
ingress:
domain: ingress.my-seed.example.com
controller:
kind: nginx
providerConfig:
<some-optional-provider-specific-config-for-the-ingressController>
⚠ Please note that if you set .spec.ingress
then .spec.dns.ingressDomain
must be nil
.
b. Self-managed DNS record and Ingress controller:
⚠️
There should exist a DNS record *.ingress.<SEED-CLUSTER-DOMAIN>
where <SEED-CLUSTER-DOMAIN>
is the value of the .dns.ingressDomain
field of a Seed cluster resource (or the respective Gardenlet configuration).
This is how it could be done for the Nginx ingress controller
Deploy nginx into the kube-system
namespace in the Kubernetes cluster that should be registered as a Seed
.
Nginx will on most cloud providers create the service with type LoadBalancer
with an external ip.
NAME TYPE CLUSTER-IP EXTERNAL-IP
nginx-ingress-controller LoadBalancer 10.0.15.46 34.200.30.30
Create a wildcard A
record (e.g *.ingress.sweet-seed.Seed
cluster.
Please configure the ingress domain in the Seed
specification as follows:
spec:
dns:
ingressDomain: ingress.sweet-seed.<my-domain>
⚠ Please note that if you set .spec.dns.ingressDomain
then .spec.ingress
must be nil
.
kubeconfig
for the Seed Cluster
The kubeconfig
is required to deploy the gardenlet Helm chart to the seed cluster.
The gardenlet requires certain privileges to be able to operate.
These privileges are described in RBAC resources in the gardenlet Helm chart (see charts/gardener/gardenlet/charts/runtime/templates).
The Helm chart contains a service account gardenlet
that the gardenlet deployment uses by default to talk to the Seed API server.
If the gardenlet isn’t deployed in the seed cluster, the gardenlet can be configured to use a
kubeconfig
, which also requires the above-mentioned privileges, from a mounted directory. Thekubeconfig
is specified in sectionseedClientConnection.kubeconfig
of the Gardenlet configuration. This configuration option isn’t used in the following, as this procedure only describes the recommended setup option where the gardenlet is running in the seed cluster itself.
Procedure Overview
Prepare the garden cluster:
Create a bootstrap token secret in the kube-system
namespace of the garden cluster
The gardenlet needs to talk to the Gardener API server residing in the garden cluster.
The gardenlet can be configured with an already existing garden cluster kubeconfig
in one of the following ways:
Either by specifying
gardenClientConnection.kubeconfig
in the Gardenlet configuration orby supplying the environment variable
GARDEN_KUBECONFIG
pointing to a mountedkubeconfig
file).
The preferred way however, is to use the gardenlets ability to request a signed certificate for the garden cluster by leveraging Kubernetes Certificate Signing Requests. The gardenlet performs a TLS bootstrapping process that is similar to the Kubelet TLS Bootstrapping. Make sure that the API server of the garden cluster has bootstrap token authentication enabled.
The client credentials required for the gardenlets TLS bootstrapping process,
need to be either token
or certificate
(OIDC isn’t supported) and have permissions
to create a Certificate Signing Request (CSR).
It’s recommended to use bootstrap tokens
due to their desirable security properties (such as a limited token lifetime).
Therefore, first create a bootstrap token secret for the garden cluster:
apiVersion: v1
kind: Secret
metadata:
# Name MUST be of form "bootstrap-token-<token id>"
name: bootstrap-token-07401b
namespace: kube-system
# Type MUST be 'bootstrap.kubernetes.io/token'
type: bootstrap.kubernetes.io/token
stringData:
# Human readable description. Optional.
description: "Token to be used by the gardenlet for Seed `sweet-seed`."
# Token ID and secret. Required.
token-id: 07401b # 6 characters
token-secret: f395accd246ae52d # 16 characters
# Expiration. Optional.
# expiration: 2017-03-10T03:22:11Z
# Allowed usages.
usage-bootstrap-authentication: "true"
usage-bootstrap-signing: "true"
When you later prepare the gardenlet Helm chart,
a kubeconfig
based on this token is shared with the gardenlet upon deployment.
Create RBAC roles for the gardenlet to allow bootstrapping in the garden cluster
This step is only required if the gardenlet you deploy is the first gardenlet in the Gardener installation. Additionally, when using the control plane chart, the following resources are already contained in the Helm chart, that is, if you use it you can skip these steps as the needed RBAC roles already exist.
The gardenlet uses the configured bootstrap kubeconfig
in gardenClientConnection.bootstrapKubeconfig
to request a signed certificate for the user gardener.cloud:system:seed:<seed-name>
in the group gardener.cloud:system:seeds
.
Create a ClusterRole
and ClusterRoleBinding
that grant full admin permissions to authenticated gardenlets.
Create the following resources in the garden cluster:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: gardener.cloud:system:seeds
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: gardener.cloud:system:seeds
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: gardener.cloud:system:seeds
subjects:
- kind: Group
name: gardener.cloud:system:seeds
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: gardener.cloud:system:seed-bootstrapper
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests
verbs:
- create
- get
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/seedclient
verbs:
- create
---
# A kubelet/gardenlet authenticating using bootstrap tokens is authenticated as a user in the group system:bootstrappers
# Allows the Gardenlet to create a CSR
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: gardener.cloud:system:seed-bootstrapper
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: gardener.cloud:system:seed-bootstrapper
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
ℹ️ After bootstrapping, the gardenlet has full administrative access to the garden cluster. You might be interested to harden this and limit its permissions to only resources related to the seed cluster it is responsible for. Please take a look into this document.
Prepare the gardenlet Helm chart
This section only describes the minimal configuration, using the global configuration values of the gardenlet Helm chart. For an overview over all values, see the configuration values. We refer to the global configuration values as gardenlet configuration in the remaining procedure.
Create a gardenlet configuration
gardenlet-values.yaml
based on this template.Create a bootstrap
kubeconfig
based on the bootstrap token created in the garden cluster.Replace the
<bootstrap-token>
withtoken-id.token-secret
(from our previous example:07401b.f395accd246ae52d
) from the bootstrap token secret.apiVersion: v1 kind: Config current-context: gardenlet-bootstrap@default clusters: - cluster: certificate-authority-data: <ca-of-garden-cluster> server: https://<endpoint-of-garden-cluster> name: default contexts: - context: cluster: default user: gardenlet-bootstrap name: gardenlet-bootstrap@default users: - name: gardenlet-bootstrap user: token: <bootstrap-token>
In section
gardenClientConnection.bootstrapKubeconfig
of your gardenlet configuration, provide the bootstrapkubeconfig
together with a name and namespace to the gardenlet Helm chart.gardenClientConnection: bootstrapKubeconfig: name: gardenlet-kubeconfig-bootstrap namespace: garden kubeconfig: | <bootstrap-kubeconfig> # will be base64 encoded by helm
The bootstrap
kubeconfig
is stored in the specified secret.In section
gardenClientConnection.kubeconfigSecret
of your gardenlet configuration, define a name and a namespace where the gardenlet stores the realkubeconfig
that it creates during the bootstrap process. If the secret doesn’t exist, the gardenlet creates it for you.gardenClientConnection: kubeconfigSecret: name: gardenlet-kubeconfig namespace: garden
Updating the garden cluster CA
The kubeconfig created by the gardenlet in step 4 will not be recreated as long as it exists, even if a new bootstrap kubeconfig is provided. To enable rotation of the garden cluster CA certificate, a new bundle can be provided via the gardenClientConnection.gardenClusterCACert
field. If the provided bundle differs from the one currently in the gardenlet’s kubeconfig secret then it will be updated. To remove the CA completely (e.g. when switching to a publicly trusted endpoint) this field can be set to either none
or null
.
Automatically register shoot cluster as a seed cluster
A seed cluster can either be registered by manually creating
the Seed
resource
or automatically by the gardenlet.
This functionality is useful for managed seed clusters,
as the gardenlet in the garden cluster deploys a copy of itself
into the cluster with automatic registration of the Seed
configured.
However, it can also be used to have a streamlined seed cluster registration process when manually deploying the gardenlet.
This procedure doesn’t describe all the possible configurations for the
Seed
resource. For more information, see:
Adjust the gardenlet component configuration
Supply the
Seed
resource in sectionseedConfig
of your gardenlet configurationgardenlet-values.yaml
.Add the
seedConfig
to your gardenlet configurationgardenlet-values.yaml
. The fieldseedConfig.spec.provider.type
specifies the infrastructure provider type (for example,aws
) of the seed cluster. For all supported infrastructure providers, see Known Extension Implementations..... seedConfig: metadata: name: sweet-seed spec: dns: ingressDomain: ingress.sweet-seed.<my-domain> # see prerequisites networks: # see prerequisites nodes: 10.240.0.0/16 pods: 100.244.0.0/16 services: 100.32.0.0/13 shootDefaults: # optional: non-overlapping default CIDRs for shoot clusters of that Seed pods: 100.96.0.0/11 services: 100.64.0.0/13 provider: region: eu-west-1 type: <provider>
Optional: Enable backup and restore
The seed cluster can be set up with backup and restore
for the main etcds
of shoot clusters.
Gardener uses etcd-backup-restore
that integrates with different storage providers
to store the shoot cluster’s main etcd
backups.
Make sure to obtain client credentials that have sufficient permissions with the chosen storage provider.
Create a secret in the garden cluster with client credentials for the storage provider. The format of the secret is cloud provider specific and can be found in the repository of the respective Gardener extension. For example, the secret for AWS S3 can be found in the AWS provider extension (30-etcd-backup-secret.yaml).
apiVersion: v1
kind: Secret
metadata:
name: sweet-seed-backup
namespace: garden
type: Opaque
data:
# client credentials format is provider specific
Configure the Seed
resource in section seedConfig
of your gardenlet configuration to use backup and restore:
...
seedConfig:
metadata:
name: sweet-seed
spec:
backup:
provider: <provider>
secretRef:
name: sweet-seed-backup
namespace: garden
Deploy the gardenlet
The gardenlet doesn’t have to run in the same Kubernetes cluster as the seed cluster it’s registering and reconciling, but it is in most cases advantageous to use in-cluster communication to talk to the Seed API server. Running a gardenlet outside of the cluster is mostly used for local development.
The gardenlet-values.yaml
looks something like this
(with automatic Seed registration and backup for shoot clusters enabled):
global:
# Gardenlet configuration values
gardenlet:
enabled: true
...
<default config>
...
config:
gardenClientConnection:
...
bootstrapKubeconfig:
name: gardenlet-bootstrap-kubeconfig
namespace: garden
kubeconfig: |
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <dummy>
server: <my-garden-cluster-endpoint>
name: my-kubernetes-cluster
....
kubeconfigSecret:
name: gardenlet-kubeconfig
namespace: garden
...
<default config>
...
seedConfig:
metadata:
name: sweet-seed
spec:
dns:
ingressDomain: ingress.sweet-seed.<my-domain>
networks:
nodes: 10.240.0.0/16
pods: 100.244.0.0/16
services: 100.32.0.0/13
shootDefaults:
pods: 100.96.0.0/11
services: 100.64.0.0/13
provider:
region: eu-west-1
type: <provider>
backup:
provider: <provider>
secretRef:
name: sweet-seed-backup
namespace: garden
Deploy the gardenlet Helm chart to the Kubernetes cluster.
helm install gardenlet charts/gardener/gardenlet \
--namespace garden \
-f gardenlet-values.yaml \
--wait
This helm chart creates:
- A service account
gardenlet
that the gardenlet can use to talk to the Seed API server. - RBAC roles for the service account (full admin rights at the moment).
- The secret (
garden
/gardenlet-bootstrap-kubeconfig
) containing the bootstrapkubeconfig
. - The gardenlet deployment in the
garden
namespace.
Check that the gardenlet is successfully deployed
Check that the gardenlets certificate bootstrap was successful.
Check if the secret
gardenlet-kubeconfig
in the namespacegarden
in the seed cluster is created and contains akubeconfig
with a valid certificate.Get the
kubeconfig
from the created secret.$ kubectl -n garden get secret gardenlet-kubeconfig -o json | jq -r .data.kubeconfig | base64 -d
Test against the garden cluster and verify it’s working.
Extract the
client-certificate-data
from the usergardenlet
.View the certificate:
$ openssl x509 -in ./gardenlet-cert -noout -text
Check that the certificate is valid for a year (that is the lifetime of new certificates).
Check that the bootstrap secret
gardenlet-bootstrap-kubeconfig
has been deleted from the seed cluster in namespacegarden
.Check that the seed cluster is registered and
READY
in the garden cluster.Check that the seed cluster
sweet-seed
exists and all conditions indicate that it’s available. If so, the Gardenlet is sending regular heartbeats and the seed bootstrapping was successful.Check that the conditions on the
Seed
resource look similar to the following:$ kubectl get seed sweet-seed -o json | jq .status.conditions [ { "lastTransitionTime": "2020-07-17T09:17:29Z", "lastUpdateTime": "2020-07-17T09:17:29Z", "message": "Gardenlet is posting ready status.", "reason": "GardenletReady", "status": "True", "type": "GardenletReady" }, { "lastTransitionTime": "2020-07-17T09:17:49Z", "lastUpdateTime": "2020-07-17T09:53:17Z", "message": "Seed cluster has been bootstrapped successfully.", "reason": "BootstrappingSucceeded", "status": "True", "type": "Bootstrapped" }, { "lastTransitionTime": "2020-07-17T09:17:49Z", "lastUpdateTime": "2020-07-17T09:53:17Z", "message": "Backup Buckets are available.", "reason": "BackupBucketsAvailable", "status": "True", "type": "BackupBucketsReady" } ]
Related Links
6 - Feature Gates
Feature Gates in Gardener
This page contains an overview of the various feature gates an administrator can specify on different Gardener components.
Overview
Feature gates are a set of key=value pairs that describe Gardener features. You can turn these features on or off using the a component configuration file for a specific component.
Each Gardener component lets you enable or disable a set of feature gates that are relevant to that component. For example this is the configuration of the gardenlet component.
The following tables are a summary of the feature gates that you can set on different Gardener components.
- The “Since” column contains the Gardener release when a feature is introduced or its release stage is changed.
- The “Until” column, if not empty, contains the last Gardener release in which you can still use a feature gate.
- If a feature is in the Alpha or Beta state, you can find the feature listed in the Alpha/Beta feature gate table.
- If a feature is stable you can find all stages for that feature listed in the Graduated/Deprecated feature gate table.
- The Graduated/Deprecated feature gate table also lists deprecated and withdrawn features.
Feature gates for Alpha or Beta features
Feature | Default | Stage | Since | Until |
---|---|---|---|---|
HVPA | false | Alpha | 0.31 | |
HVPAForShootedSeed | false | Alpha | 0.32 | |
ManagedIstio | false | Alpha | 1.5 | 1.18 |
ManagedIstio | true | Beta | 1.19 | |
APIServerSNI | false | Alpha | 1.7 | 1.18 |
APIServerSNI | true | Beta | 1.19 | |
SeedChange | false | Alpha | 1.12 | |
SeedKubeScheduler | false | Alpha | 1.15 | |
ReversedVPN | false | Alpha | 1.22 | 1.41 |
ReversedVPN | true | Beta | 1.42 | |
RotateSSHKeypairOnMaintenance | false | Alpha | 1.28 | 1.44 |
RotateSSHKeypairOnMaintenance | true | Beta | 1.45 | |
RotateSSHKeypairOnMaintenance (deprecated) | false | Beta | 1.48 | |
WorkerPoolKubernetesVersion | false | Alpha | 1.35 | 1.45 |
WorkerPoolKubernetesVersion | true | Beta | 1.46 | |
CopyEtcdBackupsDuringControlPlaneMigration | false | Alpha | 1.37 | |
SecretBindingProviderValidation | false | Alpha | 1.38 | |
ForceRestore | false | Alpha | 1.39 | |
DisableDNSProviderManagement | false | Alpha | 1.41 | |
ShootCARotation | false | Alpha | 1.42 | |
ShootSARotation | false | Alpha | 1.48 | |
ShootMaxTokenExpirationOverwrite | false | Alpha | 1.43 | 1.44 |
ShootMaxTokenExpirationOverwrite | true | Beta | 1.45 | |
ShootMaxTokenExpirationValidation | false | Alpha | 1.43 | 1.45 |
ShootMaxTokenExpirationValidation | true | Beta | 1.46 |
Feature gates for graduated or deprecated features
Feature | Default | Stage | Since | Until |
---|---|---|---|---|
NodeLocalDNS | false | Alpha | 1.7 | |
NodeLocalDNS | Removed | 1.26 | ||
KonnectivityTunnel | false | Alpha | 1.6 | |
KonnectivityTunnel | Removed | 1.27 | ||
MountHostCADirectories | false | Alpha | 1.11 | 1.25 |
MountHostCADirectories | true | Beta | 1.26 | 1.27 |
MountHostCADirectories | true | GA | 1.27 | |
MountHostCADirectories | Removed | 1.30 | ||
DisallowKubeconfigRotationForShootInDeletion | false | Alpha | 1.28 | 1.31 |
DisallowKubeconfigRotationForShootInDeletion | true | Beta | 1.32 | 1.35 |
DisallowKubeconfigRotationForShootInDeletion | true | GA | 1.36 | |
DisallowKubeconfigRotationForShootInDeletion | Removed | 1.38 | ||
Logging | false | Alpha | 0.13 | 1.40 |
Logging | false | Removed | 1.41 | |
AdminKubeconfigRequest | false | Alpha | 1.24 | 1.38 |
AdminKubeconfigRequest | true | Beta | 1.39 | 1.41 |
AdminKubeconfigRequest | true | GA | 1.42 | |
UseDNSRecords | false | Alpha | 1.27 | 1.38 |
UseDNSRecords | true | Beta | 1.39 | 1.43 |
UseDNSRecords | true | GA | 1.44 | |
CachedRuntimeClients | false | Alpha | 1.7 | 1.33 |
CachedRuntimeClients | true | Beta | 1.34 | 1.44 |
CachedRuntimeClients | true | GA | 1.45 | |
DenyInvalidExtensionResources | false | Alpha | 1.31 | 1.41 |
DenyInvalidExtensionResources | true | Beta | 1.42 | 1.44 |
DenyInvalidExtensionResources | true | GA | 1.45 |
Using a feature
A feature can be in Alpha, Beta or GA stage. An Alpha feature means:
- Disabled by default.
- Might be buggy. Enabling the feature may expose bugs.
- Support for feature may be dropped at any time without notice.
- The API may change in incompatible ways in a later software release without notice.
- Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.
A Beta feature means:
- Enabled by default.
- The feature is well tested. Enabling the feature is considered safe.
- Support for the overall feature will not be dropped, though details may change.
- The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
- Recommended for only non-critical uses because of potential for incompatible changes in subsequent releases.
Please do try Beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.
A General Availability (GA) feature is also referred to as a stable feature. It means:
- The feature is always enabled; you cannot disable it.
- The corresponding feature gate is no longer needed.
- Stable versions of features will appear in released software for many subsequent versions.
List of Feature Gates
Feature | Relevant Components | Description |
---|---|---|
HVPA | gardenlet | Enables simultaneous horizontal and vertical scaling in Seed Clusters. |
HVPAForShootedSeed | gardenlet | Enables simultaneous horizontal and vertical scaling in managed seed (aka “shooted seed”) clusters. |
ManagedIstio | gardenlet | Enables a Gardener-tailored Istio in each Seed cluster. Disable this feature if Istio is already installed in the cluster. Istio is not automatically removed if this feature is disabled. See the detailed documentation for more information. |
APIServerSNI | gardenlet | Enables only one LoadBalancer to be used for every Shoot cluster API server in a Seed. Enable this feature when ManagedIstio is enabled or Istio is manually deployed in Seed cluster. See GEP-8 for more details. |
CachedRuntimeClients | gardener-controller-manager , gardenlet | Enables a cache in the controller-runtime clients, that Gardener components use. The feature gate can be specified for gardenlet and gardener-controller-manager (and gardener-scheduler for the versions < 1.29 ). |
SeedChange | gardener-apiserver | Enables updating the spec.seedName field during shoot validation from a non-empty value in order to trigger shoot control plane migration. |
SeedKubeScheduler | gardenlet | Adds custom kube-scheduler in gardener-kube-scheduler namespace. It schedules pods with scheduler name gardener-kube-scheduler on Nodes with higher resource utilization. It requires Seed cluster with kubernetes version 1.18 or higher. |
ReversedVPN | gardenlet | Reverses the connection setup of the vpn tunnel between the Seed and the Shoot cluster(s). It allows Seed and Shoot clusters to be in different networks with only direct access in one direction (Shoot -> Seed). In addition to that, it reduces the amount of load balancers required, i.e. no load balancers are required for the vpn tunnel anymore. It requires APIServerSNI and kubernetes version 1.18 or higher to work. Details can be found in GEP-14. |
AdminKubeconfigRequest | gardener-apiserver | Enables the AdminKubeconfigRequest endpoint on Shoot resources. See GEP-16 for more details. |
UseDNSRecords | gardener-apiserver , gardener-controller-manager , gardenlet | Enables using DNSRecord resources for Gardener DNS records instead of DNSProvider , DNSEntry , and DNSOwner resources. See Contract: DNSRecord resources for more details. |
RotateSSHKeypairOnMaintenance (deprecated) | gardener-controller-manager | Enables SSH keypair rotation in the maintenance controller of the gardener-controller-manager. Details can be found in GEP-15. |
DenyInvalidExtensionResources | gardenlet | Causes the seed-admission-controller to deny invalid extension resources, instead of just logging validation errors. |
WorkerPoolKubernetesVersion | gardener-apiserver | Allows to overwrite the Kubernetes version used for shoot clusters per worker pool (see this document) |
CopyEtcdBackupsDuringControlPlaneMigration | gardenlet | Enables the copy of etcd backups from the object store of the source seed to the object store of the destination seed during control plane migration. |
SecretBindingProviderValidation | gardener-apiserver | Enables validations on Gardener API server that: - requires the provider type of a SecretBinding to be set (on SecretBinding creation) - requires the SecretBinding provider type to match the Shoot provider type (on Shoot creation) - enforces immutability on the provider type of a SecretBinding |
ForceRestore | gardenlet | Enables forcing the shoot’s restoration to the destination seed during control plane migration if the preparation for migration in the source seed is not finished after a certain grace period and is considered unlikely to succeed (falling back to the control plane migration “bad case” scenario). If you enable this feature gate, make sure to also enable UseDNSRecords and CopyEtcdBackupsDuringControlPlaneMigration . |
DisableDNSProviderManagement | gardenlet | Disables management of dns.gardener.cloud/v1alpha1.DNSProvider resources. In this case, the shoot-dns-service extension will take this over if it is installed. This feature is only effective if the feature UseDNSRecords is true . |
ShootCARotation | gardener-apiserver , gardenlet | Enables the feature to trigger automated CA rotation for shoot clusters. |
ShootSARotation | gardener-apiserver , gardenlet | Enables the feature to trigger automated service account signing key rotation for shoot clusters. |
ShootMaxTokenExpirationOverwrite | gardener-apiserver | Makes the Gardener API server overwriting values in the .spec.kubernetes.kubeAPIServer.serviceAccountConfig.maxTokenExpiration field of Shoot specifications to- be at least 720h (30d) when the current value is lower - be at most 2160h (90d) when the current value is higher before persisting the object to etcd. |
ShootMaxTokenExpirationValidation | gardener-apiserver | Enables validations on Gardener API server that enforce that the value of the .spec.kubernetes.kubeAPIServer.serviceAccountConfig.maxTokenExpiration field- is at least 720h (30d). - is at most 2160h (90d). Only enable this after ShootMaxTokenExpirationOverwrite is enabled and all shoots got updated accordingly. |
7 - Gardenlet API Access
Scoped API Access for Gardenlets
By default, gardenlet
s have administrative access in the garden cluster.
They are able to execute any API request on any object independent of whether the object is related to the seed cluster the gardenlet
is responsible for.
As RBAC is not powerful enough for fine-grained checks and for the sake of security, Gardener provides two optional but recommended configurations for your environments that scope the API access for gardenlet
s.
Similar to the Node
authorization mode in Kubernetes, Gardener features a SeedAuthorizer
plugin.
It is a special-purpose authorization plugin that specifically authorizes API requests made by the gardenlet
s.
Likewise, similar to the NodeRestriction
admission plugin in Kubernetes, Gardener features a SeedRestriction
plugin.
It is a special-purpose admission plugin that specifically limits the Kubernetes objects gardenlet
s can modify.
📚 You might be interested to look into the design proposal for scoped Kubelet API access from the Kubernetes community.
It can be translated to Gardener and Gardenlets with their Seed
and Shoot
resources.
Flow Diagram
The following diagram shows how the two plugins are included in the request flow of a gardenlet
.
When they are not enabled then the kube-apiserver
is internally authorizing the request via RBAC before forwarding the request directly to the gardener-apiserver
, i.e., the gardener-admission-controller
would not be consulted (this is not entirely correct because it also serves other admission webhook handlers, but for simplicity reasons this document focuses on the API access scope only).
When enabling the plugins, there is one additional step for each before the gardener-apiserver
responds to the request.
Please note that the example shows a request to an object (Shoot
) residing in one of the API groups served by gardener-apiserver
.
However, the gardenlet
is also interacting with objects in API groups served by the kube-apiserver
(e.g., Secret
,ConfigMap
, etc.).
In this case, the consultation of the SeedRestriction
admission plugin is performed by the kube-apiserver
itself before it forwards the request to the gardener-apiserver
.
Today, the following rules are implemented:
Resource | Verbs | Path(s) | Description |
---|---|---|---|
BackupBucket | get , list , watch , create , update , patch , delete | BackupBucket -> Seed | Allow get , list , watch requests for all BackupBucket s. Allow only create , update , patch , delete requests for BackupBucket s assigned to the gardenlet ’s Seed . |
BackupEntry | get , list , watch , create , update , patch | BackupEntry -> Seed | Allow get , list , watch requests for all BackupEntry s. Allow only create , update , patch requests for BackupEntry s assigned to the gardenlet ’s Seed and referencing BackupBucket s assigned to the gardenlet ’s Seed . |
Bastion | get , list , watch , create , update , patch | Bastion -> Seed | Allow get , list , watch requests for all Bastion s. Allow only create , update , patch requests for Bastion s assigned to the gardenlet ’s Seed . |
CertificateSigningRequest | get , create | CertificateSigningRequest -> Seed | Allow only get , create requests for CertificateSigningRequest s related to the gardenlet ’s Seed . |
CloudProfile | get | CloudProfile -> Shoot -> Seed | Allow only get requests for CloudProfile s referenced by Shoot s that are assigned to the gardenlet ’s Seed . |
ClusterRoleBinding | create , get , update , patch , delete | ClusterRoleBinding -> ManagedSeed -> Shoot -> Seed | Allow create , get , update , patch requests for ManagedSeed s in the bootstrapping phase assigned to the gardenlet’s Seed s. Allow delete requests from gardenlets bootstrapped via ManagedSeed s. |
ConfigMap | get | ConfigMap -> Shoot -> Seed | Allow only get requests for ConfigMap s referenced by Shoot s that are assigned to the gardenlet ’s Seed . Allows reading the kube-system/cluster-identity ConfigMap . |
ControllerRegistration | get , list , watch | ControllerRegistration -> ControllerInstallation -> Seed | Allow get , list , watch requests for all ControllerRegistration s. |
ControllerDeployment | get | ControllerDeployment -> ControllerInstallation -> Seed | Allow get requests for ControllerDeployments s referenced by ControllerInstallation s assigned to the gardenlet ’s Seed . |
ControllerInstallation | get , list , watch , update , patch | ControllerInstallation -> Seed | Allow get , list , watch requests for all ControllerInstallation s. Allow only update , patch requests for ControllerInstallation s assigned to the gardenlet ’s Seed . |
Event | create , patch | none | Allow to create or patch all kinds of Event s. |
ExposureClass | get | ExposureClass -> Shoot -> Seed | Allow get requests for ExposureClass es referenced by Shoot s that are assigned to the gardenlet ’s Seed . Deny get requests to other ExposureClass es. |
Lease | create , get , watch , update | Lease -> Seed | Allow create , get , update , and delete requests for Lease s of the gardenlet ’s Seed . |
ManagedSeed | get , list , watch , update , patch | ManagedSeed -> Shoot -> Seed | Allow get , list , watch requests for all ManagedSeed s. Allow only update , patch requests for ManagedSeed s referencing a Shoot assigned to the gardenlet ’s Seed . |
Namespace | get | Namespace -> Shoot -> Seed | Allow get requests for Namespace s of Shoot s that are assigned to the gardenlet ’s Seed . Always allow get requests for the garden Namespace . |
Project | get | Project -> Namespace -> Shoot -> Seed | Allow get requests for Project s referenced by the Namespace of Shoot s that are assigned to the gardenlet ’s Seed . |
SecretBinding | get | SecretBinding -> Shoot -> Seed | Allow only get requests for SecretBinding s referenced by Shoot s that are assigned to the gardenlet ’s Seed . |
Secret | create , get , update , patch , delete (, list , watch ) | Secret -> Seed , Secret -> Shoot -> Seed , Secret -> SecretBinding -> Shoot -> Seed , BackupBucket -> Seed | Allow get , list , watch requests for all Secret s in the seed-<name> namespace. Allow only create , get , update , patch , delete requests for the Secret s related to resources assigned to the gardenlet's Seed`s. |
Seed | get , list , watch , create , update , patch , delete | Seed | Allow get , list , watch requests for all Seed s. Allow only create , update , patch , delete requests for the gardenlet ’s Seed s. [1] |
ServiceAccount | create , get , update , patch , delete | ServiceAccount -> ManagedSeed -> Shoot -> Seed | Allow create , get , update , patch requests for ManagedSeed s in the bootstrapping phase assigned to the gardenlet’s Seed s. Allow delete requests from gardenlets bootstrapped via ManagedSeed s. |
Shoot | get , list , watch , update , patch | Shoot -> Seed | Allow get , list , watch requests for all Shoot s. Allow only update , patch requests for Shoot s assigned to the gardenlet ’s Seed . |
ShootState | get , create , update , patch | ShootState -> Shoot -> Seed | Allow only get , create , update , patch requests for ShootState s belonging by Shoot s that are assigned to the gardenlet ’s Seed . |
[1] If you use ManagedSeed
resources then the gardenlet reconciling them (“parent gardenlet”) may be allowed to submit certain requests for the Seed
resources resulting out of such ManagedSeed
reconciliations (even if the “parent gardenlet” is not responsible for them):
- ℹ️ It is allowed to delete the
Seed
resources if the correspondingManagedSeed
objects already have adeletionTimestamp
(this is secure as gardenlets themselves don’t have permissions for deletingManagedSeed
s). - ⚠ It is allowed to create or update
Seed
resources if the correspondingManagedSeed
objects use a seed template, i.e.,.spec.seedTemplate != nil
. In this case, there is at least one gardenlet in your system which is responsible for two or moreSeed
s. Please keep in mind that this use case is not recommended for production scenarios (you should only have one dedicated gardenlet per seed cluster), hence, the security improvements discussed in this document might be limited.
SeedAuthorizer
Authorization Webhook Enablement
The SeedAuthorizer
is implemented as Kubernetes authorization webhook and part of the gardener-admission-controller
component running in the garden cluster.
🎛 In order to activate it, you have to follow these steps:
Set the following flags for the
kube-apiserver
of the garden cluster (i.e., thekube-apiserver
whose API is extended by Gardener):--authorization-mode=RBAC,Node,Webhook
(please note thatWebhook
should appear afterRBAC
in the list [1];Node
might not be needed if you use a virtual garden cluster)--authorization-webhook-config-file=<path-to-the-webhook-config-file>
--authorization-webhook-cache-authorized-ttl=0
--authorization-webhook-cache-unauthorized-ttl=0
The webhook config file (stored at
<path-to-the-webhook-config-file>
) should look as follows:apiVersion: v1 kind: Config clusters: - name: garden cluster: certificate-authority-data: base64(CA-CERT-OF-GARDENER-ADMISSION-CONTROLLER) server: https://gardener-admission-controller.garden/webhooks/auth/seed users: - name: kube-apiserver user: {} contexts: - name: auth-webhook context: cluster: garden user: kube-apiserver current-context: auth-webhook
When deploying the Gardener
controlplane
Helm chart, set.global.rbac.seedAuthorizer.enabled=true
. This will prevent that the RBAC resources granting global access for all gardenlets will be deployed.Delete the existing RBAC resources granting global access for all gardenlets by running:
kubectl delete \ clusterrole.rbac.authorization.k8s.io/gardener.cloud:system:seeds \ clusterrolebinding.rbac.authorization.k8s.io/gardener.cloud:system:seeds \ --ignore-not-found
Please note that you should activate the SeedRestriction
admission handler as well.
[1] The reason for the fact that
Webhook
authorization plugin should appear afterRBAC
is that thekube-apiserver
will be depending on thegardener-admission-controller
(serving the webhook). However, thegardener-admission-controller
can only start whengardener-apiserver
runs, butgardener-apiserver
itself can only start whenkube-apiserver
runs. IfWebhook
is beforeRBAC
thengardener-apiserver
might not be able to start, leading to a deadlock.
Authorizer Decisions
As mentioned earlier, it’s the authorizer’s job to evaluate API requests and return one of the following decisions:
DecisionAllow
: The request is allowed, further configured authorizers won’t be consulted.DecisionDeny
: The request is denied, further configured authorizers won’t be consulted.DecisionNoOpinion
: A decision cannot be made, further configured authorizers will be consulted.
For backwards compatibility, no requests are denied at the moment, so that they are still deferred to a subsequent authorizer like RBAC. Though, this might change in the future.
First, the SeedAuthorizer
extracts the Seed
name from the API request. This requires a proper TLS certificate the gardenlet
uses to contact the API server and is automatically given if TLS bootstrapping is used.
Concretely, the authorizer checks the certificate for name gardener.cloud:system:seed:<seed-name>
and group gardener.cloud:system:seeds
.
In cases where this information is missing e.g., when a custom Kubeconfig is used, the authorizer cannot make any decision. Thus, RBAC is still a considerable option to restrict the gardenlet
’s access permission if the above explained preconditions are not given.
With the Seed
name at hand, the authorizer checks for an existing path from the resource that a request is being made for to the Seed
belonging to the gardenlet
. Take a look at the Implementation Details section for more information.
Implementation Details
Internally, the SeedAuthorizer
uses a directed, acyclic graph data structure in order to efficiently respond to authorization requests for gardenlets:
- A vertex in this graph represents a Kubernetes resource with its kind, namespace, and name (e.g.,
Shoot:garden-my-project/my-shoot
). - An edge from vertex
u
to vertexv
in this graph exists when- (1)
v
is referred byu
andv
is aSeed
, or when - (2)
u
is referred byv
, or when - (3)
u
is strictly associated withv
.
- (1)
For example, a Shoot
refers to a Seed
, a CloudProfile
, a SecretBinding
, etc., so it has an outgoing edge to the Seed
(1) and incoming edges from the CloudProfile
and SecretBinding
vertices (2).
However, there might also be a ShootState
or a BackupEntry
resource strictly associated with this Shoot
, hence, it has incoming edges from these vertices (3).
In above picture the resources that are actively watched have are shaded. Gardener resources are green while Kubernetes resources are blue. It shows the dependencies between the resources and how the graph is built based on above rules.
ℹ️ Above picture shows all resources that may be accessed by gardenlet
s, except for the Quota
resource which is only included for completeness.
Now, when a gardenlet
wants to access certain resources then the SeedAuthorizer
uses a Depth-First traversal starting from the vertex representing the resource in question, e.g., from a Project
vertex.
If there is a path from the Project
vertex to the vertex representing the Seed
the gardenlet is responsible for then it allows the request.
Metrics
The SeedAuthorizer
registers the following metrics related to the mentioned graph implementation:
Metric | Description |
---|---|
gardener_admission_controller_seed_authorizer_graph_update_duration_seconds | Histogram of duration of resource dependency graph updates in seed authorizer, i.e., how long does it take to update the graph’s vertices/edges when a resource is created, changed, or deleted. |
gardener_admission_controller_seed_authorizer_graph_path_check_duration_seconds | Histogram of duration of checks whether a path exists in the resource dependency graph in seed authorizer. |
Debug Handler
When the .server.enableDebugHandlers
field in the gardener-admission-controller
’s component configuration is set to true
then it serves a handler that can be used for debugging the resource dependency graph under /debug/resource-dependency-graph
.
🚨 Only use this setting for development purposes as it enables unauthenticated users to view all data if they have access to the gardener-admission-controller
component.
The handler renders an HTML page displaying the current graph with a list of vertices and its associated incoming and outgoing edges to other vertices.
Depending on the size of the Gardener landscape (and consequently, the size of the graph), it might not be possible to render it in its entirety.
If there are more than 2000 vertices then the default filtering will selected for kind=Seed
to prevent overloading the output.
Example output:
-------------------------------------------------------------------------------
|
| # Seed:my-seed
| <- (11)
| BackupBucket:73972fe2-3d7e-4f61-a406-b8f9e670e6b7
| BackupEntry:garden-my-project/shoot--dev--my-shoot--4656a460-1a69-4f00-9372-7452cbd38ee3
| ControllerInstallation:dns-external-mxt8m
| ControllerInstallation:extension-shoot-cert-service-4qw5j
| ControllerInstallation:networking-calico-bgrb2
| ControllerInstallation:os-gardenlinux-qvb5z
| ControllerInstallation:provider-gcp-w4mvf
| Secret:garden/backup
| Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------
|
| # Shoot:garden-my-project/my-shoot
| <- (5)
| CloudProfile:gcp
| Namespace:garden-my-project
| Secret:garden-my-project/my-dns-secret
| SecretBinding:garden-my-project/my-credentials
| ShootState:garden-my-project/my-shoot
| -> (1)
| Seed:my-seed
|
-------------------------------------------------------------------------------
|
| # ShootState:garden-my-project/my-shoot
| -> (1)
| Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------
... (etc., similarly for the other resources)
There are anchor links to easily jump from one resource to another, and the page provides means for filtering the results based on the kind
, namespace
, and/or name
.
Pitfalls
When there is a relevant update to an existing resource, i.e., when a reference to another resource is changed, then the corresponding vertex (along with all associated edges) is first deleted from the graph before it gets added again with the up-to-date edges.
However, this does only work for vertices belonging to resources that are only created in exactly one “watch handler”.
For example, the vertex for a SecretBinding
can either be created in the SecretBinding
handler itself or in the Shoot
handler.
In such cases, deleting the vertex before (re-)computing the edges might lead to race conditions and potentially renders the graph invalid.
Consequently, instead of deleting the vertex, only the edges the respective handler is responsible for are deleted.
If the vertex ends up with no remaining edges then it also gets deleted automatically.
Afterwards, the vertex can either be added again or the updated edges can be created.
SeedRestriction
Admission Webhook Enablement
The SeedRestriction
is implemented as Kubernetes admission webhook and part of the gardener-admission-controller
component running in the garden cluster.
🎛 In order to activate it, you have to set .global.admission.seedRestriction.enabled=true
when using the Gardener controlplane
Helm chart.
This will add an additional webhook in the existing ValidatingWebhookConfiguration
of the gardener-admission-controller
which contains the configuration for the SeedRestriction
handler.
Please note that it should only be activated when the SeedAuthorizer
is active as well.
Admission Decisions
The admission’s purpose is to perform extended validation on requests which require the body of the object in question.
Additionally, it handles CREATE
requests of gardenlets (above discussed resource dependency graph cannot be used in such cases because there won’t be any vertex/edge for non-existing resources).
Gardenlets are restricted to only create new resources which are somehow related to the seed clusters they are responsible for.
8 - Getting Started Locally
Deploying Gardener locally
This document will walk you through deploying Gardener on your local machine. If you encounter difficulties, please open an issue so that we can make this process easier.
Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as both garden and seed cluster (please refer to the architecture overview) for simplicity.
Based on Skaffold, the container images for all required components will be built and deployed into the cluster (via their Helm charts).
Prerequisites
Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least
8
CPUs and8Gi
memory; see here how to configure the resources for Docker for Mac).Please note that 8 CPU / 8Gi memory might not be enough for more than two
Shoot
clusters, i.e., you might need to increase these values if you want to run additionalShoot
s.Additionally, please configure at least
120Gi
of disk size for the Docker daemon.Tip: With
docker system df
anddocker system prune -a
you can cleanup unused data.
Setting up the KinD cluster (garden and seed)
make kind-up
This command sets up a new KinD cluster named gardener-local
and stores the kubeconfig in the ./example/gardener-local/kind/kubeconfig
file.
It might be helpful to copy this file to
$HOME/.kube/config
since you will need to target this KinD cluster multiple times. Alternatively, make sure to set yourKUBECONFIG
environment variable to./example/gardener-local/kind/kubeconfig
for all future steps viaexport KUBECONFIG=example/gardener-local/kind/kubeconfig
.
All following steps assume that your are using this kubeconfig.
Setting up Gardener
make gardener-up
This will first build the images based (which might take a bit if you do it for the first time). Afterwards, the Gardener resources will be deployed into the cluster.
Creating a Shoot
cluster
You can wait for the Seed
to be ready by running
kubectl wait --for=condition=gardenletready --for=condition=extensionsready --for=condition=bootstrapped seed local --timeout=5m
Alternatively, you can run kubectl get seed local
and wait for the STATUS
to indicate readiness:
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
local Ready local local 4m42s vX.Y.Z-dev v1.21.1
In order to create a first shoot cluster, just run
kubectl apply -f example/provider-local/shoot.yaml
You can wait for the Shoot
to be ready by running
kubectl wait --for=condition=apiserveravailable --for=condition=controlplanehealthy --for=condition=everynodeready --for=condition=systemcomponentshealthy shoot local -n garden-local --timeout=10m
Alternatively, you can run kubectl -n garden-local get shoot local
and wait for the LAST OPERATION
to reach 100%
:
NAME CLOUDPROFILE PROVIDER REGION K8S VERSION HIBERNATION LAST OPERATION STATUS AGE
local local local local 1.21.0 Awake Create Processing (43%) healthy 94s
(Optional): You could also execute a simple e2e test (creating and deleting a shoot) by running
make test-e2e-local-fast KUBECONFIG="$PWD/example/gardener-local/kind/kubeconfig"
Accessing the Shoot
cluster
⚠️ Please note that in this setup shoot clusters are not accessible by default when you download the kubeconfig and try to communicate with them.
The reason is that your host most probably cannot resolve the DNS names of the clusters since provider-local
extension runs inside the KinD cluster (see this for more details).
Hence, if you want to access the shoot cluster, you have to run the following command which will extend your /etc/hosts
file with the required information to make the DNS names resolvable:
cat <<EOF | sudo tee -a /etc/hosts
# Manually created to access local Gardener shoot clusters with names 'local' or 'e2e-default' in the 'garden-local' namespace.
# TODO: Remove this again when the shoot cluster access is no longer required.
127.0.0.1 api.local.local.external.local.gardener.cloud
127.0.0.1 api.local.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-default.local.external.local.gardener.cloud
127.0.0.1 api.e2e-default.local.internal.local.gardener.cloud
EOF
Now you can access it by running
kubectl -n garden-local get secret local.kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > /tmp/kubeconfig-shoot-local.yaml
kubectl --kubeconfig=/tmp/kubeconfig-shoot-local.yaml get nodes
Deleting the Shoot
cluster
./hack/usage/delete shoot local garden-local
Tear down the Gardener environment
make kind-down
Further reading
This setup makes use of the local provider extension. You can read more about it in this document.
9 - Image Vector
Image Vector
The Gardenlet is deploying several different container images into the seed and the shoot clusters. The image repositories and tags are defined in a central image vector file. Obviously, the image versions defined there must fit together with the deployment manifests (e.g., some command-line flags do only exist in certain versions).
Example
images:
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: gcr.io/google_containers/pause-amd64
tag: "3.0"
version: 1.17.x
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: gcr.io/google_containers/pause-amd64
tag: "3.1"
version: ">= 1.18"
...
That means that the Gardenlet will use the pause-container
in with tag 3.0
for all seed/shoot clusters with Kubernetes version 1.17.x
, and tag 3.1
for all clusters with Kubernetes >= 1.18
.
Overwrite image vector
In some environment it is not possible to use these “pre-defined” images that come with a Gardener release.
A prominent example for that is Alicloud in China which does not allow access to Google’s GCR.
In these cases you might want to overwrite certain images, e.g., point the pause-container
to a different registry.
⚠️ If you specify an image that does not fit to the resource manifest then the seed/shoot reconciliation might fail.
In order to overwrite the images you must provide a similar file to Gardenlet:
images:
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: my-custom-image-registry/pause-amd64
tag: "3.0"
version: 1.17.x
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: my-custom-image-registry/pause-amd64
tag: "3.1"
version: ">= 1.18"
...
During deployment of the gardenlet create a ConfigMap
containing the above content and mount it as a volume into the gardenlet pod.
Next, specify the environment variable IMAGEVECTOR_OVERWRITE
whose value must be the path to the file you just mounted:
apiVersion: v1
kind: ConfigMap
metadata:
name: gardenlet-images-overwrite
namespace: garden
data:
images_overwrite.yaml: |
images:
- ...
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: gardenlet
namespace: garden
spec:
template:
...
spec:
containers:
- name: gardenlet
env:
- name: IMAGEVECTOR_OVERWRITE
value: /charts-overwrite/images_overwrite.yaml
volumeMounts:
- name: gardenlet-images-overwrite
mountPath: /charts-overwrite
...
volumes:
- name: gardenlet-images-overwrite
configMap:
name: gardenlet-images-overwrite
...
Image vectors for dependent components
The gardenlet is deploying a lot of different components that might deploy other images themselves. These components might use an image vector as well. Operators might want to customize the image locations for these transitive images as well, hence, they might need to specify an image vector overwrite for the components directly deployed by Gardener.
It is possible to specify the IMAGEVECTOR_OVERWRITE_COMPONENTS
environment variable to the gardenlet that points to a file with the following content:
components:
- name: etcd-druid
imageVectorOverwrite: |
images:
- name: etcd
tag: v1.2.3
repository: etcd/etcd
...
The gardenlet will, if supported by the directly deployed component (etcd-druid
in this example), inject the given imageVectorOverwrite
into the Deployment
manifest.
The respective component is responsible for using the overwritten images instead of its defaults.
10 - Migration V0 To V1
Migration from Gardener v0
to v1
Please refer to the document for older Gardener versions.
11 - Secret Binding Provider Controller
SecretBinding Provider Controller
This page describes the process on how to enable the SecretBinding provider controller.
Overview
With Gardener v1.38.0 the SecretBinding resource does now contain a new optional field .provider.type
(details about the motivation can be found in https://github.com/gardener/gardener/issues/4888). To make the process of setting the new field automated and afterwards to enforce validation on the new field in backwards compatible manner, Gardener features the SecretBinding provider controller and a feature gate - SecretBindingProviderValidation
.
Process
A Gardener landscape operator can follow the following steps:
Enable the SecretBinding provider controller of Gardener Controller Manager.
The SecretBinding provider controller is responsible to populate the
.provider.type
field of a SecretBinding based on its current usage by Shoot resources. For example if a Shootcrazy-botany
with.provider.type=aws
is using a SecretBindingmy-secret-binding
, then the SecretBinding provider controller will take care to set the.provider.type
field of the SecretBinding to the same provider type (aws
). To enable the SecretBinding provider controller, in the ControllerManagerConfiguration set thecontroller.secretBindingProvider.concurentSyncs
field (e.g set it to5
). Although that it is not recommended, the API allows Shoots from different provider types to reference the same SecretBinding (assuming that backing Secret contains data for both of the provider types). To preserve the backwards compatibility for such SecretBindings, the provider controller will maintain the multiple provider types in the field (it will join them with separator,
- for exampleaws,gcp
).Disable the SecretBinding provider controller and enable
SecretBindingProviderValidation
feature gate of Gardener API server.The
SecretBindingProviderValidation
feature gate of Gardener API server enables set of validations for the SecretBinding provider field. It forbids creating a Shoot that has a different provider type from the referenced SecretBinding’s one. It also enforces immutability on the field. After making sure that SecretBinding provider controller is enabled and it populated the.provider.type
field of a majority of the SecretBindings on a Gardener landscape (the SecretBindings that are unused will have their provider type unset), a Gardener landscape operator has to disable the SecretBinding provider controller and to enable theSecretBindingProviderValidation
feature gate of Gardener API server. To disable the SecretBinding provider controller, in the ControllerManagerConfiguration set thecontroller.secretBindingProvider.concurentSyncs
field to0
.
Implementation History
- Gardener v1.38: SecretBinding resource has a new optional field
.provider.type
. SecretBinding provider controller is disabled by default.SecretBindingProviderValidation
feature gate of Gardener API server is disabled by default. - Gardener v1.42: SecretBinding provider controller is enabled by default.
12 - Setup Gardener
Deploying the Gardener into a Kubernetes cluster
Similar to Kubernetes, Gardener consists out of control plane components (Gardener API server, Gardener controller manager, Gardener scheduler), and an agent component (Gardenlet). The control plane is deployed in the so-called garden cluster while the agent is installed into every seed cluster. Please note that it is possible to use the garden cluster as seed cluster by simply deploying the Gardenlet into it.
We are providing Helm charts in order to manage the various resources of the components. Please always make sure that you use the Helm chart version that matches the Gardener version you want to deploy.
Deploying the Gardener control plane (API server, admission controller, controller manager, scheduler)
The configuration values depict the various options to configure the different components. Please consult this document for component specific configurations and this document for authentication related specifics.
Also note that all resources and deployments need to be created in the garden
namespace (not overrideable).
If you enable the Gardener admission controller as part of you setup, please make sure the garden
namespace is labelled with app: gardener
.
Otherwise, the backing service account for the admission controller Pod might not be created successfully.
No action is necessary, if you deploy the garden
namespace with the Gardener control plane Helm chart.
After preparing your values in a separate controlplane-values.yaml
file (values.yaml can be used as starting point), you can run the following command against your garden cluster:
helm install charts/gardener/controlplane \
--namespace garden \
--name gardener-controlplane \
-f controlplane-values.yaml \
--wait
Deploying Gardener extensions
Gardener is an extensible system that does not contain the logic for provider-specific things like DNS management, cloud infrastructures, network plugins, operating system configs, and many more.
You have to install extension controllers for these parts. Please consult the documentation regarding extensions to get more information.
Deploying the Gardener agent (Gardenlet)
Please refer to this document on how to deploy a Gardenlet.
13 - Version Skew Policy
Version Skew Policy
This document describes the maximum version skew supported between various Gardener components.
Supported Gardener Versions
Gardener versions are expressed as x.y.z
, where x
is the major version, y
is the minor version, and z
is the patch version, following Semantic Versioning terminology.
The Gardener project maintains release branches for the most recent three minor releases.
Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility. Patch releases are cut from those branches at a regular cadence, plus additional urgent releases when required.
For more information, see this document.
Supported Version Skew
Technically, we follow the same policy as the Kubernetes project.
However, given that our release cadence is much more frequent compared to Kubernetes (every 14d
vs. every 120d
), in many cases it is possible to skip a version.
Still, to be on the safe side, it is highly recommended to follow the described policy.
gardener-apiserver
In multi-instance setups of Gardener, the newest and oldest gardener-apiserver
instances must be within one minor version.
Example:
- newest
gardener-apiserver
is at 1.37 - other
gardener-apiserver
instances are supported at 1.37 and v1.36
gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet
gardener-controller-manager
, gardener-scheduler
, gardener-admission-controller
, and gardenlet
must not be newer than the gardener-apiserver
instances they communicate with.
They are expected to match the gardener-apiserver
minor version, but may be up to one minor version older (to allow live upgrades).
Example:
gardener-apiserver
is at v1.37gardener-controller-manager
,gardener-scheduler
,gardener-admission-controller
, andgardenlet
are supported at 1.37 and v1.36
Supported Component Upgrade Order
The supported version skew between components has implications on the order in which components must be upgraded. This section describes the order in which components must be upgraded to transition an existing Gardener installation from version 1.37 to version 1.38.
gardener-apiserver
Pre-requisites:
- In a single-instance setup, the existing
gardener-apiserver
instance is 1.37 - In a multi-instance setup, all
gardener-apiserver
instances are at 1.37 or 1.38 (this ensures maximum skew of 1 minor version between the oldest and newestgardener-apiserver
instance) - The
gardener-controller-manager
,gardener-scheduler
,gardener-admission-controller
, andgardenlet
instances that communicate with thisgardener-apiserver
are at version 1.37 (this ensures they are not newer than the existing API server version and are within 1 minor version of the new API server version)
Action:
- Upgrade
gardener-apiserver
to 1.38
gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet
Pre-requisites:
- The
gardener-apiserver
instances these components communicate with are at 1.38 (in multi-instance setups in which these components can communicate with anygardener-apiserver
instance in the cluster, allgardener-apiserver
instances must be upgraded before upgrading these components)
Action:
- Upgrade
gardener-controller-manager
,gardener-scheduler
,gardener-admission-controller
, andgardenlet
to 1.38
Supported Kubernetes Versions
Please refer to this document.