This is the multi-page printable view of this section. Click here to print.
Security
1 - Admission Configuration for the `PodSecurity` Admission Plugin
PodSecurity
plugin in .spec.kubernetes.kubeAPIServer.admissionPlugins
Admission Configuration for the PodSecurity
Admission Plugin
If you wish to add your custom configuration for the PodSecurity
plugin, you can do so in the Shoot spec under .spec.kubernetes.kubeAPIServer.admissionPlugins
by adding:
admissionPlugins:
- name: PodSecurity
config:
apiVersion: pod-security.admission.config.k8s.io/v1
kind: PodSecurityConfiguration
# Defaults applied when a mode label is not set.
#
# Level label values must be one of:
# - "privileged" (default)
# - "baseline"
# - "restricted"
#
# Version label values must be one of:
# - "latest" (default)
# - specific version like "v1.25"
defaults:
enforce: "privileged"
enforce-version: "latest"
audit: "privileged"
audit-version: "latest"
warn: "privileged"
warn-version: "latest"
exemptions:
# Array of authenticated usernames to exempt.
usernames: []
# Array of runtime class names to exempt.
runtimeClasses: []
# Array of namespaces to exempt.
namespaces: []
For proper functioning of Gardener, kube-system
namespace will also be automatically added to the exemptions.namespaces
list.
2 - Audit a Kubernetes Cluster
ConfigMap
and reference it in the shoot specAudit a Kubernetes Cluster
The shoot cluster is a Kubernetes cluster and its kube-apiserver
handles the audit events. In order to define which audit events must be logged, a proper audit policy file must be passed to the Kubernetes API server. You could find more information about auditing a kubernetes cluster in the Auditing topic.
Default Audit Policy
By default, the Gardener will deploy the shoot cluster with audit policy defined in the kube-apiserver package.
Custom Audit Policy
If you need specific audit policy for your shoot cluster, then you could deploy the required audit policy in the garden cluster as ConfigMap
resource and set up your shoot to refer this ConfigMap
. Note that the policy must be stored under the key policy
in the data section of the ConfigMap
.
For example, deploy the auditpolicy ConfigMap
in the same namespace as your Shoot
resource:
kubectl apply -f example/95-configmap-custom-audit-policy.yaml
then set your shoot to refer that ConfigMap
(only related fields are shown):
spec:
kubernetes:
kubeAPIServer:
auditConfig:
auditPolicy:
configMapRef:
name: auditpolicy
Gardener validate the Shoot
resource to refer only existing ConfigMap
containing valid audit policy, and rejects the Shoot
on failure.
If you want to switch back to the default audit policy, you have to remove the section
auditPolicy:
configMapRef:
name: <configmap-name>
from the shoot spec.
Rolling Out Changes to the Audit Policy
Gardener is not automatically rolling out changes to the Audit Policy to minimize the amount of Shoot reconciliations in order to prevent cloud provider rate limits, etc. Gardener will pick up the changes on the next reconciliation of Shoots referencing the Audit Policy ConfigMap. If users want to immediately rollout Audit Policy changes, they can manually trigger a Shoot reconciliation as described in triggering an immediate reconciliation. This is similar to changes to the cloud provider secret referenced by Shoots.
3 - Default Seccomp Profile
RuntimeDefault
as the default seccomp profile through spec.kubernetes.kubelet.seccompDefault
Default Seccomp Profile and Configuration
This is a short guide describing how to enable the defaulting of seccomp profiles for Gardener managed workloads in the seed. Running pods in Unconfined
(seccomp disabled) mode is undesirable since this is the least restrictive profile. Also, mind that any privileged container will always run as Unconfined
. More information about seccomp can be found in this Kubernetes tutorial.
Setting the Seccomp Profile to RuntimeDefault for Seed Clusters
To address the above issue, Gardener provides a webhook that is capable of mutating pods in the seed clusters, explicitly providing them with a seccomp profile type of RuntimeDefault
. This profile is defined by the container runtime and represents a set of default syscalls that are allowed or not.
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
A Pod
is mutated when all of the following preconditions are fulfilled:
- The
Pod
is created in a Gardener managed namespace. - The
Pod
is NOT labeled withseccompprofile.resources.gardener.cloud/skip
. - The
Pod
does NOT explicitly specify.spec.securityContext.seccompProfile.type
.
How to Configure
To enable this feature, the gardenlet DefaultSeccompProfile
feature gate must be set to true
.
featureGates:
DefaultSeccompProfile: true
Please refer to the examples in this yaml file for more information.
Once the feature gate is enabled, the webhook will be registered and configured for the seed cluster. Newly created pods will be mutated to have their seccomp profile set to RuntimeDefault
.
Note: Please note that this feature is still in Alpha, so you might see instabilities every now and then.
Setting the Seccomp Profile to RuntimeDefault for Shoot Clusters
You can enable the use of RuntimeDefault
as the default seccomp profile for all workloads. If enabled, the kubelet will use the RuntimeDefault
seccomp profile by default, which is defined by the container runtime, instead of using the Unconfined
mode. More information for this feature can be found in the Kubernetes documentation.
To use seccomp profile defaulting, you must run the kubelet with the SeccompDefault
feature gate enabled (this is the default).
How to Configure
To enable this feature, the kubelet seccompDefault
configuration parameter must be set to true
in the shoot’s spec.
spec:
kubernetes:
version: 1.25.0
kubelet:
seccompDefault: true
Please refer to the examples in this yaml file for more information.
4 - ETCD Encryption Config
spec.kubernetes.kubeAPIServer.encryptionConfig
ETCD Encryption Config
The spec.kubernetes.kubeAPIServer.encryptionConfig
field in the Shoot API allows users to customize encryption configurations for the API server. It provides options to specify additional resources for encryption beyond secrets.
Usage Guidelines
- The
resources
field can be used to specify resources that should be encrypted in addition to secrets. Secrets are always encrypted. - Each item is a Kubernetes resource name in plural (resource or resource.group). Wild cards are not supported.
- Adding an item to this list will cause patch requests for all the resources of that kind to encrypt them in the etcd. See Encrypting Confidential Data at Rest for more details.
- Removing an item from this list will cause patch requests for all the resources of that type to decrypt and rewrite the resource as plain text. See Decrypt Confidential Data that is Already Encrypted at Rest for more details.
ℹ️ Note that configuring encryption for a custom resource is only supported for Kubernetes versions >= 1.26.
Example Usage in a Shoot
spec:
kubernetes:
kubeAPIServer:
encryptionConfig:
resources:
- configmaps
- statefulsets.apps
- customresource.fancyoperator.io
5 - OpenIDConnect Presets
ClusterOpenIDConnectPreset and OpenIDConnectPreset
Note: OpenID Connect is deprecated in favor of Structured Authentication configuration. Setting OpenID Connect configurations is forbidden for clusters with Kubernetes version
>= 1.32
.
This page provides an overview of ClusterOpenIDConnectPresets and OpenIDConnectPresets, which are objects for injecting OpenIDConnect Configuration into Shoot
at creation time. The injected information contains configuration for the Kube API Server and optionally configuration for kubeconfig generation using said configuration.
OpenIDConnectPreset
An OpenIDConnectPreset is an API resource for injecting additional runtime OIDC requirements into a Shoot at creation time. You use label selectors to specify the Shoot
to which a given OpenIDConnectPreset applies.
Using a OpenIDConnectPresets allows project owners to not have to explicitly provide the same OIDC configuration for every Shoot
in their Project
.
For more information about the background, see the issue for OpenIDConnectPreset.
How OpenIDConnectPreset Works
Gardener provides an admission controller (OpenIDConnectPreset) which, when enabled, applies OpenIDConnectPresets to incoming Shoot
creation requests. When a Shoot
creation request occurs, the system does the following:
Retrieve all OpenIDConnectPreset available for use in the
Shoot
namespace.Check if the shoot label selectors of any OpenIDConnectPreset matches the labels on the Shoot being created.
If multiple presets are matched then only one is chosen and results are sorted based on:
.spec.weight
value.- lexicographically ordering their names (e.g.,
002preset
>001preset
)
If the
Shoot
already has a.spec.kubernetes.kubeAPIServer.oidcConfig
, then no mutation occurs.
Simple OpenIDConnectPreset Example
This is a simple example to show how a Shoot
is modified by the OpenIDConnectPreset:
apiVersion: settings.gardener.cloud/v1alpha1
kind: OpenIDConnectPreset
metadata:
name: test-1
namespace: default
spec:
shootSelector:
matchLabels:
oidc: enabled
server:
clientID: test-1
issuerURL: https://foo.bar
# caBundle: |
# -----BEGIN CERTIFICATE-----
# Li4u
# -----END CERTIFICATE-----
groupsClaim: groups-claim
groupsPrefix: groups-prefix
usernameClaim: username-claim
usernamePrefix: username-prefix
signingAlgs:
- RS256
requiredClaims:
key: value
weight: 90
Create the OpenIDConnectPreset:
kubectl apply -f preset.yaml
Examine the created OpenIDConnectPreset:
kubectl get openidconnectpresets
NAME ISSUER SHOOT-SELECTOR AGE
test-1 https://foo.bar oidc=enabled 1s
Simple Shoot
example:
This is a sample of a Shoot
with some fields omitted:
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
name: preset
namespace: default
labels:
oidc: enabled
spec:
kubernetes:
version: 1.20.2
Create the Shoot:
kubectl apply -f shoot.yaml
Examine the created Shoot:
kubectl get shoot preset -o yaml
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
name: preset
namespace: default
labels:
oidc: enabled
spec:
kubernetes:
kubeAPIServer:
oidcConfig:
clientID: test-1
groupsClaim: groups-claim
groupsPrefix: groups-prefix
issuerURL: https://foo.bar
requiredClaims:
key: value
signingAlgs:
- RS256
usernameClaim: username-claim
usernamePrefix: username-prefix
version: 1.20.2
Disable OpenIDConnectPreset
The OpenIDConnectPreset admission control is enabled by default. To disable it, use the --disable-admission-plugins
flag on the gardener-apiserver.
For example:
--disable-admission-plugins=OpenIDConnectPreset
ClusterOpenIDConnectPreset
A ClusterOpenIDConnectPreset is an API resource for injecting additional runtime OIDC requirements into a Shoot at creation time. In contrast to OpenIDConnect, it’s a cluster-scoped resource. You use label selectors to specify the Project
and Shoot
to which a given OpenIDCConnectPreset applies.
Using a OpenIDConnectPresets allows cluster owners to not have to explicitly provide the same OIDC configuration for every Shoot
in specific Project
.
For more information about the background, see the issue for ClusterOpenIDConnectPreset.
How ClusterOpenIDConnectPreset Works
Gardener provides an admission controller (ClusterOpenIDConnectPreset) which, when enabled, applies ClusterOpenIDConnectPresets to incoming Shoot
creation requests. When a Shoot
creation request occurs, the system does the following:
Retrieve all ClusterOpenIDConnectPresets available.
Check if the project label selector of any ClusterOpenIDConnectPreset matches the labels of the
Project
in which theShoot
is being created.Check if the shoot label selectors of any ClusterOpenIDConnectPreset matches the labels on the
Shoot
being created.If multiple presets are matched then only one is chosen and results are sorted based on:
.spec.weight
value.- lexicographically ordering their names ( e.g.
002preset
>001preset
)
If the
Shoot
already has a.spec.kubernetes.kubeAPIServer.oidcConfig
then no mutation occurs.
Note: Due to the previous requirement, if a
Shoot
is matched by bothOpenIDConnectPreset
andClusterOpenIDConnectPreset
, thenOpenIDConnectPreset
takes precedence overClusterOpenIDConnectPreset
.
Simple ClusterOpenIDConnectPreset Example
This is a simple example to show how a Shoot
is modified by the ClusterOpenIDConnectPreset:
apiVersion: settings.gardener.cloud/v1alpha1
kind: ClusterOpenIDConnectPreset
metadata:
name: test
spec:
shootSelector:
matchLabels:
oidc: enabled
projectSelector: {} # selects all projects.
server:
clientID: cluster-preset
issuerURL: https://foo.bar
# caBundle: |
# -----BEGIN CERTIFICATE-----
# Li4u
# -----END CERTIFICATE-----
groupsClaim: groups-claim
groupsPrefix: groups-prefix
usernameClaim: username-claim
usernamePrefix: username-prefix
signingAlgs:
- RS256
requiredClaims:
key: value
weight: 90
Create the ClusterOpenIDConnectPreset:
kubectl apply -f preset.yaml
Examine the created ClusterOpenIDConnectPreset:
kubectl get clusteropenidconnectpresets
NAME ISSUER PROJECT-SELECTOR SHOOT-SELECTOR AGE
test https://foo.bar <none> oidc=enabled 1s
This is a sample of a Shoot
, with some fields omitted:
kind: Shoot
apiVersion: core.gardener.cloud/v1beta1
metadata:
name: preset
namespace: default
labels:
oidc: enabled
spec:
kubernetes:
version: 1.20.2
Create the Shoot:
kubectl apply -f shoot.yaml
Examine the created Shoot:
kubectl get shoot preset -o yaml
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
name: preset
namespace: default
labels:
oidc: enabled
spec:
kubernetes:
kubeAPIServer:
oidcConfig:
clientID: cluster-preset
groupsClaim: groups-claim
groupsPrefix: groups-prefix
issuerURL: https://foo.bar
requiredClaims:
key: value
signingAlgs:
- RS256
usernameClaim: username-claim
usernamePrefix: username-prefix
version: 1.20.2
Disable ClusterOpenIDConnectPreset
The ClusterOpenIDConnectPreset admission control is enabled by default. To disable it, use the --disable-admission-plugins
flag on the gardener-apiserver.
For example:
--disable-admission-plugins=ClusterOpenIDConnectPreset
6 - Shoot Serviceaccounts
ServiceAccount
Configurations for Shoot Clusters
The Shoot
specification allows to configure some of the settings for the handling of ServiceAccount
s:
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
spec:
kubernetes:
kubeAPIServer:
serviceAccountConfig:
issuer: foo
acceptedIssuers:
- foo1
- foo2
extendTokenExpiration: true
maxTokenExpiration: 45d
...
Issuer and Accepted Issuers
The .spec.kubernetes.kubeAPIServer.serviceAccountConfig.{issuer,acceptedIssuers}
fields are translated to the --service-account-issuer
flag for the kube-apiserver
.
The issuer will assert its identifier in the iss
claim of the issued tokens.
According to the upstream specification, values need to meet the following requirements:
This value is a string or URI. If this option is not a valid URI per the OpenID Discovery 1.0 spec, the ServiceAccountIssuerDiscovery feature will remain disabled, even if the feature gate is set to true. It is highly recommended that this value comply with the OpenID spec: https://openid.net/specs/openid-connect-discovery-1_0.html. In practice, this means that service-account-issuer must be an https URL. It is also highly recommended that this URL be capable of serving OpenID discovery documents at {service-account-issuer}/.well-known/openid-configuration.
By default, Gardener uses the internal cluster domain as issuer (e.g., https://api.foo.bar.example.com
).
If you specify the issuer
, then this default issuer will always be part of the list of accepted issuers (you don’t need to specify it yourself).
Caution
If you change from the default issuer to a custom
issuer
, all previously issued tokens will still be valid/accepted. However, if you change from a customissuer
A
to anotherissuer
B
(custom or default), then you have to addA
to theacceptedIssuers
so that previously issued tokens are not invalidated. Otherwise, the control plane components as well as system components and your workload pods might fail. You can removeA
from theacceptedIssuers
when all currently active tokens have been issued solely byB
. This can be ensured by using projected token volumes with a short validity, or by rolling out all pods. Additionally, allServiceAccount
token secrets should be recreated. Apart from this, you should wait for at least12h
to make sure the control plane and system components have received a new token from Gardener.
Token Expirations
The .spec.kubernetes.kubeAPIServer.serviceAccountConfig.extendTokenExpiration
configures the --service-account-extend-token-expiration
flag of the kube-apiserver
.
It is enabled by default and has the following specification:
Turns on projected service account expiration extension during token generation, which helps safe transition from legacy token to bound service account token feature. If this flag is enabled, admission injected tokens would be extended up to 1 year to prevent unexpected failure during transition, ignoring value of service-account-max-token-expiration.
The .spec.kubernetes.kubeAPIServer.serviceAccountConfig.maxTokenExpiration
configures the --service-account-max-token-expiration
flag of the kube-apiserver
.
It has the following specification:
The maximum validity duration of a token created by the service account token issuer. If an otherwise valid TokenRequest with a validity duration larger than this value is requested, a token will be issued with a validity duration of this value.
Note
The value for this field must be in the
[30d,90d]
range. The background for this limitation is that all Gardener components rely on theTokenRequest
API and the Kubernetes service account token projection feature with short-lived, auto-rotating tokens. Any values lower than30d
risk impacting the SLO for shoot clusters, and any values above90d
violate security best practices with respect to maximum validity of credentials before they must be rotated. Given that the field just specifies the upper bound, end-users can still use lower values for their individual workload by specifying the.spec.volumes[].projected.sources[].serviceAccountToken.expirationSeconds
in thePodSpec
s.
Managed Service Account Issuer
Gardener also provides a way to manage the service account issuer of a shoot cluster as well as serving its OIDC discovery documents from a centrally managed server called Gardener Discovery Server.
This ability removes the need for changing the .spec.kubernetes.kubeAPIServer.serviceAccountConfig.issuer
and exposing it separately.
Prerequisites
Note
The following prerequisites are responsibility of the Gardener Administrators and are not something that end users can configure by themselves. If uncertain that these requirements are met, please contact your Gardener Administrator.
Prerequisites:
- The Garden Cluster should have the Gardener Discovery Server deployed and configured. The easiest way to handle this is by using the gardener-operator.
- The
ShootManagedIssuer
feature gate should be enabled.
Enablement
If the prerequisites are met then the feature can be enabled for a shoot cluster by annotating it with authentication.gardener.cloud/issuer=managed
. Mind that once enabled, this feature cannot be disabled. After the shoot is reconciled, you can retrieve the new shoot service account issuer value from the shoot’s status. A sample query that will retrieve the managed issuer looks like this:
kubectl -n my-project get shoot my-shoot -o jsonpath='{.status.advertisedAddresses[?(@.name=="service-account-issuer")].url}'
Once retrieved, the shoot’s OIDC discovery documents can be explored by querying the /.well-known/openid-configuration
endpoint of the issuer.
Mind that this annotation is incompatible with the .spec.kubernetes.kubeAPIServer.serviceAccountConfig.issuer
field, so if you want to enable it then the issuer
field should not be set in the shoot specification.
Caution
If you change from the default issuer to a managed issuer, all previously issued tokens will still be valid/accepted. However, if you change from a custom
issuer
A
to a managed issuer, then you have to addA
to the.spec.kubernetes.kubeAPIServer.serviceAccountConfig.acceptedIssuers
so that previously issued tokens are not invalidated. Otherwise, the control plane components as well as system components and your workload pods might fail. You can removeA
from theacceptedIssuers
when all currently active tokens have been issued solely by the managed issuer. This can be ensured by using projected token volumes with a short validity, or by rolling out all pods. Additionally, allServiceAccount
token secrets should be recreated. Apart from this, you should wait for at least12h
to make sure the control plane and system components have received a new token from Gardener.