Security

1 - Admission Configuration for the `PodSecurity` Admission Plugin

Admission Configuration for the PodSecurity Admission Plugin

If you wish to add your custom configuration for the PodSecurity plugin, you can do so in the Shoot spec under .spec.kubernetes.kubeAPIServer.admissionPlugins by adding:

admissionPlugins:
- name: PodSecurity
  config:
    apiVersion: pod-security.admission.config.k8s.io/v1
    kind: PodSecurityConfiguration
    # Defaults applied when a mode label is not set.
    #
    # Level label values must be one of:
    # - "privileged" (default)
    # - "baseline"
    # - "restricted"
    #
    # Version label values must be one of:
    # - "latest" (default) 
    # - specific version like "v1.25"
    defaults:
      enforce: "privileged"
      enforce-version: "latest"
      audit: "privileged"
      audit-version: "latest"
      warn: "privileged"
      warn-version: "latest"
    exemptions:
      # Array of authenticated usernames to exempt.
      usernames: []
      # Array of runtime class names to exempt.
      runtimeClasses: []
      # Array of namespaces to exempt.
      namespaces: []

For proper functioning of Gardener, kube-system namespace will also be automatically added to the exemptions.namespaces list.

2 - Audit a Kubernetes Cluster

Audit a Kubernetes Cluster

The shoot cluster is a Kubernetes cluster and its kube-apiserver handles the audit events. In order to define which audit events must be logged, a proper audit policy file must be passed to the Kubernetes API server. You could find more information about auditing a kubernetes cluster in the Auditing topic.

Default Audit Policy

By default, the Gardener will deploy the shoot cluster with audit policy defined in the kube-apiserver package.

Custom Audit Policy

If you need specific audit policy for your shoot cluster, then you could deploy the required audit policy in the garden cluster as ConfigMap resource and set up your shoot to refer this ConfigMap. Note that the policy must be stored under the key policy in the data section of the ConfigMap.

For example, deploy the auditpolicy ConfigMap in the same namespace as your Shoot resource:

kubectl apply -f example/95-configmap-custom-audit-policy.yaml

then set your shoot to refer that ConfigMap (only related fields are shown):

spec:
  kubernetes:
    kubeAPIServer:
      auditConfig:
        auditPolicy:
          configMapRef:
            name: auditpolicy

Gardener validate the Shoot resource to refer only existing ConfigMap containing valid audit policy, and rejects the Shoot on failure. If you want to switch back to the default audit policy, you have to remove the section

auditPolicy:
  configMapRef:
    name: <configmap-name>

from the shoot spec.

Rolling Out Changes to the Audit Policy

Gardener is not automatically rolling out changes to the Audit Policy to minimize the amount of Shoot reconciliations in order to prevent cloud provider rate limits, etc. Gardener will pick up the changes on the next reconciliation of Shoots referencing the Audit Policy ConfigMap. If users want to immediately rollout Audit Policy changes, they can manually trigger a Shoot reconciliation as described in triggering an immediate reconciliation. This is similar to changes to the cloud provider secret referenced by Shoots.

3 - Default Seccomp Profile

Default Seccomp Profile and Configuration

This is a short guide describing how to enable the defaulting of seccomp profiles for Gardener managed workloads in the seed. Running pods in Unconfined (seccomp disabled) mode is undesirable since this is the least restrictive profile. Also, mind that any privileged container will always run as Unconfined. More information about seccomp can be found in this Kubernetes tutorial.

Setting the Seccomp Profile to RuntimeDefault for Seed Clusters

To address the above issue, Gardener provides a webhook that is capable of mutating pods in the seed clusters, explicitly providing them with a seccomp profile type of RuntimeDefault. This profile is defined by the container runtime and represents a set of default syscalls that are allowed or not.

spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault

A Pod is mutated when all of the following preconditions are fulfilled:

1. The Pod is created in a Gardener managed namespace.
2. The Pod is NOT labeled with seccompprofile.resources.gardener.cloud/skip.
3. The Pod does NOT explicitly specify .spec.securityContext.seccompProfile.type.

How to Configure

To enable this feature, the gardenlet DefaultSeccompProfile feature gate must be set to true.

featureGates:
  DefaultSeccompProfile: true

Please refer to the examples in this yaml file for more information.

Once the feature gate is enabled, the webhook will be registered and configured for the seed cluster. Newly created pods will be mutated to have their seccomp profile set to RuntimeDefault.

Note: Please note that this feature is still in Alpha, so you might see instabilities every now and then.

Setting the Seccomp Profile to RuntimeDefault for Shoot Clusters

You can enable the use of RuntimeDefault as the default seccomp profile for all workloads. If enabled, the kubelet will use the RuntimeDefault seccomp profile by default, which is defined by the container runtime, instead of using the Unconfined mode. More information for this feature can be found in the Kubernetes documentation.

To use seccomp profile defaulting, you must run the kubelet with the SeccompDefault feature gate enabled (this is the default).

How to Configure

To enable this feature, the kubelet seccompDefault configuration parameter must be set to true in the shoot’s spec.

spec:
  kubernetes:
    version: 1.25.0
    kubelet:
      seccompDefault: true

Please refer to the examples in this yaml file for more information.

4 - ETCD Encryption Config

ETCD Encryption Config

The spec.kubernetes.kubeAPIServer.encryptionConfig field in the Shoot API allows users to customize encryption configurations for the API server. It provides options to specify additional resources for encryption beyond secrets.

Usage Guidelines

• The resources field can be used to specify resources that should be encrypted in addition to secrets. Secrets are always encrypted.
• Each item is a Kubernetes resource name in plural (resource or resource.group). Wild cards are not supported.
• Adding an item to this list will cause patch requests for all the resources of that kind to encrypt them in the etcd. See Encrypting Confidential Data at Rest for more details.
• Removing an item from this list will cause patch requests for all the resources of that type to decrypt and rewrite the resource as plain text. See Decrypt Confidential Data that is Already Encrypted at Rest for more details.

ℹ️ Note that configuring encryption for a custom resource is only supported for Kubernetes versions >= 1.26.

Example Usage in a Shoot

spec:
  kubernetes:
    kubeAPIServer:
      encryptionConfig:
        resources:
          - configmaps
          - statefulsets.apps
          - customresource.fancyoperator.io

5 - OpenIDConnect Presets

ClusterOpenIDConnectPreset and OpenIDConnectPreset

This page provides an overview of ClusterOpenIDConnectPresets and OpenIDConnectPresets, which are objects for injecting OpenIDConnect Configuration into Shoot at creation time. The injected information contains configuration for the Kube API Server and optionally configuration for kubeconfig generation using said configuration.

OpenIDConnectPreset

An OpenIDConnectPreset is an API resource for injecting additional runtime OIDC requirements into a Shoot at creation time. You use label selectors to specify the Shoot to which a given OpenIDConnectPreset applies.

Using a OpenIDConnectPresets allows project owners to not have to explicitly provide the same OIDC configuration for every Shoot in their Project.

For more information about the background, see the issue for OpenIDConnectPreset.

How OpenIDConnectPreset Works

Gardener provides an admission controller (OpenIDConnectPreset) which, when enabled, applies OpenIDConnectPresets to incoming Shoot creation requests. When a Shoot creation request occurs, the system does the following:

• Retrieve all OpenIDConnectPreset available for use in the Shoot namespace.

• Check if the shoot label selectors of any OpenIDConnectPreset matches the labels on the Shoot being created.

• If multiple presets are matched then only one is chosen and results are sorted based on:

  1. .spec.weight value.
  2. lexicographically ordering their names (e.g., 002preset > 001preset)

• If the Shoot already has a .spec.kubernetes.kubeAPIServer.oidcConfig, then no mutation occurs.

Simple OpenIDConnectPreset Example

This is a simple example to show how a Shoot is modified by the OpenIDConnectPreset:

apiVersion: settings.gardener.cloud/v1alpha1
kind: OpenIDConnectPreset
metadata:
  name:  test-1
  namespace: default
spec:
  shootSelector:
    matchLabels:
      oidc: enabled
  server:
    clientID: test-1
    issuerURL: https://foo.bar
    # caBundle: |
    #   -----BEGIN CERTIFICATE-----
    #   Li4u
    #   -----END CERTIFICATE-----
    groupsClaim: groups-claim
    groupsPrefix: groups-prefix
    usernameClaim: username-claim
    usernamePrefix: username-prefix
    signingAlgs:
    - RS256
    requiredClaims:
      key: value
  weight: 90

Create the OpenIDConnectPreset:

kubectl apply -f preset.yaml

Examine the created OpenIDConnectPreset:

kubectl get openidconnectpresets
NAME     ISSUER            SHOOT-SELECTOR   AGE
test-1   https://foo.bar   oidc=enabled     1s

Simple Shoot example:

This is a sample of a Shoot with some fields omitted:

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: preset
  namespace: default
  labels:
    oidc: enabled
spec:
  kubernetes:
    version: 1.20.2

Create the Shoot:

kubectl apply -f shoot.yaml

Examine the created Shoot:

kubectl get shoot preset -o yaml

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: preset
  namespace: default
  labels:
    oidc: enabled
spec:
  kubernetes:
    kubeAPIServer:
      oidcConfig:
        clientID: test-1
        groupsClaim: groups-claim
        groupsPrefix: groups-prefix
        issuerURL: https://foo.bar
        requiredClaims:
          key: value
        signingAlgs:
        - RS256
        usernameClaim: username-claim
        usernamePrefix: username-prefix
    version: 1.20.2

Disable OpenIDConnectPreset

The OpenIDConnectPreset admission control is enabled by default. To disable it, use the --disable-admission-plugins flag on the gardener-apiserver.

For example:

--disable-admission-plugins=OpenIDConnectPreset

ClusterOpenIDConnectPreset

A ClusterOpenIDConnectPreset is an API resource for injecting additional runtime OIDC requirements into a Shoot at creation time. In contrast to OpenIDConnect, it’s a cluster-scoped resource. You use label selectors to specify the Project and Shoot to which a given OpenIDCConnectPreset applies.

Using a OpenIDConnectPresets allows cluster owners to not have to explicitly provide the same OIDC configuration for every Shoot in specific Project.

For more information about the background, see the issue for ClusterOpenIDConnectPreset.

How ClusterOpenIDConnectPreset Works

Gardener provides an admission controller (ClusterOpenIDConnectPreset) which, when enabled, applies ClusterOpenIDConnectPresets to incoming Shoot creation requests. When a Shoot creation request occurs, the system does the following:

• Retrieve all ClusterOpenIDConnectPresets available.

• Check if the project label selector of any ClusterOpenIDConnectPreset matches the labels of the Project in which the Shoot is being created.

• Check if the shoot label selectors of any ClusterOpenIDConnectPreset matches the labels on the Shoot being created.

• If multiple presets are matched then only one is chosen and results are sorted based on:

  1. .spec.weight value.
  2. lexicographically ordering their names ( e.g. 002preset > 001preset )

• If the Shoot already has a .spec.kubernetes.kubeAPIServer.oidcConfig then no mutation occurs.

Note: Due to the previous requirement, if a Shoot is matched by both OpenIDConnectPreset and ClusterOpenIDConnectPreset, then OpenIDConnectPreset takes precedence over ClusterOpenIDConnectPreset.

Simple ClusterOpenIDConnectPreset Example

This is a simple example to show how a Shoot is modified by the ClusterOpenIDConnectPreset:

apiVersion: settings.gardener.cloud/v1alpha1
kind: ClusterOpenIDConnectPreset
metadata:
  name:  test
spec:
  shootSelector:
    matchLabels:
      oidc: enabled
  projectSelector: {} # selects all projects.
  server:
    clientID: cluster-preset
    issuerURL: https://foo.bar
    # caBundle: |
    #   -----BEGIN CERTIFICATE-----
    #   Li4u
    #   -----END CERTIFICATE-----
    groupsClaim: groups-claim
    groupsPrefix: groups-prefix
    usernameClaim: username-claim
    usernamePrefix: username-prefix
    signingAlgs:
    - RS256
    requiredClaims:
      key: value
  weight: 90

Create the ClusterOpenIDConnectPreset:

kubectl apply -f preset.yaml

Examine the created ClusterOpenIDConnectPreset:

kubectl get clusteropenidconnectpresets
NAME     ISSUER            PROJECT-SELECTOR   SHOOT-SELECTOR   AGE
test     https://foo.bar   <none>             oidc=enabled     1s

This is a sample of a Shoot, with some fields omitted:

kind: Shoot
apiVersion: core.gardener.cloud/v1beta1
metadata:
  name: preset
  namespace: default
  labels:
    oidc: enabled
spec:
  kubernetes:
    version: 1.20.2

Create the Shoot:

kubectl apply -f shoot.yaml

Examine the created Shoot:

kubectl get shoot preset -o yaml

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: preset
  namespace: default
  labels:
    oidc: enabled
spec:
  kubernetes:
    kubeAPIServer:
      oidcConfig:
        clientID: cluster-preset
        groupsClaim: groups-claim
        groupsPrefix: groups-prefix
        issuerURL: https://foo.bar
        requiredClaims:
          key: value
        signingAlgs:
        - RS256
        usernameClaim: username-claim
        usernamePrefix: username-prefix
    version: 1.20.2

Disable ClusterOpenIDConnectPreset

The ClusterOpenIDConnectPreset admission control is enabled by default. To disable it, use the --disable-admission-plugins flag on the gardener-apiserver.

For example:

--disable-admission-plugins=ClusterOpenIDConnectPreset

6 - Shoot Serviceaccounts

ServiceAccount Configurations for Shoot Clusters

The Shoot specification allows to configure some of the settings for the handling of ServiceAccounts:

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
spec:
  kubernetes:
    kubeAPIServer:
      serviceAccountConfig:
        issuer: foo
        acceptedIssuers:
        - foo1
        - foo2
        extendTokenExpiration: true
        maxTokenExpiration: 45d
...

Issuer and Accepted Issuers

The .spec.kubernetes.kubeAPIServer.serviceAccountConfig.{issuer,acceptedIssuers} fields are translated to the --service-account-issuer flag for the kube-apiserver. The issuer will assert its identifier in the iss claim of the issued tokens. According to the upstream specification, values need to meet the following requirements:

This value is a string or URI. If this option is not a valid URI per the OpenID Discovery 1.0 spec, the ServiceAccountIssuerDiscovery feature will remain disabled, even if the feature gate is set to true. It is highly recommended that this value comply with the OpenID spec: https://openid.net/specs/openid-connect-discovery-1_0.html. In practice, this means that service-account-issuer must be an https URL. It is also highly recommended that this URL be capable of serving OpenID discovery documents at {service-account-issuer}/.well-known/openid-configuration.

By default, Gardener uses the internal cluster domain as issuer (e.g., https://api.foo.bar.example.com). If you specify the issuer, then this default issuer will always be part of the list of accepted issuers (you don’t need to specify it yourself).

[!CAUTION] If you change from the default issuer to a custom issuer, all previously issued tokens will still be valid/accepted. However, if you change from a custom issuer A to another issuer B (custom or default), then you have to add A to the acceptedIssuers so that previously issued tokens are not invalidated. Otherwise, the control plane components as well as system components and your workload pods might fail. You can remove A from the acceptedIssuers when all currently active tokens have been issued solely by B. This can be ensured by using projected token volumes with a short validity, or by rolling out all pods. Additionally, all ServiceAccount token secrets should be recreated. Apart from this, you should wait for at least 12h to make sure the control plane and system components have received a new token from Gardener.

Token Expirations

The .spec.kubernetes.kubeAPIServer.serviceAccountConfig.extendTokenExpiration configures the --service-account-extend-token-expiration flag of the kube-apiserver. It is enabled by default and has the following specification:

Turns on projected service account expiration extension during token generation, which helps safe transition from legacy token to bound service account token feature. If this flag is enabled, admission injected tokens would be extended up to 1 year to prevent unexpected failure during transition, ignoring value of service-account-max-token-expiration.

The .spec.kubernetes.kubeAPIServer.serviceAccountConfig.maxTokenExpiration configures the --service-account-max-token-expiration flag of the kube-apiserver. It has the following specification:

The maximum validity duration of a token created by the service account token issuer. If an otherwise valid TokenRequest with a validity duration larger than this value is requested, a token will be issued with a validity duration of this value.

[!NOTE] The value for this field must be in the [30d,90d] range. The background for this limitation is that all Gardener components rely on the TokenRequest API and the Kubernetes service account token projection feature with short-lived, auto-rotating tokens. Any values lower than 30d risk impacting the SLO for shoot clusters, and any values above 90d violate security best practices with respect to maximum validity of credentials before they must be rotated. Given that the field just specifies the upper bound, end-users can still use lower values for their individual workload by specifying the .spec.volumes[].projected.sources[].serviceAccountToken.expirationSeconds in the PodSpecs.

Managed Service Account Issuer

Gardener also provides a way to manage the service account issuer of a shoot cluster as well as serving its OIDC discovery documents from a centrally managed server called Gardener Discovery Server. This ability removes the need for changing the .spec.kubernetes.kubeAPIServer.serviceAccountConfig.issuer and exposing it separately.

Prerequisites

[!NOTE] The following prerequisites are responsibility of the Gardener Administrators and are not something that end users can configure by themselves. If uncertain that these requirements are met, please contact your Gardener Administrator.

Prerequisites:

• The Garden Cluster should have the Gardener Discovery Server deployed and configured. The easiest way to handle this is by using the gardener-operator.
• The ShootManagedIssuer feature gate should be enabled.

Enablement

If the prerequisites are met then the feature can be enabled for a shoot cluster by annotating it with authentication.gardener.cloud/issuer=managed. Mind that once enabled, this feature cannot be disabled. After the shoot is reconciled, you can retrieve the new shoot service account issuer value from the shoot’s status. A sample query that will retrieve the managed issuer looks like this:

kubectl -n my-project get shoot my-shoot -o jsonpath='{.status.advertisedAddresses[?(@.name=="service-account-issuer")].url}'

Once retrieved, the shoot’s OIDC discovery documents can be explored by querying the /.well-known/openid-configuration endpoint of the issuer.

Mind that this annotation is incompatible with the .spec.kubernetes.kubeAPIServer.serviceAccountConfig.issuer field, so if you want to enable it then the issuer field should not be set in the shoot specification.

[!CAUTION] If you change from the default issuer to a managed issuer, all previously issued tokens will still be valid/accepted. However, if you change from a custom issuer A to a managed issuer, then you have to add A to the .spec.kubernetes.kubeAPIServer.serviceAccountConfig.acceptedIssuers so that previously issued tokens are not invalidated. Otherwise, the control plane components as well as system components and your workload pods might fail. You can remove A from the acceptedIssuers when all currently active tokens have been issued solely by the managed issuer. This can be ensured by using projected token volumes with a short validity, or by rolling out all pods. Additionally, all ServiceAccount token secrets should be recreated. Apart from this, you should wait for at least 12h to make sure the control plane and system components have received a new token from Gardener.