Security and Compliance

1 - Kubernetes Cluster Hardening Procedure

Overview

The Gardener team takes security seriously, which is why we mandate the Security Technical Implementation Guide (STIG) for Kubernetes as published by the Defense Information Systems Agency (DISA) here. We offer Gardener adopters the opportunity to show compliance with DISA Kubernetes STIG via the compliance checker tool diki. The latest release in machine readable format can be found in the STIGs Document Library by searching for Kubernetes.

Kubernetes Clusters Security Requirements

DISA Kubernetes STIG version 1 release 11 contains 91 rules overall. Only the following rules, however, apply to you. Some of them are secure-by-default, so your responsibility is to make sure that they are not changed. For your convenience, the requirements are grouped logically and per role:

Rules Relevant for Cluster Admins

Control Plane Configuration

Audit Configuration

End User Workload

Rules Relevant for Service Providers

2 - Run DISA K8s STIGs Ruleset

Show DISA K8s STIG Compliance for a Gardener Shoot Cluster

Introduction

This part covers the topic of showing compliance with the DISA K8s STIG for a Gardener shoot cluster. The guide features two providers - managedk8s and garden, both of which implement rules from the DISA K8s STIG ruleset.

The managedk8s provider assumes that the user running the ruleset does not have access to the environment (the seed in this particular case), in which the control plane components reside.

The garden provider is used for accessing theGarden cluster, in which the Shoot resource can be found.

Important

Since the two providers that we are going to use in this guide do not leverage access to the Shoot cluster controlplane, they only implement checks that concern configurations that cluster owners can change/modify by themselves. Compliance for configurations that cannot be influenced by cluster owners shall be ensured by the team that operates the concrete Gardener installation.

Prerequisites

Make sure you have diki installed and have a running Gardener shoot cluster.

We will be using the sample DISA K8s STIG for Shoots configuration file for this run.

Configuration

Configure the managedk8s provider

Set the following arguments:

• providers[id=="managedk8s"].args.kubeconfigPath pointing to a shoot admin kubeconfig.
• (optional) providers[id=="managedk8s"].metadata.shootName should be set to the name of the shoot cluster. The metadata field contains custom metadata from the user that will be present in the generated report.

- id: managedk8s
  name: "Managed Kubernetes"
  metadata: # custom user metadata
    # shootName: <shoot-name>
    # foo: bar
  args:
    kubeconfigPath: <shoot-kubeconfig-path>  # path to shoot admin kubeconfig

In case you need instructions on how to generate such a kubeconfig, please read Accessing Shoot Clusters.

Configure the garden provider

Set the following arguments:

• providers[id=="garden"].args.kubeconfigPath pointing to the Garden cluster kubeconfig.
• providers[id=="garden"].rulesets.args.projectNamespace should be set to the namespace in which the shoot cluster is created.
• providers[id=="garden"].rulesets.args.shootName should be set to the name of the shoot cluster.

- id: garden
  name: "Garden"
  metadata: # custom user metadata
    # foo: bar
  args:
    kubeconfigPath: <garden-kubeconfig-path>  # path to garden cluster kubeconfig
  rulesets:
  - id: security-hardened-shoot-cluster
    name: Security Hardened Shoot Cluster
    version: v0.2.1
    args:
      projectNamespace: garden-<project-name> # name of project namespace containing the shoot resource to be tested
      shootName: <shoot-name>                 # name of shoot resource to be tested

Additional configurations

Additional metadata such as the shoot’s name can also be included in the providers[id=="managedk8s|garden"].metadata section. The metadata section can be used to add additional context to different diki runs.

The provided configuration contain the recommended rule options for running the both providers, but you can modify rule options parameters according to requirements. All available options can be found in:

• managedk8s example configuration.
• garden example configuration.

Running the DISA K8s STIGs Ruleset

To run diki against a Gardener shoot cluster, run the following command:

diki run \
    --config=./example/guides/disa-k8s-stig-shoot.yaml \
    --all \
    --output=disa-k8s-stigs-report.json

Generating a Report

We can use the file generated in the previous step to create an html report by using the following command:

diki report generate \
    --output=disa-k8s-stigs-report.html \
    disa-k8s-stigs-report.json

3 - Gardener Compliance Report

Overview

Gardener aims to comply with public security standards and guidelines, such as the Security Technical Implementation Guide (STIG) for Kubernetes from Defense Information Systems Agency (DISA). The DISA Kubernetes STIG is a set of rules that provide recommendations for secure deployment and operation of Kubernetes. It covers various aspects of Kubernetes security, including the configurations of the Kubernetes API server and other components, cluster management, certificate management, handling of updates and patches.

While Gardener aims to follow this guideline, we also recognize that not all of the rules may be directly applicable or optimal for Gardener specific environment. Therefore, some of the requirements are adjusted. Rules that are not applicable to Gardener are skipped given an appropriate justification.

For every release, we check that Gardener is able of creating security hardened shoot clusters, reconfirming that the configurations which are not secure by default (as per Gardener Kubernetes Cluster Hardening Procedure) are still possible and work as expected.

In order to automate and ease this process, Gardener uses a tool called diki.

Security Hardened Shoot Configurations

The following security hardened shoot configurations were used in order to generate the compliance report.

kind: Shoot
apiVersion: core.gardener.cloud/v1beta1
metadata:
  name: aws
spec:
  cloudProfileName: aws
  kubernetes:
    kubeAPIServer:
      admissionPlugins:
        - name: PodSecurity
          config:
            apiVersion: pod-security.admission.config.k8s.io/v1beta1
            kind: PodSecurityConfiguration
            defaults:
              enforce: baseline
              audit: baseline
              warn: baseline
          disabled: false
      auditConfig:
        auditPolicy:
          configMapRef:
            name: audit-policy
    version: "1.31"
    enableStaticTokenKubeconfig: false
  networking:
    type: calico
    pods: 100.64.0.0/12
    nodes: 10.180.0.0/16
    services: 100.104.0.0/13
    ipFamilies:
      - IPv4
  provider:
    type: aws
    controlPlaneConfig:
      apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
      kind: ControlPlaneConfig
    infrastructureConfig:
      apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
      kind: InfrastructureConfig
      networks:
        vpc:
          cidr: 10.180.0.0/16
        zones:
          - internal: 10.180.48.0/20
            name: eu-west-1c
            public: 10.180.32.0/20
            workers: 10.180.0.0/19
    workers:
      - cri:
          name: containerd
        name: worker-kkfk1
        machine:
          type: m5.large
          image:
            name: gardenlinux
          architecture: amd64
        maximum: 2
        minimum: 2
        maxSurge: 1
        maxUnavailable: 0
        volume:
          type: gp3
          size: 50Gi
        zones:
          - eu-west-1c
    workersSettings:
      sshAccess:
        enabled: false
  purpose: evaluation
  region: eu-west-1
  secretBindingName: secretBindingName

kind: Shoot
apiVersion: core.gardener.cloud/v1beta1
metadata:
  name: azure
spec:
  cloudProfileName: az
  kubernetes:
    kubeAPIServer:
      admissionPlugins:
        - name: PodSecurity
          config:
            apiVersion: pod-security.admission.config.k8s.io/v1beta1
            kind: PodSecurityConfiguration
            defaults:
              enforce: baseline
              audit: baseline
              warn: baseline
          disabled: false
      auditConfig:
        auditPolicy:
          configMapRef:
            name: audit-policy
    version: "1.31"
    enableStaticTokenKubeconfig: false
  networking:
    type: calico
    pods: 100.64.0.0/12
    nodes: 10.180.0.0/16
    services: 100.104.0.0/13
    ipFamilies:
      - IPv4
  provider:
    type: azure
    controlPlaneConfig:
      apiVersion: azure.provider.extensions.gardener.cloud/v1alpha1
      kind: ControlPlaneConfig
    infrastructureConfig:
      apiVersion: azure.provider.extensions.gardener.cloud/v1alpha1
      kind: InfrastructureConfig
      networks:
        vnet:
          cidr: 10.180.0.0/16
        workers: 10.180.0.0/16
      zoned: true
    workers:
      - cri:
          name: containerd
        name: worker-g7p4p
        machine:
          type: Standard_A4_v2
          image:
            name: gardenlinux
          architecture: amd64
        maximum: 2
        minimum: 2
        maxSurge: 1
        maxUnavailable: 0
        volume:
          type: StandardSSD_LRS
          size: 50Gi
        zones:
          - '3'
    workersSettings:
      sshAccess:
        enabled: false
  purpose: evaluation
  region: westeurope
  secretBindingName: secretBindingName

kind: Shoot
apiVersion: core.gardener.cloud/v1beta1
metadata:
  name: gcp
spec:
  cloudProfileName: gcp
  kubernetes:
    kubeAPIServer:
      admissionPlugins:
        - name: PodSecurity
          config:
            apiVersion: pod-security.admission.config.k8s.io/v1beta1
            kind: PodSecurityConfiguration
            defaults:
              enforce: baseline
              audit: baseline
              warn: baseline
          disabled: false
      auditConfig:
        auditPolicy:
          configMapRef:
            name: audit-policy
    version: "1.31"
    enableStaticTokenKubeconfig: false
  networking:
    type: calico
    pods: 100.64.0.0/12
    nodes: 10.180.0.0/16
    services: 100.104.0.0/13
    ipFamilies:
      - IPv4
  provider:
    type: gcp
    controlPlaneConfig:
      apiVersion: gcp.provider.extensions.gardener.cloud/v1alpha1
      kind: ControlPlaneConfig
      zone: europe-west1-b
    infrastructureConfig:
      apiVersion: gcp.provider.extensions.gardener.cloud/v1alpha1
      kind: InfrastructureConfig
      networks:
        workers: 10.180.0.0/16
    workers:
      - cri:
          name: containerd
        name: worker-bex82
        machine:
          type: n1-standard-2
          image:
            name: gardenlinux
          architecture: amd64
        maximum: 2
        minimum: 2
        maxSurge: 1
        maxUnavailable: 0
        volume:
          type: pd-balanced
          size: 50Gi
        zones:
          - europe-west1-b
    workersSettings:
      sshAccess:
        enabled: false
  purpose: evaluation
  region: europe-west1
  secretBindingName: secretBindingName

kind: Shoot
apiVersion: core.gardener.cloud/v1beta1
metadata:
  name: openstack
spec:
  cloudProfileName: converged-cloud-cp
  kubernetes:
    kubeAPIServer:
      admissionPlugins:
        - name: PodSecurity
          config:
            apiVersion: pod-security.admission.config.k8s.io/v1beta1
            kind: PodSecurityConfiguration
            defaults:
              enforce: baseline
              audit: baseline
              warn: baseline
          disabled: false
      auditConfig:
        auditPolicy:
          configMapRef:
            name: audit-policy
    version: "1.31"
    enableStaticTokenKubeconfig: false
  networking:
    type: calico
    pods: 100.64.0.0/12
    nodes: 10.180.0.0/16
    services: 100.104.0.0/13
    ipFamilies:
      - IPv4
  provider:
    type: openstack
    controlPlaneConfig:
      apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1
      kind: ControlPlaneConfig
      loadBalancerProvider: f5
    infrastructureConfig:
      apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1
      kind: InfrastructureConfig
      networks:
        workers: 10.180.0.0/16
      floatingPoolName: FloatingIP-external-cp
    workers:
      - cri:
          name: containerd
        name: worker-dqty2
        machine:
          type: g_c2_m4
          image:
            name: gardenlinux
          architecture: amd64
        maximum: 2
        minimum: 2
        maxSurge: 1
        maxUnavailable: 0
        zones:
          - eu-de-1b
    workersSettings:
      sshAccess:
        enabled: false
  purpose: evaluation
  region: eu-de-1
  secretBindingName: secretBindingName

Diki Configuration

The following diki configuration was used in order to test each of the shoot clusters described above. Mind that the rules regarding audit logging are skipped because organizations have different requirements and Gardener can integrate with different audit logging solutions.

metadata: ...
providers:
- id: gardener
  name: Gardener
  metadata: ...
  args: ...
  rulesets:
  - id: disa-kubernetes-stig
    name: DISA Kubernetes Security Technical Implementation Guide
    version: v2r1
    args:
      maxRetries: 5
    ruleOptions: 
    - ruleID: "242402"
      skip:
        enabled: true
        justification: Gardener can integrate with different audit logging solutions
    - ruleID: "242403"
      skip:
        enabled: true
        justification: Gardener can integrate with different audit logging solutions
    - ruleID: "242414"
      args:
        acceptedPods:
        - podMatchLabels:
            k8s-app: node-local-dns
          namespaceMatchLabels:
            kubernetes.io/metadata.name: kube-system
          justification: "node local dns requires port 53 in order to operate properly"
          ports:
          - 53
    - ruleID: "242445"
      args:
        expectedFileOwner:
          users: ["0", "65532"]
          groups: ["0", "65532"]
    - ruleID: "242446"
      args:
        expectedFileOwner:
          users: ["0", "65532"]
          groups: ["0", "65532"]
    - ruleID: "242451"
      args:
        expectedFileOwner:
          users: ["0", "65532"]
          groups: ["0", "65532"]
    - ruleID: "242462"
      skip:
        enabled: true
        justification: Gardener can integrate with different audit logging solutions
    - ruleID: "242463"
      skip:
        enabled: true
        justification: Gardener can integrate with different audit logging solutions
    - ruleID: "242464"
      skip:
        enabled: true
        justification: Gardener can integrate with different audit logging solutions
    - ruleID: "245543"
      args:
        acceptedTokens:
        - user: "health-check"
          uid: "health-check"
    - ruleID: "254800"
      args:
        minPodSecurityStandardsProfile: "baseline"
output:
  minStatus: Passed

Security Compliance Report for Hardened Shoot Clusters

The report can be reviewed directly or downloaded by clicking here. <!doctype html>

4 - Credential Rotation

Keys

There are plenty of keys in Gardener. The ETCD needs one to store resources like secrets encrypted at rest. Gardener generates certificate authorities (CAs) to ensure secured communication between the various components and actors and service account tokens are signed with a dedicated key. There is also an SSH key pair to allow debugging of nodes and the observability stack has its own passwords too.

All of these keys share a common property: they are managed by Gardener. Rotating them, however, is potentially very disruptive. Hence, Gardener does not do it automatically, but offers you means to perform these tasks easily. For a single cluster, you may conveniently use the dashboard.

Where possible, the rotation happens in two phases - Preparing and Completing.

Prepare Rotation of All Credentials

The Preparing phase introduces new keys while the old ones are still valid. Users can safely exchange keys / CA bundles wherever they are used. It is possible to start the preparation by annotating the shoot resource accordingly:

kubectl -n <shoot-namespace> annotate shoot <shoot-name> gardener.cloud/operation=rotate-credentials-start

Complete Rotation of All Credentials

Afterward, the Completing phase will invalidate the old keys / CA bundles. Annotate the shoot resource accordingly:

kubectl -n <shoot-namespace> annotate shoot <shoot-name> gardener.cloud/operation=rotate-credentials-complete

Rotation Phases

At the beginning, only the old set of credentials exists. By triggering the rotation, new credentials are created in the Preparing phase and both sets are valid. Now, all clients have to update and start using the new credentials. Only afterward it is safe to trigger the Completing phase, which invalidates the old credentials.

The shoot’s status will always show the current status / phase of the rotation.

For more information, see Credentials Rotation for Shoot Clusters.

User-Provided Credentials

You grant Gardener permissions to create resources by handing over cloud provider keys. These keys are stored in a secret and referenced to a shoot via a SecretBinding. Gardener uses the keys to create the network for the cluster resources, routes, VMs, disks, and IP addresses.

When you rotate credentials, the new keys have to be stored in the same secret and the shoot needs to reconcile successfully to ensure the replication to every controller. Afterward, the old keys can be deleted safely from Gardener’s perspective.

While the reconciliation can be triggered manually, there is no need for it (if you’re not in a hurry). Each shoot reconciles once within 24h and the new keys will be picked up during the next maintenance window.

Note

It is not possible to move a shoot to a different infrastructure account (at all!).

5 - Regional Restrictions

Shared Responsibility Model

Gardener, like most cloud providers’ Kubernetes offerings, is dedicated for a global setup. And just like how most cloud providers offer means to fulfil regional restrictions, Gardener also has some means built in for this purpose. Similarly, Gardener also follows a shared responsibility model where users are obliged to use the provided Gardener means in a way which results in compliance with regional restrictions.

Regions

Gardener users need to understand that Gardener is a generic tool and has no built-in knowledge about regions as geographical or political conglomerates. For Gardener, regions are only strings. To create regional restrictions is an obligation of all Gardener users who orchestrate existing Gardener functionality to reach evidence which can be audited later on.

Support for Regional Restrictions

Gardener offers functionality to support the most important kind of regional restrictions in its global setup:

• No Restriction: All seeds in all regions can be allowed to host the control plane of all shoots.
• Restriction by Dedication: Shoots running in a region can be configured so that only dedicated seeds in dedicated regions are allowed to host the shoot’s control plane. This can be achieved by adding labels to a seed and subsequently restricting shoot control plane placement to appropriately labeled seeds by using the field spec.seedSelector (example).
• Restriction by Tainting: Some seeds running in some dedicated regions are not allowed to host the control plane of any shoots unless explicitly allowed. This can be achieved by tainting seeds appropriately (example) which in turn requires explicit tolerations if a shoot’s control plane should be placed on such tainted seeds (example).