Egress filtering

Gardener extension controller for egress filtering for shoot clusters

Gardener Extension for Networking Filter

Project Gardener implements the automated management and operation of Kubernetes clusters as a service. Its main principle is to leverage Kubernetes concepts for all of its tasks.

Recently, most of the vendor specific logic has been developed in-tree. However, the project has grown to a size where it is very hard to extend, maintain, and test. With GEP-1 we have proposed how the architecture can be changed in a way to support external controllers that contain their very own vendor specifics. This way, we can keep Gardener core clean and independent.

This controller implements Gardener’s extension contract for the shoot-networking-filter extension.

An example for a ControllerRegistration resource that can be used to register this controller to Gardener can be found here.

Please find more information regarding the extensibility concepts and a detailed proposal here.

Extension Resources

Currently there is nothing to specify in the extension spec.

Example extension resource:

apiVersion: extensions.gardener.cloud/v1alpha1
kind: Extension
metadata:
  name: extension-shoot-networking-filter
  namespace: shoot--project--abc
spec:

When an extension resource is reconciled, the extension controller will create a daemonset egress-filter-applier on the shoot containing a Dockerfile container.

Please note, this extension controller relies on the Gardener-Resource-Manager to deploy k8s resources to seed and shoot clusters.

How to start using or developing this extension controller locally

You can run the controller locally on your machine by executing make start.

We are using Go modules for Golang package dependency management and Ginkgo/Gomega for testing.

Feedback and Support

Feedback and contributions are always welcome. Please report bugs or suggestions as GitHub issues or join our Slack channel #gardener (please invite yourself to the Kubernetes workspace here).

Learn more!

Please find further resources about out project here:

1 - Deployment

Gardener Networking Policy Filter for Shoots

Introduction

Gardener allows shoot clusters to filter egress traffic on node level. To support this the Gardener must be installed with the shoot-networking-filter extension.

Configuration

To generally enable the networking filter for shoot objects the shoot-networking-filter extension must be registered by providing an appropriate extension registration in the garden cluster.

Here it is possible to decide whether the extension should be always available for all shoots or whether the extension must be separately enabled per shoot.

If the extension should be used for all shoots the globallyEnabled flag should be set to true.

apiVersion: core.gardener.cloud/v1beta1
kind: ControllerRegistration
...
spec:
  resources:
    - kind: Extension
      type: shoot-networking-filter
      globallyEnabled: true

ControllerRegistration

An example of a ControllerRegistration for the shoot-networking-filter can be found at controller-registration.yaml.

The ControllerRegistration contains a Helm chart which eventually deploys the shoot-networking-filter to seed clusters. It offers some configuration options, mainly to set up a static filter list or provide the configuration for downloading the filter list from a service endpoint.

apiVersion: core.gardener.cloud/v1beta1
kind: ControllerDeployment
...
  values:
    egressFilter:
      blackholingEnabled: true

      filterListProviderType: static
      staticFilterList:
        - network: 1.2.3.4/31
          policy: BLOCK_ACCESS
        - network: 5.6.7.8/32
          policy: BLOCK_ACCESS
        - network: ::2/128
          policy: BLOCK_ACCESS

      #filterListProviderType: download
      #downloaderConfig:
      #  endpoint: https://my.filter.list.server/lists/policy
      #  oauth2Endpoint: https://my.auth.server/oauth2/token
      #  refreshPeriod: 1h

      ## if the downloader needs an OAuth2 access token, client credentials can be provided with oauth2Secret
      #oauth2Secret:
      # clientID: 1-2-3-4
      # clientSecret: secret!!
      ## either clientSecret of client certificate is required
      # client.crt.pem: |
      #   -----BEGIN CERTIFICATE-----
      #   ...
      #   -----END CERTIFICATE-----
      # client.key.pem: |
      #   -----BEGIN PRIVATE KEY-----
      #   ...
      #   -----END PRIVATE KEY-----

Enablement for a Shoot

If the shoot networking filter is not globally enabled by default (depends on the extension registration on the garden cluster), it can be enabled per shoot. To enable the service for a shoot, the shoot manifest must explicitly add the shoot-networking-filter extension.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
...
spec:
  extensions:
    - type: shoot-networking-filter
...

If the shoot networking filter is globally enabled by default, it can be disabled per shoot. To disable the service for a shoot, the shoot manifest must explicitly state it.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
...
spec:
  extensions:
    - type: shoot-networking-filter
      disabled: true
...

2 - Shoot Networking Filter

Register Shoot Networking Filter Extension in Shoot Clusters

Introduction

Within a shoot cluster, it is possible to enable the networking filter. It is necessary that the Gardener installation your shoot cluster runs in is equipped with a shoot-networking-filter extension. Please ask your Gardener operator if the extension is available in your environment.

Shoot Feature Gate

In most of the Gardener setups the shoot-networking-filter extension is not enabled globally and thus must be configured per shoot cluster. Please adapt the shoot specification by the configuration shown below to activate the extension individually.

kind: Shoot
...
spec:
  extensions:
    - type: shoot-networking-filter
...

Opt-out

If the shoot networking filter is globally enabled by default, it can be disabled per shoot. To disable the service for a shoot, the shoot manifest must explicitly state it.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
...
spec:
  extensions:
    - type: shoot-networking-filter
      disabled: true
...

Ingress Filtering

By default, the networking filter only filters egress traffic. However, if you enable blackholing, incoming traffic will also be blocked. You can enable blackholing on a per-shoot basis.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
...
spec:
  extensions:
    - type: shoot-networking-filter
      providerConfig:
        egressFilter:
          blackholingEnabled: true
...

Ingress traffic can only be blocked by blackhole routing, if the source IP address is preserved. On Azure, GCP and AliCloud this works by default. The default on AWS is a classic load balancer that replaces the source IP by it’s own IP address. Here, a network load balancer has to be configured adding the annotation service.beta.kubernetes.io/aws-load-balancer-type: "nlb" to the service. On OpenStack, load balancers don’t preserve the source address.

Please note that if you disable blackholing in an existing shoot, the associated blackhole routes will not be removed automatically. To remove these routes, you can either replace the affected nodes or delete the routes manually.

Custom IP

It is possible to add custom IP addresses to the network filter. This can be useful for testing purposes.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
...
spec:
  extensions:
    - type: shoot-networking-filter
      providerConfig:
        egressFilter:
          staticFilterList:
          - network: 1.2.3.4/31
            policy: BLOCK_ACCESS
          - network: 5.6.7.8/32
            policy: BLOCK_ACCESS
          - network: ::2/128
            policy: BLOCK_ACCESS
...