그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그
5 minute read
The Gardener API server provides a cluster-scoped
This resource is used to allow exposing the control plane of a Shoot cluster in various network environments like restricted corporate networks, DMZ etc.
ExposureClass resource is based on the concept for the
RuntimeClass resource in Kubernetes.
RuntimeClass abstracts the installation of a certain container runtime (e.g. gVisor, Kata Containers) on all nodes or a subset of the nodes in a Kubernetes cluster.
In contrast, an
ExposureClass abstracts the ability to expose a Shoot clusters control plane in certain network environments (e.g. corporate networks, DMZ, internet) on all Seeds or a subset of the Seeds.
apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: gvisor handler: gvisorconfig # scheduling: # nodeSelector: # env: prod --- kind: ExposureClass metadata: name: internet handler: internet-config # scheduling: # seedSelector: # matchLabels: # network/env: internet
ExposureClasses also define a
.handler field reflecting the name reference for the corresponding CRI configuration of the
RuntimeClass and the control plane exposure configuration for the
The CRI handler for
RuntimeClasses is usually installed by an administrator (e.g. via a
DaemonSet which installs the corresponding container runtime on the nodes).
The control plane exposure configuration for
ExposureClasses will be also provided by an administrator.
This exposure configuration is part of the Gardenlet configuration as this component is responsible to configure the control plane accordingly.
RuntimeClass also supports the selection of a node subset (which have the respective controller runtime binaries installed) for pod scheduling via its
ExposureClass also supports the selection of a subset of available Seed clusters whose Gardenlet is capable of applying the exposure configuration for the Shoot control plane accordingly via its
Usage by a
Shoot can reference an
ExposureClass via the
⚠️ When creating a
Shoot resource, the Gardener scheduler will try to assign the
Shoot to a
Seed which will host its control plane.
The scheduling behaviour can be influenced via the
.spec.tolerations fields in the
ExposureClasses can contain also scheduling instructions.
Shoot is referencing an
ExposureClass then the scheduling instructions of both will be merged into the
Those unions of scheduling instructions might lead to a selection of a
Seed which is not able to deal with the
handler of the
ExposureClass and the
Shoot creation might end up in an error.
In such case, the
Shoot scheduling instructions should be revisited to check that they are not interfere with the ones from the
If this is not feasible then the combination with the
ExposureClass is might not possible and you need to contact your Gardener administrator.
Example: Shoot and ExposureClass scheduling instructions merge flow
- Assuming there is the following
Shootwhich is referencing the
apiVersion: core.gardener.cloud/v1beta1 kind: Shoot metadata: name: abc namespace: garden-dev spec: exposureClassName: abc seedSelectors: matchLabels: env: prod --- apiVersion: core.gardener.cloud/v1alpha1 kind: ExposureClass metadata: name: abc handler: abc scheduling: seedSelector: matchLabels: network: internal
seedSelectorswould be merged into the
Shoot. The result would be the following:
apiVersion: core.gardener.cloud/v1alpha1 kind: Shoot metadata: name: abc namespace: garden-dev spec: exposureClassName: abc seedSelectors: matchLabels: env: prod network: internal
- Now the Gardener Scheduler would try to find a
Seedwith those labels.
- If there are no Seeds with matching labels for the seed selector then the
Shootwill be unschedulable
- If there are Seeds with matching labels for the seed selector then the Shoot will be assigned to the best candidate after the scheduling strategy is applied, see here
- If the
Seedis not able to serve the
abcthen the Shoot will end up in error state
- If the
Seedis able to serve the
Shootwill be created
- If the
The Gardenlet is responsible to realize the control plane exposure strategy defined in the referenced
ExposureClass of a
GardenletConfiguration can contain an
.exposureClassHandlers list with the respective configuration.
Example of the
exposureClassHandlers: - name: internet-config loadBalancerService: annotations: loadbalancer/network: internet - name: internal-config loadBalancerService: annotations: loadbalancer/network: internal sni: ingress: namespace: ingress-internal labels: network: internal
Each Gardenlet can define how the handler of a certain
ExposureClass needs to be implemented for the Seed(s) where it is responsible for.
.name is the name of the handler config and it must match to the
.handler in the
All control planes on a
Seed are exposed via a load balancer.
Either a dedicated one or a central shared one.
The load balancer service needs to be configured in a way that it is reachable from the target network environment.
Therefore, the configuration of load balancer service need to be specified which can be done via the
The common way to influence load balancer service behaviour is via annotations where the respective cloud-controller-manager will react on and configure the infrastructure load balancer accordingly.
In case the Gardenlet runs with activated
APIServerSNI feature flag (default), the control planes on a
Seed will be exposed via a central load balancer and with Envoy via TLS SNI passthrough proxy.
In this case, the Gardenlet will install a dedicated ingress gateway (Envoy + load balancer + respective configuration) for each handler on the
The configuration of the ingress gateways can be controlled via the
.sni section in the same way like for the default ingress gateways.