그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그
4 minute read
Gardener Network Extension
Gardener is an open-source project that provides a nested user model. Basically, there are two types of services provided by Gardener to its users:
- Managed: end-users only request a Kubernetes cluster (Clusters-as-a-Service)
- Hosted: operators utilize Gardener to provide their own managed version of Kubernetes (Cluster-Provisioner-as-a-service)
Whether a user is an operator or an end-user, it makes sense to provide choice. For example, for an end-user it might make sense to choose a network-plugin that would support enforcing network policies (some plugins does not come with network-policy support by default). For operators however, choice only matters for delegation purposes i.e., when providing an own managed-service, it becomes important to also provide choice over which network-plugins to use.
Furthermore, Gardener provisions clusters on different cloud-providers with different networking requirements. For example, Azure does not support Calico Networking. This leads to the introduction of manual exceptions in static add-on charts which is error prone and can lead to failures during upgrades.
Finally, every provider is different, and thus the network always needs to adapt to the infrastructure needs to provide better performance. Consistency does not necessarily lie in the implementation but in the interface.
Prior to the
Network Extensibility concept, Gardener followed a mono network-plugin support model (i.e., Calico). Although this seemed to be the easier approach, it did not completely reflect the real use-case.
The goal of the Gardener Network Extensions is to support different network plugins, therefore, the specification for the network resource won’t be fixed and will be customized based on the underlying network plugin.
To do so, a
ProviderConfig field in the spec will be provided where each plugin will define. Below is an example for how to deploy Calico as the cluster network plugin.
The Network Extensions Resource
Here is what a typical
Network resource would look-like:
--- apiVersion: extensions.gardener.cloud/v1alpha1 kind: Network metadata: name: my-network spec: ipFamilies: - IPv4 podCIDR: 22.214.171.124/16 serviceCIDR: 126.96.36.199/13 type: calico providerConfig: apiVersion: calico.networking.extensions.gardener.cloud/v1alpha1 kind: NetworkConfig backend: bird ipam: cidr: usePodCIDR type: host-local
The above resources is divided into two parts (more information can be found at Using the Networking Calico Extension):
- global configuration (e.g., podCIDR, serviceCIDR, and type)
- provider specific config (e.g., for calico we can choose to configure a
Note: Certain cloud-provider extensions might have webhooks that would modify the network-resource to fit into their network specific context. As previously mentioned, Azure does not support IPIP, as a result, the Azure provider extension implements a webhook to mutate the backend and set it to
Supporting a New Network Extension Provider
To add support for another networking provider (e.g., weave, Cilium, Flannel) a network extension controller needs to be implemented which would optionally have its own custom configuration specified in the
spec.providerConfig in the
Network resource. For example, if support for a network plugin named
gardenet is required, the following
Network resource would be created:
--- apiVersion: extensions.gardener.cloud/v1alpha1 kind: Network metadata: name: my-network spec: ipFamilies: - IPv4 podCIDR: 188.8.131.52/16 serviceCIDR: 184.108.40.206/13 type: gardenet providerConfig: apiVersion: gardenet.networking.extensions.gardener.cloud/v1alpha1 kind: NetworkConfig gardenetCustomConfigField: <value> ipam: cidr: usePodCIDR type: host-local
Once applied, the presumably implemented
Gardenet extension controller would pick the configuration up, parse the
providerConfig, and create the necessary resources in the shoot.
For additional reference, please have a look at the networking-calico provider extension, which provides more information on how to configure the necessary charts, as well as the actuators required to reconcile networking inside the
Shoot cluster to the desired state.
kube-proxy-less Service Routing
Some networking extensions support service routing without the
kube-proxy component. This is why Gardener supports disabling of
kube-proxy for service routing by setting
false in the
Shoot specification. The implicit contract of the flag is:
kube-proxy is disabled, then the networking extension is responsible for the service routing.
The networking extensions need to handle this twofold:
- During the reconciliation of the networking resources, the extension needs to check whether
kube-proxytakes care of the service routing or the networking extension itself should handle it. In case the networking extension should be responsible according to
.spec.kubernetes.kubeproxy.enabled(but is unable to perform the service routing), it should raise an error during the reconciliation. If the networking extension should handle the service routing, it may reconfigure itself accordingly.
- (Optional) In case the networking extension does not support taking over the service routing (in some scenarios), it is recommended to also provide a validating admission webhook to reject corresponding changes early on. The validation may take the current operating mode of the networking extension into consideration.