Deploying Gardener Locally

This document will walk you through deploying Gardener on your local machine. If you encounter difficulties, please open an issue so that we can make this process easier.

Overview

Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as both garden and seed cluster (please refer to the architecture overview) for simplicity.

Based on Skaffold, the container images for all required components will be built and deployed into the cluster (via their Helm charts).

Alternatives

When deploying Gardener on your local machine you might face several limitations:

• Your machine doesn't have enough compute resources (see prerequisites) for hosting a second seed cluster or multiple shoot clusters.
• Testing Gardener's IPv6 features requires a Linux machine and native IPv6 connectivity to the internet, but you're on macOS or don't have IPv6 connectivity in your office environment or via your home ISP.

In these cases, you might want to check out one of the following options that run the setup described in this guide elsewhere for circumventing these limitations:

• remote local setup: deploy on a remote pod for more compute resources
• dev box on Google Cloud: deploy on a Google Cloud machine for more compute resource and/or simple IPv4/IPv6 dual-stack networking

Prerequisites

• Make sure that you have followed the Local Setup guide up until the Get the sources step.
• Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least 8 CPUs and 16Gi memory; see here how to configure the resources for Docker for Mac).

  Please note that 8 CPU / 16Gi memory might not be enough for more than two Shoot clusters, i.e., you might need to increase these values if you want to run additional Shoots. If you plan on following the optional steps to create a second seed cluster, the required resources will be more - at least 10 CPUs and 32Gi memory. Additionally, please configure at least 120Gi of disk size for the Docker daemon. Tip: You can clean up unused data with docker system df and docker system prune -a.

Setting Up the KinD Cluster (Garden and Seed)

make kind-up

If you want to set up an IPv6 KinD cluster, use make kind-up IPFAMILY=ipv6 instead.

This command sets up a new KinD cluster named gardener-local and stores the kubeconfig in the ./dev-setup/kubeconfigs/runtime/kubeconfig file.

It might be helpful to copy this file to $HOME/.kube/config, since you will need to target this KinD cluster multiple times. Alternatively, make sure to set your KUBECONFIG environment variable to ./dev-setup/kubeconfigs/runtime/kubeconfig for all future steps via export KUBECONFIG=$PWD/dev-setup/kubeconfigs/runtime/kubeconfig.

After setting up Gardener, the kubeconfig for the virtual garden cluster will be available at ./dev-setup/kubeconfigs/virtual-garden/kubeconfig.

All following steps assume that you are using the runtime cluster kubeconfig.

Additionally, this command also deploys a local container registry to the cluster, as well as a few registry mirrors, that are set up as a pull-through cache for all upstream registries Gardener uses by default. This is done to speed up image pulls across local clusters.

The local registry can now be accessed either via localhost:5001 or registry.local.gardener.cloud:5001 for pushing and pulling. The storage directories of the registries are mounted to the host machine under dev/local-registry. With this, mirrored images don't have to be pulled again after recreating the cluster.

It may also be necessary to mark the registry in Docker as an insecure registry to ensure that Docker establishes the connection via HTTP. This can be achieved by adding the registry accordingly in the /etc/docker/daemon.json file:

{ "insecure-registries":["registry.local.gardener.cloud:5001"] }

The command also deploys a default calico installation as the cluster's CNI implementation with NetworkPolicy support (the default kindnet CNI doesn't provide NetworkPolicy support). Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.

After a restart of your host, the loopback IP addresses that were created for the kind cluster could be removed. To avoid recreating the cluster, there is a script that can be used to easily bring back the addresses.

./dev-setup/infra.sh setup-loopback-devices

Depending on the cluster you are using for your local dev setup you might have to:

• Replace the value for --cluster-name with the name of the kind cluster you are using for your local dev setup.
• Add the --ip-family <ipv4|ipv6|dual> flag if you want to set up ipv6 or dual stack IPs (ipv4 is the default value).
• Add the --multi-zonal flag if you want to set up IPs for a multi-zonal cluster.

Setting Up IPv6 Single-Stack Networking (optional)

We need to configure NAT for outgoing traffic from the kind network to the internet. After executing make kind-up IPFAMILY=ipv6, execute the following command to set up the corresponding iptables rules:

ip6tables -t nat -A POSTROUTING -o $(ip route | grep '^default') -s fd00:10::/64 -j MASQUERADE

Configure Feature Gates (Optional)

You can optionally configure feature gates on different components of your local Gardener deployment. The feature gates can be set by editing respective resource manifests before setting up Gardener. They can also be configured dynamically on the running deployment by editing respective resources on the appropriate clusters, or by running relevant make targets after updating the featureGates configurations in respective files. The files, make targets and the resources to edit for configuring feature gates on respective Gardener components are detailed in the table below.

Gardener components	Resource manifests	Respective make targets	Respective resources
gardener-operator	charts/gardener/operator/values.yaml ( config.featureGates )	operator-up / operator-dev	N/A
gardener-apiserver and gardener-controller-manager	dev-setup/garden/base/garden.yaml	garden-up	garden resource on the runtime cluster
gardenlet	dev-setup/gardenlet/base/gardenlet.yaml	seed-up / seed-dev	gardenlet resource on the virtual garden cluster

Setting Up Gardener

make gardener-up

If you want to set up an IPv6 ready Gardener, use make gardener-up IPFAMILY=ipv6 instead.

This will first build the base images (which might take a bit if you do it for the first time). Afterwards, the Gardener resources will be deployed into the cluster.

Developing Gardener

make gardener-dev

This is similar to make gardener-up but additionally starts a skaffold dev loop. After the initial deployment, skaffold starts watching source files. Once it has detected changes, press any key to trigger a new build and deployment of the changed components.

Tip: you can set the SKAFFOLD_MODULE environment variable to select specific modules of the skaffold configuration (see skaffold.yaml) that skaffold should watch, build, and deploy. This significantly reduces turnaround times during development.

For example, if you want to develop changes to gardenlet:

# initial deployment of all components
make gardener-up
# start iterating on gardenlet without deploying other components
make gardener-dev SKAFFOLD_MODULE=gardenlet

Debugging Gardener

To debug individual components, you need to build the respective -debug targets first. The available debug targets are:

make seed-debug

for the gardenlet,

make operator-debug

for the gardener-operator, and

make cloud-provider-local-debug

for the local cloud provider extension.

The latter two make targets use Skaffold debugging features, which inject a Delve entrypoint into the image. The make seed-debug target does not change the entrypoint, so you must manually attach dlv after the pod is deployed. You can do so via a helper script provided in the hack directory:

./hack/attach-debugger-gardenlet.sh

To step through the code with a debugger, attach your IDE to the port forwarded to localhost on your host machine. The default port forwarded by the attach-debugger-gardenlet.sh script is 2345 (can be controlled via the -p option). For the gardener-operator and the local cloud provider extensions, the default port is 56268. If you debug multiple pods at the same time (e.g. by running make operator-debug first, followed by make cloud-provider-local-debug in a second terminal), the port of the second pod will be forwarded to 56269. This is done automatically by skaffold, which will write the port to your console. See the Skaffold debugging documentation for details on how to set up your IDE, or check the examples below (GoLand, VS Code).

Note: Resuming or stopping only a single goroutine is currently not supported (Go issues 25578, 31132), so the action causes all goroutines to be resumed or paused. See the vscode-go wiki.

This means that when a goroutine of the gardenlet, or any other gardener-core component you are debugging, is paused on a breakpoint, all other goroutines are paused as well. Consequently, when the whole gardenlet process is paused, it cannot renew its lease and cannot respond to the liveness and readiness probes.

Skaffold automatically increases the timeoutSeconds of liveness and readiness probes to 600. However, we have still encountered problems where pods were killed after a while during debugging.

Therefore, leader election, health checks, and readiness checks for the respective components are disabled when debugging.

If you have similar problems with other components, you can temporarily turn off leader election and disable liveness and readiness probes there as well.

Debugging in GoLand

1. Edit your Run/Debug Configurations.
2. Add a new Go Remote configuration.
3. Set the port to 56268 (or any increment of it when debugging multiple components).
4. Recommended: Change the behavior of On disconnect to Leave it running.

Debugging in VS Code

1. Create or edit your .vscode/launch.json configuration.
2. Add the following configuration:

{
  "name": "go remote",
  "type": "go",
  "request": "attach",
  "mode": "remote",
  "port": 56268, // or any increment of it when debugging multiple components
  "host": "127.0.0.1"
}

Since the ko builder is used in Skaffold to build the images, it's not necessary to specify the cwd and remotePath options as they match the workspace folder (ref).

Creating a Shoot Cluster

NOTE

The following steps assume that you are using the kubeconfig that points to the virtual garden cluster: export KUBECONFIG=$PWD/dev-setup/kubeconfigs/virtual-garden/kubeconfig.

You can wait for the Seed to be ready by running:

./hack/usage/wait-for.sh seed local GardenletReady SeedSystemComponentsHealthy ExtensionsReady

Alternatively, you can run kubectl get seed local and wait for the STATUS to indicate readiness:

NAME    STATUS   PROVIDER   REGION   AGE     VERSION       K8S VERSION
local   Ready    local      local    4m42s   vX.Y.Z-dev    v1.28.1

In order to create a first shoot cluster, just run:

kubectl apply -f example/provider-local/shoot.yaml

You can wait for the Shoot to be ready by running:

NAMESPACE=garden-local ./hack/usage/wait-for.sh shoot local APIServerAvailable ControlPlaneHealthy ObservabilityComponentsHealthy EveryNodeReady SystemComponentsHealthy

Alternatively, you can run kubectl -n garden-local get shoot local and wait for the LAST OPERATION to reach 100%:

NAME    CLOUDPROFILE   PROVIDER   REGION   K8S VERSION   HIBERNATION   LAST OPERATION            STATUS    AGE
local   local          local      local   1.28.1        Awake         Create Processing (43%)   healthy   94s

If you don't need any worker pools, you can create a workerless Shoot by running:

kubectl apply -f example/provider-local/shoot-workerless.yaml

(Optional): You could also execute a simple e2e test (creating and deleting a shoot) by running:

make test-e2e-local-simple

Accessing the Shoot Cluster

To access the Shoot, you can acquire a kubeconfig by using the shoots/adminkubeconfig subresource.

For convenience a helper script is provided in the hack directory. By default, the script will generate an admin kubeconfig for a Shoot named "local" in the garden-local namespace valid for one hour.

./hack/usage/generate-kubeconfig.sh > admin-kubeconf.yaml

To generate a viewer kubeconfig instead of an admin kubeconfig, use the --viewer flag:

./hack/usage/generate-kubeconfig.sh --viewer > viewer-kubeconf.yaml

NOTE

Keep in mind that using a VPN on your local machine could cause problems with the setup, and the shoot's kubeconfig could fail with connection issues. If you experience connection problems using the shoot's kubeconfig, try disabling the VPN first.

If you want to change the default namespace or shoot name, you can do so by passing different values as arguments.

./hack/usage/generate-kubeconfig.sh --namespace <namespace> --shoot-name <shootname> > admin-kubeconf.yaml

To access an Ingress resource from the Seed, use the Ingress host with port 8448 (https://<ingress-host>:8448, for example https://gu-local--local.ingress.local.seed.local.gardener.cloud:8448).

(Optional): Setting Up a Second Seed Cluster

There are cases where you would want to create a second seed cluster in your local setup. For example, if you want to test the control plane migration feature. The following steps describe how to do that.

Start by setting up the second KinD cluster:

make kind2-up

This command sets up a new KinD cluster named gardener-local2 and stores its kubeconfig in the ./dev-setup/kubeconfigs/seed2/kubeconfig file.

In order to deploy required resources in the KinD cluster that you just created, run:

make seed2-up

The following steps assume that you are using the kubeconfig that points to the virtual garden cluster: export KUBECONFIG=$PWD/dev-setup/kubeconfigs/virtual-garden/kubeconfig.

You can wait for the local2 Seed to be ready by running:

./hack/usage/wait-for.sh seed local2 GardenletReady SeedSystemComponentsHealthy ExtensionsReady

Alternatively, you can run kubectl get seed local2 and wait for the STATUS to indicate readiness:

NAME    STATUS   PROVIDER   REGION   AGE     VERSION       K8S VERSION
local2  Ready    local      local    4m42s   vX.Y.Z-dev    v1.25.1

If you want to perform control plane migration, you can follow the steps outlined in Control Plane Migration to migrate the shoot cluster to the second seed you just created.

Deleting the Shoot Cluster

./hack/usage/delete shoot local garden-local

(Optional): Tear Down the Second Seed Cluster

make kind2-down

Tear Down the Gardener Environment

make kind-down

Remote Local Setup

Just like Prow is executing the KinD-based e2e tests in a K8s pod, it is possible to interactively run this KinD based Gardener development environment, aka "local setup", in a "remote" K8s pod.

k apply -f docs/deployment/content/remote-local-setup.yaml
k exec -it remote-local-setup-0 -- sh

tmux a

Caveats

Please refer to the TMUX documentation for working effectively inside the remote-local-setup pod.

To access Plutono, Prometheus or other components in a browser, two port forwards are needed:

The port forward from the laptop to the pod:

k port-forward remote-local-setup-0 3000

The port forward in the remote-local-setup pod to the respective component:

k port-forward -n shoot--local--local deployment/plutono 3000

Related Links

• Local Provider Extension