4 minute read
Deploying Autonomous Shoot Clusters Locally
Caution
The
gardenadm
tool is currently under development and considered highly experimental. Do not use it in production environments. Read more about it in GEP-28.
This document walks you through deploying Autonomous Shoot Clusters using gardenadm
on your local machine.
This setup can be used for trying out and developing gardenadm
locally without additional infrastructure.
The setup is also used for running e2e tests for gardenadm
in CI (Prow).
If you encounter difficulties, please open an issue so that we can make this process easier.
Overview
gardenadm
is a command line tool for bootstrapping Kubernetes clusters called “Autonomous Shoot Clusters”. Read the gardenadm
documentation for more details on its concepts.
In this guide, we will start a KinD cluster which hosts pods serving as machines for the autonomous shoot cluster – just as for shoot clusters of provider-local.
The setup supports both the high-touch and medium-touch scenario of gardenadm
.
Based on Skaffold, the container images for all required components will be built and deployed into the cluster.
This also includes the gardenadm
CLI, which is installed on the machine pods by pulling the container image and extracting the binary.
Prerequisites
- Make sure that you have followed the Local Setup guide up until the Get the sources step.
- Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least
8
CPUs and8Gi
memory; see here how to configure the resources for Docker for Mac).Additionally, please configure at least
120Gi
of disk size for the Docker daemon.
Tip
You can clean up unused data with
docker system df
anddocker system prune -a
.
Setting Up the KinD Cluster
make kind-up
Please see this documentation section for more details.
All following steps assume that you are using the kubeconfig for this KinD cluster:
export KUBECONFIG=$PWD/example/gardener-local/kind/local/kubeconfig
High-Touch Scenario
Use the following command to prepare the gardenadm
high-touch scenario:
make gardenadm-high-touch-up
This will first build the needed images, deploy 2 machine pods using the gardener-extension-provider-local-node
image, install the gardenadm
binary on both of them, and copy the needed manifests to the /gardenadm/resources
directory.
Afterward, you can use kubectl exec
to execute gardenadm
commands on the machines.
Let’s start with exec’ing into the machine-0
pod:
$ kubectl -n gardenadm-high-touch exec -it machine-0 -- bash
root@machine-0:/# gardenadm -h
gardenadm bootstraps and manages autonomous shoot clusters in the Gardener project.
...
root@machine-0:/# cat /gardenadm/resources/manifests.yaml
apiVersion: core.gardener.cloud/v1beta1
kind: CloudProfile
metadata:
name: local
...
Bootstrapping a Single-Node Control Plane
Use gardenadm init
to bootstrap the first control plane node using the provided manifests:
root@machine-0:/# gardenadm init -d /gardenadm/resources
...
Your Shoot cluster control-plane has initialized successfully!
...
Connecting to the Autonomous Shoot Cluster
The machine pod’s shell environment is configured for easily connecting to the autonomous shoot cluster.
Just execute kubectl
within a bash
shell in the machine pod:
$ kubectl -n gardenadm-high-touch exec -it machine-0 -- bash
root@machine-0:/# kubectl get node
NAME STATUS ROLES AGE VERSION
machine-0 Ready <none> 4m11s v1.32.0
You can also copy the kubeconfig to your local machine and use a port-forward to connect to the cluster’s API server:
$ kubectl -n gardenadm-high-touch exec -it machine-0 -- cat /etc/kubernetes/admin.conf | sed 's/api.root.garden.internal.gardenadm.local/localhost:6443/' > /tmp/shoot--garden--root.conf
$ kubectl -n gardenadm-high-touch port-forward pod/machine-0 6443:443
# in a new terminal
$ export KUBECONFIG=/tmp/shoot--garden--root.conf
$ kubectl get no
NAME STATUS ROLES AGE VERSION
machine-0 Ready <none> 10m v1.32.0
Joining a Worker Node
If you would like to join a worker node to the cluster, generate a bootstrap token and the corresponding gardenadm join
command on machine-0
(the control plane node).
Then exec into the machine-1
pod to run the command:
root@machine-0:/# gardenadm token create --print-join-command
# now copy the output, terminate the exec session and start a new one for machine-1
$ kubectl -n gardenadm-high-touch exec -it machine-1 -- bash
# paste the copied 'gardenadm join' command here and execute it
root@machine-1:/# gardenadm join ...
...
Your node has successfully been instructed to join the cluster as a worker!
...
Using the kubeconfig as described in this section, you should now be able to see the new node in the cluster:
$ kubectl get no
NAME STATUS ROLES AGE VERSION
machine-0 Ready <none> 10m v1.32.0
machine-1 Ready <none> 37s v1.32.0
Medium-Touch Scenario
Use the following command to prepare the gardenadm
medium-touch scenario:
make gardenadm-medium-touch-up
This will first build the needed images and then render the needed manifests for gardenadm bootstrap
to the ./example/gardenadm-local/medium-touch
directory.
Afterward, you can use go run
to execute gardenadm
commands on your machine:
$ go run ./cmd/gardenadm -h
gardenadm bootstraps and manages autonomous shoot clusters in the Gardener project.
...
Running E2E Tests for gardenadm
Based on the described setup, you can execute the e2e test suite for gardenadm
:
make gardenadm-high-touch-up gardenadm-medium-touch-up
make test-e2e-local-gardenadm
You can also selectively run the e2e tests for one of the scenarios:
make gardenadm-high-touch-up
./hack/test-e2e-local.sh gardenadm --label-filter="high-touch" ./test/e2e/gardenadm/...
Tear Down the KinD Cluster
make kind-down