그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그
4 minute read
Control Plane Migration
Prerequisites
The Seed
s involved in the control plane migration must have backups enabled - their .spec.backup
fields cannot be nil.
ShootState
ShootState
is an API resource which stores non-reconstructible state and data required to completely recreate a Shoot
’s control plane on a new Seed
. The ShootState
resource is created on Shoot
creation in its Project
namespace and the required state/data is persisted during Shoot
creation or reconciliation.
Shoot Control Plane Migration
Triggering the migration is done by changing the Shoot
’s .spec.seedName
to a Seed
that differs from the .status.seedName
, we call this Seed
a "Destination Seed"
. This action can only be performed by an operator (see Triggering the Migration). If the Destination Seed
does not have a backup and restore configuration, the change to spec.seedName
is rejected. Additionally, this Seed must not be set for deletion and must be healthy.
If the Shoot
has different .spec.seedName
and .status.seedName
, a process is started to prepare the Control Plane for migration:
.status.lastOperation
is changed toMigrate
.- Kubernetes API Server is stopped and the extension resources are annotated with
gardener.cloud/operation=migrate
. - Full snapshot of the ETCD is created and terminating of the Control Plane in the
Source Seed
is initiated.
If the process is successful, we update the status of the Shoot
by setting the .status.seedName
to the null value. That way, a restoration is triggered in the Destination Seed
and .status.lastOperation
is changed to Restore
. The control plane migration is completed when the Restore
operation has completed successfully.
The etcd backups will be copied over to the BackupBucket
of the Destination Seed
during control plane migration and any future backups will be uploaded there.
Triggering the Migration
For control plane migration, operators with the necessary RBAC can use the shoots/binding
subresource to change the .spec.seedName
, with the following commands:
NAMESPACE=my-namespace
SHOOT_NAME=my-shoot
DEST_SEED_NAME=destination-seed
kubectl get --raw /apis/core.gardener.cloud/v1beta1/namespaces/${NAMESPACE}/shoots/${SHOOT_NAME} | jq -c '.spec.seedName = "'${DEST_SEED_NAME}'"' | kubectl replace --raw /apis/core.gardener.cloud/v1beta1/namespaces/${NAMESPACE}/shoots/${SHOOT_NAME}/binding -f - | jq -r '.spec.seedName'
Important
When migrating
Shoot
s to aDestination Seed
with different provider type from theSource Seed
, make sure of the following:Pods running in the
Destination Seed
must have network connectivity to the backup storage provider of theSource Seed
so that etcd backups can be copied successfully. Otherwise, theRestore
operation will get stuck at theWaiting until etcd backups are copied
step. However, if you do end up in this case, you can still finish the control plane migration by following the guide to manually copy etcd backups.The nodes of your
Shoot
cluster must have network connectivity to theShoot
’skube-apiserver
and thevpn-seed-server
once they are migrated to theDestination Seed
. Otherwise, theRestore
operation will get stuck at theWaiting until the Kubernetes API server can connect to the Shoot workers
step. However, if you do end up in this case and cannot allow network traffic from the nodes to theShoot
’s control plane, you can annotate theShoot
with theshoot.gardener.cloud/skip-readiness
annotation so that theRestore
operation finishes, and then use theshoots/binding
subresource to migrate the control plane back to theSource Seed
.
Copying ETCD Backups Manually During the Restore
Operation
Following is a workaround that can be used to copy etcd backups manually in situations where a Shoot
’s control plane has been moved to a Destination Seed
and the pods running in it lack network connectivity to the Source Seed
’s storage provider:
- Follow the instructions in the
etcd-backup-restore
getting started documentation on how to run theetcdbrctl
command locally or in a container. - Follow the instructions in the passing-credentials guide on how to set up the required credentials for the copy operation depending on the storage providers for which you want to perform it.
- Use the
etcdbrctl copy
command to copy the backups by following the instructions in theetcdbrctl copy
guide - After you have successfully copied the etcd backups, wait for the
EtcdCopyBackupsTask
custom resource to be created in theShoot
’s control plane on theDestination Seed
, if it does not already exist. Afterwards, mark it as successful by patching it using the following command:SHOOT_NAME=my-shoot PROJECT_NAME=my-project kubectl patch -n shoot--${PROJECT_NAME}--${SHOOT_NAME} etcdcopybackupstask ${SHOOT_NAME} --subresource status --type merge -p "{\"status\":{\"conditions\":[{\"type\":\"Succeeded\",\"status\":\"True\",\"reason\":\"manual copy successful\",\"message\":\"manual copy successful\",\"lastTransitionTime\":\"$(date -Iseconds)\",\"lastUpdateTime\":\"$(date -Iseconds)\"}]}}"
- After the
main-etcd
becomesReady
, and thesource-etcd-backup
secret is deleted from theShoot
’s control plane, remove the finalizer on the sourceextensions.gardener.cloud/v1alpha1.BackupEntry
in theDestination Seed
so that it can be deleted successfully (the resource name uses the following format:source-shoot--<project-name>--<shoot-name>--<uid>
). This is necessary as theDestination Seed
will not have network connectivity to theSource Seed
’s storage provider and the deletion will fail. - Once the control plane migration has finished successfully, make sure to manually clean up the source backup directory in the
Source Seed
’s storage provider.