그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그
6 minute read
This document provides an overview of the ShootStatus.
The Shoot status consists of a set of conditions. A Condition has the following fields:
|Name of the condition.|
|Indicates whether the condition is applicable, with possible values |
|Timestamp for when the condition last transitioned from one status to another.|
|Timestamp for when the condition was updated. Usually changes when |
|Machine-readable, UpperCamelCase text indicating the reason for the condition’s last transition.|
|Human-readable message indicating details about the last status transition.|
|Well-defined error codes in case the condition reports a problem.|
Currently the available Shoot condition types are:
The condition checks are executed periodically at interval which is configurable in the
.controllers.shootCare.syncPeriod, defaults to
GardenletConfiguration also allows configuring condition thresholds (
controllers.shootCare.conditionThresholds). Condition threshold is the amount of time to consider condition as
Processing on condition status changes.
Let’s check the following example to get better understanding. Let’s say that the
APIServerAvailable condition of our Shoot is with status
True. If the next condition check fails (for example kube-apiserver becomes unreachable), then the condition first goes to
Processing state. Only if this state remains for condition threshold amount of time, then the condition finally is updated to
Constraints represent conditions of a Shoot’s current state that constraint some operations on it. The current constraints are:
This constraint indicates whether a Shoot is allowed to be hibernated.
The rationale behind this constraint is that a Shoot can have
MutatingWebhookConfigurations acting on resources that are critical for waking up a cluster.
For example, if a webhook has rules for
CREATE/UPDATE Pods or Nodes and
failurePolicy=Fail, the webhook will block joining
Nodes and creating critical system component Pods and thus block the entire wakeup operation, because the server backing the webhook is not running.
Even if the
failurePolicy is set to
Ignore, high timeouts (
>15s) can lead to blocking requests of control plane components.
That’s because most control-plane API calls are made with a client-side timeout of
30s, so if a webhook has
the overall request might still fail as there is overhead in communication with the API server and potential other webhooks.
Generally, it’s best pratice to specify low timeouts in WebhookConfigs.
Also, it’s best practice
to exclude the
kube-system namespace from webhooks to avoid blocking critical operations on system components of the cluster.
Shoot owners can do so by adding a
namespaceSelector similar to this one to their webhook configurations:
namespaceSelector: matchExpressions: - key: gardener.cloud/purpose operator: NotIn values: - kube-system
If the Shoot still has webhooks with either
failurePolicy=Ignore && timeoutSeconds>15 that act on critical resources in the
kube-system namespace, Gardener will set the
False indicating, that the Shoot can probably not be woken up again after hibernation without manual intervention of the Gardener Operator.
gardener-apiserver will prevent any Shoot with the
HibernationPossible constraint set to
False from being hibernated, that is via manual hibernation as well as scheduled hibernation.
.controllers.shootCare.webhookRemediatorEnabled=truein the gardenlet configuration, the auto-remediation of webhooks not following the best practices can be turned on in the shoot clusters. Concretely, missing
objectSelectors will be added and too high
timeoutSecondswill be lowered. In some cases, the
failurePolicywill be set from
Ignore. Gardenlet will also add an annotation to make it visible to end-users that their webhook configurations were mutated and should be fixed by them in the first place. Note that all of this is no perfect solution and just done on a best effort basis. Only the owner of the webhook can know whether it indeed is problematic and configured correctly.
Webhooks labeled with
remediation.webhook.shoot.gardener.cloud/exclude=truewill be excluded from auto-remediation.
This constraint indicates whether all preconditions for a safe maintenance operation are satisfied (see also this document for more information about what happens during a shoot maintenance).
As of today, the same checks as in the
HibernationPossible constraint are being performed (user-deployed webhooks that might interfere with potential rolling updates of shoot worker nodes).
There is no further action being performed on this constraint’s status (maintenance is still being performed).
It is meant to make the user aware of potential problems that might occur due to his configurations.
This constraints indicates that there is at least one CA certificate which expires in less than
It will not be added to the
.status.constraints if there is no such CA certificate.
However, if it’s visible, then a credentials rotation operation should be considered.
The Shoot status holds information about the last operation that is performed on the Shoot. The last operation field reflects overall progress and the tasks that are currently being executed. Allowed operation types are
Restore. Allowed operation states are
Aborted. An operation in
Error state is an operation that will be retried for a configurable amount of time (
controllers.shoot.retryDuration field in
GardenletConfiguration, defaults to
12h). If the operation cannot complete successfully for the configured retry duration, it will be marked as
Failed. An operation in
Failed state is an operation that won’t be retried automatically (to retry such an operation, see Retry failed operation).
The Shoot status also contains information about the last occurred error(s) (if any) during an operation. A LastError consists of identifier of the task returned error, human-readable message of the error and error codes (if any) associated with the error.
Known error codes are:
ERR_INFRA_UNAUTHENTICATED- indicates that the last error occurred due to the client request not being completed because it lacks valid authentication credentials for the requested resource. It is classified as a non-retryable error code.
ERR_INFRA_UNAUTHORIZED- indicates that the last error occurred due to the server understanding the request but refusing to authorize it. It is classified as a non-retryable error code.
ERR_INFRA_QUOTA_EXCEEDED- indicates that the last error occurred due to infrastructure quota limits. It is classified as a non-retryable error code.
ERR_INFRA_RATE_LIMITS_EXCEEDED- indicates that the last error occurred due to exceeded infrastructure request rate limits.
ERR_INFRA_DEPENDENCIES- indicates that the last error occurred due to dependent objects on the infrastructure level. It is classified as a non-retryable error code.
ERR_RETRYABLE_INFRA_DEPENDENCIES- indicates that the last error occurred due to dependent objects on the infrastructure level, but the operation should be retried.
ERR_INFRA_RESOURCES_DEPLETED- indicates that the last error occurred due to depleted resource in the infrastructure.
ERR_CLEANUP_CLUSTER_RESOURCES- indicates that the last error occurred due to resources in the cluster that are stuck in deletion.
ERR_CONFIGURATION_PROBLEM- indicates that the last error occurred due to a configuration problem. It is classified as a non-retryable error code.
ERR_RETRYABLE_CONFIGURATION_PROBLEM- indicates that the last error occurred due to a retryable configuration problem. “Retryable” means that the occurred error is likely to be resolved in a ungraceful manner after given period of time.
ERR_PROBLEMATIC_WEBHOOK- indicates that the last error occurred due to a webhook not following the Kubernetes best practices (https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#best-practices-and-warnings).
Shoots will be automatically labeled with the
Its value might either be
unknown depending on the
status.lastErrors of the
This can be used as an easy filter method to find shoots based on their “health” status.