This is the multi-page printable view of this section. Click here to print.
Docs
1 - Gardener
Gardener
Gardener implements the automated management and operation of Kubernetes clusters as a service and provides a fully validated extensibility framework that can be adjusted to any programmatic cloud or infrastructure provider.
Gardener is 100% Kubernetes-native and exposes its own Cluster API to create homogeneous clusters on all supported infrastructures. This API differs from SIG Cluster Lifecycle’s Cluster API that only harmonizes how to get to clusters, while Gardener’s Cluster API goes one step further and also harmonizes the make-up of the clusters themselves. That means, Gardener gives you homogeneous clusters with exactly the same bill of material, configuration and behavior on all supported infrastructures, which you can see further down below in the section on our K8s Conformance Test Coverage.
In 2020, SIG Cluster Lifecycle’s Cluster API made a huge step forward with v1alpha3
and the newly added support for declarative control plane management. This made it possible to integrate managed services like GKE or Gardener. We would be more than happy, if the community would be interested, to contribute a Gardener control plane provider. For more information on the relation between Gardener API and SIG Cluster Lifecycle’s Cluster API, please see here.
Gardener’s main principle is to leverage Kubernetes concepts for all of its tasks.
In essence, Gardener is an extension API server that comes along with a bundle of custom controllers. It introduces new API objects in an existing Kubernetes cluster (which is called garden cluster) in order to use them for the management of end-user Kubernetes clusters (which are called shoot clusters). These shoot clusters are described via declarative cluster specifications which are observed by the controllers. They will bring up the clusters, reconcile their state, perform automated updates and make sure they are always up and running.
To accomplish these tasks reliably and to offer a high quality of service, Gardener controls the main components of a Kubernetes cluster (etcd, API server, controller manager, scheduler). These so-called control plane components are hosted in Kubernetes clusters themselves (which are called seed clusters). This is the main difference compared to many other OSS cluster provisioning tools: The shoot clusters do not have dedicated master VMs. Instead, the control plane is deployed as a native Kubernetes workload into the seeds (the architecture is commonly referred to as kubeception or inception design). This does not only effectively reduce the total cost of ownership but also allows easier implementations for “day-2 operations” (like cluster updates or robustness) by relying on all the mature Kubernetes features and capabilities.
Gardener reuses the identical Kubernetes design to span a scalable multi-cloud and multi-cluster landscape. Such familiarity with known concepts has proven to quickly ease the initial learning curve and accelerate developer productivity:
- Kubernetes API Server = Gardener API Server
- Kubernetes Controller Manager = Gardener Controller Manager
- Kubernetes Scheduler = Gardener Scheduler
- Kubelet = Gardenlet
- Node = Seed cluster
- Pod = Shoot cluster
Please find more information regarding the concepts and a detailed description of the architecture in our Gardener Wiki and our blog posts on kubernetes.io: Gardener - the Kubernetes Botanist (17.5.2018) and Gardener Project Update (2.12.2019).
K8s Conformance Test Coverage 
Gardener takes part in the Certified Kubernetes Conformance Program to attest its compatibility with the K8s conformance testsuite. Currently Gardener is certified for K8s versions up to v1.25, see the conformance spreadsheet.
Continuous conformance test results of the latest stable Gardener release are uploaded regularly to the CNCF test grid:
Provider/K8s | v1.26 | v1.25 | v1.24 | v1.23 | v1.22 | v1.21 | v1.20 |
---|---|---|---|---|---|---|---|
AWS | N/A | ||||||
Azure | N/A | ||||||
GCP | N/A | ||||||
OpenStack | N/A | ||||||
Alicloud | N/A | ||||||
Equinix Metal | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
vSphere | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
Get an overview of the test results at testgrid.
Start using or developing the Gardener locally
See our documentation in the /docs
repository, please find the index here.
Setting up your own Gardener landscape in the Cloud
The quickest way to test drive Gardener is to install it virtually onto an existing Kubernetes cluster, just like you would install any other Kubernetes-ready application. You can do this with our Gardener Helm Chart.
Alternatively you can use our garden setup project to create a fully configured Gardener landscape which also includes our Gardener Dashboard.
Feedback and Support
Feedback and contributions are always welcome!
All channels for getting in touch or learning about our project are listed under the community section. We are cordially inviting interested parties to join our bi-weekly meetings.
Please report bugs or suggestions about our Kubernetes clusters as such or the Gardener itself as GitHub issues or join our Slack channel #gardener (please invite yourself to the Kubernetes workspace here).
Learn More!
Please find further resources about our project here:
- Our landing page gardener.cloud
- “Gardener Project Update” blog on kubernetes.io.
- “Gardener, the Kubernetes Botanist” blog on kubernetes.io
- SAP news article about “Project Gardener”
- Introduction movie: “Gardener - Planting the Seeds of Success in the Cloud”
- “Thinking Cloud Native” talk at EclipseCon 2018
- Blog - “Showcase of Gardener at OSCON 2018”
1.1 - API Reference
Gardener API Reference
1.1.1 - Authentication
Packages:
authentication.gardener.cloud/v1alpha1
Package v1alpha1 is a version of the API.
Resource Types:AdminKubeconfigRequest
AdminKubeconfigRequest can be used to request a kubeconfig with admin credentials for a Shoot cluster.
Field | Description | ||
---|---|---|---|
apiVersion string | authentication.gardener.cloud/v1alpha1 | ||
kind string | AdminKubeconfigRequest | ||
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||
spec AdminKubeconfigRequestSpec | Spec is the specification of the AdminKubeconfigRequest.
| ||
status AdminKubeconfigRequestStatus | Status is the status of the AdminKubeconfigRequest. |
AdminKubeconfigRequestSpec
(Appears on: AdminKubeconfigRequest)
AdminKubeconfigRequestSpec contains the expiration time of the kubeconfig.
Field | Description |
---|---|
expirationSeconds int64 | (Optional) ExpirationSeconds is the requested validity duration of the credential. The credential issuer may return a credential with a different validity duration so a client needs to check the ‘expirationTimestamp’ field in a response. Defaults to 1 hour. |
AdminKubeconfigRequestStatus
(Appears on: AdminKubeconfigRequest)
AdminKubeconfigRequestStatus is the status of the AdminKubeconfigRequest containing the kubeconfig and expiration of the credential.
Field | Description |
---|---|
kubeconfig []byte | Kubeconfig contains the kubeconfig with cluster-admin privileges for the shoot cluster. |
expirationTimestamp Kubernetes meta/v1.Time | ExpirationTimestamp is the expiration timestamp of the returned credential. |
Generated with gen-crd-api-reference-docs
1.1.2 - Core
Packages:
core.gardener.cloud/v1beta1
Package v1beta1 is a version of the API.
Resource Types:- BackupBucket
- BackupEntry
- CloudProfile
- ControllerDeployment
- ControllerInstallation
- ControllerRegistration
- Project
- Quota
- SecretBinding
- Seed
- Shoot
BackupBucket
BackupBucket holds details about backup bucket
Field | Description | ||||||||
---|---|---|---|---|---|---|---|---|---|
apiVersion string | core.gardener.cloud/v1beta1 | ||||||||
kind string | BackupBucket | ||||||||
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||||
spec BackupBucketSpec | Specification of the Backup Bucket.
| ||||||||
status BackupBucketStatus | Most recently observed status of the Backup Bucket. |
BackupEntry
BackupEntry holds details about shoot backup.
Field | Description | ||||
---|---|---|---|---|---|
apiVersion string | core.gardener.cloud/v1beta1 | ||||
kind string | BackupEntry | ||||
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||
spec BackupEntrySpec | (Optional) Spec contains the specification of the Backup Entry.
| ||||
status BackupEntryStatus | (Optional) Status contains the most recently observed status of the Backup Entry. |
CloudProfile
CloudProfile represents certain properties about a provider environment.
Field | Description | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion string | core.gardener.cloud/v1beta1 | ||||||||||||||||||
kind string | CloudProfile | ||||||||||||||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional) Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||||||||||||||
spec CloudProfileSpec | (Optional) Spec defines the provider environment properties.
|
ControllerDeployment
ControllerDeployment contains information about how this controller is deployed.
Field | Description |
---|---|
apiVersion string | core.gardener.cloud/v1beta1 |
kind string | ControllerDeployment |
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. |
type string | Type is the deployment type. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | ProviderConfig contains type-specific configuration. It contains assets that deploy the controller. |
ControllerInstallation
ControllerInstallation represents an installation request for an external controller.
Field | Description | ||||||
---|---|---|---|---|---|---|---|
apiVersion string | core.gardener.cloud/v1beta1 | ||||||
kind string | ControllerInstallation | ||||||
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||
spec ControllerInstallationSpec | Spec contains the specification of this installation. If the object’s deletion timestamp is set, this field is immutable.
| ||||||
status ControllerInstallationStatus | Status contains the status of this installation. |
ControllerRegistration
ControllerRegistration represents a registration of an external controller.
Field | Description | ||||
---|---|---|---|---|---|
apiVersion string | core.gardener.cloud/v1beta1 | ||||
kind string | ControllerRegistration | ||||
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||
spec ControllerRegistrationSpec | Spec contains the specification of this registration. If the object’s deletion timestamp is set, this field is immutable.
|
Project
Project holds certain properties about a Gardener project.
Field | Description | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion string | core.gardener.cloud/v1beta1 | ||||||||||||||
kind string | Project | ||||||||||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional) Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||||||||||
spec ProjectSpec | (Optional) Spec defines the project properties.
| ||||||||||||||
status ProjectStatus | (Optional) Most recently observed status of the Project. |
Quota
Quota represents a quota on resources consumed by shoot clusters either per project or per provider secret.
Field | Description | ||||||
---|---|---|---|---|---|---|---|
apiVersion string | core.gardener.cloud/v1beta1 | ||||||
kind string | Quota | ||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional) Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||
spec QuotaSpec | (Optional) Spec defines the Quota constraints.
|
SecretBinding
SecretBinding represents a binding to a secret in the same or another namespace.
Field | Description |
---|---|
apiVersion string | core.gardener.cloud/v1beta1 |
kind string | SecretBinding |
metadata Kubernetes meta/v1.ObjectMeta | (Optional) Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. |
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a secret object in the same or another namespace. This field is immutable. |
quotas []Kubernetes core/v1.ObjectReference | (Optional) Quotas is a list of references to Quota objects in the same or another namespace. This field is immutable. |
provider SecretBindingProvider | (Optional) Provider defines the provider type of the SecretBinding. This field is immutable. |
Seed
Seed represents an installation request for an external controller.
Field | Description | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion string | core.gardener.cloud/v1beta1 | ||||||||||||||||||
kind string | Seed | ||||||||||||||||||
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||||||||||||||
spec SeedSpec | Spec contains the specification of this installation.
| ||||||||||||||||||
status SeedStatus | Status contains the status of this installation. |
Shoot
Shoot represents a Shoot cluster created and managed by Gardener.
Field | Description | ||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion string | core.gardener.cloud/v1beta1 | ||||||||||||||||||||||||||||||||||||||||
kind string | Shoot | ||||||||||||||||||||||||||||||||||||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional) Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||||||||||||||||||||||||||||||||||||
spec ShootSpec | (Optional) Specification of the Shoot cluster. If the object’s deletion timestamp is set, this field is immutable.
| ||||||||||||||||||||||||||||||||||||||||
status ShootStatus | (Optional) Most recently observed status of the Shoot cluster. |
Addon
(Appears on: KubernetesDashboard, NginxIngress)
Addon allows enabling or disabling a specific addon and is used to derive from.
Field | Description |
---|---|
enabled bool | Enabled indicates whether the addon is enabled or not. |
Addons
(Appears on: ShootSpec)
Addons is a collection of configuration for specific addons which are managed by the Gardener.
Field | Description |
---|---|
kubernetesDashboard KubernetesDashboard | (Optional) KubernetesDashboard holds configuration settings for the kubernetes dashboard addon. |
nginxIngress NginxIngress | (Optional) NginxIngress holds configuration settings for the nginx-ingress addon. |
AdmissionPlugin
(Appears on: KubeAPIServerConfig)
AdmissionPlugin contains information about a specific admission plugin and its corresponding configuration.
Field | Description |
---|---|
name string | Name is the name of the plugin. |
config k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) Config is the configuration of the plugin. |
disabled bool | (Optional) Disabled specifies whether this plugin should be disabled. |
Alerting
(Appears on: Monitoring)
Alerting contains information about how alerting will be done (i.e. who will receive alerts and how).
Field | Description |
---|---|
emailReceivers []string | (Optional) MonitoringEmailReceivers is a list of recipients for alerts |
AuditConfig
(Appears on: KubeAPIServerConfig)
AuditConfig contains settings for audit of the api server
Field | Description |
---|---|
auditPolicy AuditPolicy | (Optional) AuditPolicy contains configuration settings for audit policy of the kube-apiserver. |
AuditPolicy
(Appears on: AuditConfig)
AuditPolicy contains audit policy for kube-apiserver
Field | Description |
---|---|
configMapRef Kubernetes core/v1.ObjectReference | (Optional) ConfigMapRef is a reference to a ConfigMap object in the same namespace, which contains the audit policy for the kube-apiserver. |
AvailabilityZone
(Appears on: Region)
AvailabilityZone is an availability zone.
Field | Description |
---|---|
name string | Name is an an availability zone name. |
unavailableMachineTypes []string | (Optional) UnavailableMachineTypes is a list of machine type names that are not availability in this zone. |
unavailableVolumeTypes []string | (Optional) UnavailableVolumeTypes is a list of volume type names that are not availability in this zone. |
BackupBucketProvider
(Appears on: BackupBucketSpec)
BackupBucketProvider holds the details of cloud provider of the object store.
Field | Description |
---|---|
type string | Type is the type of provider. |
region string | Region is the region of the bucket. |
BackupBucketSpec
(Appears on: BackupBucket)
BackupBucketSpec is the specification of a Backup Bucket.
Field | Description |
---|---|
provider BackupBucketProvider | Provider holds the details of cloud provider of the object store. This field is immutable. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig is the configuration passed to BackupBucket resource. |
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a secret that contains the credentials to access object store. |
seedName string | (Optional) SeedName holds the name of the seed allocated to BackupBucket for running controller. This field is immutable. |
BackupBucketStatus
(Appears on: BackupBucket)
BackupBucketStatus holds the most recently observed status of the Backup Bucket.
Field | Description |
---|---|
providerStatus k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderStatus is the configuration passed to BackupBucket resource. |
lastOperation LastOperation | (Optional) LastOperation holds information about the last operation on the BackupBucket. |
lastError LastError | (Optional) LastError holds information about the last occurred error during an operation. |
observedGeneration int64 | (Optional) ObservedGeneration is the most recent generation observed for this BackupBucket. It corresponds to the BackupBucket’s generation, which is updated on mutation by the API Server. |
generatedSecretRef Kubernetes core/v1.SecretReference | (Optional) GeneratedSecretRef is reference to the secret generated by backup bucket, which will have object store specific credentials. |
BackupEntrySpec
(Appears on: BackupEntry)
BackupEntrySpec is the specification of a Backup Entry.
Field | Description |
---|---|
bucketName string | BucketName is the name of backup bucket for this Backup Entry. |
seedName string | (Optional) SeedName holds the name of the seed to which this BackupEntry is scheduled |
BackupEntryStatus
(Appears on: BackupEntry)
BackupEntryStatus holds the most recently observed status of the Backup Entry.
Field | Description |
---|---|
lastOperation LastOperation | (Optional) LastOperation holds information about the last operation on the BackupEntry. |
lastError LastError | (Optional) LastError holds information about the last occurred error during an operation. |
observedGeneration int64 | (Optional) ObservedGeneration is the most recent generation observed for this BackupEntry. It corresponds to the BackupEntry’s generation, which is updated on mutation by the API Server. |
seedName string | (Optional) SeedName is the name of the seed to which this BackupEntry is currently scheduled. This field is populated at the beginning of a create/reconcile operation. It is used when moving the BackupEntry between seeds. |
migrationStartTime Kubernetes meta/v1.Time | (Optional) MigrationStartTime is the time when a migration to a different seed was initiated. |
CARotation
(Appears on: ShootCredentialsRotation)
CARotation contains information about the certificate authority credential rotation.
Field | Description |
---|---|
phase CredentialsRotationPhase | Phase describes the phase of the certificate authority credential rotation. |
lastCompletionTime Kubernetes meta/v1.Time | (Optional) LastCompletionTime is the most recent time when the certificate authority credential rotation was successfully completed. |
lastInitiationTime Kubernetes meta/v1.Time | (Optional) LastInitiationTime is the most recent time when the certificate authority credential rotation was initiated. |
lastInitiationFinishedTime Kubernetes meta/v1.Time | (Optional) LastInitiationFinishedTime is the recent time when the certificate authority credential rotation initiation was completed. |
lastCompletionTriggeredTime Kubernetes meta/v1.Time | (Optional) LastCompletionTriggeredTime is the recent time when the certificate authority credential rotation completion was triggered. |
CRI
(Appears on: MachineImageVersion, Worker)
CRI contains information about the Container Runtimes.
Field | Description |
---|---|
name CRIName | The name of the CRI library. Supported values are |
containerRuntimes []ContainerRuntime | (Optional) ContainerRuntimes is the list of the required container runtimes supported for a worker pool. |
CRIName
(string
alias)
(Appears on: CRI)
CRIName is a type alias for the CRI name string.
CloudProfileSpec
(Appears on: CloudProfile)
CloudProfileSpec is the specification of a CloudProfile. It must contain exactly one of its defined keys.
Field | Description |
---|---|
caBundle string | (Optional) CABundle is a certificate bundle which will be installed onto every host machine of shoot cluster targeting this profile. |
kubernetes KubernetesSettings | Kubernetes contains constraints regarding allowed values of the ‘kubernetes’ block in the Shoot specification. |
machineImages []MachineImage | MachineImages contains constraints regarding allowed values for machine images in the Shoot specification. |
machineTypes []MachineType | MachineTypes contains constraints regarding allowed values for machine types in the ‘workers’ block in the Shoot specification. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig contains provider-specific configuration for the profile. |
regions []Region | Regions contains constraints regarding allowed values for regions and zones. |
seedSelector SeedSelector | (Optional) SeedSelector contains an optional list of labels on |
type string | Type is the name of the provider. |
volumeTypes []VolumeType | (Optional) VolumeTypes contains constraints regarding allowed values for volume types in the ‘workers’ block in the Shoot specification. |
ClusterAutoscaler
(Appears on: Kubernetes)
ClusterAutoscaler contains the configuration flags for the Kubernetes cluster autoscaler.
Field | Description |
---|---|
scaleDownDelayAfterAdd Kubernetes meta/v1.Duration | (Optional) ScaleDownDelayAfterAdd defines how long after scale up that scale down evaluation resumes (default: 1 hour). |
scaleDownDelayAfterDelete Kubernetes meta/v1.Duration | (Optional) ScaleDownDelayAfterDelete how long after node deletion that scale down evaluation resumes, defaults to scanInterval (default: 0 secs). |
scaleDownDelayAfterFailure Kubernetes meta/v1.Duration | (Optional) ScaleDownDelayAfterFailure how long after scale down failure that scale down evaluation resumes (default: 3 mins). |
scaleDownUnneededTime Kubernetes meta/v1.Duration | (Optional) ScaleDownUnneededTime defines how long a node should be unneeded before it is eligible for scale down (default: 30 mins). |
scaleDownUtilizationThreshold float64 | (Optional) ScaleDownUtilizationThreshold defines the threshold in fraction (0.0 - 1.0) under which a node is being removed (default: 0.5). |
scanInterval Kubernetes meta/v1.Duration | (Optional) ScanInterval how often cluster is reevaluated for scale up or down (default: 10 secs). |
expander ExpanderMode | (Optional) Expander defines the algorithm to use during scale up (default: least-waste). See: https://github.com/gardener/autoscaler/blob/machine-controller-manager-provider/cluster-autoscaler/FAQ.md#what-are-expanders. |
maxNodeProvisionTime Kubernetes meta/v1.Duration | (Optional) MaxNodeProvisionTime defines how long CA waits for node to be provisioned (default: 20 mins). |
maxGracefulTerminationSeconds int32 | (Optional) MaxGracefulTerminationSeconds is the number of seconds CA waits for pod termination when trying to scale down a node (default: 600). |
ignoreTaints []string | (Optional) IgnoreTaints specifies a list of taint keys to ignore in node templates when considering to scale a node group. |
Condition
(Appears on: ControllerInstallationStatus, SeedStatus, ShootStatus)
Condition holds the information about the state of a resource.
Field | Description |
---|---|
type ConditionType | Type of the condition. |
status ConditionStatus | Status of the condition, one of True, False, Unknown. |
lastTransitionTime Kubernetes meta/v1.Time | Last time the condition transitioned from one status to another. |
lastUpdateTime Kubernetes meta/v1.Time | Last time the condition was updated. |
reason string | The reason for the condition’s last transition. |
message string | A human readable message indicating details about the transition. |
codes []ErrorCode | (Optional) Well-defined error codes in case the condition reports a problem. |
ConditionStatus
(string
alias)
(Appears on: Condition)
ConditionStatus is the status of a condition.
ConditionType
(string
alias)
(Appears on: Condition)
ConditionType is a string alias.
ContainerRuntime
(Appears on: CRI)
ContainerRuntime contains information about worker’s available container runtime
Field | Description |
---|---|
type string | Type is the type of the Container Runtime. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig is the configuration passed to container runtime resource. |
ControlPlane
(Appears on: ShootSpec)
ControlPlane holds information about the general settings for the control plane of a shoot.
Field | Description |
---|---|
highAvailability HighAvailability | (Optional) HighAvailability holds the configuration settings for high availability of the control plane of a shoot. |
ControllerDeploymentPolicy
(string
alias)
(Appears on: ControllerRegistrationDeployment)
ControllerDeploymentPolicy is a string alias.
ControllerInstallationSpec
(Appears on: ControllerInstallation)
ControllerInstallationSpec is the specification of a ControllerInstallation.
Field | Description |
---|---|
registrationRef Kubernetes core/v1.ObjectReference | RegistrationRef is used to reference a ControllerRegistration resource. The name field of the RegistrationRef is immutable. |
seedRef Kubernetes core/v1.ObjectReference | SeedRef is used to reference a Seed resource. The name field of the SeedRef is immutable. |
deploymentRef Kubernetes core/v1.ObjectReference | (Optional) DeploymentRef is used to reference a ControllerDeployment resource. |
ControllerInstallationStatus
(Appears on: ControllerInstallation)
ControllerInstallationStatus is the status of a ControllerInstallation.
Field | Description |
---|---|
conditions []Condition | (Optional) Conditions represents the latest available observations of a ControllerInstallations’s current state. |
providerStatus k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderStatus contains type-specific status. |
ControllerRegistrationDeployment
(Appears on: ControllerRegistrationSpec)
ControllerRegistrationDeployment contains information for how this controller is deployed.
Field | Description |
---|---|
policy ControllerDeploymentPolicy | (Optional) Policy controls how the controller is deployed. It defaults to ‘OnDemand’. |
seedSelector Kubernetes meta/v1.LabelSelector | (Optional) SeedSelector contains an optional label selector for seeds. Only if the labels match then this controller will be considered for a deployment. An empty list means that all seeds are selected. |
deploymentRefs []DeploymentRef | (Optional) DeploymentRefs holds references to |
ControllerRegistrationSpec
(Appears on: ControllerRegistration)
ControllerRegistrationSpec is the specification of a ControllerRegistration.
Field | Description |
---|---|
resources []ControllerResource | (Optional) Resources is a list of combinations of kinds (DNSProvider, Infrastructure, Generic, …) and their actual types (aws-route53, gcp, auditlog, …). |
deployment ControllerRegistrationDeployment | (Optional) Deployment contains information for how this controller is deployed. |
ControllerResource
(Appears on: ControllerRegistrationSpec)
ControllerResource is a combination of a kind (DNSProvider, Infrastructure, Generic, …) and the actual type for this kind (aws-route53, gcp, auditlog, …).
Field | Description |
---|---|
kind string | Kind is the resource kind, for example “OperatingSystemConfig”. |
type string | Type is the resource type, for example “coreos” or “ubuntu”. |
globallyEnabled bool | (Optional) GloballyEnabled determines if this ControllerResource is required by all Shoot clusters. This field is defaulted to false when kind is “Extension”. |
reconcileTimeout Kubernetes meta/v1.Duration | (Optional) ReconcileTimeout defines how long Gardener should wait for the resource reconciliation. This field is defaulted to 3m0s when kind is “Extension”. |
primary bool | (Optional) Primary determines if the controller backed by this ControllerRegistration is responsible for the extension resource’s lifecycle. This field defaults to true. There must be exactly one primary controller for this kind/type combination. This field is immutable. |
lifecycle ControllerResourceLifecycle | (Optional) Lifecycle defines a strategy that determines when different operations on a ControllerResource should be performed. This field is defaulted in the following way when kind is “Extension”. Reconcile: “AfterKubeAPIServer” Delete: “BeforeKubeAPIServer” Migrate: “BeforeKubeAPIServer” |
ControllerResourceLifecycle
(Appears on: ControllerResource)
ControllerResourceLifecycle defines the lifecycle of a controller resource.
Field | Description |
---|---|
reconcile ControllerResourceLifecycleStrategy | (Optional) Reconcile defines the strategy during reconciliation. |
delete ControllerResourceLifecycleStrategy | (Optional) Delete defines the strategy during deletion. |
migrate ControllerResourceLifecycleStrategy | (Optional) Migrate defines the strategy during migration. |
ControllerResourceLifecycleStrategy
(string
alias)
(Appears on: ControllerResourceLifecycle)
ControllerResourceLifecycleStrategy is a string alias.
CoreDNS
(Appears on: SystemComponents)
CoreDNS contains the settings of the Core DNS components running in the data plane of the Shoot cluster.
Field | Description |
---|---|
autoscaling CoreDNSAutoscaling | (Optional) Autoscaling contains the settings related to autoscaling of the Core DNS components running in the data plane of the Shoot cluster. |
rewriting CoreDNSRewriting | (Optional) Rewriting contains the setting related to rewriting of requests, which are obviously incorrect due to the unnecessary application of the search path. |
CoreDNSAutoscaling
(Appears on: CoreDNS)
CoreDNSAutoscaling contains the settings related to autoscaling of the Core DNS components running in the data plane of the Shoot cluster.
Field | Description |
---|---|
mode CoreDNSAutoscalingMode | The mode of the autoscaling to be used for the Core DNS components running in the data plane of the Shoot cluster.
Supported values are |
CoreDNSAutoscalingMode
(string
alias)
(Appears on: CoreDNSAutoscaling)
CoreDNSAutoscalingMode is a type alias for the Core DNS autoscaling mode string.
CoreDNSRewriting
(Appears on: CoreDNS)
CoreDNSRewriting contains the setting related to rewriting requests, which are obviously incorrect due to the unnecessary application of the search path.
Field | Description |
---|---|
commonSuffixes []string | (Optional) CommonSuffixes are expected to be the suffix of a fully qualified domain name. Each suffix should contain at least one or two dots (‘.’) to prevent accidental clashes. |
CredentialsRotationPhase
(string
alias)
(Appears on: CARotation, ShootETCDEncryptionKeyRotation, ShootServiceAccountKeyRotation)
CredentialsRotationPhase is a string alias.
DNS
(Appears on: ShootSpec)
DNS holds information about the provider, the hosted zone id and the domain.
Field | Description |
---|---|
domain string | (Optional) Domain is the external available domain of the Shoot cluster. This domain will be written into the kubeconfig that is handed out to end-users. This field is immutable. |
providers []DNSProvider | (Optional) Providers is a list of DNS providers that shall be enabled for this shoot cluster. Only relevant if not a default domain is used. |
DNSIncludeExclude
(Appears on: DNSProvider, SeedDNSProvider)
DNSIncludeExclude contains information about which domains shall be included/excluded.
Field | Description |
---|---|
include []string | (Optional) Include is a list of domains that shall be included. |
exclude []string | (Optional) Exclude is a list of domains that shall be excluded. |
DNSProvider
(Appears on: DNS)
DNSProvider contains information about a DNS provider.
Field | Description |
---|---|
domains DNSIncludeExclude | (Optional) Domains contains information about which domains shall be included/excluded for this provider. |
primary bool | (Optional) Primary indicates that this DNSProvider is used for shoot related domains. |
secretName string | (Optional) SecretName is a name of a secret containing credentials for the stated domain and the provider. When not specified, the Gardener will use the cloud provider credentials referenced by the Shoot and try to find respective credentials there (primary provider only). Specifying this field may override this behavior, i.e. forcing the Gardener to only look into the given secret. |
type string | (Optional) Type is the DNS provider type. |
zones DNSIncludeExclude | (Optional) Zones contains information about which hosted zones shall be included/excluded for this provider. |
DataVolume
(Appears on: Worker)
DataVolume contains information about a data volume.
Field | Description |
---|---|
name string | Name of the volume to make it referencable. |
type string | (Optional) Type is the type of the volume. |
size string | VolumeSize is the size of the volume. |
encrypted bool | (Optional) Encrypted determines if the volume should be encrypted. |
DeploymentRef
(Appears on: ControllerRegistrationDeployment)
DeploymentRef contains information about ControllerDeployment
references.
Field | Description |
---|---|
name string | Name is the name of the |
ErrorCode
(string
alias)
(Appears on: Condition, LastError)
ErrorCode is a string alias.
ExpanderMode
(string
alias)
(Appears on: ClusterAutoscaler)
ExpanderMode is type used for Expander values
ExpirableVersion
(Appears on: KubernetesSettings, MachineImageVersion)
ExpirableVersion contains a version and an expiration date.
Field | Description |
---|---|
version string | Version is the version identifier. |
expirationDate Kubernetes meta/v1.Time | (Optional) ExpirationDate defines the time at which this version expires. |
classification VersionClassification | (Optional) Classification defines the state of a version (preview, supported, deprecated) |
Extension
(Appears on: ShootSpec)
Extension contains type and provider information for Shoot extensions.
Field | Description |
---|---|
type string | Type is the type of the extension resource. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig is the configuration passed to extension resource. |
disabled bool | (Optional) Disabled allows to disable extensions that were marked as ‘globally enabled’ by Gardener administrators. |
FailureTolerance
(Appears on: HighAvailability)
FailureTolerance describes information about failure tolerance level of a highly available resource.
Field | Description |
---|---|
type FailureToleranceType | Type specifies the type of failure that the highly available resource can tolerate |
FailureToleranceType
(string
alias)
(Appears on: FailureTolerance)
FailureToleranceType specifies the type of failure that a highly available shoot control plane that can tolerate.
Gardener
(Appears on: SeedStatus, ShootStatus)
Gardener holds the information about the Gardener version that operated a resource.
Field | Description |
---|---|
id string | ID is the Docker container id of the Gardener which last acted on a resource. |
name string | Name is the hostname (pod name) of the Gardener which last acted on a resource. |
version string | Version is the version of the Gardener which last acted on a resource. |
Hibernation
(Appears on: ShootSpec)
Hibernation contains information whether the Shoot is suspended or not.
Field | Description |
---|---|
enabled bool | (Optional) Enabled specifies whether the Shoot needs to be hibernated or not. If it is true, the Shoot’s desired state is to be hibernated. If it is false or nil, the Shoot’s desired state is to be awakened. |
schedules []HibernationSchedule | (Optional) Schedules determine the hibernation schedules. |
HibernationSchedule
(Appears on: Hibernation)
HibernationSchedule determines the hibernation schedule of a Shoot. A Shoot will be regularly hibernated at each start time and will be woken up at each end time. Start or End can be omitted, though at least one of each has to be specified.
Field | Description |
---|---|
start string | (Optional) Start is a Cron spec at which time a Shoot will be hibernated. |
end string | (Optional) End is a Cron spec at which time a Shoot will be woken up. |
location string | (Optional) Location is the time location in which both start and and shall be evaluated. |
HighAvailability
(Appears on: ControlPlane)
HighAvailability specifies the configuration settings for high availability for a resource. Typical usages could be to configure HA for shoot control plane or for seed system components.
Field | Description |
---|---|
failureTolerance FailureTolerance | FailureTolerance holds information about failure tolerance level of a highly available resource. |
HorizontalPodAutoscalerConfig
(Appears on: KubeControllerManagerConfig)
HorizontalPodAutoscalerConfig contains horizontal pod autoscaler configuration settings for the kube-controller-manager. Note: Descriptions were taken from the Kubernetes documentation.
Field | Description |
---|---|
cpuInitializationPeriod Kubernetes meta/v1.Duration | (Optional) The period after which a ready pod transition is considered to be the first. |
downscaleStabilization Kubernetes meta/v1.Duration | (Optional) The configurable window at which the controller will choose the highest recommendation for autoscaling. |
initialReadinessDelay Kubernetes meta/v1.Duration | (Optional) The configurable period at which the horizontal pod autoscaler considers a Pod “not yet ready” given that it’s unready and it has transitioned to unready during that time. |
syncPeriod Kubernetes meta/v1.Duration | (Optional) The period for syncing the number of pods in horizontal pod autoscaler. |
tolerance float64 | (Optional) The minimum change (from 1.0) in the desired-to-actual metrics ratio for the horizontal pod autoscaler to consider scaling. |
IPFamily
(string
alias)
(Appears on: Networking, SeedNetworks)
IPFamily is a type for specifying an IP protocol version to use in Gardener clusters.
Ingress
(Appears on: SeedSpec)
Ingress configures the Ingress specific settings of the Seed cluster
Field | Description |
---|---|
domain string | Domain specifies the IngressDomain of the Seed cluster pointing to the ingress controller endpoint. It will be used to construct ingress URLs for system applications running in Shoot clusters. Once set this field is immutable. |
controller IngressController | Controller configures a Gardener managed Ingress Controller listening on the ingressDomain |
IngressController
(Appears on: Ingress)
IngressController enables a Gardener managed Ingress Controller listening on the ingressDomain
Field | Description |
---|---|
kind string | Kind defines which kind of IngressController to use, for example |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig specifies infrastructure specific configuration for the ingressController |
KubeAPIServerConfig
(Appears on: Kubernetes)
KubeAPIServerConfig contains configuration settings for the kube-apiserver.
Field | Description |
---|---|
KubernetesConfig KubernetesConfig | (Members of |
admissionPlugins []AdmissionPlugin | (Optional) AdmissionPlugins contains the list of user-defined admission plugins (additional to those managed by Gardener), and, if desired, the corresponding configuration. |
apiAudiences []string | (Optional) APIAudiences are the identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. Defaults to [“kubernetes”]. |
auditConfig AuditConfig | (Optional) AuditConfig contains configuration settings for the audit of the kube-apiserver. |
enableBasicAuthentication bool | (Optional) EnableBasicAuthentication defines whether basic authentication should be enabled for this cluster or not. Defaults to false. Deprecated: basic authentication has been removed in Kubernetes v1.19+. This field will be removed in a future version. |
oidcConfig OIDCConfig | (Optional) OIDCConfig contains configuration settings for the OIDC provider. |
runtimeConfig map[string]bool | (Optional) RuntimeConfig contains information about enabled or disabled APIs. |
serviceAccountConfig ServiceAccountConfig | (Optional) ServiceAccountConfig contains configuration settings for the service account handling of the kube-apiserver. |
watchCacheSizes WatchCacheSizes | (Optional) WatchCacheSizes contains configuration of the API server’s watch cache sizes.
Configuring these flags might be useful for large-scale Shoot clusters with a lot of parallel update requests
and a lot of watching controllers (e.g. large ManagedSeed clusters). When the API server’s watch cache’s
capacity is too small to cope with the amount of update requests and watchers for a particular resource, it
might happen that controller watches are permanently stopped with |
requests KubeAPIServerRequests | (Optional) Requests contains configuration for request-specific settings for the kube-apiserver. |
enableAnonymousAuthentication bool | (Optional) EnableAnonymousAuthentication defines whether anonymous requests to the secure port
of the API server should be allowed (flag |
eventTTL Kubernetes meta/v1.Duration | (Optional) EventTTL controls the amount of time to retain events. Defaults to 1h. |
logging KubeAPIServerLogging | (Optional) Logging contains configuration for the log level and HTTP access logs. |
defaultNotReadyTolerationSeconds int64 | (Optional) DefaultNotReadyTolerationSeconds indicates the tolerationSeconds of the toleration for notReady:NoExecute
that is added by default to every pod that does not already have such a toleration (flag |
defaultUnreachableTolerationSeconds int64 | (Optional) DefaultUnreachableTolerationSeconds indicates the tolerationSeconds of the toleration for unreachable:NoExecute
that is added by default to every pod that does not already have such a toleration (flag |
KubeAPIServerLogging
(Appears on: KubeAPIServerConfig)
KubeAPIServerLogging contains configuration for the logs level and http access logs
Field | Description |
---|---|
verbosity int32 | (Optional) Verbosity is the kube-apiserver log verbosity level Defaults to 2. |
httpAccessVerbosity int32 | (Optional) HTTPAccessVerbosity is the kube-apiserver access logs level |
KubeAPIServerRequests
(Appears on: KubeAPIServerConfig)
KubeAPIServerRequests contains configuration for request-specific settings for the kube-apiserver.
Field | Description |
---|---|
maxNonMutatingInflight int32 | (Optional) MaxNonMutatingInflight is the maximum number of non-mutating requests in flight at a given time. When the server exceeds this, it rejects requests. |
maxMutatingInflight int32 | (Optional) MaxMutatingInflight is the maximum number of mutating requests in flight at a given time. When the server exceeds this, it rejects requests. |
KubeControllerManagerConfig
(Appears on: Kubernetes)
KubeControllerManagerConfig contains configuration settings for the kube-controller-manager.
Field | Description |
---|---|
KubernetesConfig KubernetesConfig | (Members of |
horizontalPodAutoscaler HorizontalPodAutoscalerConfig | (Optional) HorizontalPodAutoscalerConfig contains horizontal pod autoscaler configuration settings for the kube-controller-manager. |
nodeCIDRMaskSize int32 | (Optional) NodeCIDRMaskSize defines the mask size for node cidr in cluster (default is 24). This field is immutable. |
podEvictionTimeout Kubernetes meta/v1.Duration | (Optional) PodEvictionTimeout defines the grace period for deleting pods on failed nodes. Defaults to 2m. Deprecated: The corresponding kube-controller-manager flag |
nodeMonitorGracePeriod Kubernetes meta/v1.Duration | (Optional) NodeMonitorGracePeriod defines the grace period before an unresponsive node is marked unhealthy. |
KubeProxyConfig
(Appears on: Kubernetes)
KubeProxyConfig contains configuration settings for the kube-proxy.
Field | Description |
---|---|
KubernetesConfig KubernetesConfig | (Members of |
mode ProxyMode | (Optional) Mode specifies which proxy mode to use. defaults to IPTables. |
enabled bool | (Optional) Enabled indicates whether kube-proxy should be deployed or not. Depending on the networking extensions switching kube-proxy off might be rejected. Consulting the respective documentation of the used networking extension is recommended before using this field. defaults to true if not specified. |
KubeSchedulerConfig
(Appears on: Kubernetes)
KubeSchedulerConfig contains configuration settings for the kube-scheduler.
Field | Description |
---|---|
KubernetesConfig KubernetesConfig | (Members of |
kubeMaxPDVols string | (Optional) KubeMaxPDVols allows to configure the |
profile SchedulingProfile | (Optional) Profile configures the scheduling profile for the cluster. If not specified, the used profile is “balanced” (provides the default kube-scheduler behavior). |
KubeletConfig
(Appears on: Kubernetes, WorkerKubernetes)
KubeletConfig contains configuration settings for the kubelet.
Field | Description |
---|---|
KubernetesConfig KubernetesConfig | (Members of |
cpuCFSQuota bool | (Optional) CPUCFSQuota allows you to disable/enable CPU throttling for Pods. |
cpuManagerPolicy string | (Optional) CPUManagerPolicy allows to set alternative CPU management policies (default: none). |
evictionHard KubeletConfigEviction | (Optional) EvictionHard describes a set of eviction thresholds (e.g. memory.available<1Gi) that if met would trigger a Pod eviction. Default: memory.available: “100Mi/1Gi/5%” nodefs.available: “5%” nodefs.inodesFree: “5%” imagefs.available: “5%” imagefs.inodesFree: “5%” |
evictionMaxPodGracePeriod int32 | (Optional) EvictionMaxPodGracePeriod describes the maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. Default: 90 |
evictionMinimumReclaim KubeletConfigEvictionMinimumReclaim | (Optional) EvictionMinimumReclaim configures the amount of resources below the configured eviction threshold that the kubelet attempts to reclaim whenever the kubelet observes resource pressure. Default: 0 for each resource |
evictionPressureTransitionPeriod Kubernetes meta/v1.Duration | (Optional) EvictionPressureTransitionPeriod is the duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. Default: 4m0s |
evictionSoft KubeletConfigEviction | (Optional) EvictionSoft describes a set of eviction thresholds (e.g. memory.available<1.5Gi) that if met over a corresponding grace period would trigger a Pod eviction. Default: memory.available: “200Mi/1.5Gi/10%” nodefs.available: “10%” nodefs.inodesFree: “10%” imagefs.available: “10%” imagefs.inodesFree: “10%” |
evictionSoftGracePeriod KubeletConfigEvictionSoftGracePeriod | (Optional) EvictionSoftGracePeriod describes a set of eviction grace periods (e.g. memory.available=1m30s) that correspond to how long a soft eviction threshold must hold before triggering a Pod eviction. Default: memory.available: 1m30s nodefs.available: 1m30s nodefs.inodesFree: 1m30s imagefs.available: 1m30s imagefs.inodesFree: 1m30s |
maxPods int32 | (Optional) MaxPods is the maximum number of Pods that are allowed by the Kubelet. Default: 110 |
podPidsLimit int64 | (Optional) PodPIDsLimit is the maximum number of process IDs per pod allowed by the kubelet. |
imagePullProgressDeadline Kubernetes meta/v1.Duration | (Optional) ImagePullProgressDeadline describes the time limit under which if no pulling progress is made, the image pulling will be cancelled. Default: 1m |
failSwapOn bool | (Optional) FailSwapOn makes the Kubelet fail to start if swap is enabled on the node. (default true). |
kubeReserved KubeletConfigReserved | (Optional) KubeReserved is the configuration for resources reserved for kubernetes node components (mainly kubelet and container runtime). When updating these values, be aware that cgroup resizes may not succeed on active worker nodes. Look for the NodeAllocatableEnforced event to determine if the configuration was applied. Default: cpu=80m,memory=1Gi,pid=20k |
systemReserved KubeletConfigReserved | (Optional) SystemReserved is the configuration for resources reserved for system processes not managed by kubernetes (e.g. journald). When updating these values, be aware that cgroup resizes may not succeed on active worker nodes. Look for the NodeAllocatableEnforced event to determine if the configuration was applied. |
imageGCHighThresholdPercent int32 | (Optional) ImageGCHighThresholdPercent describes the percent of the disk usage which triggers image garbage collection. Default: 50 |
imageGCLowThresholdPercent int32 | (Optional) ImageGCLowThresholdPercent describes the percent of the disk to which garbage collection attempts to free. Default: 40 |
serializeImagePulls bool | (Optional) SerializeImagePulls describes whether the images are pulled one at a time. Default: true |
registryPullQPS int32 | (Optional) RegistryPullQPS is the limit of registry pulls per second. The value must not be a negative number. Setting it to 0 means no limit. Default: 5 |
registryBurst int32 | (Optional) RegistryBurst is the maximum size of bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registryPullQPS. The value must not be a negative number. Only used if registryPullQPS is greater than 0. Default: 10 |
seccompDefault bool | (Optional) SeccompDefault enables the use of |
containerLogMaxSize k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) A quantity defines the maximum size of the container log file before it is rotated. For example: “5Mi” or “256Ki”. Default: 100Mi |
containerLogMaxFiles int32 | (Optional) Maximum number of container log files that can be present for a container. |
protectKernelDefaults bool | (Optional) ProtectKernelDefaults ensures that the kernel tunables are equal to the kubelet defaults. Defaults to true for Kubernetes v1.26 or later. |
streamingConnectionIdleTimeout Kubernetes meta/v1.Duration | (Optional) StreamingConnectionIdleTimeout is the maximum time a streaming connection can be idle before the connection is automatically closed. This field cannot be set lower than “30s” or greater than “4h”. Default: “4h” for Kubernetes < v1.26. “5m” for Kubernetes >= v1.26. |
KubeletConfigEviction
(Appears on: KubeletConfig)
KubeletConfigEviction contains kubelet eviction thresholds supporting either a resource.Quantity or a percentage based value.
Field | Description |
---|---|
memoryAvailable string | (Optional) MemoryAvailable is the threshold for the free memory on the host server. |
imageFSAvailable string | (Optional) ImageFSAvailable is the threshold for the free disk space in the imagefs filesystem (docker images and container writable layers). |
imageFSInodesFree string | (Optional) ImageFSInodesFree is the threshold for the available inodes in the imagefs filesystem. |
nodeFSAvailable string | (Optional) NodeFSAvailable is the threshold for the free disk space in the nodefs filesystem (docker volumes, logs, etc). |
nodeFSInodesFree string | (Optional) NodeFSInodesFree is the threshold for the available inodes in the nodefs filesystem. |
KubeletConfigEvictionMinimumReclaim
(Appears on: KubeletConfig)
KubeletConfigEvictionMinimumReclaim contains configuration for the kubelet eviction minimum reclaim.
Field | Description |
---|---|
memoryAvailable k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) MemoryAvailable is the threshold for the memory reclaim on the host server. |
imageFSAvailable k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) ImageFSAvailable is the threshold for the disk space reclaim in the imagefs filesystem (docker images and container writable layers). |
imageFSInodesFree k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) ImageFSInodesFree is the threshold for the inodes reclaim in the imagefs filesystem. |
nodeFSAvailable k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) NodeFSAvailable is the threshold for the disk space reclaim in the nodefs filesystem (docker volumes, logs, etc). |
nodeFSInodesFree k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) NodeFSInodesFree is the threshold for the inodes reclaim in the nodefs filesystem. |
KubeletConfigEvictionSoftGracePeriod
(Appears on: KubeletConfig)
KubeletConfigEvictionSoftGracePeriod contains grace periods for kubelet eviction thresholds.
Field | Description |
---|---|
memoryAvailable Kubernetes meta/v1.Duration | (Optional) MemoryAvailable is the grace period for the MemoryAvailable eviction threshold. |
imageFSAvailable Kubernetes meta/v1.Duration | (Optional) ImageFSAvailable is the grace period for the ImageFSAvailable eviction threshold. |
imageFSInodesFree Kubernetes meta/v1.Duration | (Optional) ImageFSInodesFree is the grace period for the ImageFSInodesFree eviction threshold. |
nodeFSAvailable Kubernetes meta/v1.Duration | (Optional) NodeFSAvailable is the grace period for the NodeFSAvailable eviction threshold. |
nodeFSInodesFree Kubernetes meta/v1.Duration | (Optional) NodeFSInodesFree is the grace period for the NodeFSInodesFree eviction threshold. |
KubeletConfigReserved
(Appears on: KubeletConfig)
KubeletConfigReserved contains reserved resources for daemons
Field | Description |
---|---|
cpu k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) CPU is the reserved cpu. |
memory k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) Memory is the reserved memory. |
ephemeralStorage k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) EphemeralStorage is the reserved ephemeral-storage. |
pid k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) PID is the reserved process-ids. |
Kubernetes
(Appears on: ShootSpec)
Kubernetes contains the version and configuration variables for the Shoot control plane.
Field | Description |
---|---|
allowPrivilegedContainers bool | (Optional) AllowPrivilegedContainers indicates whether privileged containers are allowed in the Shoot. Defaults to true for Kubernetes versions below v1.25. Unusable for Kubernetes versions v1.25 and higher. |
clusterAutoscaler ClusterAutoscaler | (Optional) ClusterAutoscaler contains the configuration flags for the Kubernetes cluster autoscaler. |
kubeAPIServer KubeAPIServerConfig | (Optional) KubeAPIServer contains configuration settings for the kube-apiserver. |
kubeControllerManager KubeControllerManagerConfig | (Optional) KubeControllerManager contains configuration settings for the kube-controller-manager. |
kubeScheduler KubeSchedulerConfig | (Optional) KubeScheduler contains configuration settings for the kube-scheduler. |
kubeProxy KubeProxyConfig | (Optional) KubeProxy contains configuration settings for the kube-proxy. |
kubelet KubeletConfig | (Optional) Kubelet contains configuration settings for the kubelet. |
version string | Version is the semantic Kubernetes version to use for the Shoot cluster. |
verticalPodAutoscaler VerticalPodAutoscaler | (Optional) VerticalPodAutoscaler contains the configuration flags for the Kubernetes vertical pod autoscaler. |
enableStaticTokenKubeconfig bool | (Optional) EnableStaticTokenKubeconfig indicates whether static token kubeconfig secret will be created for the Shoot cluster. Defaults to true for Shoots with Kubernetes versions < 1.26. Defaults to false for Shoots with Kubernetes versions >= 1.26. Starting Kubernetes 1.27 the field will be locked to false. |
KubernetesConfig
(Appears on: KubeAPIServerConfig, KubeControllerManagerConfig, KubeProxyConfig, KubeSchedulerConfig, KubeletConfig)
KubernetesConfig contains common configuration fields for the control plane components.
Field | Description |
---|---|
featureGates map[string]bool | (Optional) FeatureGates contains information about enabled feature gates. |
KubernetesDashboard
(Appears on: Addons)
KubernetesDashboard describes configuration values for the kubernetes-dashboard addon.
Field | Description |
---|---|
Addon Addon | (Members of |
authenticationMode string | (Optional) AuthenticationMode defines the authentication mode for the kubernetes-dashboard. |
KubernetesSettings
(Appears on: CloudProfileSpec)
KubernetesSettings contains constraints regarding allowed values of the ‘kubernetes’ block in the Shoot specification.
Field | Description |
---|---|
versions []ExpirableVersion | (Optional) Versions is the list of allowed Kubernetes versions with optional expiration dates for Shoot clusters. |
LastError
(Appears on: BackupBucketStatus, BackupEntryStatus, ShootStatus)
LastError indicates the last occurred error for an operation on a resource.
Field | Description |
---|---|
description string | A human readable message indicating details about the last error. |
taskID string | (Optional) ID of the task which caused this last error |
codes []ErrorCode | (Optional) Well-defined error codes of the last error(s). |
lastUpdateTime Kubernetes meta/v1.Time | (Optional) Last time the error was reported |
LastMaintenance
(Appears on: ShootStatus)
LastMaintenance holds information about a maintenance operation on the Shoot.
Field | Description |
---|---|
description string | A human-readable message containing details about the operations performed in the last maintenance. |
triggeredTime Kubernetes meta/v1.Time | TriggeredTime is the time when maintenance was triggered. |
state LastOperationState | Status of the last maintenance operation, one of Processing, Succeeded, Error. |
failureReason string | (Optional) FailureReason holds the information about the last maintenance operation failure reason. |
LastOperation
(Appears on: BackupBucketStatus, BackupEntryStatus, ShootStatus)
LastOperation indicates the type and the state of the last operation, along with a description message and a progress indicator.
Field | Description |
---|---|
description string | A human readable message indicating details about the last operation. |
lastUpdateTime Kubernetes meta/v1.Time | Last time the operation state transitioned from one to another. |
progress int32 | The progress in percentage (0-100) of the last operation. |
state LastOperationState | Status of the last operation, one of Aborted, Processing, Succeeded, Error, Failed. |
type LastOperationType | Type of the last operation, one of Create, Reconcile, Delete, Migrate, Restore. |
LastOperationState
(string
alias)
(Appears on: LastMaintenance, LastOperation)
LastOperationState is a string alias.
LastOperationType
(string
alias)
(Appears on: LastOperation)
LastOperationType is a string alias.
Machine
(Appears on: Worker)
Machine contains information about the machine type and image.
Field | Description |
---|---|
type string | Type is the machine type of the worker group. |
image ShootMachineImage | (Optional) Image holds information about the machine image to use for all nodes of this pool. It will default to the latest version of the first image stated in the referenced CloudProfile if no value has been provided. |
architecture string | (Optional) Architecture is CPU architecture of machines in this worker pool. |
MachineControllerManagerSettings
(Appears on: Worker)
MachineControllerManagerSettings contains configurations for different worker-pools. Eg. MachineDrainTimeout, MachineHealthTimeout.
Field | Description |
---|---|
machineDrainTimeout Kubernetes meta/v1.Duration | (Optional) MachineDrainTimeout is the period after which machine is forcefully deleted. |
machineHealthTimeout Kubernetes meta/v1.Duration | (Optional) MachineHealthTimeout is the period after which machine is declared failed. |
machineCreationTimeout Kubernetes meta/v1.Duration | (Optional) MachineCreationTimeout is the period after which creation of the machine is declared failed. |
maxEvictRetries int32 | (Optional) MaxEvictRetries are the number of eviction retries on a pod after which drain is declared failed, and forceful deletion is triggered. |
nodeConditions []string | (Optional) NodeConditions are the set of conditions if set to true for the period of MachineHealthTimeout, machine will be declared failed. |
MachineImage
(Appears on: CloudProfileSpec)
MachineImage defines the name and multiple versions of the machine image in any environment.
Field | Description |
---|---|
name string | Name is the name of the image. |
versions []MachineImageVersion | Versions contains versions, expiration dates and container runtimes of the machine image |
MachineImageVersion
(Appears on: MachineImage)
MachineImageVersion is an expirable version with list of supported container runtimes and interfaces
Field | Description |
---|---|
ExpirableVersion ExpirableVersion | (Members of |
cri []CRI | (Optional) CRI list of supported container runtime and interfaces supported by this version |
architectures []string | (Optional) Architectures is the list of CPU architectures of the machine image in this version. |
kubeletVersionConstraint string | (Optional) KubeletVersionConstraint is a constraint describing the supported kubelet versions by the machine image in this version. If the field is not specified, it is assumed that the machine image in this version supports all kubelet versions. Examples: - ‘>= 1.26’ - supports only kubelet versions greater than or equal to 1.26 - ‘< 1.26’ - supports only kubelet versions less than 1.26 |
MachineType
(Appears on: CloudProfileSpec)
MachineType contains certain properties of a machine type.
Field | Description |
---|---|
cpu k8s.io/apimachinery/pkg/api/resource.Quantity | CPU is the number of CPUs for this machine type. |
gpu k8s.io/apimachinery/pkg/api/resource.Quantity | GPU is the number of GPUs for this machine type. |
memory k8s.io/apimachinery/pkg/api/resource.Quantity | Memory is the amount of memory for this machine type. |
name string | Name is the name of the machine type. |
storage MachineTypeStorage | (Optional) Storage is the amount of storage associated with the root volume of this machine type. |
usable bool | (Optional) Usable defines if the machine type can be used for shoot clusters. |
architecture string | (Optional) Architecture is the CPU architecture of this machine type. |
MachineTypeStorage
(Appears on: MachineType)
MachineTypeStorage is the amount of storage associated with the root volume of this machine type.
Field | Description |
---|---|
class string | Class is the class of the storage type. |
size k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) StorageSize is the storage size. |
type string | Type is the type of the storage. |
minSize k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) MinSize is the minimal supported storage size.
This overrides any other common minimum size configuration from |
Maintenance
(Appears on: ShootSpec)
Maintenance contains information about the time window for maintenance operations and which operations should be performed.
Field | Description |
---|---|
autoUpdate MaintenanceAutoUpdate | (Optional) AutoUpdate contains information about which constraints should be automatically updated. |
timeWindow MaintenanceTimeWindow | (Optional) TimeWindow contains information about the time window for maintenance operations. |
confineSpecUpdateRollout bool | (Optional) ConfineSpecUpdateRollout prevents that changes/updates to the shoot specification will be rolled out immediately. Instead, they are rolled out during the shoot’s maintenance time window. There is one exception that will trigger an immediate roll out which is changes to the Spec.Hibernation.Enabled field. |
MaintenanceAutoUpdate
(Appears on: Maintenance)
MaintenanceAutoUpdate contains information about which constraints should be automatically updated.
Field | Description |
---|---|
kubernetesVersion bool | KubernetesVersion indicates whether the patch Kubernetes version may be automatically updated (default: true). |
machineImageVersion bool | MachineImageVersion indicates whether the machine image version may be automatically updated (default: true). |
MaintenanceTimeWindow
(Appears on: Maintenance)
MaintenanceTimeWindow contains information about the time window for maintenance operations.
Field | Description |
---|---|
begin string | Begin is the beginning of the time window in the format HHMMSS+ZONE, e.g. “220000+0100”. If not present, a random value will be computed. |
end string | End is the end of the time window in the format HHMMSS+ZONE, e.g. “220000+0100”. If not present, the value will be computed based on the “Begin” value. |
Monitoring
(Appears on: ShootSpec)
Monitoring contains information about the monitoring configuration for the shoot.
Field | Description |
---|---|
alerting Alerting | (Optional) Alerting contains information about the alerting configuration for the shoot cluster. |
NamedResourceReference
(Appears on: ShootSpec)
NamedResourceReference is a named reference to a resource.
Field | Description |
---|---|
name string | Name of the resource reference. |
resourceRef Kubernetes autoscaling/v1.CrossVersionObjectReference | ResourceRef is a reference to a resource. |
Networking
(Appears on: ShootSpec)
Networking defines networking parameters for the shoot cluster.
Field | Description |
---|---|
type string | Type identifies the type of the networking plugin. This field is immutable. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig is the configuration passed to network resource. |
pods string | (Optional) Pods is the CIDR of the pod network. This field is immutable. |
nodes string | (Optional) Nodes is the CIDR of the entire node network. This field is immutable. |
services string | (Optional) Services is the CIDR of the service network. This field is immutable. |
ipFamilies []IPFamily | (Optional) IPFamilies specifies the IP protocol versions to use for shoot networking. This field is immutable. See https://github.com/gardener/gardener/blob/master/docs/usage/ipv6.md. Defaults to [“IPv4”]. |
NginxIngress
(Appears on: Addons)
NginxIngress describes configuration values for the nginx-ingress addon.
Field | Description |
---|---|
Addon Addon | (Members of |
loadBalancerSourceRanges []string | (Optional) LoadBalancerSourceRanges is list of allowed IP sources for NginxIngress |
config map[string]string | (Optional) Config contains custom configuration for the nginx-ingress-controller configuration. See https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/configmap.md#configuration-options |
externalTrafficPolicy Kubernetes core/v1.ServiceExternalTrafficPolicyType | (Optional) ExternalTrafficPolicy controls the |
NodeLocalDNS
(Appears on: SystemComponents)
NodeLocalDNS contains the settings of the node local DNS components running in the data plane of the Shoot cluster.
Field | Description |
---|---|
enabled bool | Enabled indicates whether node local DNS is enabled or not. |
forceTCPToClusterDNS bool | (Optional) ForceTCPToClusterDNS indicates whether the connection from the node local DNS to the cluster DNS (Core DNS) will be forced to TCP or not. Default, if unspecified, is to enforce TCP. |
forceTCPToUpstreamDNS bool | (Optional) ForceTCPToUpstreamDNS indicates whether the connection from the node local DNS to the upstream DNS (infrastructure DNS) will be forced to TCP or not. Default, if unspecified, is to enforce TCP. |
disableForwardToUpstreamDNS bool | (Optional) DisableForwardToUpstreamDNS indicates whether requests from node local DNS to upstream DNS should be disabled. Default, if unspecified, is to forward requests for external domains to upstream DNS |
OIDCConfig
(Appears on: KubeAPIServerConfig)
OIDCConfig contains configuration settings for the OIDC provider. Note: Descriptions were taken from the Kubernetes documentation.
Field | Description |
---|---|
caBundle string | (Optional) If set, the OpenID server’s certificate will be verified by one of the authorities in the oidc-ca-file, otherwise the host’s root CA set will be used. |
clientAuthentication OpenIDConnectClientAuthentication | (Optional) ClientAuthentication can optionally contain client configuration used for kubeconfig generation. |
clientID string | (Optional) The client ID for the OpenID Connect client, must be set if oidc-issuer-url is set. |
groupsClaim string | (Optional) If provided, the name of a custom OpenID Connect claim for specifying user groups. The claim value is expected to be a string or array of strings. This flag is experimental, please see the authentication documentation for further details. |
groupsPrefix string | (Optional) If provided, all groups will be prefixed with this value to prevent conflicts with other authentication strategies. |
issuerURL string | (Optional) The URL of the OpenID issuer, only HTTPS scheme will be accepted. If set, it will be used to verify the OIDC JSON Web Token (JWT). |
requiredClaims map[string]string | (Optional) key=value pairs that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value. |
signingAlgs []string | (Optional) List of allowed JOSE asymmetric signing algorithms. JWTs with a ‘alg’ header value not in this list will be rejected. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1 |
usernameClaim string | (Optional) The OpenID claim to use as the user name. Note that claims other than the default (‘sub’) is not guaranteed to be unique and immutable. This flag is experimental, please see the authentication documentation for further details. (default “sub”) |
usernamePrefix string | (Optional) If provided, all usernames will be prefixed with this value. If not provided, username claims other than ‘email’ are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value ‘-’. |
OpenIDConnectClientAuthentication
(Appears on: OIDCConfig)
OpenIDConnectClientAuthentication contains configuration for OIDC clients.
Field | Description |
---|---|
extraConfig map[string]string | (Optional) Extra configuration added to kubeconfig’s auth-provider. Must not be any of idp-issuer-url, client-id, client-secret, idp-certificate-authority, idp-certificate-authority-data, id-token or refresh-token |
secret string | (Optional) The client Secret for the OpenID Connect client. |
ProjectMember
(Appears on: ProjectSpec)
ProjectMember is a member of a project.
Field | Description |
---|---|
Subject Kubernetes rbac/v1.Subject | (Members of Subject is representing a user name, an email address, or any other identifier of a user, group, or service account that has a certain role. |
role string | Role represents the role of this member.
IMPORTANT: Be aware that this field will be removed in the |
roles []string | (Optional) Roles represents the list of roles of this member. |
ProjectPhase
(string
alias)
(Appears on: ProjectStatus)
ProjectPhase is a label for the condition of a project at the current time.
ProjectSpec
(Appears on: Project)
ProjectSpec is the specification of a Project.
Field | Description |
---|---|
createdBy Kubernetes rbac/v1.Subject | (Optional) CreatedBy is a subject representing a user name, an email address, or any other identifier of a user who created the project. This field is immutable. |
description string | (Optional) Description is a human-readable description of what the project is used for. |
owner Kubernetes rbac/v1.Subject | (Optional) Owner is a subject representing a user name, an email address, or any other identifier of a user owning
the project.
IMPORTANT: Be aware that this field will be removed in the |
purpose string | (Optional) Purpose is a human-readable explanation of the project’s purpose. |
members []ProjectMember | (Optional) Members is a list of subjects representing a user name, an email address, or any other identifier of a user, group, or service account that has a certain role. |
namespace string | (Optional) Namespace is the name of the namespace that has been created for the Project object. A nil value means that Gardener will determine the name of the namespace. This field is immutable. |
tolerations ProjectTolerations | (Optional) Tolerations contains the tolerations for taints on seed clusters. |
ProjectStatus
(Appears on: Project)
ProjectStatus holds the most recently observed status of the project.
Field | Description |
---|---|
observedGeneration int64 | (Optional) ObservedGeneration is the most recent generation observed for this project. |
phase ProjectPhase | Phase is the current phase of the project. |
staleSinceTimestamp Kubernetes meta/v1.Time | (Optional) StaleSinceTimestamp contains the timestamp when the project was first discovered to be stale/unused. |
staleAutoDeleteTimestamp Kubernetes meta/v1.Time | (Optional) StaleAutoDeleteTimestamp contains the timestamp when the project will be garbage-collected/automatically deleted because it’s stale/unused. |
lastActivityTimestamp Kubernetes meta/v1.Time | (Optional) LastActivityTimestamp contains the timestamp from the last activity performed in this project. |
ProjectTolerations
(Appears on: ProjectSpec)
ProjectTolerations contains the tolerations for taints on seed clusters.
Field | Description |
---|---|
defaults []Toleration | (Optional) Defaults contains a list of tolerations that are added to the shoots in this project by default. |
whitelist []Toleration | (Optional) Whitelist contains a list of tolerations that are allowed to be added to the shoots in this project. Please note
that this list may only be added by users having the |
Provider
(Appears on: ShootSpec)
Provider contains provider-specific information that are handed-over to the provider-specific extension controller.
Field | Description |
---|---|
type string | Type is the type of the provider. This field is immutable. |
controlPlaneConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ControlPlaneConfig contains the provider-specific control plane config blob. Please look up the concrete definition in the documentation of your provider extension. |
infrastructureConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) InfrastructureConfig contains the provider-specific infrastructure config blob. Please look up the concrete definition in the documentation of your provider extension. |
workers []Worker | Workers is a list of worker groups. |
workersSettings WorkersSettings | (Optional) WorkersSettings contains settings for all workers. |
ProxyMode
(string
alias)
(Appears on: KubeProxyConfig)
ProxyMode available in Linux platform: ‘userspace’ (older, going to be EOL), ‘iptables’ (newer, faster), ‘ipvs’ (newest, better in performance and scalability). As of now only ‘iptables’ and ‘ipvs’ is supported by Gardener. In Linux platform, if the iptables proxy is selected, regardless of how, but the system’s kernel or iptables versions are insufficient, this always falls back to the userspace proxy. IPVS mode will be enabled when proxy mode is set to ‘ipvs’, and the fall back path is firstly iptables and then userspace.
QuotaSpec
(Appears on: Quota)
QuotaSpec is the specification of a Quota.
Field | Description |
---|---|
clusterLifetimeDays int32 | (Optional) ClusterLifetimeDays is the lifetime of a Shoot cluster in days before it will be terminated automatically. |
metrics Kubernetes core/v1.ResourceList | Metrics is a list of resources which will be put under constraints. |
scope Kubernetes core/v1.ObjectReference | Scope is the scope of the Quota object, either ‘project’ or ‘secret’. This field is immutable. |
Region
(Appears on: CloudProfileSpec)
Region contains certain properties of a region.
Field | Description |
---|---|
name string | Name is a region name. |
zones []AvailabilityZone | (Optional) Zones is a list of availability zones in this region. |
labels map[string]string | (Optional) Labels is an optional set of key-value pairs that contain certain administrator-controlled labels for this region. It can be used by Gardener administrators/operators to provide additional information about a region, e.g. wrt quality, reliability, access restrictions, etc. |
ResourceWatchCacheSize
(Appears on: WatchCacheSizes)
ResourceWatchCacheSize contains configuration of the API server’s watch cache size for one specific resource.
Field | Description |
---|---|
apiGroup string | (Optional) APIGroup is the API group of the resource for which the watch cache size should be configured.
An unset value is used to specify the legacy core API (e.g. for |
resource string | Resource is the name of the resource for which the watch cache size should be configured
(in lowercase plural form, e.g. |
size int32 | CacheSize specifies the watch cache size that should be configured for the specified resource. |
SSHAccess
(Appears on: WorkersSettings)
SSHAccess contains settings regarding ssh access to the worker nodes.
Field | Description |
---|---|
enabled bool | Enabled indicates whether the SSH access to the worker nodes is ensured to be enabled or disabled in systemd. Defaults to true. |
SchedulingProfile
(string
alias)
(Appears on: KubeSchedulerConfig)
SchedulingProfile is a string alias used for scheduling profile values.
SecretBindingProvider
(Appears on: SecretBinding)
SecretBindingProvider defines the provider type of the SecretBinding.
Field | Description |
---|---|
type string | Type is the type of the provider. For backwards compatibility, the field can contain multiple providers separated by a comma. However the usage of single SecretBinding (hence Secret) for different cloud providers is strongly discouraged. |
SeedBackup
(Appears on: SeedSpec)
SeedBackup contains the object store configuration for backups for shoot (currently only etcd).
Field | Description |
---|---|
provider string | Provider is a provider name. This field is immutable. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig is the configuration passed to BackupBucket resource. |
region string | (Optional) Region is a region name. This field is immutable. |
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a Secret object containing the cloud provider credentials for the object store where backups should be stored. It should have enough privileges to manipulate the objects as well as buckets. |
SeedDNS
(Appears on: SeedSpec)
SeedDNS contains DNS-relevant information about this seed cluster.
Field | Description |
---|---|
ingressDomain string | (Optional) IngressDomain is the domain of the Seed cluster pointing to the ingress controller endpoint. It will be used to construct ingress URLs for system applications running in Shoot clusters. This field is immutable. This will be removed in the next API version and replaced by spec.ingress.domain. |
provider SeedDNSProvider | (Optional) Provider configures a DNSProvider |
SeedDNSProvider
(Appears on: SeedDNS)
SeedDNSProvider configures a DNSProvider for Seeds
Field | Description |
---|---|
type string | Type describes the type of the dns-provider, for example |
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a Secret object containing cloud provider credentials used for registering external domains. |
domains DNSIncludeExclude | (Optional) Domains contains information about which domains shall be included/excluded for this provider. |
zones DNSIncludeExclude | (Optional) Zones contains information about which hosted zones shall be included/excluded for this provider. |
SeedNetworks
(Appears on: SeedSpec)
SeedNetworks contains CIDRs for the pod, service and node networks of a Kubernetes cluster.
Field | Description |
---|---|
nodes string | (Optional) Nodes is the CIDR of the node network. This field is immutable. |
pods string | Pods is the CIDR of the pod network. This field is immutable. |
services string | Services is the CIDR of the service network. This field is immutable. |
shootDefaults ShootNetworks | (Optional) ShootDefaults contains the default networks CIDRs for shoots. |
blockCIDRs []string | (Optional) BlockCIDRs is a list of network addresses that should be blocked for shoot control plane components running in the seed cluster. |
ipFamilies []IPFamily | (Optional) IPFamilies specifies the IP protocol versions to use for seed networking. This field is immutable. See https://github.com/gardener/gardener/blob/master/docs/usage/ipv6.md. Defaults to [“IPv4”]. |
SeedProvider
(Appears on: SeedSpec)
SeedProvider defines the provider-specific information of this Seed cluster.
Field | Description |
---|---|
type string | Type is the name of the provider. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig is the configuration passed to Seed resource. |
region string | Region is a name of a region. |
zones []string | (Optional) Zones is the list of availability zones the seed cluster is deployed to. |
SeedSelector
(Appears on: CloudProfileSpec, ShootSpec)
SeedSelector contains constraints for selecting seed to be usable for shoots using a profile
Field | Description |
---|---|
LabelSelector Kubernetes meta/v1.LabelSelector | (Members of LabelSelector is optional and can be used to select seeds by their label settings |
providerTypes []string | (Optional) Providers is optional and can be used by restricting seeds by their provider type. ‘*’ can be used to enable seeds regardless of their provider type. |
SeedSettingDependencyWatchdog
(Appears on: SeedSettings)
SeedSettingDependencyWatchdog controls the dependency-watchdog settings for the seed.
Field | Description |
---|---|
endpoint SeedSettingDependencyWatchdogEndpoint | (Optional) Endpoint controls the endpoint settings for the dependency-watchdog for the seed. |
probe SeedSettingDependencyWatchdogProbe | (Optional) Probe controls the probe settings for the dependency-watchdog for the seed. |
SeedSettingDependencyWatchdogEndpoint
(Appears on: SeedSettingDependencyWatchdog)
SeedSettingDependencyWatchdogEndpoint controls the endpoint settings for the dependency-watchdog for the seed.
Field | Description |
---|---|
enabled bool | Enabled controls whether the endpoint controller of the dependency-watchdog should be enabled. This controller helps to alleviate the delay where control plane components remain unavailable by finding the respective pods in CrashLoopBackoff status and restarting them once their dependants become ready and available again. |
SeedSettingDependencyWatchdogProbe
(Appears on: SeedSettingDependencyWatchdog)
SeedSettingDependencyWatchdogProbe controls the probe settings for the dependency-watchdog for the seed.
Field | Description |
---|---|
enabled bool | Enabled controls whether the probe controller of the dependency-watchdog should be enabled. This controller scales down the kube-controller-manager of shoot clusters in case their respective kube-apiserver is not reachable via its external ingress in order to avoid melt-down situations. |
SeedSettingExcessCapacityReservation
(Appears on: SeedSettings)
SeedSettingExcessCapacityReservation controls the excess capacity reservation for shoot control planes in the seed.
Field | Description |
---|---|
enabled bool | Enabled controls whether the excess capacity reservation should be enabled. |
SeedSettingLoadBalancerServices
(Appears on: SeedSettings)
SeedSettingLoadBalancerServices controls certain settings for services of type load balancer that are created in the seed.
Field | Description |
---|---|
annotations map[string]string | (Optional) Annotations is a map of annotations that will be injected/merged into every load balancer service object. |
externalTrafficPolicy Kubernetes core/v1.ServiceExternalTrafficPolicyType | (Optional) ExternalTrafficPolicy describes how nodes distribute service traffic they receive on one of the service’s “externally-facing” addresses. Defaults to “Cluster”. |
zones []SeedSettingLoadBalancerServicesZones | (Optional) Zones controls settings, which are specific to the single-zone load balancers in a multi-zonal setup. Can be empty for single-zone seeds. Each specified zone has to relate to one of the zones in seed.spec.provider.zones. |
SeedSettingLoadBalancerServicesZones
(Appears on: SeedSettingLoadBalancerServices)
SeedSettingLoadBalancerServicesZones controls settings, which are specific to the single-zone load balancers in a multi-zonal setup.
Field | Description |
---|---|
name string | Name is the name of the zone as specified in seed.spec.provider.zones. |
annotations map[string]string | (Optional) Annotations is a map of annotations that will be injected/merged into the zone-specific load balancer service object. |
externalTrafficPolicy Kubernetes core/v1.ServiceExternalTrafficPolicyType | (Optional) ExternalTrafficPolicy describes how nodes distribute service traffic they receive on one of the service’s “externally-facing” addresses. Defaults to “Cluster”. |
SeedSettingOwnerChecks
(Appears on: SeedSettings)
SeedSettingOwnerChecks controls certain owner checks settings for shoots scheduled on this seed.
Field | Description |
---|---|
enabled bool | Enabled controls whether owner checks are enabled for shoots scheduled on this seed. It is enabled by default because it is a prerequisite for control plane migration. |
SeedSettingScheduling
(Appears on: SeedSettings)
SeedSettingScheduling controls settings for scheduling decisions for the seed.
Field | Description |
---|---|
visible bool | Visible controls whether the gardener-scheduler shall consider this seed when scheduling shoots. Invisible seeds are not considered by the scheduler. |
SeedSettingShootDNS
(Appears on: SeedSettings)
SeedSettingShootDNS controls the shoot DNS settings for the seed.
Field | Description |
---|---|
enabled bool | Enabled controls whether the DNS for shoot clusters should be enabled. When disabled then all shoots using the seed won’t get any DNS providers, DNS records, and no DNS extension controller is required to be installed here. This is useful for environments where DNS is not required. |
SeedSettingVerticalPodAutoscaler
(Appears on: SeedSettings)
SeedSettingVerticalPodAutoscaler controls certain settings for the vertical pod autoscaler components deployed in the seed.
Field | Description |
---|---|
enabled bool | Enabled controls whether the VPA components shall be deployed into the garden namespace in the seed cluster. It is enabled by default because Gardener heavily relies on a VPA being deployed. You should only disable this if your seed cluster already has another, manually/custom managed VPA deployment. |
SeedSettings
(Appears on: SeedSpec)
SeedSettings contains certain settings for this seed cluster.
Field | Description |
---|---|
excessCapacityReservation SeedSettingExcessCapacityReservation | (Optional) ExcessCapacityReservation controls the excess capacity reservation for shoot control planes in the seed. |
scheduling SeedSettingScheduling | (Optional) Scheduling controls settings for scheduling decisions for the seed. |
shootDNS SeedSettingShootDNS | (Optional) ShootDNS controls the shoot DNS settings for the seed. Deprecated: This field is deprecated and will be removed in a future version of Gardener. Do not use it. |
loadBalancerServices SeedSettingLoadBalancerServices | (Optional) LoadBalancerServices controls certain settings for services of type load balancer that are created in the seed. |
verticalPodAutoscaler SeedSettingVerticalPodAutoscaler | (Optional) VerticalPodAutoscaler controls certain settings for the vertical pod autoscaler components deployed in the seed. |
ownerChecks SeedSettingOwnerChecks | (Optional) SeedSettingOwnerChecks controls certain owner checks settings for shoots scheduled on this seed. |
dependencyWatchdog SeedSettingDependencyWatchdog | (Optional) DependencyWatchdog controls certain settings for the dependency-watchdog components deployed in the seed. |
SeedSpec
(Appears on: Seed, SeedTemplate)
SeedSpec is the specification of a Seed.
Field | Description |
---|---|
backup SeedBackup | (Optional) Backup holds the object store configuration for the backups of shoot (currently only etcd). If it is not specified, then there won’t be any backups taken for shoots associated with this seed. If backup field is present in seed, then backups of the etcd from shoot control plane will be stored under the configured object store. |
dns SeedDNS | DNS contains DNS-relevant information about this seed cluster. |
networks SeedNetworks | Networks defines the pod, service and worker network of the Seed cluster. |
provider SeedProvider | Provider defines the provider type and region for this Seed cluster. |
secretRef Kubernetes core/v1.SecretReference | (Optional) SecretRef is a reference to a Secret object containing the Kubeconfig of the Kubernetes cluster to be registered as Seed. |
taints []SeedTaint | (Optional) Taints describes taints on the seed. |
volume SeedVolume | (Optional) Volume contains settings for persistentvolumes created in the seed cluster. |
settings SeedSettings | (Optional) Settings contains certain settings for this seed cluster. |
ingress Ingress | (Optional) Ingress configures Ingress specific settings of the Seed cluster. This field is immutable. |
SeedStatus
(Appears on: Seed)
SeedStatus is the status of a Seed.
Field | Description |
---|---|
gardener Gardener | (Optional) Gardener holds information about the Gardener which last acted on the Shoot. |
kubernetesVersion string | (Optional) KubernetesVersion is the Kubernetes version of the seed cluster. |
conditions []Condition | (Optional) Conditions represents the latest available observations of a Seed’s current state. |
observedGeneration int64 | (Optional) ObservedGeneration is the most recent generation observed for this Seed. It corresponds to the Seed’s generation, which is updated on mutation by the API Server. |
clusterIdentity string | (Optional) ClusterIdentity is the identity of the Seed cluster. This field is immutable. |
capacity Kubernetes core/v1.ResourceList | (Optional) Capacity represents the total resources of a seed. |
allocatable Kubernetes core/v1.ResourceList | (Optional) Allocatable represents the resources of a seed that are available for scheduling. Defaults to Capacity. |
clientCertificateExpirationTimestamp Kubernetes meta/v1.Time | (Optional) ClientCertificateExpirationTimestamp is the timestamp at which gardenlet’s client certificate expires. |
SeedTaint
(Appears on: SeedSpec)
SeedTaint describes a taint on a seed.
Field | Description |
---|---|
key string | Key is the taint key to be applied to a seed. |
value string | (Optional) Value is the taint value corresponding to the taint key. |
SeedTemplate
SeedTemplate is a template for creating a Seed object.
Field | Description | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
metadata Kubernetes meta/v1.ObjectMeta | (Optional) Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||||||||||||||
spec SeedSpec | (Optional) Specification of the desired behavior of the Seed.
|
SeedVolume
(Appears on: SeedSpec)
SeedVolume contains settings for persistentvolumes created in the seed cluster.
Field | Description |
---|---|
minimumSize k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) MinimumSize defines the minimum size that should be used for PVCs in the seed. |
providers []SeedVolumeProvider | (Optional) Providers is a list of storage class provisioner types for the seed. |
SeedVolumeProvider
(Appears on: SeedVolume)
SeedVolumeProvider is a storage class provisioner type.
Field | Description |
---|---|
purpose string | Purpose is the purpose of this provider. |
name string | Name is the name of the storage class provisioner type. |
ServiceAccountConfig
(Appears on: KubeAPIServerConfig)
ServiceAccountConfig is the kube-apiserver configuration for service accounts.
Field | Description |
---|---|
issuer string | (Optional) Issuer is the identifier of the service account token issuer. The issuer will assert this identifier in “iss” claim of issued tokens. This value is used to generate new service account tokens. This value is a string or URI. Defaults to URI of the API server. |
extendTokenExpiration bool | (Optional) ExtendTokenExpiration turns on projected service account expiration extension during token generation, which helps safe transition from legacy token to bound service account token feature. If this flag is enabled, admission injected tokens would be extended up to 1 year to prevent unexpected failure during transition, ignoring value of service-account-max-token-expiration. |
maxTokenExpiration Kubernetes meta/v1.Duration | (Optional) MaxTokenExpiration is the maximum validity duration of a token created by the service account token issuer. If an otherwise valid TokenRequest with a validity duration larger than this value is requested, a token will be issued with a validity duration of this value. This field must be within [30d,90d]. |
acceptedIssuers []string | (Optional) AcceptedIssuers is an additional set of issuers that are used to determine which service account tokens are accepted. These values are not used to generate new service account tokens. Only useful when service account tokens are also issued by another external system or a change of the current issuer that is used for generating tokens is being performed. This field is only available for Kubernetes v1.22 or later. |
ShootAdvertisedAddress
(Appears on: ShootStatus)
ShootAdvertisedAddress contains information for the shoot’s Kube API server.
Field | Description |
---|---|
name string | Name of the advertised address. e.g. external |
url string | The URL of the API Server. e.g. https://api.foo.bar or https://1.2.3.4 |
ShootCredentials
(Appears on: ShootStatus)
ShootCredentials contains information about the shoot credentials.
Field | Description |
---|---|
rotation ShootCredentialsRotation | (Optional) Rotation contains information about the credential rotations. |
ShootCredentialsRotation
(Appears on: ShootCredentials)
ShootCredentialsRotation contains information about the rotation of credentials.
Field | Description |
---|---|
certificateAuthorities CARotation | (Optional) CertificateAuthorities contains information about the certificate authority credential rotation. |
kubeconfig ShootKubeconfigRotation | (Optional) Kubeconfig contains information about the kubeconfig credential rotation. |
sshKeypair ShootSSHKeypairRotation | (Optional) SSHKeypair contains information about the ssh-keypair credential rotation. |
observability ShootObservabilityRotation | (Optional) Observability contains information about the observability credential rotation. |
serviceAccountKey ShootServiceAccountKeyRotation | (Optional) ServiceAccountKey contains information about the service account key credential rotation. |
etcdEncryptionKey ShootETCDEncryptionKeyRotation | (Optional) ETCDEncryptionKey contains information about the ETCD encryption key credential rotation. |
ShootETCDEncryptionKeyRotation
(Appears on: ShootCredentialsRotation)
ShootETCDEncryptionKeyRotation contains information about the ETCD encryption key credential rotation.
Field | Description |
---|---|
phase CredentialsRotationPhase | Phase describes the phase of the ETCD encryption key credential rotation. |
lastCompletionTime Kubernetes meta/v1.Time | (Optional) LastCompletionTime is the most recent time when the ETCD encryption key credential rotation was successfully completed. |
lastInitiationTime Kubernetes meta/v1.Time | (Optional) LastInitiationTime is the most recent time when the ETCD encryption key credential rotation was initiated. |
lastInitiationFinishedTime Kubernetes meta/v1.Time | (Optional) LastInitiationFinishedTime is the recent time when the certificate authority credential rotation initiation was completed. |
lastCompletionTriggeredTime Kubernetes meta/v1.Time | (Optional) LastCompletionTriggeredTime is the recent time when the certificate authority credential rotation completion was triggered. |
ShootKubeconfigRotation
(Appears on: ShootCredentialsRotation)
ShootKubeconfigRotation contains information about the kubeconfig credential rotation.
Field | Description |
---|---|
lastInitiationTime Kubernetes meta/v1.Time | (Optional) LastInitiationTime is the most recent time when the kubeconfig credential rotation was initiated. |
lastCompletionTime Kubernetes meta/v1.Time | (Optional) LastCompletionTime is the most recent time when the kubeconfig credential rotation was successfully completed. |
ShootMachineImage
(Appears on: Machine)
ShootMachineImage defines the name and the version of the shoot’s machine image in any environment. Has to be defined in the respective CloudProfile.
Field | Description |
---|---|
name string | Name is the name of the image. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig is the shoot’s individual configuration passed to an extension resource. |
version string | (Optional) Version is the version of the shoot’s image. If version is not provided, it will be defaulted to the latest version from the CloudProfile. |
ShootNetworks
(Appears on: SeedNetworks)
ShootNetworks contains the default networks CIDRs for shoots.
Field | Description |
---|---|
pods string | (Optional) Pods is the CIDR of the pod network. |
services string | (Optional) Services is the CIDR of the service network. |
ShootObservabilityRotation
(Appears on: ShootCredentialsRotation)
ShootObservabilityRotation contains information about the observability credential rotation.
Field | Description |
---|---|
lastInitiationTime Kubernetes meta/v1.Time | (Optional) LastInitiationTime is the most recent time when the observability credential rotation was initiated. |
lastCompletionTime Kubernetes meta/v1.Time | (Optional) LastCompletionTime is the most recent time when the observability credential rotation was successfully completed. |
ShootPurpose
(string
alias)
(Appears on: ShootSpec)
ShootPurpose is a type alias for string.
ShootSSHKeypairRotation
(Appears on: ShootCredentialsRotation)
ShootSSHKeypairRotation contains information about the ssh-keypair credential rotation.
Field | Description |
---|---|
lastInitiationTime Kubernetes meta/v1.Time | (Optional) LastInitiationTime is the most recent time when the ssh-keypair credential rotation was initiated. |
lastCompletionTime Kubernetes meta/v1.Time | (Optional) LastCompletionTime is the most recent time when the ssh-keypair credential rotation was successfully completed. |
ShootServiceAccountKeyRotation
(Appears on: ShootCredentialsRotation)
ShootServiceAccountKeyRotation contains information about the service account key credential rotation.
Field | Description |
---|---|
phase CredentialsRotationPhase | Phase describes the phase of the service account key credential rotation. |
lastCompletionTime Kubernetes meta/v1.Time | (Optional) LastCompletionTime is the most recent time when the service account key credential rotation was successfully completed. |
lastInitiationTime Kubernetes meta/v1.Time | (Optional) LastInitiationTime is the most recent time when the service account key credential rotation was initiated. |
lastInitiationFinishedTime Kubernetes meta/v1.Time | (Optional) LastInitiationFinishedTime is the recent time when the certificate authority credential rotation initiation was completed. |
lastCompletionTriggeredTime Kubernetes meta/v1.Time | (Optional) LastCompletionTriggeredTime is the recent time when the certificate authority credential rotation completion was triggered. |
ShootSpec
(Appears on: Shoot, ShootTemplate)
ShootSpec is the specification of a Shoot.
Field | Description |
---|---|
addons Addons | (Optional) Addons contains information about enabled/disabled addons and their configuration. |
cloudProfileName string | CloudProfileName is a name of a CloudProfile object. This field is immutable. |
dns DNS | (Optional) DNS contains information about the DNS settings of the Shoot. |
extensions []Extension | (Optional) Extensions contain type and provider information for Shoot extensions. |
hibernation Hibernation | (Optional) Hibernation contains information whether the Shoot is suspended or not. |
kubernetes Kubernetes | Kubernetes contains the version and configuration settings of the control plane components. |
networking Networking | Networking contains information about cluster networking such as CNI Plugin type, CIDRs, …etc. |
maintenance Maintenance | (Optional) Maintenance contains information about the time window for maintenance operations and which operations should be performed. |
monitoring Monitoring | (Optional) Monitoring contains information about custom monitoring configurations for the shoot. |
provider Provider | Provider contains all provider-specific and provider-relevant information. |
purpose ShootPurpose | (Optional) Purpose is the purpose class for this cluster. |
region string | Region is a name of a region. This field is immutable. |
secretBindingName string | SecretBindingName is the name of the a SecretBinding that has a reference to the provider secret. The credentials inside the provider secret will be used to create the shoot in the respective account. This field is immutable. |
seedName string | (Optional) SeedName is the name of the seed cluster that runs the control plane of the Shoot. This field is immutable when the SeedChange feature gate is disabled. |
seedSelector SeedSelector | (Optional) SeedSelector is an optional selector which must match a seed’s labels for the shoot to be scheduled on that seed. |
resources []NamedResourceReference | (Optional) Resources holds a list of named resource references that can be referred to in extension configs by their names. |
tolerations []Toleration | (Optional) Tolerations contains the tolerations for taints on seed clusters. |
exposureClassName string | (Optional) ExposureClassName is the optional name of an exposure class to apply a control plane endpoint exposure strategy. This field is immutable. |
systemComponents SystemComponents | (Optional) SystemComponents contains the settings of system components in the control or data plane of the Shoot cluster. |
controlPlane ControlPlane | (Optional) ControlPlane contains general settings for the control plane of the shoot. |
ShootStatus
(Appears on: Shoot)
ShootStatus holds the most recently observed status of the Shoot cluster.
Field | Description |
---|---|
conditions []Condition | (Optional) Conditions represents the latest available observations of a Shoots’s current state. |
constraints []Condition | (Optional) Constraints represents conditions of a Shoot’s current state that constraint some operations on it. |
gardener Gardener | Gardener holds information about the Gardener which last acted on the Shoot. |
hibernated bool | IsHibernated indicates whether the Shoot is currently hibernated. |
lastOperation LastOperation | (Optional) LastOperation holds information about the last operation on the Shoot. |
lastErrors []LastError | (Optional) LastErrors holds information about the last occurred error(s) during an operation. |
observedGeneration int64 | (Optional) ObservedGeneration is the most recent generation observed for this Shoot. It corresponds to the Shoot’s generation, which is updated on mutation by the API Server. |
retryCycleStartTime Kubernetes meta/v1.Time | (Optional) RetryCycleStartTime is the start time of the last retry cycle (used to determine how often an operation must be retried until we give up). |
seedName string | (Optional) SeedName is the name of the seed cluster that runs the control plane of the Shoot. This value is only written after a successful create/reconcile operation. It will be used when control planes are moved between Seeds. |
technicalID string | TechnicalID is the name that is used for creating the Seed namespace, the infrastructure resources, and basically everything that is related to this particular Shoot. This field is immutable. |
uid k8s.io/apimachinery/pkg/types.UID | UID is a unique identifier for the Shoot cluster to avoid portability between Kubernetes clusters. It is used to compute unique hashes. This field is immutable. |
clusterIdentity string | (Optional) ClusterIdentity is the identity of the Shoot cluster. This field is immutable. |
advertisedAddresses []ShootAdvertisedAddress | (Optional) List of addresses on which the Kube API server can be reached. |
migrationStartTime Kubernetes meta/v1.Time | (Optional) MigrationStartTime is the time when a migration to a different seed was initiated. |
credentials ShootCredentials | (Optional) Credentials contains information about the shoot credentials. |
lastHibernationTriggerTime Kubernetes meta/v1.Time | (Optional) LastHibernationTriggerTime indicates the last time when the hibernation controller managed to change the hibernation settings of the cluster |
lastMaintenance LastMaintenance | (Optional) LastMaintenance holds information about the last maintenance operations on the Shoot. |
ShootTemplate
ShootTemplate is a template for creating a Shoot object.
Field | Description | ||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
metadata Kubernetes meta/v1.ObjectMeta | (Optional) Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||||||||||||||||||||||||||||||||||||
spec ShootSpec | (Optional) Specification of the desired behavior of the Shoot.
|
SystemComponents
(Appears on: ShootSpec)
SystemComponents contains the settings of system components in the control or data plane of the Shoot cluster.
Field | Description |
---|---|
coreDNS CoreDNS | (Optional) CoreDNS contains the settings of the Core DNS components running in the data plane of the Shoot cluster. |
nodeLocalDNS NodeLocalDNS | (Optional) NodeLocalDNS contains the settings of the node local DNS components running in the data plane of the Shoot cluster. |
Toleration
(Appears on: ProjectTolerations, ShootSpec)
Toleration is a toleration for a seed taint.
Field | Description |
---|---|
key string | Key is the toleration key to be applied to a project or shoot. |
value string | (Optional) Value is the toleration value corresponding to the toleration key. |
VersionClassification
(string
alias)
(Appears on: ExpirableVersion)
VersionClassification is the logical state of a version.
VerticalPodAutoscaler
(Appears on: Kubernetes)
VerticalPodAutoscaler contains the configuration flags for the Kubernetes vertical pod autoscaler.
Field | Description |
---|---|
enabled bool | Enabled specifies whether the Kubernetes VPA shall be enabled for the shoot cluster. |
evictAfterOOMThreshold Kubernetes meta/v1.Duration | (Optional) EvictAfterOOMThreshold defines the threshold that will lead to pod eviction in case it OOMed in less than the given threshold since its start and if it has only one container (default: 10m0s). |
evictionRateBurst int32 | (Optional) EvictionRateBurst defines the burst of pods that can be evicted (default: 1) |
evictionRateLimit float64 | (Optional) EvictionRateLimit defines the number of pods that can be evicted per second. A rate limit set to 0 or -1 will disable the rate limiter (default: -1). |
evictionTolerance float64 | (Optional) EvictionTolerance defines the fraction of replica count that can be evicted for update in case more than one pod can be evicted (default: 0.5). |
recommendationMarginFraction float64 | (Optional) RecommendationMarginFraction is the fraction of usage added as the safety margin to the recommended request (default: 0.15). |
updaterInterval Kubernetes meta/v1.Duration | (Optional) UpdaterInterval is the interval how often the updater should run (default: 1m0s). |
recommenderInterval Kubernetes meta/v1.Duration | (Optional) RecommenderInterval is the interval how often metrics should be fetched (default: 1m0s). |
Volume
(Appears on: Worker)
Volume contains information about the volume type, size, and encryption.
Field | Description |
---|---|
name string | (Optional) Name of the volume to make it referencable. |
type string | (Optional) Type is the type of the volume. |
size string | VolumeSize is the size of the volume. |
encrypted bool | (Optional) Encrypted determines if the volume should be encrypted. |
VolumeType
(Appears on: CloudProfileSpec)
VolumeType contains certain properties of a volume type.
Field | Description |
---|---|
class string | Class is the class of the volume type. |
name string | Name is the name of the volume type. |
usable bool | (Optional) Usable defines if the volume type can be used for shoot clusters. |
minSize k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) MinSize is the minimal supported storage size. |
WatchCacheSizes
(Appears on: KubeAPIServerConfig)
WatchCacheSizes contains configuration of the API server’s watch cache sizes.
Field | Description |
---|---|
default int32 | (Optional) Default configures the default watch cache size of the kube-apiserver
(flag |
resources []ResourceWatchCacheSize | (Optional) Resources configures the watch cache size of the kube-apiserver per resource
(flag |
Worker
(Appears on: Provider)
Worker is the base definition of a worker group.
Field | Description |
---|---|
annotations map[string]string | (Optional) Annotations is a map of key/value pairs for annotations for all the |
caBundle string | (Optional) CABundle is a certificate bundle which will be installed onto every machine of this worker pool. |
cri CRI | (Optional) CRI contains configurations of CRI support of every machine in the worker pool.
Defaults to a CRI with name |
kubernetes WorkerKubernetes | (Optional) Kubernetes contains configuration for Kubernetes components related to this worker pool. |
labels map[string]string | (Optional) Labels is a map of key/value pairs for labels for all the |
name string | Name is the name of the worker group. |
machine Machine | Machine contains information about the machine type and image. |
maximum int32 | Maximum is the maximum number of VMs to create. |
minimum int32 | Minimum is the minimum number of VMs to create. |
maxSurge k8s.io/apimachinery/pkg/util/intstr.IntOrString | (Optional) MaxSurge is maximum number of VMs that are created during an update. |
maxUnavailable k8s.io/apimachinery/pkg/util/intstr.IntOrString | (Optional) MaxUnavailable is the maximum number of VMs that can be unavailable during an update. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig is the provider-specific configuration for this worker pool. |
taints []Kubernetes core/v1.Taint | (Optional) Taints is a list of taints for all the |
volume Volume | (Optional) Volume contains information about the volume type and size. |
dataVolumes []DataVolume | (Optional) DataVolumes contains a list of additional worker volumes. |
kubeletDataVolumeName string | (Optional) KubeletDataVolumeName contains the name of a dataVolume that should be used for storing kubelet state. |
zones []string | (Optional) Zones is a list of availability zones that are used to evenly distribute this worker pool. Optional as not every provider may support availability zones. |
systemComponents WorkerSystemComponents | (Optional) SystemComponents contains configuration for system components related to this worker pool |
machineControllerManager MachineControllerManagerSettings | (Optional) MachineControllerManagerSettings contains configurations for different worker-pools. Eg. MachineDrainTimeout, MachineHealthTimeout. |
WorkerKubernetes
(Appears on: Worker)
WorkerKubernetes contains configuration for Kubernetes components related to this worker pool.
Field | Description |
---|---|
kubelet KubeletConfig | (Optional) Kubelet contains configuration settings for all kubelets of this worker pool.
If set, all |
version string | (Optional) Version is the semantic Kubernetes version to use for the Kubelet in this Worker Group. If not specified the kubelet version is derived from the global shoot cluster kubernetes version. version must be equal or lower than the version of the shoot kubernetes version. Only one minor version difference to other worker groups and global kubernetes version is allowed. |
WorkerSystemComponents
(Appears on: Worker)
WorkerSystemComponents contains configuration for system components related to this worker pool
Field | Description |
---|---|
allow bool | Allow determines whether the pool should be allowed to host system components or not (defaults to true) |
WorkersSettings
(Appears on: Provider)
WorkersSettings contains settings for all workers.
Field | Description |
---|---|
sshAccess SSHAccess | (Optional) SSHAccess contains settings regarding ssh access to the worker nodes. |
Generated with gen-crd-api-reference-docs
1.1.3 - Extensions
Packages:
extensions.gardener.cloud/v1alpha1
Package v1alpha1 is the v1alpha1 version of the API.
Resource Types:- BackupBucket
- BackupEntry
- Bastion
- Cluster
- ContainerRuntime
- ControlPlane
- DNSRecord
- Extension
- Infrastructure
- Network
- OperatingSystemConfig
- Worker
BackupBucket
BackupBucket is a specification for backup bucket.
Field | Description | ||||||
---|---|---|---|---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||||||
kind string | BackupBucket | ||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional)
Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||||||
spec BackupBucketSpec | Specification of the BackupBucket. If the object’s deletion timestamp is set, this field is immutable.
| ||||||
status BackupBucketStatus | (Optional) |
BackupEntry
BackupEntry is a specification for backup Entry.
Field | Description | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||||||||||
kind string | BackupEntry | ||||||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional)
Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||||||||||
spec BackupEntrySpec | Specification of the BackupEntry. If the object’s deletion timestamp is set, this field is immutable.
| ||||||||||
status BackupEntryStatus | (Optional) |
Bastion
Bastion is a bastion or jump host that is dynamically created to provide SSH access to shoot nodes.
Field | Description | ||||||
---|---|---|---|---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||||||
kind string | Bastion | ||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional)
Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||||||
spec BastionSpec | Spec is the specification of this Bastion. If the object’s deletion timestamp is set, this field is immutable.
| ||||||
status BastionStatus | (Optional) Status is the bastion’s status. |
Cluster
Cluster is a specification for a Cluster resource.
Field | Description | ||||||
---|---|---|---|---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||||||
kind string | Cluster | ||||||
metadata Kubernetes meta/v1.ObjectMeta | Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||||||
spec ClusterSpec |
|
ContainerRuntime
ContainerRuntime is a specification for a container runtime resource.
Field | Description | ||||||
---|---|---|---|---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||||||
kind string | ContainerRuntime | ||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional)
Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||||||
spec ContainerRuntimeSpec | Specification of the ContainerRuntime. If the object’s deletion timestamp is set, this field is immutable.
| ||||||
status ContainerRuntimeStatus | (Optional) |
ControlPlane
ControlPlane is a specification for a ControlPlane resource.
Field | Description | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||||||||||
kind string | ControlPlane | ||||||||||
metadata Kubernetes meta/v1.ObjectMeta | Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||||||||||
spec ControlPlaneSpec | Specification of the ControlPlane. If the object’s deletion timestamp is set, this field is immutable.
| ||||||||||
status ControlPlaneStatus | (Optional) |
DNSRecord
DNSRecord is a specification for a DNSRecord resource.
Field | Description | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||||||||||||||||
kind string | DNSRecord | ||||||||||||||||
metadata Kubernetes meta/v1.ObjectMeta | Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||||||||||||||||
spec DNSRecordSpec | Specification of the DNSRecord. If the object’s deletion timestamp is set, this field is immutable.
| ||||||||||||||||
status DNSRecordStatus | (Optional) |
Extension
Extension is a specification for a Extension resource.
Field | Description | ||
---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||
kind string | Extension | ||
metadata Kubernetes meta/v1.ObjectMeta | (Optional)
Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||
spec ExtensionSpec | Specification of the Extension. If the object’s deletion timestamp is set, this field is immutable.
| ||
status ExtensionStatus | (Optional) |
Infrastructure
Infrastructure is a specification for cloud provider infrastructure.
Field | Description | ||||||||
---|---|---|---|---|---|---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||||||||
kind string | Infrastructure | ||||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional)
Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||||||||
spec InfrastructureSpec | Specification of the Infrastructure. If the object’s deletion timestamp is set, this field is immutable.
| ||||||||
status InfrastructureStatus | (Optional) |
Network
Network is the specification for cluster networking.
Field | Description | ||||||||
---|---|---|---|---|---|---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||||||||
kind string | Network | ||||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional)
Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||||||||
spec NetworkSpec | Specification of the Network. If the object’s deletion timestamp is set, this field is immutable.
| ||||||||
status NetworkStatus | (Optional) |
OperatingSystemConfig
OperatingSystemConfig is a specification for a OperatingSystemConfig resource
Field | Description | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||||||||||||
kind string | OperatingSystemConfig | ||||||||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional)
Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||||||||||||
spec OperatingSystemConfigSpec | Specification of the OperatingSystemConfig. If the object’s deletion timestamp is set, this field is immutable.
| ||||||||||||
status OperatingSystemConfigStatus | (Optional) |
Worker
Worker is a specification for a Worker resource.
Field | Description | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion string | extensions.gardener.cloud/v1alpha1 | ||||||||||||
kind string | Worker | ||||||||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional)
Refer to the Kubernetes API documentation for the fields of the
metadata field. | ||||||||||||
spec WorkerSpec | Specification of the Worker. If the object’s deletion timestamp is set, this field is immutable.
| ||||||||||||
status WorkerStatus | (Optional) |
BackupBucketSpec
(Appears on: BackupBucket)
BackupBucketSpec is the spec for an BackupBucket resource.
Field | Description |
---|---|
DefaultSpec DefaultSpec | (Members of DefaultSpec is a structure containing common fields used by all extension resources. |
region string | Region is the region of this bucket. This field is immutable. |
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a secret that contains the credentials to access object store. |
BackupBucketStatus
(Appears on: BackupBucket)
BackupBucketStatus is the status for an BackupBucket resource.
Field | Description |
---|---|
DefaultStatus DefaultStatus | (Members of DefaultStatus is a structure containing common fields used by all extension resources. |
generatedSecretRef Kubernetes core/v1.SecretReference | (Optional) GeneratedSecretRef is reference to the secret generated by backup bucket, which will have object store specific credentials. |
BackupEntrySpec
(Appears on: BackupEntry)
BackupEntrySpec is the spec for an BackupEntry resource.
Field | Description |
---|---|
DefaultSpec DefaultSpec | (Members of DefaultSpec is a structure containing common fields used by all extension resources. |
backupBucketProviderStatus k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) BackupBucketProviderStatus contains the provider status that has
been generated by the controller responsible for the |
region string | Region is the region of this Entry. This field is immutable. |
bucketName string | BucketName is the name of backup bucket for this Backup Entry. |
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a secret that contains the credentials to access object store. |
BackupEntryStatus
(Appears on: BackupEntry)
BackupEntryStatus is the status for an BackupEntry resource.
Field | Description |
---|---|
DefaultStatus DefaultStatus | (Members of DefaultStatus is a structure containing common fields used by all extension resources. |
BastionIngressPolicy
(Appears on: BastionSpec)
BastionIngressPolicy represents an ingress policy for SSH bastion hosts.
Field | Description |
---|---|
ipBlock Kubernetes networking/v1.IPBlock | IPBlock defines an IP block that is allowed to access the bastion. |
BastionSpec
(Appears on: Bastion)
BastionSpec contains the specification for an SSH bastion host.
Field | Description |
---|---|
DefaultSpec DefaultSpec | (Members of DefaultSpec is a structure containing common fields used by all extension resources. |
userData []byte | UserData is the base64-encoded user data for the bastion instance. This should contain code to provision the SSH key on the bastion instance. This field is immutable. |
ingress []BastionIngressPolicy | Ingress controls from where the created bastion host should be reachable. |
BastionStatus
(Appears on: Bastion)
BastionStatus holds the most recently observed status of the Bastion.
Field | Description |
---|---|
DefaultStatus DefaultStatus | (Members of DefaultStatus is a structure containing common fields used by all extension resources. |
ingress Kubernetes core/v1.LoadBalancerIngress | (Optional) Ingress is the external IP and/or hostname of the bastion host. |
CRIConfig
(Appears on: OperatingSystemConfigSpec)
CRIConfig contains configurations of the CRI library.
Field | Description |
---|---|
name CRIName | Name is a mandatory string containing the name of the CRI library. Supported values are |
CRIName
(string
alias)
(Appears on: CRIConfig)
CRIName is a type alias for the CRI name string.
CloudConfig
(Appears on: OperatingSystemConfigStatus)
CloudConfig contains the generated output for the given operating system config spec. It contains a reference to a secret as the result may contain confidential data.
Field | Description |
---|---|
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a secret that contains the actual result of the generated cloud config. |
ClusterSpec
(Appears on: Cluster)
ClusterSpec is the spec for a Cluster resource.
Field | Description |
---|---|
cloudProfile k8s.io/apimachinery/pkg/runtime.RawExtension | CloudProfile is a raw extension field that contains the cloudprofile resource referenced by the shoot that has to be reconciled. |
seed k8s.io/apimachinery/pkg/runtime.RawExtension | Seed is a raw extension field that contains the seed resource referenced by the shoot that has to be reconciled. |
shoot k8s.io/apimachinery/pkg/runtime.RawExtension | Shoot is a raw extension field that contains the shoot resource that has to be reconciled. |
ContainerRuntimeSpec
(Appears on: ContainerRuntime)
ContainerRuntimeSpec is the spec for a ContainerRuntime resource.
Field | Description |
---|---|
binaryPath string | BinaryPath is the Worker’s machine path where container runtime extensions should copy the binaries to. |
workerPool ContainerRuntimeWorkerPool | WorkerPool identifies the worker pool of the Shoot. For each worker pool and type, Gardener deploys a ContainerRuntime CRD. |
DefaultSpec DefaultSpec | (Members of DefaultSpec is a structure containing common fields used by all extension resources. |
ContainerRuntimeStatus
(Appears on: ContainerRuntime)
ContainerRuntimeStatus is the status for a ContainerRuntime resource.
Field | Description |
---|---|
DefaultStatus DefaultStatus | (Members of DefaultStatus is a structure containing common fields used by all extension resources. |
ContainerRuntimeWorkerPool
(Appears on: ContainerRuntimeSpec)
ContainerRuntimeWorkerPool identifies a Shoot worker pool by its name and selector.
Field | Description |
---|---|
name string | Name specifies the name of the worker pool the container runtime should be available for. This field is immutable. |
selector Kubernetes meta/v1.LabelSelector | Selector is the label selector used by the extension to match the nodes belonging to the worker pool. |
ControlPlaneSpec
(Appears on: ControlPlane)
ControlPlaneSpec is the spec of a ControlPlane resource.
Field | Description |
---|---|
DefaultSpec DefaultSpec | (Members of DefaultSpec is a structure containing common fields used by all extension resources. |
purpose Purpose | (Optional) Purpose contains the data if a cloud provider needs additional components in order to expose the control plane. This field is immutable. |
infrastructureProviderStatus k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) InfrastructureProviderStatus contains the provider status that has
been generated by the controller responsible for the |
region string | Region is the region of this control plane. This field is immutable. |
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a secret that contains the cloud provider specific credentials. |
ControlPlaneStatus
(Appears on: ControlPlane)
ControlPlaneStatus is the status of a ControlPlane resource.
Field | Description |
---|---|
DefaultStatus DefaultStatus | (Members of DefaultStatus is a structure containing common fields used by all extension resources. |
DNSRecordSpec
(Appears on: DNSRecord)
DNSRecordSpec is the spec of a DNSRecord resource.
Field | Description |
---|---|
DefaultSpec DefaultSpec | (Members of DefaultSpec is a structure containing common fields used by all extension resources. |
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a secret that contains the cloud provider specific credentials. |
region string | (Optional) Region is the region of this DNS record. If not specified, the region specified in SecretRef will be used. If that is also not specified, the extension controller will use its default region. |
zone string | (Optional) Zone is the DNS hosted zone of this DNS record. If not specified, it will be determined automatically by getting all hosted zones of the account and searching for the longest zone name that is a suffix of Name. |
name string | Name is the fully qualified domain name, e.g. “api. |
recordType DNSRecordType | RecordType is the DNS record type. Only A, CNAME, and TXT records are currently supported. This field is immutable. |
values []string | Values is a list of IP addresses for A records, a single hostname for CNAME records, or a list of texts for TXT records. |
ttl int64 | (Optional) TTL is the time to live in seconds. Defaults to 120. |
DNSRecordStatus
(Appears on: DNSRecord)
DNSRecordStatus is the status of a DNSRecord resource.
Field | Description |
---|---|
DefaultStatus DefaultStatus | (Members of DefaultStatus is a structure containing common fields used by all extension resources. |
zone string | (Optional) Zone is the DNS hosted zone of this DNS record. |
DNSRecordType
(string
alias)
(Appears on: DNSRecordSpec)
DNSRecordType is a string alias.
DataVolume
(Appears on: WorkerPool)
DataVolume contains information about a data volume.
Field | Description |
---|---|
name string | Name of the volume to make it referencable. |
type string | (Optional) Type is the type of the volume. |
size string | Size is the of the root volume. |
encrypted bool | (Optional) Encrypted determines if the volume should be encrypted. |
DefaultSpec
(Appears on: BackupBucketSpec, BackupEntrySpec, BastionSpec, ContainerRuntimeSpec, ControlPlaneSpec, DNSRecordSpec, ExtensionSpec, InfrastructureSpec, NetworkSpec, OperatingSystemConfigSpec, WorkerSpec)
DefaultSpec contains common status fields for every extension resource.
Field | Description |
---|---|
type string | Type contains the instance of the resource’s kind. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig is the provider specific configuration. |
DefaultStatus
(Appears on: BackupBucketStatus, BackupEntryStatus, BastionStatus, ContainerRuntimeStatus, ControlPlaneStatus, DNSRecordStatus, ExtensionStatus, InfrastructureStatus, NetworkStatus, OperatingSystemConfigStatus, WorkerStatus)
DefaultStatus contains common status fields for every extension resource.
Field | Description |
---|---|
providerStatus k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderStatus contains provider-specific status. |
conditions []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition | (Optional) Conditions represents the latest available observations of a Seed’s current state. |
lastError github.com/gardener/gardener/pkg/apis/core/v1beta1.LastError | (Optional) LastError holds information about the last occurred error during an operation. |
lastOperation github.com/gardener/gardener/pkg/apis/core/v1beta1.LastOperation | (Optional) LastOperation holds information about the last operation on the resource. |
observedGeneration int64 | ObservedGeneration is the most recent generation observed for this resource. |
state k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) State can be filled by the operating controller with what ever data it needs. |
resources []github.com/gardener/gardener/pkg/apis/core/v1beta1.NamedResourceReference | (Optional) Resources holds a list of named resource references that can be referred to in the state by their names. |
DropIn
(Appears on: Unit)
DropIn is a drop-in configuration for a systemd unit.
Field | Description |
---|---|
name string | Name is the name of the drop-in. |
content string | Content is the content of the drop-in. |
ExtensionSpec
(Appears on: Extension)
ExtensionSpec is the spec for a Extension resource.
Field | Description |
---|---|
DefaultSpec DefaultSpec | (Members of DefaultSpec is a structure containing common fields used by all extension resources. |
ExtensionStatus
(Appears on: Extension)
ExtensionStatus is the status for a Extension resource.
Field | Description |
---|---|
DefaultStatus DefaultStatus | (Members of DefaultStatus is a structure containing common fields used by all extension resources. |
File
(Appears on: OperatingSystemConfigSpec)
File is a file that should get written to the host’s file system. The content can either be inlined or referenced from a secret in the same namespace.
Field | Description |
---|---|
path string | Path is the path of the file system where the file should get written to. |
permissions int32 | (Optional) Permissions describes with which permissions the file should get written to the file system. Should be defaulted to octal 0644. |
content FileContent | Content describe the file’s content. |
FileCodecID
(string
alias)
FileCodecID is the id of a FileCodec for cloud-init scripts.
FileContent
(Appears on: File)
FileContent can either reference a secret or contain inline configuration.
Field | Description |
---|---|
secretRef FileContentSecretRef | (Optional) SecretRef is a struct that contains information about the referenced secret. |
inline FileContentInline | (Optional) Inline is a struct that contains information about the inlined data. |
transmitUnencoded bool | (Optional) TransmitUnencoded set to true will ensure that the os-extension does not encode the file content when sent to the node. This for example can be used to manipulate the clear-text content before it reaches the node. |
FileContentInline
(Appears on: FileContent)
FileContentInline contains keys for inlining a file content’s data and encoding.
Field | Description |
---|---|
encoding string | Encoding is the file’s encoding (e.g. base64). |
data string | Data is the file’s data. |
FileContentSecretRef
(Appears on: FileContent)
FileContentSecretRef contains keys for referencing a file content’s data from a secret in the same namespace.
Field | Description |
---|---|
name string | Name is the name of the secret. |
dataKey string | DataKey is the key in the secret’s |
IPFamily
(string
alias)
(Appears on: NetworkSpec)
IPFamily is a type for specifying an IP protocol version to use in Gardener clusters.
InfrastructureSpec
(Appears on: Infrastructure)
InfrastructureSpec is the spec for an Infrastructure resource.
Field | Description |
---|---|
DefaultSpec DefaultSpec | (Members of DefaultSpec is a structure containing common fields used by all extension resources. |
region string | Region is the region of this infrastructure. This field is immutable. |
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a secret that contains the cloud provider credentials. |
sshPublicKey []byte | (Optional) SSHPublicKey is the public SSH key that should be used with this infrastructure. |
InfrastructureStatus
(Appears on: Infrastructure)
InfrastructureStatus is the status for an Infrastructure resource.
Field | Description |
---|---|
DefaultStatus DefaultStatus | (Members of DefaultStatus is a structure containing common fields used by all extension resources. |
nodesCIDR string | (Optional) NodesCIDR is the CIDR of the node network that was optionally created by the acting extension controller. This might be needed in environments in which the CIDR for the network for the shoot worker node cannot be statically defined in the Shoot resource but must be computed dynamically. |
MachineDeployment
(Appears on: WorkerStatus)
MachineDeployment is a created machine deployment.
Field | Description |
---|---|
name string | Name is the name of the |
minimum int32 | Minimum is the minimum number for this machine deployment. |
maximum int32 | Maximum is the maximum number for this machine deployment. |
MachineImage
(Appears on: WorkerPool)
MachineImage contains logical information about the name and the version of the machie image that should be used. The logical information must be mapped to the provider-specific information (e.g., AMIs, …) by the provider itself.
Field | Description |
---|---|
name string | Name is the logical name of the machine image. |
version string | Version is the version of the machine image. |
NetworkSpec
(Appears on: Network)
NetworkSpec is the spec for an Network resource.
Field | Description |
---|---|
DefaultSpec DefaultSpec | (Members of DefaultSpec is a structure containing common fields used by all extension resources. |
podCIDR string | PodCIDR defines the CIDR that will be used for pods. This field is immutable. |
serviceCIDR string | ServiceCIDR defines the CIDR that will be used for services. This field is immutable. |
ipFamilies []IPFamily | (Optional) IPFamilies specifies the IP protocol versions to use for shoot networking. This field is immutable. See https://github.com/gardener/gardener/blob/master/docs/usage/ipv6.md |
NetworkStatus
(Appears on: Network)
NetworkStatus is the status for an Network resource.
Field | Description |
---|---|
DefaultStatus DefaultStatus | (Members of DefaultStatus is a structure containing common fields used by all extension resources. |
NodeTemplate
(Appears on: WorkerPool)
NodeTemplate contains information about the expected node properties.
Field | Description |
---|---|
capacity Kubernetes core/v1.ResourceList | Capacity represents the expected Node capacity. |
Object
Object is an extension object resource.
OperatingSystemConfigPurpose
(string
alias)
(Appears on: OperatingSystemConfigSpec)
OperatingSystemConfigPurpose is a string alias.
OperatingSystemConfigSpec
(Appears on: OperatingSystemConfig)
OperatingSystemConfigSpec is the spec for a OperatingSystemConfig resource.
Field | Description |
---|---|
criConfig CRIConfig | (Optional) CRI config is a structure contains configurations of the CRI library |
DefaultSpec DefaultSpec | (Members of DefaultSpec is a structure containing common fields used by all extension resources. |
purpose OperatingSystemConfigPurpose | Purpose describes how the result of this OperatingSystemConfig is used by Gardener. Either it
gets sent to the |
reloadConfigFilePath string | (Optional) ReloadConfigFilePath is the path to the generated operating system configuration. If set, controllers are asked to use it when determining the .status.command of this resource. For example, if for CoreOS the reload-path might be “/var/lib/config”; then the controller shall set .status.command to “/usr/bin/coreos-cloudinit –from-file=/var/lib/config”. |
units []Unit | (Optional) Units is a list of unit for the operating system configuration (usually, a systemd unit). |
files []File | (Optional) Files is a list of files that should get written to the host’s file system. |
OperatingSystemConfigStatus
(Appears on: OperatingSystemConfig)
OperatingSystemConfigStatus is the status for a OperatingSystemConfig resource.
Field | Description |
---|---|
DefaultStatus DefaultStatus | (Members of DefaultStatus is a structure containing common fields used by all extension resources. |
cloudConfig CloudConfig | (Optional) CloudConfig is a structure for containing the generated output for the given operating system config spec. It contains a reference to a secret as the result may contain confidential data. |
command string | (Optional) Command is the command whose execution renews/reloads the cloud config on an existing VM, e.g.
“/usr/bin/reload-cloud-config -from-file= |
units []string | (Optional) Units is a list of systemd unit names that are part of the generated Cloud Config and shall be restarted when a new version has been downloaded. |
Purpose
(string
alias)
(Appears on: ControlPlaneSpec)
Purpose is a string alias.
Spec
Spec is the spec section of an Object.
Status
Status is the status of an Object.
Unit
(Appears on: OperatingSystemConfigSpec)
Unit is a unit for the operating system configuration (usually, a systemd unit).
Field | Description |
---|---|
name string | Name is the name of a unit. |
command string | (Optional) Command is the unit’s command. |
enable bool | (Optional) Enable describes whether the unit is enabled or not. |
content string | (Optional) Content is the unit’s content. |
dropIns []DropIn | (Optional) DropIns is a list of drop-ins for this unit. |
Volume
(Appears on: WorkerPool)
Volume contains information about the root disks that should be used for worker pools.
Field | Description |
---|---|
name string | (Optional) Name of the volume to make it referencable. |
type string | (Optional) Type is the type of the volume. |
size string | Size is the of the root volume. |
encrypted bool | (Optional) Encrypted determines if the volume should be encrypted. |
WorkerPool
(Appears on: WorkerSpec)
WorkerPool is the definition of a specific worker pool.
Field | Description |
---|---|
machineType string | MachineType contains information about the machine type that should be used for this worker pool. |
maximum int32 | Maximum is the maximum size of the worker pool. |
maxSurge k8s.io/apimachinery/pkg/util/intstr.IntOrString | MaxSurge is maximum number of VMs that are created during an update. |
maxUnavailable k8s.io/apimachinery/pkg/util/intstr.IntOrString | MaxUnavailable is the maximum number of VMs that can be unavailable during an update. |
annotations map[string]string | (Optional) Annotations is a map of key/value pairs for annotations for all the |
labels map[string]string | (Optional) Labels is a map of key/value pairs for labels for all the |
taints []Kubernetes core/v1.Taint | (Optional) Taints is a list of taints for all the |
machineImage MachineImage | MachineImage contains logical information about the name and the version of the machie image that should be used. The logical information must be mapped to the provider-specific information (e.g., AMIs, …) by the provider itself. |
minimum int32 | Minimum is the minimum size of the worker pool. |
name string | Name is the name of this worker pool. |
providerConfig k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) ProviderConfig is a provider specific configuration for the worker pool. |
userData []byte | UserData is a base64-encoded string that contains the data that is sent to the provider’s APIs when a new machine/VM that is part of this worker pool shall be spawned. |
volume Volume | (Optional) Volume contains information about the root disks that should be used for this worker pool. |
dataVolumes []DataVolume | (Optional) DataVolumes contains a list of additional worker volumes. |
kubeletDataVolumeName string | (Optional) KubeletDataVolumeName contains the name of a dataVolume that should be used for storing kubelet state. |
zones []string | (Optional) Zones contains information about availability zones for this worker pool. |
machineControllerManager github.com/gardener/gardener/pkg/apis/core/v1beta1.MachineControllerManagerSettings | (Optional) MachineControllerManagerSettings contains configurations for different worker-pools. Eg. MachineDrainTimeout, MachineHealthTimeout. |
kubernetesVersion string | (Optional) KubernetesVersion is the kubernetes version in this worker pool |
nodeTemplate NodeTemplate | (Optional) NodeTemplate contains resource information of the machine which is used by Cluster Autoscaler to generate nodeTemplate during scaling a nodeGroup from zero |
architecture string | (Optional) Architecture is the CPU architecture of the worker pool machines and machine image. |
WorkerSpec
(Appears on: Worker)
WorkerSpec is the spec for a Worker resource.
Field | Description |
---|---|
DefaultSpec DefaultSpec | (Members of DefaultSpec is a structure containing common fields used by all extension resources. |
infrastructureProviderStatus k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) InfrastructureProviderStatus is a raw extension field that contains the provider status that has
been generated by the controller responsible for the |
region string | Region is the name of the region where the worker pool should be deployed to. This field is immutable. |
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a secret that contains the cloud provider specific credentials. |
sshPublicKey []byte | (Optional) SSHPublicKey is the public SSH key that should be used with these workers. |
pools []WorkerPool | Pools is a list of worker pools. |
WorkerStatus
(Appears on: Worker)
WorkerStatus is the status for a Worker resource.
Field | Description |
---|---|
DefaultStatus DefaultStatus | (Members of DefaultStatus is a structure containing common fields used by all extension resources. |
machineDeployments []MachineDeployment | MachineDeployments is a list of created machine deployments. It will be used to e.g. configure the cluster-autoscaler properly. |
Generated with gen-crd-api-reference-docs
1.1.4 - Operations
Packages:
operations.gardener.cloud/v1alpha1
Package v1alpha1 is a version of the API.
Resource Types:Bastion
Bastion holds details about an SSH bastion for a shoot cluster.
Field | Description | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion string | operations.gardener.cloud/v1alpha1 | ||||||||||
kind string | Bastion | ||||||||||
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||||||
spec BastionSpec | Specification of the Bastion.
| ||||||||||
status BastionStatus | (Optional) Most recently observed status of the Bastion. |
BastionIngressPolicy
(Appears on: BastionSpec)
BastionIngressPolicy represents an ingress policy for SSH bastion hosts.
Field | Description |
---|---|
ipBlock Kubernetes networking/v1.IPBlock | IPBlock defines an IP block that is allowed to access the bastion. |
BastionSpec
(Appears on: Bastion)
BastionSpec is the specification of a Bastion.
Field | Description |
---|---|
shootRef Kubernetes core/v1.LocalObjectReference | ShootRef defines the target shoot for a Bastion. The name field of the ShootRef is immutable. |
seedName string | (Optional) SeedName is the name of the seed to which this Bastion is currently scheduled. This field is populated at the beginning of a create/reconcile operation. |
providerType string | (Optional) ProviderType is cloud provider used by the referenced Shoot. |
sshPublicKey string | SSHPublicKey is the user’s public key. This field is immutable. |
ingress []BastionIngressPolicy | Ingress controls from where the created bastion host should be reachable. |
BastionStatus
(Appears on: Bastion)
BastionStatus holds the most recently observed status of the Bastion.
Field | Description |
---|---|
ingress Kubernetes core/v1.LoadBalancerIngress | (Optional) Ingress holds the public IP and/or hostname of the bastion instance. |
conditions []github.com/gardener/gardener/pkg/apis/core/v1alpha1.Condition | (Optional) Conditions represents the latest available observations of a Bastion’s current state. |
lastHeartbeatTimestamp Kubernetes meta/v1.Time | (Optional) LastHeartbeatTimestamp is the time when the bastion was last marked as not to be deleted. When this is set, the ExpirationTimestamp is advanced as well. |
expirationTimestamp Kubernetes meta/v1.Time | (Optional) ExpirationTimestamp is the time after which a Bastion is supposed to be garbage collected. |
observedGeneration int64 | (Optional) ObservedGeneration is the most recent generation observed for this Bastion. It corresponds to the Bastion’s generation, which is updated on mutation by the API Server. |
Generated with gen-crd-api-reference-docs
1.1.5 - Operator
Packages:
operator.gardener.cloud/v1alpha1
Package v1alpha1 contains the configuration of the Gardener Operator.
Resource Types:Backup
(Appears on: ETCDMain)
Backup contains the object store configuration for backups for the virtual garden etcd.
Field | Description |
---|---|
provider string | Provider is a provider name. This field is immutable. |
bucketName string | BucketName is the name of the backup bucket. |
secretRef Kubernetes core/v1.SecretReference | SecretRef is a reference to a Secret object containing the cloud provider credentials for the object store where backups should be stored. It should have enough privileges to manipulate the objects as well as buckets. |
ControlPlane
(Appears on: VirtualCluster)
ControlPlane holds information about the general settings for the control plane of the virtual garden cluster.
Field | Description |
---|---|
highAvailability HighAvailability | (Optional) HighAvailability holds the configuration settings for high availability settings. |
Credentials
(Appears on: GardenStatus)
Credentials contains information about the virtual garden cluster credentials.
Field | Description |
---|---|
rotation CredentialsRotation | (Optional) Rotation contains information about the credential rotations. |
CredentialsRotation
(Appears on: Credentials)
CredentialsRotation contains information about the rotation of credentials.
Field | Description |
---|---|
certificateAuthorities github.com/gardener/gardener/pkg/apis/core/v1beta1.CARotation | (Optional) CertificateAuthorities contains information about the certificate authority credential rotation. |
ETCD
(Appears on: VirtualCluster)
ETCD contains configuration for the etcds of the virtual garden cluster.
Field | Description |
---|---|
main ETCDMain | (Optional) Main contains configuration for the main etcd. |
events ETCDEvents | (Optional) Events contains configuration for the events etcd. |
ETCDEvents
(Appears on: ETCD)
ETCDEvents contains configuration for the events etcd.
Field | Description |
---|---|
storage Storage | (Optional) Storage contains storage configuration. |
ETCDMain
(Appears on: ETCD)
ETCDMain contains configuration for the main etcd.
Field | Description |
---|---|
backup Backup | (Optional) Backup contains the object store configuration for backups for the virtual garden etcd. |
storage Storage | (Optional) Storage contains storage configuration. |
Garden
Garden describes a list of gardens.
Field | Description | ||||
---|---|---|---|---|---|
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||
spec GardenSpec | Spec contains the specification of this garden.
| ||||
status GardenStatus | Status contains the status of this garden. |
GardenSpec
(Appears on: Garden)
GardenSpec contains the specification of a garden environment.
Field | Description |
---|---|
runtimeCluster RuntimeCluster | RuntimeCluster contains configuration for the runtime cluster. |
virtualCluster VirtualCluster | VirtualCluster contains configuration for the virtual cluster. |
GardenStatus
(Appears on: Garden)
GardenStatus is the status of a garden environment.
Field | Description |
---|---|
gardener github.com/gardener/gardener/pkg/apis/core/v1beta1.Gardener | (Optional) Gardener holds information about the Gardener which last acted on the Garden. |
conditions []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition | Conditions is a list of conditions. |
observedGeneration int64 | ObservedGeneration is the most recent generation observed for this resource. |
credentials Credentials | (Optional) Credentials contains information about the virtual garden cluster credentials. |
HighAvailability
(Appears on: ControlPlane)
HighAvailability specifies the configuration settings for high availability for a resource.
Maintenance
(Appears on: VirtualCluster)
Maintenance contains information about the time window for maintenance operations.
Field | Description |
---|---|
timeWindow github.com/gardener/gardener/pkg/apis/core/v1beta1.MaintenanceTimeWindow | TimeWindow contains information about the time window for maintenance operations. |
Provider
(Appears on: RuntimeCluster)
Provider defines the provider-specific information for this cluster.
Field | Description |
---|---|
zones []string | (Optional) Zones is the list of availability zones the cluster is deployed to. |
RuntimeCluster
(Appears on: GardenSpec)
RuntimeCluster contains configuration for the runtime cluster.
Field | Description |
---|---|
provider Provider | Provider defines the provider-specific information for this cluster. |
settings Settings | (Optional) Settings contains certain settings for this cluster. |
SettingLoadBalancerServices
(Appears on: Settings)
SettingLoadBalancerServices controls certain settings for services of type load balancer that are created in the runtime cluster.
Field | Description |
---|---|
annotations map[string]string | (Optional) Annotations is a map of annotations that will be injected/merged into every load balancer service object. |
SettingVerticalPodAutoscaler
(Appears on: Settings)
SettingVerticalPodAutoscaler controls certain settings for the vertical pod autoscaler components deployed in the seed.
Field | Description |
---|---|
enabled bool | (Optional) Enabled controls whether the VPA components shall be deployed into this cluster. It is true by default because the operator (and Gardener) heavily rely on a VPA being deployed. You should only disable this if your runtime cluster already has another, manually/custom managed VPA deployment. If this is not the case, but you still disable it, then reconciliation will fail. |
Settings
(Appears on: RuntimeCluster)
Settings contains certain settings for this cluster.
Field | Description |
---|---|
loadBalancerServices SettingLoadBalancerServices | (Optional) LoadBalancerServices controls certain settings for services of type load balancer that are created in the runtime cluster. |
verticalPodAutoscaler SettingVerticalPodAutoscaler | (Optional) VerticalPodAutoscaler controls certain settings for the vertical pod autoscaler components deployed in the cluster. |
Storage
(Appears on: ETCDEvents, ETCDMain)
Storage contains storage configuration.
Field | Description |
---|---|
capacity k8s.io/apimachinery/pkg/api/resource.Quantity | (Optional) Capacity is the storage capacity for the volumes. |
className string | (Optional) ClassName is the name of a storage class. |
VirtualCluster
(Appears on: GardenSpec)
VirtualCluster contains configuration for the virtual cluster.
Field | Description |
---|---|
controlPlane ControlPlane | (Optional) ControlPlane holds information about the general settings for the control plane of the virtual cluster. |
etcd ETCD | (Optional) ETCD contains configuration for the etcds of the virtual garden cluster. |
maintenance Maintenance | Maintenance contains information about the time window for maintenance operations. |
Generated with gen-crd-api-reference-docs
1.1.6 - Provider Local
Packages:
local.provider.extensions.gardener.cloud/v1alpha1
Package v1alpha1 contains the local provider API resources.
Resource Types:CloudProfileConfig
CloudProfileConfig contains provider-specific configuration that is embedded into Gardener’s CloudProfile
resource.
Field | Description |
---|---|
apiVersion string | local.provider.extensions.gardener.cloud/v1alpha1 |
kind string | CloudProfileConfig |
machineImages []MachineImages | MachineImages is the list of machine images that are understood by the controller. It maps logical names and versions to provider-specific identifiers. |
WorkerStatus
WorkerStatus contains information about created worker resources.
Field | Description |
---|---|
apiVersion string | local.provider.extensions.gardener.cloud/v1alpha1 |
kind string | WorkerStatus |
machineImages []MachineImage | (Optional) MachineImages is a list of machine images that have been used in this worker. Usually, the extension controller
gets the mapping from name/version to the provider-specific machine image data from the CloudProfile. However, if
a version that is still in use gets removed from this componentconfig it cannot reconcile anymore existing |
MachineImage
(Appears on: WorkerStatus)
MachineImage is a mapping from logical names and versions to provider-specific machine image data.
Field | Description |
---|---|
name string | Name is the logical name of the machine image. |
version string | Version is the logical version of the machine image. |
image string | Image is the image for the machine image. |
MachineImageVersion
(Appears on: MachineImages)
MachineImageVersion contains a version and a provider-specific identifier.
Field | Description |
---|---|
version string | Version is the version of the image. |
image string | Image is the image for the machine image. |
MachineImages
(Appears on: CloudProfileConfig)
MachineImages is a mapping from logical names and versions to provider-specific identifiers.
Field | Description |
---|---|
name string | Name is the logical name of the machine image. |
versions []MachineImageVersion | Versions contains versions and a provider-specific identifier. |
Generated with gen-crd-api-reference-docs
1.1.7 - Resources
Packages:
resources.gardener.cloud/v1alpha1
Package v1alpha1 contains the configuration of the Gardener Resource Manager.
Resource Types:ManagedResource
ManagedResource describes a list of managed resources.
Field | Description | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||||||||||||
spec ManagedResourceSpec | Spec contains the specification of this managed resource.
| ||||||||||||||||
status ManagedResourceStatus | Status contains the status of this managed resource. |
ManagedResourceSpec
(Appears on: ManagedResource)
ManagedResourceSpec contains the specification of this managed resource.
Field | Description |
---|---|
class string | (Optional) Class holds the resource class used to control the responsibility for multiple resource manager instances |
secretRefs []Kubernetes core/v1.LocalObjectReference | SecretRefs is a list of secret references. |
injectLabels map[string]string | (Optional) InjectLabels injects the provided labels into every resource that is part of the referenced secrets. |
forceOverwriteLabels bool | (Optional) ForceOverwriteLabels specifies that all existing labels should be overwritten. Defaults to false. |
forceOverwriteAnnotations bool | (Optional) ForceOverwriteAnnotations specifies that all existing annotations should be overwritten. Defaults to false. |
keepObjects bool | (Optional) KeepObjects specifies whether the objects should be kept although the managed resource has already been deleted. Defaults to false. |
equivalences [][]k8s.io/apimachinery/pkg/apis/meta/v1.GroupKind | (Optional) Equivalences specifies possible group/kind equivalences for objects. |
deletePersistentVolumeClaims bool | (Optional) DeletePersistentVolumeClaims specifies if PersistentVolumeClaims created by StatefulSets, which are managed by this resource, should also be deleted when the corresponding StatefulSet is deleted (defaults to false). |
ManagedResourceStatus
(Appears on: ManagedResource)
ManagedResourceStatus is the status of a managed resource.
Field | Description |
---|---|
conditions []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition | |
observedGeneration int64 | ObservedGeneration is the most recent generation observed for this resource. |
resources []ObjectReference | (Optional) Resources is a list of objects that have been created. |
secretsDataChecksum string | (Optional) SecretsDataChecksum is the checksum of referenced secrets data. |
ObjectReference
(Appears on: ManagedResourceStatus)
ObjectReference is a reference to another object.
Field | Description |
---|---|
ObjectReference Kubernetes core/v1.ObjectReference | (Members of |
labels map[string]string | Labels is a map of labels that were used during last update of the resource. |
annotations map[string]string | Annotations is a map of annotations that were used during last update of the resource. |
Generated with gen-crd-api-reference-docs
1.1.8 - Seedmanagement
Packages:
seedmanagement.gardener.cloud/v1alpha1
Package v1alpha1 is a version of the API.
Resource Types:ManagedSeed
ManagedSeed represents a Shoot that is registered as Seed.
Field | Description | ||||
---|---|---|---|---|---|
apiVersion string | seedmanagement.gardener.cloud/v1alpha1 | ||||
kind string | ManagedSeed | ||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional) Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||
spec ManagedSeedSpec | (Optional) Specification of the ManagedSeed.
| ||||
status ManagedSeedStatus | (Optional) Most recently observed status of the ManagedSeed. |
ManagedSeedSet
ManagedSeedSet represents a set of identical ManagedSeeds.
Field | Description | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion string | seedmanagement.gardener.cloud/v1alpha1 | ||||||||||||
kind string | ManagedSeedSet | ||||||||||||
metadata Kubernetes meta/v1.ObjectMeta | (Optional) Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||||||||
spec ManagedSeedSetSpec | (Optional) Spec defines the desired identities of ManagedSeeds and Shoots in this set.
| ||||||||||||
status ManagedSeedSetStatus | (Optional) Status is the current status of ManagedSeeds and Shoots in this ManagedSeedSet. |
Bootstrap
(string
alias)
(Appears on: Gardenlet)
Bootstrap describes a mechanism for bootstrapping gardenlet connection to the Garden cluster.
Gardenlet
(Appears on: ManagedSeedSpec)
Gardenlet specifies gardenlet deployment parameters and the GardenletConfiguration used to configure gardenlet.
Field | Description |
---|---|
deployment GardenletDeployment | (Optional) Deployment specifies certain gardenlet deployment parameters, such as the number of replicas, the image, etc. |
config k8s.io/apimachinery/pkg/runtime.RawExtension | (Optional) Config is the GardenletConfiguration used to configure gardenlet. |
bootstrap Bootstrap | (Optional) Bootstrap is the mechanism that should be used for bootstrapping gardenlet connection to the Garden cluster. One of ServiceAccount, BootstrapToken, None. If set to ServiceAccount or BootstrapToken, a service account or a bootstrap token will be created in the garden cluster and used to compute the bootstrap kubeconfig. If set to None, the gardenClientConnection.kubeconfig field will be used to connect to the Garden cluster. Defaults to BootstrapToken. This field is immutable. |
mergeWithParent bool | (Optional) MergeWithParent specifies whether the GardenletConfiguration of the parent gardenlet should be merged with the specified GardenletConfiguration. Defaults to true. This field is immutable. |
GardenletDeployment
(Appears on: Gardenlet)
GardenletDeployment specifies certain gardenlet deployment parameters, such as the number of replicas, the image, etc.
Field | Description |
---|---|
replicaCount int32 | (Optional) ReplicaCount is the number of gardenlet replicas. Defaults to 2. |
revisionHistoryLimit int32 | (Optional) RevisionHistoryLimit is the number of old gardenlet ReplicaSets to retain to allow rollback. Defaults to 2. |
serviceAccountName string | (Optional) ServiceAccountName is the name of the ServiceAccount to use to run gardenlet pods. |
image Image | (Optional) Image is the gardenlet container image. |
resources Kubernetes core/v1.ResourceRequirements | (Optional) Resources are the compute resources required by the gardenlet container. |
podLabels map[string]string | (Optional) PodLabels are the labels on gardenlet pods. |
podAnnotations map[string]string | (Optional) PodAnnotations are the annotations on gardenlet pods. |
additionalVolumes []Kubernetes core/v1.Volume | (Optional) AdditionalVolumes is the list of additional volumes that should be mounted by gardenlet containers. |
additionalVolumeMounts []Kubernetes core/v1.VolumeMount | (Optional) AdditionalVolumeMounts is the list of additional pod volumes to mount into the gardenlet container’s filesystem. |
env []Kubernetes core/v1.EnvVar | (Optional) Env is the list of environment variables to set in the gardenlet container. |
vpa bool | (Optional) VPA specifies whether to enable VPA for gardenlet. Defaults to true. |
Image
(Appears on: GardenletDeployment)
Image specifies container image parameters.
Field | Description |
---|---|
repository string | (Optional) Repository is the image repository. |
tag string | (Optional) Tag is the image tag. |
pullPolicy Kubernetes core/v1.PullPolicy | (Optional) PullPolicy is the image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if latest tag is specified, or IfNotPresent otherwise. |
ManagedSeedSetSpec
(Appears on: ManagedSeedSet)
ManagedSeedSetSpec is the specification of a ManagedSeedSet.
Field | Description |
---|---|
replicas int32 | (Optional) Replicas is the desired number of replicas of the given Template. Defaults to 1. |
selector Kubernetes meta/v1.LabelSelector | Selector is a label query over ManagedSeeds and Shoots that should match the replica count. It must match the ManagedSeeds and Shoots template’s labels. This field is immutable. |
template ManagedSeedTemplate | Template describes the ManagedSeed that will be created if insufficient replicas are detected. Each ManagedSeed created / updated by the ManagedSeedSet will fulfill this template. |
shootTemplate github.com/gardener/gardener/pkg/apis/core/v1beta1.ShootTemplate | ShootTemplate describes the Shoot that will be created if insufficient replicas are detected for hosting the corresponding ManagedSeed. Each Shoot created / updated by the ManagedSeedSet will fulfill this template. |
updateStrategy UpdateStrategy | (Optional) UpdateStrategy specifies the UpdateStrategy that will be employed to update ManagedSeeds / Shoots in the ManagedSeedSet when a revision is made to Template / ShootTemplate. |
revisionHistoryLimit int32 | (Optional) RevisionHistoryLimit is the maximum number of revisions that will be maintained in the ManagedSeedSet’s revision history. Defaults to 10. This field is immutable. |
ManagedSeedSetStatus
(Appears on: ManagedSeedSet)
ManagedSeedSetStatus represents the current state of a ManagedSeedSet.
Field | Description |
---|---|
observedGeneration int64 | ObservedGeneration is the most recent generation observed for this ManagedSeedSet. It corresponds to the ManagedSeedSet’s generation, which is updated on mutation by the API Server. |
replicas int32 | Replicas is the number of replicas (ManagedSeeds and their corresponding Shoots) created by the ManagedSeedSet controller. |
readyReplicas int32 | ReadyReplicas is the number of ManagedSeeds created by the ManagedSeedSet controller that have a Ready Condition. |
nextReplicaNumber int32 | NextReplicaNumber is the ordinal number that will be assigned to the next replica of the ManagedSeedSet. |
currentReplicas int32 | CurrentReplicas is the number of ManagedSeeds created by the ManagedSeedSet controller from the ManagedSeedSet version indicated by CurrentRevision. |
updatedReplicas int32 | UpdatedReplicas is the number of ManagedSeeds created by the ManagedSeedSet controller from the ManagedSeedSet version indicated by UpdateRevision. |
currentRevision string | CurrentRevision, if not empty, indicates the version of the ManagedSeedSet used to generate ManagedSeeds with smaller ordinal numbers during updates. |
updateRevision string | UpdateRevision, if not empty, indicates the version of the ManagedSeedSet used to generate ManagedSeeds with larger ordinal numbers during updates |
collisionCount int32 | (Optional) CollisionCount is the count of hash collisions for the ManagedSeedSet. The ManagedSeedSet controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest ControllerRevision. |
conditions []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition | (Optional) Conditions represents the latest available observations of a ManagedSeedSet’s current state. |
pendingReplica PendingReplica | (Optional) PendingReplica, if not empty, indicates the replica that is currently pending creation, update, or deletion. This replica is in a state that requires the controller to wait for it to change before advancing to the next replica. |
ManagedSeedSpec
(Appears on: ManagedSeed, ManagedSeedTemplate)
ManagedSeedSpec is the specification of a ManagedSeed.
Field | Description |
---|---|
shoot Shoot | (Optional) Shoot references a Shoot that should be registered as Seed. This field is immutable. |
gardenlet Gardenlet | (Optional) Gardenlet specifies that the ManagedSeed controller should deploy a gardenlet into the cluster with the given deployment parameters and GardenletConfiguration. |
ManagedSeedStatus
(Appears on: ManagedSeed)
ManagedSeedStatus is the status of a ManagedSeed.
Field | Description |
---|---|
conditions []github.com/gardener/gardener/pkg/apis/core/v1beta1.Condition | (Optional) Conditions represents the latest available observations of a ManagedSeed’s current state. |
observedGeneration int64 | ObservedGeneration is the most recent generation observed for this ManagedSeed. It corresponds to the ManagedSeed’s generation, which is updated on mutation by the API Server. |
ManagedSeedTemplate
(Appears on: ManagedSeedSetSpec)
ManagedSeedTemplate is a template for creating a ManagedSeed object.
Field | Description | ||||
---|---|---|---|---|---|
metadata Kubernetes meta/v1.ObjectMeta | (Optional) Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||
spec ManagedSeedSpec | (Optional) Specification of the desired behavior of the ManagedSeed.
|
PendingReplica
(Appears on: ManagedSeedSetStatus)
PendingReplica contains information about a replica that is currently pending creation, update, or deletion.
Field | Description |
---|---|
name string | Name is the replica name. |
reason PendingReplicaReason | Reason is the reason for the replica to be pending. |
since Kubernetes meta/v1.Time | Since is the moment in time since the replica is pending with the specified reason. |
retries int32 | (Optional) Retries is the number of times the shoot operation (reconcile or delete) has been retried after having failed. Only applicable if Reason is ShootReconciling or ShootDeleting. |
PendingReplicaReason
(string
alias)
(Appears on: PendingReplica)
PendingReplicaReason is a string enumeration type that enumerates all possible reasons for a replica to be pending.
RollingUpdateStrategy
(Appears on: UpdateStrategy)
RollingUpdateStrategy is used to communicate parameters for RollingUpdateStrategyType.
Field | Description |
---|---|
partition int32 | (Optional) Partition indicates the ordinal at which the ManagedSeedSet should be partitioned. Defaults to 0. |
Shoot
(Appears on: ManagedSeedSpec)
Shoot identifies the Shoot that should be registered as Seed.
Field | Description |
---|---|
name string | Name is the name of the Shoot that will be registered as Seed. |
UpdateStrategy
(Appears on: ManagedSeedSetSpec)
UpdateStrategy specifies the strategy that the ManagedSeedSet controller will use to perform updates. It includes any additional parameters necessary to perform the update for the indicated strategy.
Field | Description |
---|---|
type UpdateStrategyType | (Optional) Type indicates the type of the UpdateStrategy. Defaults to RollingUpdate. |
rollingUpdate RollingUpdateStrategy | (Optional) RollingUpdate is used to communicate parameters when Type is RollingUpdateStrategyType. |
UpdateStrategyType
(string
alias)
(Appears on: UpdateStrategy)
UpdateStrategyType is a string enumeration type that enumerates all possible update strategies for the ManagedSeedSet controller.
Generated with gen-crd-api-reference-docs
1.1.9 - Settings
Packages:
settings.gardener.cloud/v1alpha1
Package v1alpha1 is a version of the API.
Resource Types:ClusterOpenIDConnectPreset
ClusterOpenIDConnectPreset is a OpenID Connect configuration that is applied to a Shoot objects cluster-wide.
Field | Description | ||||
---|---|---|---|---|---|
apiVersion string | settings.gardener.cloud/v1alpha1 | ||||
kind string | ClusterOpenIDConnectPreset | ||||
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||
spec ClusterOpenIDConnectPresetSpec | Spec is the specification of this OpenIDConnect preset.
|
OpenIDConnectPreset
OpenIDConnectPreset is a OpenID Connect configuration that is applied to a Shoot in a namespace.
Field | Description | ||||||||
---|---|---|---|---|---|---|---|---|---|
apiVersion string | settings.gardener.cloud/v1alpha1 | ||||||||
kind string | OpenIDConnectPreset | ||||||||
metadata Kubernetes meta/v1.ObjectMeta | Standard object metadata. Refer to the Kubernetes API documentation for the fields of themetadata field. | ||||||||
spec OpenIDConnectPresetSpec | Spec is the specification of this OpenIDConnect preset.
|
ClusterOpenIDConnectPresetSpec
(Appears on: ClusterOpenIDConnectPreset)
ClusterOpenIDConnectPresetSpec contains the OpenIDConnect specification and project selector matching Shoots in Projects.
Field | Description |
---|---|
OpenIDConnectPresetSpec OpenIDConnectPresetSpec | (Members of |
projectSelector Kubernetes meta/v1.LabelSelector | (Optional) Project decides whether to apply the configuration if the Shoot is in a specific Project matching the label selector. Use the selector only if the OIDC Preset is opt-in, because end users may skip the admission by setting the labels. Defaults to the empty LabelSelector, which matches everything. |
KubeAPIServerOpenIDConnect
(Appears on: OpenIDConnectPresetSpec)
KubeAPIServerOpenIDConnect contains configuration settings for the OIDC provider. Note: Descriptions were taken from the Kubernetes documentation.
Field | Description |
---|---|
caBundle string | (Optional) If set, the OpenID server’s certificate will be verified by one of the authorities in the oidc-ca-file, otherwise the host’s root CA set will be used. |
clientID string | The client ID for the OpenID Connect client. Required. |
groupsClaim string | (Optional) If provided, the name of a custom OpenID Connect claim for specifying user groups. The claim value is expected to be a string or array of strings. This field is experimental, please see the authentication documentation for further details. |
groupsPrefix string | (Optional) If provided, all groups will be prefixed with this value to prevent conflicts with other authentication strategies. |
issuerURL string | The URL of the OpenID issuer, only HTTPS scheme will be accepted. If set, it will be used to verify the OIDC JSON Web Token (JWT). Required. |
requiredClaims map[string]string | (Optional) key=value pairs that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value. |
signingAlgs []string | (Optional) List of allowed JOSE asymmetric signing algorithms. JWTs with a ‘alg’ header value not in this list will be rejected. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1 Defaults to [RS256] |
usernameClaim string | (Optional) The OpenID claim to use as the user name. Note that claims other than the default (‘sub’) is not guaranteed to be unique and immutable. This field is experimental, please see the authentication documentation for further details. Defaults to “sub”. |
usernamePrefix string | (Optional) If provided, all usernames will be prefixed with this value. If not provided, username claims other than ‘email’ are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value ‘-’. |
OpenIDConnectClientAuthentication
(Appears on: OpenIDConnectPresetSpec)
OpenIDConnectClientAuthentication contains configuration for OIDC clients.
Field | Description |
---|---|
secret string | (Optional) The client Secret for the OpenID Connect client. |
extraConfig map[string]string | (Optional) Extra configuration added to kubeconfig’s auth-provider. Must not be any of idp-issuer-url, client-id, client-secret, idp-certificate-authority, idp-certificate-authority-data, id-token or refresh-token |
OpenIDConnectPresetSpec
(Appears on: OpenIDConnectPreset, ClusterOpenIDConnectPresetSpec)
OpenIDConnectPresetSpec contains the Shoot selector for which a specific OpenID Connect configuration is applied.
Field | Description |
---|---|
server KubeAPIServerOpenIDConnect | Server contains the kube-apiserver’s OpenID Connect configuration. This configuration is not overwritting any existing OpenID Connect configuration already set on the Shoot object. |
client OpenIDConnectClientAuthentication | (Optional) Client contains the configuration used for client OIDC authentication of Shoot clusters. This configuration is not overwritting any existing OpenID Connect client authentication already set on the Shoot object. |
shootSelector Kubernetes meta/v1.LabelSelector | (Optional) ShootSelector decides whether to apply the configuration if the Shoot has matching labels. Use the selector only if the OIDC Preset is opt-in, because end users may skip the admission by setting the labels. Default to the empty LabelSelector, which matches everything. |
weight int32 | Weight associated with matching the corresponding preset, in the range 1-100. Required. |
Generated with gen-crd-api-reference-docs
1.2 - Concepts
1.2.1 - Admission Controller
Gardener Admission Controller
While the Gardener API server works with admission plugins to validate and mutate resources belonging to Gardener related API groups, e.g. core.gardener.cloud
, the same is needed for resources belonging to non-Gardener API groups as well, e.g. secrets in the core
API group.
Therefore, the Gardener Admission Controller runs a http(s) server with the following handlers which serve as validating/mutating endpoints for admission webhooks.
It is also used to serve http(s) handlers for authorization webhooks.
Admission Webhook Handlers
This section describes the admission webhook handlers that are currently served.
Kubeconfig Secret Validator
Malicious Kubeconfigs applied by end users may cause a leakage of sensitive data.
This handler checks if the incoming request contains a Kubernetes secret with a .data.kubeconfig
field and denies the request if the Kubeconfig structure violates Gardener’s security standards.
Namespace Validator
Namespaces are the backing entities of Gardener projects in which shoot cluster objects reside.
This validation handler protects active namespaces against premature deletion requests.
Therefore, it denies deletion requests if a namespace still contains shoot clusters or if it belongs to a non-deleting Gardener project (without .metadata.deletionTimestamp
).
Resource Size Validator
Since users directly apply Kubernetes native objects to the Garden cluster, it also involves the risk of being vulnerable to DoS attacks because these resources are continuously watched and read by controllers. One example is the creation of shoot resources with large annotation values (up to 256 kB per value), which can cause severe out-of-memory issues for the gardenlet component. Vertical autoscaling can help to mitigate such situations, but we cannot expect to scale infinitely, and thus need means to block the attack itself.
The Resource Size Validator checks arbitrary incoming admission requests against a configured maximum size for the resource’s group-version-kind combination. It denies the request if the object exceeds the quota.
Example for Gardener Admission Controller configuration:
server:
resourceAdmissionConfiguration:
limits:
- apiGroups: ["core.gardener.cloud"]
apiVersions: ["*"]
resources: ["shoots"]
size: 100k
- apiGroups: [""]
apiVersions: ["v1"]
resources: ["secrets"]
size: 100k
unrestrictedSubjects:
- kind: Group
name: gardener.cloud:system:seeds
apiGroup: rbac.authorization.k8s.io
# - kind: User
# name: admin
# apiGroup: rbac.authorization.k8s.io
# - kind: ServiceAccount
# name: "*"
# namespace: garden
# apiGroup: ""
operationMode: block #log
With the configuration above, the Resource Size Validator denies requests for shoots with Gardener’s core API group which exceed a size of 100 kB. The same is done for Kubernetes secrets.
As this feature is meant to protect the system from malicious requests sent by users, it is recommended to exclude trusted groups, users or service accounts from the size restriction via resourceAdmissionConfiguration.unrestrictedSubjects
.
For example, the backing user for the gardenlet should always be capable of changing the shoot resource instead of being blocked due to size restrictions.
This is because the gardenlet itself occasionally changes the shoot specification, labels or annotations, and might violate the quota if the existing resource is already close to the quota boundary.
Also, operators are supposed to be trusted users and subjecting them to a size limitation can inhibit important operational tasks.
Wildcard ("*") in subject name
is supported.
Size limitations depend on the individual Gardener setup and choosing the wrong values can affect the availability of your Gardener service.
resourceAdmissionConfiguration.operationMode
allows to control if a violating request is actually denied (default) or only logged.
It’s recommended to start with log
, check the logs for exceeding requests, adjust the limits if necessary and finally switch to block
.
SeedRestriction
Please refer to Scoped API Access for Gardenlets for more information.
Authorization Webhook Handlers
This section describes the authorization webhook handlers that are currently served.
SeedAuthorization
Please refer to Scoped API Access for Gardenlets for more information.
1.2.2 - API Server
Gardener API Server
The Gardener API server is a Kubernetes-native extension based on its aggregation layer.
It is registered via an APIService
object and designed to run inside a Kubernetes cluster whose API it wants to extend.
After registration, it exposes the following resources:
CloudProfile
s
CloudProfile
s are resources that describe a specific environment of an underlying infrastructure provider, e.g. AWS, Azure, etc.
Each shoot has to reference a CloudProfile
to declare the environment it should be created in.
In a CloudProfile
, the gardener operator specifies certain constraints like available machine types, regions, which Kubernetes versions they want to offer, etc.
End-users can read CloudProfile
s to see these values, but only operators can change the content or create/delete them.
When a shoot is created or updated, then an admission plugin checks that only allowed values are used via the referenced CloudProfile
.
Additionally, a CloudProfile
may contain a providerConfig
, which is a special configuration dedicated for the infrastructure provider.
Gardener does not evaluate or understand this config, but extension controllers might need it for declaration of provider-specific constraints, or global settings.
Please see this example manifest and consult the documentation of your provider extension controller to get information about its providerConfig
.
Seed
s
Seed
s are resources that represent seed clusters.
Gardener does not care about how a seed cluster got created - the only requirement is that it is of at least Kubernetes v1.20 and passes the Kubernetes conformance tests.
The Gardener operator has to either deploy the gardenlet into the cluster they want to use as seed (recommended, then the gardenlet will create the Seed
object itself after bootstrapping) or provide the kubeconfig to the cluster inside a secret (that is referenced by the Seed
resource) and create the Seed
resource themselves.
Please see this, this, and optionally this example manifests.
ShootQuota
s
To allow end-users not having their dedicated infrastructure account to try out Gardener, the operator can register an account owned by them that they allow to be used for trial clusters. Trial clusters can be put under quota so that they don’t consume too many resources (resulting in costs) and that one user cannot consume all resources on their own. These clusters are automatically terminated after a specified time, but end-users may extend the lifetime manually if needed.
Please see this example manifest.
Project
s
The first thing before creating a shoot cluster is to create a Project
.
A project is used to group multiple shoot clusters together.
End-users can invite colleagues to the project to enable collaboration, and they can either make them admin
or viewer
.
After an end-user has created a project, they will get a dedicated namespace in the garden cluster for all their shoots.
Please see this example manifest.
SecretBinding
s
Now that the end-user has a namespace the next step is registering their infrastructure provider account.
Please see this example manifest and consult the documentation of the extension controller for the respective infrastructure provider to get information about which keys are required in this secret.
After the secret has been created, the end-user has to create a special SecretBinding
resource that binds this secret.
Later, when creating shoot clusters, they will reference such binding.
Please see this example manifest.
Shoot
s
Shoot cluster contain various settings that influence how end-user Kubernetes clusters will look like in the end. As Gardener heavily relies on extension controllers for operating system configuration, networking, and infrastructure specifics, the end-user has the possibility (and responsibility) to provide these provider-specific configurations as well. Such configurations are not evaluated by Gardener (because it doesn’t know/understand them), but they are only transported to the respective extension controller.
⚠️ This means that any configuration issues/mistake on the end-user side that relates to a provider-specific flag or setting cannot be caught during the update request itself but only later during the reconciliation (unless a validator webhook has been registered in the garden cluster by an operator).
Please see this example manifest and consult the documentation of the provider extension controller to get information about its spec.provider.controlPlaneConfig
, .spec.provider.infrastructureConfig
, and .spec.provider.workers[].providerConfig
.
(Cluster)OpenIDConnectPreset
s
Please see this separate documentation file.
Overview Data Model
1.2.3 - APIServer Admission Plugins
Admission Plugins
Similar to the kube-apiserver, the gardener-apiserver comes with a few in-tree managed admission plugins. If you want to get an overview of the what and why of admission plugins then this document might be a good start.
This document lists all existing admission plugins with a short explanation of what it is responsible for.
ClusterOpenIDConnectPreset
, OpenIDConnectPreset
(both enabled by default)
These admission controllers react on CREATE
operations for Shoot
s.
If the Shoot
does not specify any OIDC configuration (.spec.kubernetes.kubeAPIServer.oidcConfig=nil
), then it tries to find a matching ClusterOpenIDConnectPreset
or OpenIDConnectPreset
, respectively.
If there are multiple matches, then the one with the highest weight “wins”.
In this case, the admission controller will default the OIDC configuration in the Shoot
.
ControllerRegistrationResources
(enabled by default)
This admission controller reacts on CREATE
and UPDATE
operations for ControllerRegistration
s.
It validates that there exists only one ControllerRegistration
in the system that is primarily responsible for a given kind/type resource combination.
This prevents misconfiguration by the Gardener administrator/operator.
CustomVerbAuthorizer
(enabled by default)
This admission controller reacts on CREATE
and UPDATE
operations for Project
s.
It validates whether the user is bound to a RBAC role with the modify-spec-tolerations-whitelist
verb in case the user tries to change the .spec.tolerations.whitelist
field of the respective Project
resource.
Usually, regular project members are not bound to this custom verb, allowing the Gardener administrator to manage certain toleration whitelists on Project
basis.
DeletionConfirmation
(enabled by default)
This admission controller reacts on DELETE
operations for Project
s and Shoot
s and ShootState
s.
It validates that the respective resource is annotated with a deletion confirmation annotation, namely confirmation.gardener.cloud/deletion=true
.
Only if this annotation is present it allows the DELETE
operation to pass.
This prevents users from accidental/undesired deletions.
ExposureClass
(enabled by default)
This admission controller reacts on Create
operations for Shoots
s.
It mutates Shoot
resources which have an ExposureClass
referenced by merging both their shootSelectors
and/or tolerations
into the Shoot
resource.
ExtensionValidator
(enabled by default)
This admission controller reacts on CREATE
and UPDATE
operations for BackupEntry
s, BackupBucket
s, Seed
s, and Shoot
s.
For all the various extension types in the specifications of these objects, it validates whether there exists a ControllerRegistration
in the system that is primarily responsible for the stated extension type(s).
This prevents misconfigurations that would otherwise allow users to create such resources with extension types that don’t exist in the cluster, effectively leading to failing reconciliation loops.
ExtensionLabels
(enabled by default)
This admission controller reacts on CREATE
and UPDATE
operations for BackupBucket
s, BackupEntry
s, CloudProfile
s, Seed
s, SecretBinding
s and Shoot
s. For all the various extension types in the specifications of these objects, it adds a corresponding label in the resource. This would allow extension admission webhooks to filter out the resources they are responsible for and ignore all others. This label is of the form <extension-type>.extensions.gardener.cloud/<extension-name> : "true"
. For example, an extension label for provider extension type aws
, looks like provider.extensions.gardener.cloud/aws : "true"
.
ProjectValidator
(enabled by default)
This admission controller reacts on CREATE
operations for Project
s.
It prevents creating Project
s with a non-empty .spec.namespace
if the value in .spec.namespace
does not start with garden-
.
⚠️ This admission plugin will be removed in a future release and its business logic will be incorporated into the static validation of the gardener-apiserver
.
ResourceQuota
(enabled by default)
This admission controller enables object count ResourceQuotas for Gardener resources, e.g. Shoots
, SecretBindings
, Projects
, etc.
⚠️ In addition to this admission plugin, the ResourceQuota controller must be enabled for the Kube-Controller-Manager of your Garden cluster.
ResourceReferenceManager
(enabled by default)
This admission controller reacts on CREATE
and UPDATE
operations for CloudProfile
s, Project
s, SecretBinding
s, Seed
s, and Shoot
s.
Generally, it checks whether referred resources stated in the specifications of these objects exist in the system (e.g., if a referenced Secret
exists).
However, it also has some special behaviours for certain resources:
CloudProfile
s: It rejects removing Kubernetes or machine image versions if there is at least oneShoot
that refers to them.Project
s: It sets the.spec.createdBy
field for newly createdProject
resources, and defaults the.spec.owner
field in case it is empty (to the same value of.spec.createdBy
).Shoot
s: It sets thegardener.cloud/created-by=<username>
annotation for newly createdShoot
resources.
SeedValidator
(enabled by default)
This admission controller reacts on DELETE
operations for Seed
s.
Rejects the deletion if Shoot
(s) reference the seed cluster.
ShootDNS
(enabled by default)
This admission controller reacts on CREATE
and UPDATE
operations for Shoot
s.
It tries to assign a default domain to the Shoot
.
It also validates the DNS configuration (.spec.dns
) for shoots.
ShootNodeLocalDNSEnabledByDefault
(disabled by default)
This admission controller reacts on CREATE
operations for Shoot
s.
If enabled, it will enable node local dns within the shoot cluster (for more information, see NodeLocalDNS Configuration) by setting spec.systemComponents.nodeLocalDNS.enabled=true
for newly created Shoots.
Already existing Shoots and new Shoots that explicitly disable node local dns (spec.systemComponents.nodeLocalDNS.enabled=false
)
will not be affected by this admission plugin.
ShootQuotaValidator
(enabled by default)
This admission controller reacts on CREATE
and UPDATE
operations for Shoot
s.
It validates the resource consumption declared in the specification against applicable Quota
resources.
Only if the applicable Quota
resources admit the configured resources in the Shoot
then it allows the request.
Applicable Quota
s are referred in the SecretBinding
that is used by the Shoot
.
ShootVPAEnabledByDefault
(disabled by default)
This admission controller reacts on CREATE
operations for Shoot
s.
If enabled, it will enable the managed VerticalPodAutoscaler
components (for more information, see Vertical Pod Auto-Scaling)
by setting spec.kubernetes.verticalPodAutoscaler.enabled=true
for newly created Shoots.
Already existing Shoots and new Shoots that explicitly disable VPA (spec.kubernetes.verticalPodAutoscaler.enabled=false
)
will not be affected by this admission plugin.
ShootTolerationRestriction
(enabled by default)
This admission controller reacts on CREATE
and UPDATE
operations for Shoot
s.
It validates the .spec.tolerations
used in Shoot
s against the whitelist of its Project
, or against the whitelist configured in the admission controller’s configuration, respectively.
Additionally, it defaults the .spec.tolerations
in Shoot
s with those configured in its Project
, and those configured in the admission controller’s configuration, respectively.
ShootValidator
(enabled by default)
This admission controller reacts on CREATE
, UPDATE
and DELETE
operations for Shoot
s.
It validates certain configurations in the specification against the referred CloudProfile
(e.g., machine images, machine types, used Kubernetes version, …).
Generally, it performs validations that cannot be handled by the static API validation due to their dynamic nature (e.g., when something needs to be checked against referred resources).
Additionally, it takes over certain defaulting tasks (e.g., default machine image for worker pools).
ShootManagedSeed
(enabled by default)
This admission controller reacts on UPDATE
and DELETE
operations for Shoot
s.
It validates certain configuration values in the specification that are specific to ManagedSeed
s (e.g. the nginx-addon of the Shoot has to be disabled, the Shoot VPA has to be enabled).
It rejects the deletion if the Shoot
is referred to by a ManagedSeed
.
ManagedSeedValidator
(enabled by default)
This admission controller reacts on CREATE
and UPDATE
operations for ManagedSeeds
s.
It validates certain configuration values in the specification against the referred Shoot
, for example Seed provider, network ranges, DNS domain, etc.
Similarly to ShootValidator
, it performs validations that cannot be handled by the static API validation due to their dynamic nature.
Additionally, it performs certain defaulting tasks, making sure that configuration values that are not specified are defaulted to the values of the referred Shoot
, for example Seed provider, network ranges, DNS domain, etc.
ManagedSeedShoot
(enabled by default)
This admission controller reacts on DELETE
operations for ManagedSeed
s.
It rejects the deletion if there are Shoot
s that are scheduled onto the Seed
that is registered by the ManagedSeed
.
ShootDNSRewriting
(disabled by default)
This admission controller reacts on CREATE
operations for Shoot
s.
If enabled, it adds a set of common suffixes configured in its admission plugin configuration to the Shoot
(spec.systemComponents.coreDNS.rewriting.commonSuffixes
) (for more information, see DNS Search Path Optimization).
Already existing Shoot
s will not be affected by this admission plugin.
1.2.4 - Architecture
Official Definition - What is Kubernetes?
“Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.”
Introduction - Basic Principle
The foundation of the Gardener (providing Kubernetes Clusters as a Service) is Kubernetes itself, because Kubernetes is the go-to solution to manage software in the Cloud, even when it’s Kubernetes itself (see also OpenStack which is provisioned more and more on top of Kubernetes as well).
While self-hosting, meaning to run Kubernetes components inside Kubernetes, is a popular topic in the community, we apply a special pattern catering to the needs of our cloud platform to provision hundreds or even thousands of clusters. We take a so-called “seed” cluster and seed the control plane (such as the API server, scheduler, controllers, etcd persistence and others) of an end-user cluster, which we call “shoot” cluster, as pods into the “seed” cluster. That means that one “seed” cluster, of which we will have one per IaaS and region, hosts the control planes of multiple “shoot” clusters. That allows us to avoid dedicated hardware/virtual machines for the “shoot” cluster control planes. We simply put the control plane into pods/containers and since the “seed” cluster watches them, they can be deployed with a replica count of 1 and only need to be scaled out when the control plane gets under pressure, but no longer for HA reasons. At the same time, the deployments get simpler (standard Kubernetes deployment) and easier to update (standard Kubernetes rolling update). The actual “shoot” cluster consists only of the worker nodes (no control plane) and therefore the users may get full administrative access to their clusters.
Setting The Scene - Components and Procedure
We provide a central operator UI, which we call the “Gardener Dashboard”. It talks to a dedicated cluster, which we call the “Garden” cluster and uses custom resources managed by an aggregated API server, one of the general extension concepts of Kubernetes) to represent “shoot” clusters. In this “Garden” cluster runs the “Gardener”, which is basically a Kubernetes controller that watches the custom resources and acts upon them, i.e. creates, updates/modifies, or deletes “shoot” clusters. The creation follows basically these steps:
- Create a namespace in the “seed” cluster for the “shoot” cluster, which will host the “shoot” cluster control plane.
- Generate secrets and credentials, which the worker nodes will need to talk to the control plane.
- Create the infrastructure (using Terraform), which basically consists out of the network setup).
- Deploy the “shoot” cluster control plane into the “shoot” namespace in the “seed” cluster, containing the “machine-controller-manager” pod.
- Create machine CRDs in the “seed” cluster, describing the configuration and the number of worker machines for the “shoot” (the machine-controller-manager watches the CRDs and creates virtual machines out of it).
- Wait for the “shoot” cluster API server to become responsive (pods will be scheduled, persistent volumes and load balancers are created by Kubernetes via the respective cloud provider).
- Finally, we deploy
kube-system
daemons likekube-proxy
and further add-ons like thedashboard
into the “shoot” cluster and the cluster becomes active.
Overview Architecture Diagram
Detailed Architecture Diagram
Note: The kubelet
, as well as the pods inside the “shoot” cluster, talks through the front-door (load balancer IP; public Internet) to its “shoot” cluster API server running in the “seed” cluster. The reverse communication from the API server to the pod, service, and node networks happens through a VPN connection that we deploy into the “seed” and “shoot” clusters.
1.2.5 - Backup and Restore
Backup and Restore
Kubernetes uses etcd as the key-value store for its resource definitions. Gardener supports the backup and restore of etcd. It is the responsibility of the shoot owners to backup the workload data.
Gardener uses an etcd-backup-restore component to backup the etcd backing the Shoot cluster regularly and restore it in case of disaster. It is deployed as sidecar via etcd-druid. This doc mainly focuses on the backup and restore configuration used by Gardener when deploying these components. For more details on the design and internal implementation details, please refer to GEP-06 and the documentation on individual repositories.
Bucket Provisioning
Refer to the backup bucket extension document to find out details about configuring the backup bucket.
Backup Policy
etcd-backup-restore supports full snapshot and delta snapshots over full snapshot. In Gardener, this configuration is currently hard-coded to the following parameters:
- Full Snapshot schedule:
- Daily,
24hr
interval. - For each Shoot, the schedule time in a day is randomized based on the configured Shoot maintenance window.
- Daily,
- Delta Snapshot schedule:
- At
5min
interval. - If aggregated events size since last snapshot goes beyond
100Mib
.
- At
- Backup History / Garbage backup deletion policy:
- Gardener configures backup restore to have
Exponential
garbage collection policy. - As per policy, the following backups are retained:
- All full backups and delta backups for the previous hour.
- Latest full snapshot of each previous hour for the day.
- Latest full snapshot of each previous day for 7 days.
- Latest full snapshot of the previous 4 weeks.
- Garbage Collection is configured at
12hr
interval.
- Gardener configures backup restore to have
- Listing:
- Gardener doesn’t have any API to list out the backups.
- To find the backups list, an admin can checkout the
BackupEntry
resource associated with the Shoot which holds the bucket and prefix details on the object store.
Restoration
The restoration process of etcd is automated through the etcd-backup-restore component from the latest snapshot. Gardener doesn’t support Point-In-Time-Recovery (PITR) of etcd. In case of an etcd disaster, the etcd is recovered from the latest backup automatically. For further details, please refer the Restoration topic. Post restoration of etcd, the Shoot reconciliation loop brings the cluster back to its previous state.
Again, the Shoot owner is responsible for maintaining the backup/restore of his workload. Gardener only takes care of the cluster’s etcd.
1.2.6 - Cluster API
Relation Between Gardener API and Cluster API (SIG Cluster Lifecycle)
In essence, the Cluster API harmonizes how to get to clusters, while Gardener goes one step further and also harmonizes the clusters themselves. The Cluster API delegates the specifics to so-called providers for infrastructures or control planes via specific CR(D)s, while Gardener only has one cluster CR(D). Different Cluster API providers, e.g. for AWS, Azure, GCP, etc., give you vastly different Kubernetes clusters. In contrast, Gardener gives you the exact same clusters with the exact same K8s version, operating system, control plane configuration like for API server or kubelet, add-ons like overlay network, HPA/VPA, DNS and certificate controllers, ingress and network policy controllers, control plane monitoring and logging stacks, down to the behavior of update procedures, auto-scaling, self-healing, etc., on all supported infrastructures. These homogeneous clusters are an essential goal for Gardener, as its main purpose is to simplify operations for teams that need to develop and ship software on Kubernetes clusters on a plethora of infrastructures (a.k.a. multi-cloud).
Incidentally, Gardener influenced the Machine API in the Cluster API with its Machine Controller Manager and was the first to adopt it. You can find more information on that in the joint SIG Cluster Lifecycle KubeCon talk where @hardikdr from our Gardener team in India spoke.
That means that we follow the Cluster API with great interest and are active members. It was completely overhauled from v1alpha1
to v1alpha2
. But because v1alpha2
made too many assumptions about the bring-up of masters and was enforcing master machine operations (for more information, see The Cluster API Book: “As of v1alpha2
, Machine-Based is the only control plane type that Cluster API supports”), services that managed their control planes differently like GKE or Gardener couldn’t adopt it (e.g. Google only supports v1alpha1
). In 2020 v1alpha3
was introduced and made it possible (again) to integrate managed services like GKE or Gardener. The mapping from the Gardener API to the Cluster API is mostly syntactic.
To wrap it up, while the Cluster API knows about clusters, it doesn’t know about their make-up. With Gardener, we wanted to go beyond that and harmonize the make-up of the clusters themselves and make them homogeneous across all supported infrastructures. Gardener can therefore deliver homogeneous clusters with exactly the same configuration and behavior on all infrastructures (see also Gardener’s coverage in the official conformance test grid).
With Cluster API v1alpha3
and the support for declarative control plane management, it has became possible (again) to enable Kubernetes managed services like GKE or Gardener. We would be more than happy if the community would be interested to contribute a Gardener control plane provider.
1.2.7 - Controller Manager
Gardener Controller Manager
The gardener-controller-manager
(often refered to as “GCM”) is a component that runs next to the Gardener API server, similar to the Kubernetes Controller Manager.
It runs several controllers that do not require talking to any seed or shoot cluster.
Also, as of today, it exposes an HTTP server that is serving several health check endpoints and metrics.
This document explains the various functionalities of the gardener-controller-manager
and their purpose.
Controllers
Bastion
Controller
Bastion
resources have a limited lifetime which can be extended up to a certain amount by performing a heartbeat on them.
The Bastion
controller is responsible for deleting expired or rotten Bastion
s.
- “expired” means a
Bastion
has exceeded itsstatus.expirationTimestamp
. - “rotten” means a
Bastion
is older than the configuredmaxLifetime
.
The maxLifetime
defaults to 24 hours and is an option in the BastionControllerConfiguration
which is part of gardener-controller-manager
s ControllerManagerControllerConfiguration
, see the example config file for details.
The controller also deletes Bastion
s in case the referenced Shoot
:
- no longer exists
- is marked for deletion (i.e., have a non-
nil
.metadata.deletionTimestamp
) - was migrated to another seed (i.e.,
Shoot.spec.seedName
is different thanBastion.spec.seedName
).
The deletion of Bastion
s triggers the gardenlet
to perform the necessary cleanups in the Seed cluster, so some time can pass between deletion and the Bastion
actually disappearing.
Clients like gardenctl
are advised to not re-use Bastion
s whose deletion timestamp has been set already.
Refer to GEP-15 for more information on the lifecycle of
Bastion
resources.
CertificateSigningRequest
Controller
After the gardenlet gets deployed on the Seed cluster, it needs to establish itself as a trusted party to communicate with the Gardener API server. It runs through a bootstrap flow similar to the kubelet bootstrap process.
On startup, the gardenlet uses a kubeconfig
with a bootstrap token which authenticates it as being part of the system:bootstrappers
group. This kubeconfig is used to create a CertificateSigningRequest
(CSR) against the Gardener API server.
The controller in gardener-controller-manager
checks whether the CertificateSigningRequest
has the expected organisation, common name and usages which the gardenlet would request.
It only auto-approves the CSR if the client making the request is allowed to “create” the
certificatesigningrequests/seedclient
subresource. Clients with the system:bootstrappers
group are bound to the gardener.cloud:system:seed-bootstrapper
ClusterRole
, hence, they have such privileges. As the bootstrap kubeconfig for the gardenlet contains a bootstrap token which is authenticated as being part of the systems:bootstrappers
group, its created CSR gets auto-approved.
CloudProfile
Controller
CloudProfile
s are essential when it comes to reconciling Shoot
s since they contain constraints (like valid machine types, Kubernetes versions, or machine images) and sometimes also some global configuration for the respective environment (typically via provider-specific configuration in .spec.providerConfig
).
Consequently, to ensure that CloudProfile
s in-use are always present in the system until the last referring Shoot
gets deleted, the controller adds a finalizer which is only released when there is no Shoot
referencing the CloudProfile
anymore.
ControllerDeployment
Controller
Extensions are registered in the garden cluster via ControllerRegistration
and deployment of respective extensions are specified via ControllerDeployment
. For more info refer to Registering Extension Controllers.
This controller ensures that ControllerDeployment
in-use always exists until the last ControllerRegistration
referencing them gets deleted. The controller adds a finalizer which is only released when there is no ControllerRegistration
referencing the ControllerDeployment
anymore.
ControllerRegistration
Controller
The ControllerRegistration
controller makes sure that the required Gardener Extensions specified by the ControllerRegistration
resources are present in the seed clusters.
It also takes care of the creation and deletion of ControllerInstallation
objects for a given seed cluster.
The controller has three reconciliation loops.
“Main” Reconciler
This reconciliation loop watches the Seed
objects and determines which ControllerRegistrations
are required for them and reconciles the corresponding ControllerInstallation
resources to reach the determined state.
To begin with, it computes the kind/type combinations of extensions required for the seed.
For this, the controller examines a live list of ControllerRegistration
s, ControllerInstallation
s, BackupBucket
s, BackupEntry
s, Shoot
s, and Secret
s from the garden cluster.
For example, it examines the shoots running on the seed and deducts the kind/type, like Infrastructure/gcp
.
It also decides whether they should always be deployed based on the .spec.deployment.policy
.
For the configuration options, please see this section.
Based on these required combinations, each of them are mapped to ControllerRegistration
objects and then to their corresponding ControllerInstallation
objects (if existing).
The controller then creates or updates the required ControllerInstallation
objects for the given seed.
It also deletes every existing ControllerInstallation
whose referenced ControllerRegistration
is not part of the required list.
For example, if the shoots in the seed are no longer using the DNS provider aws-route53
, then the controller proceeds to delete the respective ControllerInstallation
object.
“ControllerRegistration” Reconciler
This reconciliation loop watches the ControllerRegistration
resource and adds finalizers to it when they are created.
In case a deletion request comes in for the resource, i.e., if a .metadata.deletionTimestamp
is set, it actively scans for a ControllerInstallation
resource using this ControllerRegistration
, and decides whether the deletion can be allowed.
In case no related ControllerInstallation
is present, it removes the finalizer and marks it for deletion.
“Seed” Reconciler
This loop also watches the Seed
object and adds finalizers to it at creation.
If a .metadata.deletionTimestamp
is set for the seed, then the controller checks for existing ControllerInstallation
objects which reference this seed.
If no such objects exist, then it removes the finalizer and allows the deletion.
Event
Controller
With the Gardener Event Controller, you can prolong the lifespan of events related to Shoot clusters. This is an optional controller which will become active once you provide the below mentioned configuration.
All events in K8s are deleted after a configurable time-to-live (controlled via a kube-apiserver argument called --event-ttl
(defaulting to 1 hour)).
The need to prolong the time-to-live for Shoot cluster events frequently arises when debugging customer issues on live systems.
This controller leaves events involving Shoots untouched, while deleting all other events after a configured time.
In order to activate it, provide the following configuration:
concurrentSyncs
: The amount of goroutines scheduled for reconciling events.ttlNonShootEvents
: When an event reaches this time-to-live it gets deleted unless it is a Shoot-related event (defaults to1h
, equivalent to theevent-ttl
default).
⚠️ In addition, you should also configure the
--event-ttl
for the kube-apiserver to define an upper-limit of how long Shoot-related events should be stored. The--event-ttl
should be larger than thettlNonShootEvents
or this controller will have no effect.
ExposureClass
Controller
ExposureClass
abstracts the ability to expose a Shoot clusters control plane in certain network environments (e.g. corporate networks, DMZ, internet) on all Seeds or a subset of the Seeds. For more information, see ExposureClasses.
Consequently, to ensure that ExposureClass
es in-use are always present in the system until the last referring Shoot
gets deleted, the controller adds a finalizer which is only released when there is no Shoot
referencing the ExposureClass
anymore.
ManagedSeedSet
Controller
ManagedSeedSet
objects maintain a stable set of replicas of ManagedSeed
s, i.e. they guarantee the availability of a specified number of identical ManagedSeed
s on an equal number of identical Shoot
s.
The ManagedSeedSet
controller creates and deletes ManagedSeed
s and Shoot
s in response to changes to the replicas and selector fields. For more information, refer to the ManagedSeedSet
proposal document.
- The reconciler first gets all the replicas of the given
ManagedSeedSet
in theManagedSeedSet
’s namespace and with the matching selector. Each replica is a struct that contains aManagedSeed
, its correspondingSeed
andShoot
objects. - Then the pending replica is retrieved, if it exists.
- Next it determines the ready, postponed, and deletable replicas.
- A replica is considered
ready
when aSeed
owned by aManagedSeed
has been registered either directly or by deployinggardenlet
into aShoot
, theSeed
isReady
and theShoot
’s status isHealthy
. - If a replica is not ready and it is not pending, i.e. it is not specified in the
ManagedSeed
’sstatus.pendingReplica
field, then it is added to thepostponed
replicas. - A replica is deletable if it has no scheduled
Shoot
s and the replica’sShoot
andManagedSeed
do not have theseedmanagement.gardener.cloud/protect-from-deletion
annotation.
- A replica is considered
- Finally, it checks the actual and target replica counts. If the actual count is less than the target count, the controller scales up the replicas by creating new replicas to match the desired target count. If the actual count is more than the target, the controller deletes replicas to match the desired count. Before scale-out or scale-in, the controller first reconciles the pending replica (there can always only be one) and makes sure the replica is ready before moving on to the next one.
Scale-out
(actual count < target count)- During the scale-out phase, the controller first creates the
Shoot
object from theManagedSeedSet
’sspec.shootTemplate
field and adds the replica to thestatus.pendingReplica
of theManagedSeedSet
. - For the subsequent reconciliation steps, the controller makes sure that the pending replica is ready before proceeding to the next replica. Once the
Shoot
is created successfully, theManagedSeed
object is created from theManagedSeedSet
’sspec.template
. TheManagedSeed
object is reconciled by theManagedSeed
controller and aSeed
object is created for the replica. Once the replica’sSeed
becomes ready and theShoot
becomes healthy, the replica also becomes ready.
- During the scale-out phase, the controller first creates the
Scale-in
(actual count > target count)- During the scale-in phase, the controller first determines the replica that can be deleted. From the deletable replicas, it chooses the one with the lowest priority and deletes it. Priority is determined in the following order:
- First, compare replica statuses. Replicas with “less advanced” status are considered lower priority. For example, a replica with
StatusShootReconciling
status has a lower value than a replica withStatusShootReconciled
status. Hence, in this case, a replica with aStatusShootReconciling
status will have lower priority and will be considered for deletion. - Then, the replicas are compared with the readiness of their
Seed
s. Replicas with non-readySeed
s are considered lower priority. - Then, the replicas are compared with the health statuses of their
Shoot
s. Replicas with “worse” statuses are considered lower priority. - Finally, the replica ordinals are compared. Replicas with lower ordinals are considered lower priority.
- First, compare replica statuses. Replicas with “less advanced” status are considered lower priority. For example, a replica with
- During the scale-in phase, the controller first determines the replica that can be deleted. From the deletable replicas, it chooses the one with the lowest priority and deletes it. Priority is determined in the following order:
Quota
Controller
Quota
object limits the resources consumed by shoot clusters either per provider secret or per project/namespace.
Consequently, to ensure that Quota
s in-use are always present in the system until the last SecretBinding
that references them gets deleted, the controller adds a finalizer which is only released when there is no SecretBinding
referencing the Quota
anymore.
Project
Controller
There are multiple controllers responsible for different aspects of Project
objects.
Please also refer to the Project
documentation.
“Main” Reconciler
This reconciler manages a dedicated Namespace
for each Project
.
The namespace name can either be specified explicitly in .spec.namespace
(must be prefixed with garden-
) or it will be determined by the controller.
If .spec.namespace
is set, it tries to create it. If it already exists, it tries to adopt it.
This will only succeed if the Namespace
was previously labeled with gardener.cloud/role=project
and project.gardener.cloud/name=<project-name>
.
This is to prevent end-users from being able to adopt arbitrary namespaces and escalate their privileges, e.g. the kube-system
namespace.
After the namespace was created/adopted, the controller creates several ClusterRole
s and ClusterRoleBinding
s that allow the project members to access related resources based on their roles.
These RBAC resources are prefixed with gardener.cloud:system:project{-member,-viewer}:<project-name>
.
Gardener administrators and extension developers can define their own roles. For more information, see Extending Project Roles for more information.
In addition, operators can configure the Project controller to maintain a default ResourceQuota for project namespaces.
Quotas can especially limit the creation of user facing resources, e.g. Shoots
, SecretBindings
, Secrets
and thus protect the Garden cluster from massive resource exhaustion but also enable operators to align quotas with respective enterprise policies.
⚠️ Gardener itself is not exempted from configured quotas. For example, Gardener creates
Secrets
for every shoot cluster in the project namespace and at the same time increases the available quota count. Please mind this additional resource consumption.
The controller configuration provides a template section controllers.project.quotas
where such a ResourceQuota (see the example below) can be deposited.
controllers:
project:
quotas:
- config:
apiVersion: v1
kind: ResourceQuota
spec:
hard:
count/shoots.core.gardener.cloud: "100"
count/secretbindings.core.gardener.cloud: "10"
count/secrets: "800"
projectSelector: {}
The Project controller takes the specified config
and creates a ResourceQuota
with the name gardener
in the project namespace.
If a ResourceQuota
resource with the name gardener
already exists, the controller will only update fields in spec.hard
which are unavailable at that time.
This is done to configure a default Quota
in all projects but to allow manual quota increases as the projects’ demands increase.
spec.hard
fields in the ResourceQuota
object that are not present in the configuration are removed from the object.
Labels and annotations on the ResourceQuota
config
get merged with the respective fields on existing ResourceQuota
s.
An optional projectSelector
narrows down the amount of projects that are equipped with the given config
.
If multiple configs match for a project, then only the first match in the list is applied to the project namespace.
The .status.phase
of the Project
resources is set to Ready
or Failed
by the reconciler to indicate whether the reconciliation loop was performed successfully.
Also, it generates Event
s to provide further information about its operations.
When a Project
is marked for deletion, the controller ensures that there are no Shoots
left in the project namespace.
Once all Shoots
are gone, the Namespace
and Project
are released.
“Stale Projects” Reconciler
As Gardener is a large-scale Kubernetes as a Service, it is designed for being used by a large amount of end-users.
Over time, it is likely to happen that some of the hundreds or thousands of Project
resources are no longer actively used.
Gardener offers the “stale projects” reconciler which will take care of identifying such stale projects, marking them with a “warning”, and eventually deleting them after a certain time period. This reconciler is enabled by default and works as follows:
- Projects are considered as “stale”/not actively used when all of the following conditions apply: The namespace associated with the
Project
does not have any…Shoot
resources.BackupEntry
resources.Secret
resources that are referenced by aSecretBinding
that is in use by aShoot
(not necessarily in the same namespace).Quota
resources that are referenced by aSecretBinding
that is in use by aShoot
(not necessarily in the same namespace).- The time period when the project was used for the last time (
status.lastActivityTimestamp
) is longer than the configuredminimumLifetimeDays
If a project is considered “stale”, then its .status.staleSinceTimestamp
will be set to the time when it was first detected to be stale.
If it gets actively used again, this timestamp will be removed.
After some time, the .status.staleAutoDeleteTimestamp
will be set to a timestamp after which Gardener will auto-delete the Project
resource if it still is not actively used.
The component configuration of the gardener-controller-manager
offers to configure the following options:
minimumLifetimeDays
: Don’t consider newly createdProject
s as “stale” too early to give people/end-users some time to onboard and get familiar with the system. The “stale project” reconciler won’t set any timestamp forProject
s younger thanminimumLifetimeDays
. When you change this value, then projects marked as “stale” may be no longer marked as “stale” in case they are young enough, or vice versa.staleGracePeriodDays
: Don’t compute auto-delete timestamps for staleProject
s that are unused for less thanstaleGracePeriodDays
. This is to not unnecessarily make people/end-users nervous “just because” they haven’t actively used theirProject
for a given amount of time. When you change this value, then already assigned auto-delete timestamps may be removed if the new grace period is not yet exceeded.staleExpirationTimeDays
: Expiration time after which staleProject
s are finally auto-deleted (after.status.staleSinceTimestamp
). If this value is changed and an auto-delete timestamp got already assigned to the projects, then the new value will only take effect if it’s increased. Hence, decreasing thestaleExpirationTimeDays
will not decrease already assigned auto-delete timestamps.
Gardener administrators/operators can exclude specific
Project
s from the stale check by annotating the relatedNamespace
resource withproject.gardener.cloud/skip-stale-check=true
.
“Activity” Reconciler
Since the other two reconcilers are unable to actively monitor the relevant objects that are used in a Project
(Shoot
, Secret
, etc.), there could be a situation where the user creates and deletes objects in a short period of time. In that case, the Stale Project Reconciler
could not see that there was any activity on that project and it will still mark it as a Stale
, even though it is actively used.
The Project Activity Reconciler
is implemented to take care of such cases. An event handler will notify the reconciler for any acitivity and then it will update the status.lastActivityTimestamp
. This update will also trigger the Stale Project Reconciler
.
SecretBinding
Controller
SecretBinding
s reference Secret
s and Quota
s and are themselves referenced by Shoot
s.
The controller adds finalizers to the referenced objects to ensure they don’t get deleted while still being referenced.
Similarly, to ensure that SecretBinding
s in-use are always present in the system until the last referring Shoot
gets deleted, the controller adds a finalizer which is only released when there is no Shoot
referencing the SecretBinding
anymore.
Referenced Secret
s will also be labeled with provider.shoot.gardener.cloud/<type>=true
, where <type>
is the value of the .provider.type
of the SecretBinding
.
Also, all referenced Secret
s, as well as Quota
s, will be labeled with reference.gardener.cloud/secretbinding=true
to allow for easily filtering for objects referenced by SecretBinding
s.
Seed
Controller
The Seed controller in the gardener-controller-manager
reconciles Seed
objects with the help of the following reconcilers.
“Main” Reconciler
This reconciliation loop takes care of seed related operations in the Garden cluster. When a new Seed
object is created,
the reconciler creates a new Namespace
in the garden cluster seed-<seed-name>
. Namespaces
dedicated to single
seed clusters allow us to segregate access permissions i.e., a gardenlet
must not have permissions to access objects in
all Namespaces
in the Garden cluster.
There are objects in a Garden environment which are created once by the operator e.g., default domain secret,
alerting credentials, and are required for operations happening in the gardenlet
. Therefore, we not only need a seed specific
Namespace
but also a copy of these “shared” objects.
The “main” reconciler takes care about this replication:
Kind | Namespace | Label Selector |
---|---|---|
Secret | garden | gardener.cloud/role |
“Backup Buckets Check” Reconciler
Every time a BackupBucket
object is created or updated, the referenced Seed
object is enqueued for reconciliation.
It’s the reconciler’s task to check the status
subresource of all existing BackupBucket
s that reference this Seed
.
If at least one BackupBucket
has .status.lastError != nil
, the BackupBucketsReady
condition on the Seed
will be set to False
, and consequently the Seed
is considered as NotReady
.
If the SeedBackupBucketsCheckControllerConfiguration
(which is part of gardener-controller-manager
s component configuration) contains a conditionThreshold
for the BackupBucketsReady
, the condition will instead first be set to Progressing
and eventually to False
once the conditionThreshold
expires. See the example config file for details.
Once the BackupBucket
is healthy again, the seed will be re-queued and the condition will turn true
.
“Extensions Check” Reconciler
This reconciler reconciles Seed
objects and checks whether all ControllerInstallation
s referencing them are in a healthy state.
Concretely, all three conditions Valid
, Installed
, and Healthy
must have status True
and the Progressing
condition must have status False
.
Based on this check, it maintains the ExtensionsReady
condition in the respective Seed
’s .status.conditions
list.
“Lifecycle” Reconciler
The “Lifecycle” reconciler processes Seed
objects which are enqueued every 10 seconds in order to check if the responsible
gardenlet
is still responding and operable. Therefore, it checks renewals via Lease
objects of the seed in the garden cluster
which are renewed regularly by the gardenlet
.
In case a Lease
is not renewed for the configured amount in config.controllers.seed.monitorPeriod.duration
:
- The reconciler assumes that the
gardenlet
stopped operating and updates theGardenletReady
condition toUnknown
. - Additionally, the conditions and constraints of all
Shoot
resources scheduled on the affected seed are set toUnknown
as well, because a strikinggardenlet
won’t be able to maintain these conditions any more. - If the gardenlet’s client certificate has expired (identified based on the
.status.clientCertificateExpirationTimestamp
field in theSeed
resource) and if it is managed by aManagedSeed
, then this will be triggered for a reconciliation. This will trigger the bootstrapping process again and allows gardenlets to obtain a fresh client certificate.
Shoot
Controller
“Conditions” Reconciler
In case the reconciled Shoot
is registered via a ManagedSeed
as a seed cluster, this reconciler merges the conditions in the respective Seed
’s .status.conditions
into the .status.conditions
of the Shoot
.
This is to provide a holistic view on the status of the registered seed cluster by just looking at the Shoot
resource.
“Hibernation” Reconciler
This reconciler is responsible for hibernating or awakening shoot clusters based on the schedules defined in their .spec.hibernation.schedules
.
It ignores failed Shoot
s and those marked for deletion.
“Maintenance” Reconciler
This reconciler is responsible for maintaining shoot clusters based on the time window defined in their .spec.maintenance.timeWindow
.
It might auto-update the Kubernetes version or the operating system versions specified in the worker pools (.spec.provider.workers
).
It could also add some operation or task annotations. For more information, see Shoot Maintenance.
“Quota” Reconciler
This reconciler might auto-delete shoot clusters in case their referenced SecretBinding
is itself referencing a Quota
with .spec.clusterLifetimeDays != nil
.
If the shoot cluster is older than the configured lifetime, then it gets deleted.
It maintains the expiration time of the Shoot
in the value of the shoot.gardener.cloud/expiration-timestamp
annotation.
This annotation might be overridden, however only by at most twice the value of the .spec.clusterLifetimeDays
.
“Reference” Reconciler
Shoot objects may specify references to other objects in the Garden cluster which are required for certain features.
For example, users can configure various DNS providers via .spec.dns.providers
and usually need to refer to a corresponding Secret
with valid DNS provider credentials inside.
Such objects need a special protection against deletion requests as long as they are still being referenced by one or multiple shoots.
Therefore, this reconciler checks Shoot
s for referenced objects and adds the finalizer gardener.cloud/reference-protection
to their .metadata.finalizers
list.
The reconciled Shoot
also gets this finalizer to enable a proper garbage collection in case the gardener-controller-manager
is offline at the moment of an incoming deletion request.
When an object is not actively referenced anymore because the Shoot
specification has changed or all related shoots were deleted (are in deletion), the controller will remove the added finalizer again so that the object can safely be deleted or garbage collected.
This reconciler inspects the following references:
- DNS provider secrets (
.spec.dns.provider
) - Audit policy configmaps (
.spec.kubernetes.kubeAPIServer.auditConfig.auditPolicy.configMapRef
)
Further checks might be added in the future.
“Retry” Reconciler
This reconciler is responsible for retrying certain failed Shoot
s.
Currently, the reconciler retries only failed Shoot
s with an error code ERR_INFRA_RATE_LIMITS_EXCEEDED
. See Shoot Status for more details.
“Status Label” Reconciler
This reconciler is responsible for maintaining the shoot.gardener.cloud/status
label on Shoot
s. See Shoot Status for more details.
1.2.8 - etcd
etcd - Key-Value Store for Kubernetes
etcd is a strongly consistent key-value store and the most prevalent choice for the Kubernetes
persistence layer. All API cluster objects like Pod
s, Deployment
s, Secret
s, etc., are stored in etcd
, which
makes it an essential part of a Kubernetes control plane.
Garden or Shoot Cluster Persistence
Each garden or shoot cluster gets its very own persistence for the control plane.
It runs in the shoot namespace on the respective seed cluster (or in the garden
namespace in the garden cluster, respectively).
Concretely, there are two etcd instances per shoot cluster, which the kube-apiserver
is configured to use in the following way:
etcd-main
A store that contains all “cluster critical” or “long-term” objects. These object kinds are typically considered for a backup to prevent any data loss.
etcd-events
A store that contains all Event
objects (events.k8s.io
) of a cluster.
Events
usually have a short retention period and occur frequently, but are not essential for a disaster recovery.
The setup above prevents both, the critical etcd-main
is not flooded by Kubernetes Events
, as well as backup space is not occupied by non-critical data.
This separation saves time and resources.
etcd Operator
Configuring, maintaining, and health-checking etcd is outsourced to a dedicated operator called etcd Druid.
When a gardenlet
reconciles a Shoot
resource or a gardener-operator
reconciles a Garden
resource, they manage an Etcd
resource in the seed or garden cluster, containing necessary information (backup information, defragmentation schedule, resources, etc.).
etcd-druid
needs to manage the lifecycle of the desired etcd instance (today main
or events
).
Likewise, when the Shoot
or Garden
is deleted, gardenlet
or gardener-operator
deletes the Etcd
resources and etcd Druid takes care of cleaning up all related objects, e.g. the backing StatefulSet
s.
Autoscaling
Gardenlet maintains HVPA
objects for etcd StatefulSet
s if the corresponding feature gate is enabled.
This enables a vertical scaling for etcd.
Downscaling is handled more pessimistically to prevent many subsequent etcd restarts.
Thus, for production
and infrastructure
shoot clusters (or all garden clusters), downscaling is deactivated for the main etcd.
For all other shoot clusters, lower advertised requests/limits are only applied during a shoot’s maintenance time window.
Backup
If Seed
s specify backups for etcd (example), then Gardener and the respective provider extensions are responsible for creating a bucket on the cloud provider’s side (modelled through a BackupBucket resource).
The bucket stores backups of Shoot
s scheduled on that Seed
.
Furthermore, Gardener creates a BackupEntry, which subdivides the bucket and thus makes it possible to store backups of multiple shoot clusters.
How long backups are stored in the bucket after a shoot has been deleted depends on the configured retention period in the Seed
resource.
Please see this example configuration for more information.
For Garden
s specifying backups for etcd (example), the bucket must be pre-created externally and provided via the Garden
specification.
Both etcd instances are configured to run with a special backup-restore sidecar. It takes care about regularly backing up etcd data and restoring it in case of data loss (in the main etcd only). The sidecar also performs defragmentation and other house-keeping tasks. More information can be found in the component’s GitHub repository.
Housekeeping
etcd maintenance tasks must be performed from time to time in order to re-gain database storage and to ensure the system’s reliability. The backup-restore sidecar takes care about this job as well.
For both Shoot
s and Garden
s, a random time within the shoot’s maintenance time is chosen for scheduling these tasks.
1.2.9 - gardenelet
gardenlet
Gardener is implemented using the operator pattern: It uses custom controllers that act on our own custom resources, and apply Kubernetes principles to manage clusters instead of containers. Following this analogy, you can recognize components of the Gardener architecture as well-known Kubernetes components, for example, shoot clusters can be compared with pods, and seed clusters can be seen as worker nodes.
The following Gardener components play a similar role as the corresponding components in the Kubernetes architecture:
Gardener Component | Kubernetes Component |
---|---|
gardener-apiserver | kube-apiserver |
gardener-controller-manager | kube-controller-manager |
gardener-scheduler | kube-scheduler |
gardenlet | kubelet |
Similar to how the kube-scheduler
of Kubernetes finds an appropriate node
for newly created pods, the gardener-scheduler
of Gardener finds an appropriate seed cluster
to host the control plane for newly ordered clusters.
By providing multiple seed clusters for a region or provider, and distributing the workload,
Gardener also reduces the blast radius of potential issues.
Kubernetes runs a primary “agent” on every node, the kubelet, which is responsible for managing pods and containers on its particular node. Decentralizing the responsibility to the kubelet has the advantage that the overall system is scalable. Gardener achieves the same for cluster management by using a gardenlet as а primary “agent” on every seed cluster, and is only responsible for shoot clusters located in its particular seed cluster:
The gardener-controller-manager
has controllers to manage resources of the Gardener API. However, instead of letting the gardener-controller-manager
talk directly to seed clusters or shoot clusters, the responsibility isn’t only delegated to the gardenlet, but also managed using a reversed control flow: It’s up to the gardenlet to contact the Gardener API server, for example, to share a status for its managed seed clusters.
Reversing the control flow allows placing seed clusters or shoot clusters behind firewalls without the necessity of direct access via VPN tunnels anymore.
TLS Bootstrapping
Kubernetes doesn’t manage worker nodes itself, and it’s also not responsible for the lifecycle of the kubelet running on the workers. Similarly, Gardener doesn’t manage seed clusters itself, so it is also not responsible for the lifecycle of the gardenlet running on the seeds. As a consequence, both the gardenlet and the kubelet need to prepare a trusted connection to the Gardener API server and the Kubernetes API server correspondingly.
To prepare a trusted connection between the gardenlet and the Gardener API server, the gardenlet initializes a bootstrapping process after you deployed it into your seed clusters:
The gardenlet starts up with a bootstrap
kubeconfig
having a bootstrap token that allows to createCertificateSigningRequest
(CSR) resources.After the CSR is signed, the gardenlet downloads the created client certificate, creates a new
kubeconfig
with it, and stores it inside aSecret
in the seed cluster.The gardenlet deletes the bootstrap
kubeconfig
secret, and starts up with its newkubeconfig
.The gardenlet starts normal operation.
The gardener-controller-manager
runs a control loop
that automatically signs CSRs created by gardenlets.
The gardenlet bootstrapping process is based on the kubelet bootstrapping process. More information: Kubelet’s TLS bootstrapping.
If you don’t want to run this bootstrap process, you can create
a kubeconfig
pointing to the garden cluster for the gardenlet yourself,
and use the field gardenClientConnection.kubeconfig
in the
gardenlet configuration to share it with the gardenlet.
gardenlet Certificate Rotation
The certificate used to authenticate the gardenlet against the API server
has a certain validity based on the configuration of the garden cluster
(--cluster-signing-duration
flag of the kube-controller-manager
(default 1y
)).
If your garden cluster is of at least Kubernetes v1.22, then you can also configure the validity for the client certificate by specifying
.gardenClientConnection.kubeconfigValidity.validity
in the gardenlet’s component configuration. Note that changing this value will only take effect when the kubeconfig is rotated again (it is not picked up immediately). The minimum validity is10m
(that’s what is enforced by theCertificateSigningRequest
API in Kubernetes which is used by the gardenlet).
By default, after about 70-90% of the validity has expired, the gardenlet tries to automatically replace the current certificate with a new one (certificate rotation).
You can change these boundaries by specifying
.gardenClientConnection.kubeconfigValidity.autoRotationJitterPercentage{Min,Max}
in the gardenlet’s component configuration.
To use a certificate rotation, you need to specify the secret to store
the kubeconfig
with the rotated certificate in the field
.gardenClientConnection.kubeconfigSecret
of the
gardenlet component configuration.
Rotate Certificates Using Bootstrap kubeconfig
If the gardenlet created the certificate during the initial TLS Bootstrapping
using the Bootstrap kubeconfig
, certificates can be rotated automatically.
The same control loop in the gardener-controller-manager
that signs
the CSRs during the initial TLS Bootstrapping also automatically signs
the CSR during a certificate rotation.
ℹ️ You can trigger an immediate renewal by annotating the Secret
in the seed
cluster stated in the .gardenClientConnection.kubeconfigSecret
field with
gardener.cloud/operation=renew
and restarting the gardenlet. After it has booted
up again, gardenlet will issue a new certificate independent of the remaining
validity of the existing one.
Rotate Certificates Using Custom kubeconfig
When trying to rotate a custom certificate that wasn’t created by gardenlet
as part of the TLS Bootstrap, the x509 certificate’s Subject
field
needs to conform to the following:
- the Common Name (CN) is prefixed with
gardener.cloud:system:seed:
- the Organization (O) equals
gardener.cloud:system:seeds
Otherwise, the gardener-controller-manager
doesn’t automatically
sign the CSR.
In this case, an external component or user needs to approve the CSR manually,
for example, using the command kubectl certificate approve seed-csr-<...>
).
If that doesn’t happen within 15 minutes,
the gardenlet repeats the process and creates another CSR.
Configuring the Seed to Work with gardenlet
The gardenlet works with a single seed, which must be configured in the
GardenletConfiguration
under .seedConfig
. This must be a copy of the
Seed
resource, for example:
apiVersion: gardenlet.config.gardener.cloud/v1alpha1
kind: GardenletConfiguration
seedConfig:
metadata:
name: my-seed
spec:
provider:
type: aws
# ...
secretRef:
name: my-seed-secret
namespace: garden
(see this yaml file for a more complete example)
When using make start-gardenlet
, the corresponding script will automatically
fetch the seed cluster’s kubeconfig
based on the seedConfig.spec.secretRef
and set the environment accordingly.
On startup, gardenlet registers a Seed
resource using the given template
in the seedConfig
if it’s not present already.
Component Configuration
In the component configuration for the gardenlet, it’s possible to define:
- settings for the Kubernetes clients interacting with the various clusters
- settings for the controllers inside the gardenlet
- settings for leader election and log levels, feature gates, and seed selection or seed configuration.
More information: Example gardenlet Component Configuration.
Heartbeats
Similar to how Kubernetes uses Lease
objects for node heart beats
(see KEP),
the gardenlet is using Lease
objects for heart beats of the seed cluster.
Every two seconds, the gardenlet checks that the seed cluster’s /healthz
endpoint returns HTTP status code 200.
If that is the case, the gardenlet renews the lease in the Garden cluster in the gardener-system-seed-lease
namespace and updates
the GardenletReady
condition in the status.conditions
field of the Seed
resource. For more information, see this section.
Similarly to the node-lifecycle-controller
inside the kube-controller-manager
,
the gardener-controller-manager
features a seed-lifecycle-controller
that sets
the GardenletReady
condition to Unknown
in case the gardenlet fails to renew the lease.
As a consequence, the gardener-scheduler
doesn’t consider this seed cluster for newly created shoot clusters anymore.
/healthz
Endpoint
The gardenlet includes an HTTP server that serves a /healthz
endpoint.
It’s used as a liveness probe in the Deployment
of the gardenlet.
If the gardenlet fails to renew its lease,
then the endpoint returns 500 Internal Server Error
, otherwise it returns 200 OK
.
Please note that the /healthz
only indicates whether the gardenlet
could successfully probe the Seed’s API server and renew the lease with
the Garden cluster.
It does not show that the Gardener extension API server (with the Gardener resource groups)
is available.
However, the gardenlet is designed to withstand such connection outages and
retries until the connection is reestablished.
Controllers
The gardenlet consists out of several controllers which are now described in more detail.
BackupBucket
Controller
The BackupBucket
controller reconciles those core.gardener.cloud/v1beta1.BackupBucket
resources whose .spec.seedName
value is equal to the name of the Seed
the respective gardenlet
is responsible for.
A core.gardener.cloud/v1beta1.BackupBucket
resource is created by the Seed
controller if .spec.backup
is defined in the Seed
.
The controller adds finalizers to the BackupBucket
and the secret mentioned in the .spec.secretRef
of the BackupBucket
. The controller also copies this secret to the seed cluster. Additionally, it creates an extensions.gardener.cloud/v1alpha1.BackupBucket
resource (non-namespaced) in the seed cluster and waits until the responsible extension controller reconciles it (see Contract: BackupBucket Resource for more details).
The status from the reconciliation is reported in the .status.lastOperation
field. Once the extension resource is ready and the .status.generatedSecretRef
is set by the extension controller, the gardenlet
copies the referenced secret to the garden
namespace in the garden cluster. An owner reference to the core.gardener.cloud/v1beta1.BackupBucket
is added to this secret.
If the core.gardener.cloud/v1beta1.BackupBucket
is deleted, the controller deletes the generated secret in the garden cluster and the extensions.gardener.cloud/v1alpha1.BackupBucket
resource in the seed cluster and it waits for the respective extension controller to remove its finalizers from the extensions.gardener.cloud/v1alpha1.BackupBucket
. Then it deletes the secret in the seed cluster and finally removes the finalizers from the core.gardener.cloud/v1beta1.BackupBucket
and the referred secret.
BackupEntry
Controller
The BackupEntry
controller reconciles those core.gardener.cloud/v1beta1.BackupEntry
resources whose .spec.seedName
value is equal to the name of a Seed
the respective gardenlet is responsible for.
Those resources are created by the Shoot
controller (only if backup is enabled for the respective Seed
) and there is exactly one BackupEntry
per Shoot
.
“Main” Reconciler
The controller creates an extensions.gardener.cloud/v1alpha1.BackupEntry
resource (non-namespaced) in the seed cluster and waits until the responsible extension controller reconciled it (see Contract: BackupEntry Resource for more details).
The status is populated in the .status.lastOperation
field.
The core.gardener.cloud/v1beta1.BackupEntry
resource has an owner reference pointing to the corresponding Shoot
.
Hence, if the Shoot
is deleted, the BackupEntry
resource also gets deleted.
In this case, the controller deletes the extensions.gardener.cloud/v1alpha1.BackupEntry
resource in the seed cluster and waits until the responsible extension controller has deleted it.
Afterwards, the finalizer of the core.gardener.cloud/v1beta1.BackupEntry
resource is released so that it finally disappears from the system.
If the spec.seedName
and .status.seedName
of the core.gardener.cloud/v1beta1.BackupEntry
are different, the controller will migrate it by annotating the extensions.gardener.cloud/v1alpha1.BackupEntry
in the Source Seed
with gardener.cloud/operation: migrate
, waiting for it to be migrated successfully and eventually deleting it from the Source Seed
cluster. Afterwards, the controller will recreate the extensions.gardener.cloud/v1alpha1.BackupEntry
in the Destination Seed
, annotate it with gardener.cloud/operation: restore
and wait for the restore operation to finish. For more details about control plane migration, please read Shoot Control Plane Migration.
Keep Backup for Deleted Shoots
In some scenarios it might be beneficial to not immediately delete the BackupEntry
s (and with them, the etcd backup) for deleted Shoot
s.
In this case you can configure the .controllers.backupEntry.deletionGracePeriodHours
field in the component configuration of the gardenlet.
For example, if you set it to 48
, then the BackupEntry
s for deleted Shoot
s will only be deleted 48
hours after the Shoot
was deleted.
Additionally, you can limit the shoot purposes for which this applies by setting .controllers.backupEntry.deletionGracePeriodShootPurposes[]
.
For example, if you set it to [production]
then only the BackupEntry
s for Shoot
s with .spec.purpose=production
will be deleted after the configured grace period. All others will be deleted immediately after the Shoot
deletion.
“Migration” Reconciler
This reconciler is only active if the ForceRestore
feature gate is enabled in the gardenlet
and if the seed has owner checks enabled (i.e., spec.setttings.ownerChecks.enabled=true
).
It checks if the source Seed
also has owner checks enabled. If not or if the BackupEntry
is being deleted, it sets the status.migrationStartTime
to nil
.
The controller updates the status to force restoration in the following cases:
- If the
BackupEntry
is annotated withshoot.gardener.cloud/force-restore=true
. - If the grace period for migration has elapsed (which is set in the
BackupEntryMigrationControllerConfiguration
in the gardenlet’s component configuration). - If the last operation is
Migrate
and ifLastOperationStaleDuration
(which is also set in theBackupEntryMigrationControllerConfiguration
in the gardenlet’s component configuration) has passed since thelastUpdateTime
of the operation.
Bastion
Controller
The Bastion
controller reconciles those operations.gardener.cloud/v1alpha1.Bastion
resources whose .spec.seedName
value is equal to the name of a Seed
the respective gardenlet is responsible for.
The controller creates an extensions.gardener.cloud/v1alpha1.Bastion
resource in the seed cluster in the shoot namespace with the same name as operations.gardener.cloud/v1alpha1.Bastion
. Then it waits until the responsible extension controller has reconciled it (see Contract: Bastion Resource for more details). The status is populated in the .status.conditions
and .status.ingress
fields.
During the deletion of operations.gardener.cloud/v1alpha1.Bastion
resources, the controller first sets the Ready
condition to False
and then deletes the extensions.gardener.cloud/v1alpha1.Bastion
resource in the seed cluster.
Once this resource is gone, the finalizer of the operations.gardener.cloud/v1alpha1.Bastion
resource is released, so it finally disappears from the system.
ControllerInstallation
Controller
The ControllerInstallation
controller in the gardenlet
reconciles ControllerInstallation
objects with the help of the following reconcilers.
“Main” Reconciler
This reconciler is responsible for ControllerInstallation
s referencing a ControllerDeployment
whose type=helm
.
It is responsible for unpacking the Helm chart tarball in the ControllerDeployment
s .providerConfig.chart
field and deploying the rendered resources to the seed cluster.
The Helm chart values in .providerConfig.values
will be used and extended with some information about the Gardener environment and the seed cluster:
gardener:
version: <gardenlet-version>
garden:
clusterIdentity: <identity-of-garden-cluster>
seed:
identity: <seed-name>
clusterIdentity: <identity-of-seed-cluster>
annotations: <seed-annotations>
labels: <seed-labels>
spec: <seed-specification>
As of today, there are a few more fields in .gardener.seed
, but it is recommended to use the .gardener.seed.spec
if the Helm chart needs more information about the seed configuration.
The rendered chart will be deployed via a ManagedResource
created in the garden
namespace of the seed cluster.
It is labeled with controllerinstallation-name=<name>
so that one can easily find the owning ControllerInstallation
for an existing ManagedResource
.
The reconciler maintains the Installed
condition of the ControllerInstallation
and sets it to False
if the rendering or deployment fails.
“Care” Reconciler
This reconciler reconciles ControllerInstallation
objects and checks whether they are in a healthy state.
It checks the .status.conditions
of the backing ManagedResource
created in the garden
namespace of the seed cluster.
- If the
ResourcesApplied
condition of theManagedResource
isTrue
, then theInstalled
condition of theControllerInstallation
will be set toTrue
. - If the
ResourcesHealthy
condition of theManagedResource
isTrue
, then theHealthy
condition of theControllerInstallation
will be set toTrue
. - If the
ResourcesProgressing
condition of theManagedResource
isTrue
, then theProgressing
condition of theControllerInstallation
will be set toTrue
.
A ControllerInstallation
is considered “healthy” if Applied=Healthy=True
and Progressing=False
.
“Required” Reconciler
This reconciler watches all resources in the extensions.gardener.cloud
API group in the seed cluster.
It is responsible for maintaining the Required
condition on ControllerInstallation
s.
Concretely, when there is at least one extension resource in the seed cluster a ControllerInstallation
is responsible for, then the status of the Required
condition will be True
.
If there are no extension resources anymore, its status will be False
.
This condition is taken into account by the ControllerRegistration
controller part of gardener-controller-manager
when it computes which extensions have to be deployed to which seed cluster. See Gardener Controller Manager for more details.
NetworkPolicy
Controller
The NetworkPolicy
controller reconciles NetworkPolicy
s in shoot namespaces in order to ensure access to the Kubernetes API server.
The controller resolves the IP address of the Kubernetes service in the default
namespace and creates an egress NetworkPolicy
s for it.
For more details about NetworkPolicy
s in Gardener, please see Network Policies in Gardener.
Seed
Controller
The Seed
controller in the gardenlet
reconciles Seed
objects with the help of the following reconcilers.
“Main Reconciler”
This reconciler is responsible for managing the seed’s system components.
Those comprise CA certificates, the various CustomResourceDefinition
s, the logging and monitoring stacks, and few central components like gardener-resource-manager
, etcd-druid
, istio
, etc.
The reconciler also deploys a BackupBucket
resource in the garden cluster in case the Seed'
s .spec.backup
is set.
It also checks whether the seed cluster’s Kubernetes version is at least the minimum supported version and errors in case this constraint is not met.
This reconciler maintains the Bootstrapped
condition, i.e. it sets it:
- to
Progressing
before it executes its reconciliation flow. - to
False
in case an error occurs. - to
True
in case the reconciliation succeeded.
“Care” Reconciler
This reconciler checks whether the seed system components (deployed by the “main” reconciler) are healthy.
It checks the .status.conditions
of the backing ManagedResource
created in the garden
namespace of the seed cluster.
A ManagedResource
is considered “healthy” if the conditions ResourcesApplied=ResourcesHealthy=True
and ResourcesProgressing=False
.
If all ManagedResource
s are healthy, then the SeedSystemComponentsHealthy
condition of the Seed
will be set to True
.
Otherwise, it will be set to False
.
If at least one ManagedResource
is unhealthy and there is threshold configuration for the conditions (in .controllers.seedCare.conditionThresholds
), then the status of the SeedSystemComponentsHealthy
condition will be set:
- to
Progressing
if it wasTrue
before. - to
Progressing
if it wasProgressing
before and thelastUpdateTime
of the condition does not exceed the configured threshold duration yet. - to
False
if it wasProgressing
before and thelastUpdateTime
of the condition exceeds the configured threshold duration.
The condition thresholds can be used to prevent reporting issues too early just because there is a rollout or a short disruption.
Only if the unhealthiness persists for at least the configured threshold duration, then the issues will be reported (by setting the status to False
).
“Lease” Reconciler
This reconciler checks whether the connection to the seed cluster’s /healthz
endpoint works.
If this succeeds, then it renews a Lease
resource in the garden cluster’s gardener-system-seed-lease
namespace.
This indicates a heartbeat to the external world, and internally the gardenlet
sets its health status to true
.
In addition, the GardenletReady
condition in the status
of the Seed
is set to True
.
The whole process is similar to what the kubelet
does to report heartbeats for its Node
resource and its KubeletReady
condition. For more information, see this section.
If the connection to the /healthz
endpoint or the update of the Lease
fails, then the internal health status of gardenlet
is set to false
.
Also, this internal health status is set to false
automatically after some time, in case the controller gets stuck for whatever reason.
This internal health status is available via the gardenlet
’s /healthz
endpoint and is used for the livenessProbe
in the gardenlet
pod.
Shoot
Controller
The Shoot
controller in the gardenlet
reconciles Shoot
objects with the help of the following reconcilers.
“Main” Reconciler
This reconciler is responsible for managing all shoot cluster components and implements the core logic for creating, updating, hibernating, deleting, and migrating shoot clusters.
It is also responsible for syncing the Cluster
cluster to the seed cluster before and after each successful shoot reconciliation.
The main reconciliation logic is performed in 3 different task flows dedicated to specific operation types:
reconcile
(operations: create, reconcile, restore): this is the main flow responsible for creation and regular reconciliation of shoots. Hibernating a shoot also triggers this flow. It is also used for restoration of the shoot control plane on the new seed (second half of a Control Plane Migration)migrate
: this flow is triggered whenspec.seedName
specifies a different seed thanstatus.seedName
. It performs the first half of the Control Plane Migration, i.e., a backup (migrate
operation) of all control plane components followed by a “shallow delete”.delete
: this flow is triggered when the shoot’sdeletionTimestamp
is set, i.e., when it is deleted.
The gardenlet takes special care to prevent unnecessary shoot reconciliations. This is important for several reasons, e.g., to not overload the seed API servers and to not exhaust infrastructure rate limits too fast. The gardenlet performs shoot reconciliations according to the following rules:
- If
status.observedGeneration
is less thanmetadata.generation
: this is the case, e.g., when the spec was changed, a manual reconciliation operation was triggered, or the shoot was deleted. - If the last operation was not successful.
- If the shoot is in a failed state, the gardenlet does not perform any reconciliation on the shoot (unless the retry operation was triggered). However, it syncs the
Cluster
resource to the seed in order to inform the extension controllers about the failed state. - Regular reconciliations are performed with every
GardenletConfiguration.controllers.shoot.syncPeriod
(defaults to1h
). - Shoot reconciliations are not performed if the assigned seed cluster is not healthy or has not been reconciled by the current gardenlet version yet (determined by the
Seed.status.gardener
section). This is done to make sure that shoots are reconciled with fully rolled out seed system components after a Gardener upgrade. Otherwise, the gardenlet might perform operations of the new version that doesn’t match the old version of the deployed seed system components, which might lead to unspecified behavior.
There are a few special cases that overwrite or confine how often and under which circumstances periodic shoot reconciliations are performed:
- In case the gardenlet config allows it (
controllers.shoot.respectSyncPeriodOverwrite
, disabled by default), the sync period for a shoot can be increased individually by setting theshoot.gardener.cloud/sync-period
annotation. This is always allowed for shoots in thegarden
namespace. Shoots are not reconciled with a higher frequency than specified inGardenletConfiguration.controllers.shoot.syncPeriod
. - In case the gardenlet config allows it (
controllers.shoot.respectSyncPeriodOverwrite
, disabled by default), shoots can be marked as “ignored” by setting theshoot.gardener.cloud/ignore
annotation. In this case, the gardenlet does not perform any reconciliation for the shoot. - In case
GardenletConfiguration.controllers.shoot.reconcileInMaintenanceOnly
is enabled (disabled by default), the gardenlet performs regular shoot reconciliations only once in the respective maintenance time window (GardenletConfiguration.controllers.shoot.syncPeriod
is ignored). The gardenlet randomly distributes shoot reconciliations over the maintenance time window to avoid high bursts of reconciliations (see Shoot Maintenance). - In case
Shoot.spec.maintenance.confineSpecUpdateRollout
is enabled (disabled by default), changes to the shoot specification are not rolled out immediately but only during the respective maintenance time window (see Shoot Maintenance).
“Care” Reconciler
This reconciler performs three “care” actions related to Shoot
s.
Conditions
It maintains four conditions and performs the following checks:
APIServerAvailable
: The/healthz
endpoint of the shoot’skube-apiserver
is called and considered healthy when it responds with200 OK
.ControlPlaneHealthy
: The control plane is considered healthy when the respectiveDeployment
s (for examplekube-apiserver
,kube-controller-manager
), andEtcd
s (for exampleetcd-main
) exist and are healthy.ObservabilityComponentsHealthy
: This condition is considered healthy when the respectiveDeployment
s (for examplegrafana
),StatefulSet
s (for exampleprometheus
,loki
), exist and are healthy.EveryNodyReady
: The conditions of the worker nodes are checked (e.g.,Ready
,MemoryPressure
, etc.). Also, it’s checked whether the Kubernetes version of the installedkubelet
matches the desired version specified in theShoot
resource.SystemComponentsHealthy
: The conditions of theManagedResource
s are checked (e.g.ResourcesApplied
, etc.). Also, it is verified whether the VPN tunnel connection is established (which is required for thekube-apiserver
to communicate with the worker nodes).
Sometimes, ManagedResource
s can have both Healthy
and Progressing
conditions set to True
(e.g., when a DaemonSet
rolls out one-by-one on a large cluster with many nodes) while this is not reflected in the Shoot
status. In order to catch issues where the rollout gets stuck, one can set .controllers.shootCare.managedResourceProgressingThreshold
in the gardenlet
’s component configuration. If the Progressing
condition is still True
for more than the configured duration, the SystemComponentsHealthy
condition in the Shoot
is set to False
, eventually.
Each condition can optionally also have error codes
in order to indicate which type of issue was detected (see Shoot Status for more details).
Apart from the above, extension controllers can also contribute to the status
or error codes
of these conditions (see Contributing to Shoot Health Status Conditions for more details).
If all checks for a certain conditions are succeeded, then its status
will be set to True
.
Otherwise, it will be set to False
.
If at least one check fails and there is threshold configuration for the conditions (in .controllers.seedCare.conditionThresholds
), then the status will be set:
- to
Progressing
if it wasTrue
before. - to
Progressing
if it wasProgressing
before and thelastUpdateTime
of the condition does not exceed the configured threshold duration yet. - to
False
if it wasProgressing
before and thelastUpdateTime
of the condition exceeds the configured threshold duration.
The condition thresholds can be used to prevent reporting issues too early just because there is a rollout or a short disruption.
Only if the unhealthiness persists for at least the configured threshold duration, then the issues will be reported (by setting the status to False
).
Constraints And Automatic Webhook Remediation
Please see Shoot Status for more details.
Garbage Collection
Stale pods in the shoot namespace in the seed cluster and in the kube-system
namespace in the shoot cluster are deleted.
A pod is considered stale when:
- it was terminated with reason
Evicted
. - it was terminated with reason starting with
OutOf
(e.g.,OutOfCpu
). - it is stuck in termination (i.e., if its
deletionTimestamp
is more than5m
ago).
“Migration” Reconciler
This reconciler is only active if the ForceRestore
feature gate is enabled in the gardenlet
and if the Seed
has owner checks enabled (i.e., spec.setttings.ownerChecks.enabled=true
).
It checks if the source Seed
also has owner checks enabled.
If not or if the Shoot
is being deleted, it sets the status.migrationStartTime
to nil
.
The controller updates the status to force restoration in the following cases:
- If the
Shoot
is annotated withshoot.gardener.cloud/force-restore=true
. - If the grace period for migration has elapsed (which is set in the
ShootMigrationControllerConfiguration
in the gardenlet’s component configuration). - If the last operation is
Migrate
and ifLastOperationStaleDuration
(which is also set in theShootMigrationControllerConfiguration
in the gardenlet’s component configuration) has passed since thelastUpdateTime
of the operation.
ShootState
Controller
The ShootState
controller in the gardenlet
reconciles resources containing information that has to be synced to the ShootState
.
This information is used when a control plane migration is performed.
“Extensions” Reconciler
This reconciler watches resources in the extensions.gardener.cloud
API group in the seed cluster which contain a Shoot
-specific state or data.
Those are BackupEntry
s, ContainerRuntime
s, ControlPlane
s, DNSRecord
s, Extension
s, Infrastructure
s, Network
s, OperatingSystemConfig
s, and Worker
s.
When there is a change in the .status.state
or .status.resources[]
fields of these resources, then this information is synced into the ShootState
resource in the garden cluster.
“Secret” Reconciler
This reconciler reconciles Secret
s having labels managed-by=secrets-manager
and persist=true
in the shoot namespaces in the seed cluster.
It syncs them to the ShootState
so that the secrets can be restored from there in case a shoot control plane has to be restored to another seed cluster (in case of migration).
Managed Seeds
Gardener users can use shoot clusters as seed clusters, so-called “managed seeds” (aka “shooted seeds”),
by creating ManagedSeed
resources.
By default, the gardenlet that manages this shoot cluster then automatically
creates a clone of itself with the same version and the same configuration
that it currently has.
Then it deploys the gardenlet clone into the managed seed cluster.
For more information, see Register Shoot as Seed.
Migrating from Previous Gardener Versions
If your Gardener version doesn’t support gardenlets yet, no special migration is required, but the following prerequisites must be met:
- Your Gardener version is at least 0.31 before upgrading to v1.
- You have to make sure that your garden cluster is exposed in a way that it’s reachable from all your seed clusters.
With previous Gardener versions, you had deployed the Gardener Helm chart
(incorporating the API server, controller-manager
, and scheduler).
With v1, this stays the same, but you now have to deploy the gardenlet Helm chart as well
into all of your seeds (if they aren’t managed, as mentioned earlier).
See Deploy a gardenlet for all instructions.
Related Links
1.2.10 - Network Policies
Network Policies in Gardener
As Seed
clusters can host the Kubernetes control planes of many Shoot
clusters, it is necessary to isolate the control planes from each other for security reasons.
Besides deploying each control plane in its own namespace, Gardener creates network policies to also isolate the networks.
Essentially, network policies make sure that pods can only talk to other pods over the network they are supposed to.
As such, network policies are an important part of Gardener’s tenant isolation.
Gardener deploys network policies into:
- each namespace hosting the Kubernetes control plane of the Shoot cluster.
- the namespace dedicated to Gardener seed-wide global controllers. This namespace is often called
garden
and contains e.g. the Gardenlet. - the
kube-system
namespace in the Shoot.
The aforementioned namespaces in the Seed contain a deny-all
network policy that denies all ingress and egress traffic.
This secure by default setting requires pods to allow network traffic.
This is done by pods having labels matching to the selectors of the network policies deployed by Gardener.
More details on the deployed network policies can be found in the development and usage sections.
1.2.11 - Operator
Gardener Operator
The gardener-operator
is meant to be responsible for the garden cluster environment.
Without this component, users must deploy ETCD, the Gardener control plane, etc., manually and with separate mechanisms (not maintained in this repository).
This is quite unfortunate since this requires separate tooling, processes, etc.
A lot of production- and enterprise-grade features were built into Gardener for managing the seed and shoot clusters, so it makes sense to re-use them as much as possible also for the garden cluster.
⚠️ Consider this component highly experimental and DO NOT use it in production.
Deployment
There is a Helm chart which can be used to deploy the gardener-operator
.
Once deployed and ready, you can create a Garden
resource.
Note that there can only be one Garden
resource per system at a time.
ℹ️ Similar to seed clusters, garden runtime clusters require a VPA, see this section. By default,
gardener-operator
deploys the VPA components. However, when there already is a VPA available, then set.spec.runtimeCluster.settings.verticalPodAutoscaler.enabled=false
in theGarden
resource.
Using Garden Runtime Cluster As Seed Cluster
In production scenarios, you probably wouldn’t use the Kubernetes cluster running gardener-operator
and the Gardener control plane (called “runtime cluster”) as seed cluster at the same time.
However, such setup is technically possible and might simplify certain situations (e.g., development, evaluation, …).
If the runtime cluster is a seed cluster at the same time, gardenlet
’s Seed
controller will not manage the components which were already deployed (and reconciled) by gardener-operator
.
As of today, this applies to:
gardener-resource-manager
vpa-{admission-controller,recommender,updater}
hvpa-controller
(whenHVPA
feature gate is enabled)etcd-druid
Those components are so-called “seed system components”.
As they were already made available by gardener-operator
, the gardenlet
just skips them.
ℹ️ There is no need to configure anything - the
gardenlet
will automatically detect when its seed cluster is the garden runtime cluster at the same time.
⚠️ Note that such setup requires that you upgrade the versions of gardener-operator
and gardenlet
in lock-step.
Otherwise, you might experience unexpected behaviour or issues with your seed or shoot clusters.
Garden
Resources
Please find an exemplary Garden
resource here.
Settings For Runtime Cluster
The Garden
resource offers a few settings that are used to control the behaviour of gardener-operator
in the runtime cluster.
This section provides an overview over the available settings:
Load Balancer Services
gardener-operator
creates a Kubernetes Service
object of type LoadBalancer
in the runtime cluster.
It is used for exposing the virtual garden control planes, namely the virtual-garden-kube-apiserver
.
In most cases, the cloud-controller-manager
(responsible for managing these load balancers on the respective underlying infrastructure) supports certain customization and settings via annotations.
This document provides a good overview and many examples.
By setting the .spec.settings.loadBalancerServices.annotations
field the Gardener administrator can specify a list of annotations which will be injected into the Service
s of type LoadBalancer
.
Note that we might switch to exposing the virtual-garden-kube-apiserver
via Istio in the future (similar to how the kube-apiservers
of shoot clusters are exposed).
The load balancer service settings might still be relevant, though.
Vertical Pod Autoscaler
gardener-operator
heavily relies on the Kubernetes vertical-pod-autoscaler
component.
By default, the Garden
controller deploys the VPA components into the garden
namespace of the respective runtime cluster.
In case you want to manage the VPA deployment on your own or have a custom one, then you might want to disable the automatic deployment of gardener-operator
.
Otherwise, you might end up with two VPAs which will cause erratic behaviour.
By setting the .spec.settings.verticalPodAutoscaler.enabled=false
you can disable the automatic deployment.
⚠️ In any case, there must be a VPA available for your runtime cluster. Using a runtime cluster without VPA is not supported.
Credentials Rotation
The credentials rotation works in the same way like it does for Shoot
resources, i.e. there are gardener.cloud/operation
annotation values for starting or completing the rotation procedures.
For certificate authorities, gardener-operator
generates one which is automatically rotated roughly each month (ca-garden-runtime
) and several CAs which are NOT automatically rotated but only on demand.
🚨 Hence, it is the responsibility of the operator to regularly perform the credentials rotation.
Please refer to this document for more details. As of today, gardener-operator
only creates the following types of credentials (i.e., some sections of the document don’t apply for Garden
s and can be ignored):
- certificate authorities (and related server and client certificates)
Local Development
The easiest setup is using a local KinD cluster and the Skaffold based approach to deploy the gardener-operator
.
make kind-operator-up
make operator-up
# now you can create Garden resources, for example
kubectl create -f example/operator/20-garden.yaml
# alternatively, you can run the e2e test
make test-e2e-local-operator
make operator-down
make kind-operator-down
Generally, any Kubernetes cluster can be used.
An alternative approach is to start the process locally and manually deploy the CustomResourceDefinition
for the Garden
resources into the targeted cluster (potentially remote):
kubectl create -f example/operator/10-crd-operator.gardener.cloud_gardens.yaml
make KUBECONFIG=... start-operator
# now you can create Garden resources, for example
kubectl create -f example/operator/20-garden.yaml
# alternatively, you can run the e2e test
make KUBECONFIG=... test-e2e-local-operator
Implementation Details
Controllers
As of today, the gardener-operator
only has one controller which is now described in more detail.
Garden
Controller
The reconciler first generates a general CA certificate which is valid for ~30d
and auto-rotated when 80% of its lifetime is reached.
Afterwards, it brings up the so-called “garden system components”.
The gardener-resource-manager
is deployed first since its ManagedResource
controller will be used to bring up the remainders.
Other system components are:
- garden system resources (
PriorityClass
es for the workload resources) - Vertical Pod Autoscaler (if enabled via
.spec.runtimeCluster.settings.verticalPodAutoscaler.enabled=true
in theGarden
) - HVPA controller (when
HVPA
feature gate is enabled) - ETCD Druid
As soon as all system components are up, the reconciler deploys the virtual garden cluster.
It comprises out of two ETCDs (one “main” etcd, one “events” etcd) which are managed by ETCD Druid via druid.gardener.cloud/v1alpha1.Etcd
custom resources.
The whole management works similar to how it works for Shoot
s, so you can take a look at this document for more information in general.
The virtual garden control plane components are:
virtual-garden-etcd-main
virtual-garden-etcd-events
If the .spec.virtualCluster.controlPlane.highAvailability={}
is set then these components will be deployed in a “highly available” mode.
For ETCD, this means that there will be 3 replicas each.
This works similar like for Shoot
s (see this document) except for the fact that there is no failure tolerance type configurability.
The gardener-resource-manager
’s HighAvailabilityConfig webhook makes sure that all pods with multiple replicas are spread on nodes, and if there are at least two zones in .spec.runtimeCluster.provider.zones
then they also get spread across availability zones.
If once set, removing
.spec.virtualCluster.controlPlane.highAvailability
again is not supported.
The virtual-garden-kube-apiserver
Deployment
(not yet managed by the operator) is exposed via a Service
of type LoadBalancer
with the same name.
In the future, we might switch to exposing it via Istio, similar to how the kube-apiservers
of shoot clusters are exposed.
The controller maintains the Reconciled
condition which indicates the status of an operation.
Webhooks
As of today, the gardener-operator
only has one webhook handler which is now described in more detail.
Validation
This webhook handler validates CREATE
/UPDATE
/DELETE
operations on Garden
resources.
Simple validation is performed via standard CRD validation.
However, more advanced validation is hard to express via these means and is performed by this webhook handler.
Furthermore, for deletion requests, it is validated that the Garden
is annotated with a deletion confirmation annotation, namely confirmation.gardener.cloud/deletion=true
.
Only if this annotation is present it allows the DELETE
operation to pass.
This prevents users from accidental/undesired deletions.
Another validation is to check that there is only one Garden
resource at a time.
It prevents creating a second Garden
when there is already one in the system.
1.2.12 - Resource Manager
Gardener Resource Manager
Initially, the gardener-resource-manager
was a project similar to the kube-addon-manager.
It manages Kubernetes resources in a target cluster which means that it creates, updates, and deletes them.
Also, it makes sure that manual modifications to these resources are reconciled back to the desired state.
In the Gardener project we were using the kube-addon-manager since more than two years.
While we have progressed with our extensibility story (moving cloud providers out-of-tree), we had decided that the kube-addon-manager is no longer suitable for this use-case.
The problem with it is that it needs to have its managed resources on its file system.
This requires storing the resources in ConfigMap
s or Secret
s and mounting them to the kube-addon-manager pod during deployment time.
The gardener-resource-manager
uses CustomResourceDefinition
s which allows to dynamically add, change, and remove resources with immediate action and without the need to reconfigure the volume mounts/restarting the pod.
Meanwhile, the gardener-resource-manager
has evolved to a more generic component comprising several controllers and webhook handlers.
It is deployed by gardenlet once per seed (in the garden
namespace) and once per shoot (in the respective shoot namespaces in the seed).
Component Configuration
Similar to other Gardener components, the gardener-resource-manager
uses a so-called component configuration file.
It allows specifying certain central settings like log level and formatting, client connection configuration, server ports and bind addresses, etc.
In addition, controllers and webhooks can be configured and sometimes even disabled.
Note that the very basic ManagedResource
, secret, and health controllers cannot be disabled.
You can find an example configuration file here.
Controllers
ManagedResource
Controller
This controller watches custom objects called ManagedResource
s in the resources.gardener.cloud/v1alpha1
API group.
These objects contain references to secrets, which itself contain the resources to be managed.
The reason why a Secret
is used to store the resources is that they could contain confidential information like credentials.
---
apiVersion: v1
kind: Secret
metadata:
name: managedresource-example1
namespace: default
type: Opaque
data:
objects.yaml: YXBpVmVyc2lvbjogdjEKa2luZDogQ29uZmlnTWFwCm1ldGFkYXRhOgogIG5hbWU6IHRlc3QtMTIzNAogIG5hbWVzcGFjZTogZGVmYXVsdAotLS0KYXBpVmVyc2lvbjogdjEKa2luZDogQ29uZmlnTWFwCm1ldGFkYXRhOgogIG5hbWU6IHRlc3QtNTY3OAogIG5hbWVzcGFjZTogZGVmYXVsdAo=
# apiVersion: v1
# kind: ConfigMap
# metadata:
# name: test-1234
# namespace: default
# ---
# apiVersion: v1
# kind: ConfigMap
# metadata:
# name: test-5678
# namespace: default
---
apiVersion: resources.gardener.cloud/v1alpha1
kind: ManagedResource
metadata:
name: example
namespace: default
spec:
secretRefs:
- name: managedresource-example1
In the above example, the controller creates two ConfigMap
s in the default
namespace.
When a user is manually modifying them, they will be reconciled back to the desired state stored in the managedresource-example
secret.
It is also possible to inject labels into all the resources:
---
apiVersion: v1
kind: Secret
metadata:
name: managedresource-example2
namespace: default
type: Opaque
data:
other-objects.yaml: YXBpVmVyc2lvbjogYXBwcy92MSAjIGZvciB2ZXJzaW9ucyBiZWZvcmUgMS45LjAgdXNlIGFwcHMvdjFiZXRhMgpraW5kOiBEZXBsb3ltZW50Cm1ldGFkYXRhOgogIG5hbWU6IG5naW54LWRlcGxveW1lbnQKc3BlYzoKICBzZWxlY3RvcjoKICAgIG1hdGNoTGFiZWxzOgogICAgICBhcHA6IG5naW54CiAgcmVwbGljYXM6IDIgIyB0ZWxscyBkZXBsb3ltZW50IHRvIHJ1biAyIHBvZHMgbWF0Y2hpbmcgdGhlIHRlbXBsYXRlCiAgdGVtcGxhdGU6CiAgICBtZXRhZGF0YToKICAgICAgbGFiZWxzOgogICAgICAgIGFwcDogbmdpbngKICAgIHNwZWM6CiAgICAgIGNvbnRhaW5lcnM6CiAgICAgIC0gbmFtZTogbmdpbngKICAgICAgICBpbWFnZTogbmdpbng6MS43LjkKICAgICAgICBwb3J0czoKICAgICAgICAtIGNvbnRhaW5lclBvcnQ6IDgwCg==
# apiVersion: apps/v1
# kind: Deployment
# metadata:
# name: nginx-deployment
# spec:
# selector:
# matchLabels:
# app: nginx
# replicas: 2 # tells deployment to run 2 pods matching the template
# template:
# metadata:
# labels:
# app: nginx
# spec:
# containers:
# - name: nginx
# image: nginx:1.7.9
# ports:
# - containerPort: 80
---
apiVersion: resources.gardener.cloud/v1alpha1
kind: ManagedResource
metadata:
name: example
namespace: default
spec:
secretRefs:
- name: managedresource-example2
injectLabels:
foo: bar
In this example, the label foo=bar
will be injected into the Deployment
, as well as into all created ReplicaSet
s and Pod
s.
Preventing Reconciliations
If a ManagedResource
is annotated with resources.gardener.cloud/ignore=true
, then it will be skipped entirely by the controller (no reconciliations or deletions of managed resources at all).
However, when the ManagedResource
itself is deleted (for example when a shoot is deleted), then the annotation is not respected and all resources will be deleted as usual.
This feature can be helpful to temporarily patch/change resources managed as part of such ManagedResource
.
Condition checks will be skipped for such ManagedResource
s.
Modes
The gardener-resource-manager
can manage a resource in the following supported modes:
Ignore
- The corresponding resource is removed from the
ManagedResource
status (.status.resources
). No action is performed on the cluster. - The resource is no longer “managed” (updated or deleted).
- The primary use case is a migration of a resource from one
ManagedResource
to another one.
- The corresponding resource is removed from the
The mode for a resource can be specified with the resources.gardener.cloud/mode
annotation. The annotation should be specified in the encoded resource manifest in the Secret that is referenced by the ManagedResource
.
Skipping Health Check
If a resource in the ManagedResource
is annotated with resources.gardener.cloud/skip-health-check=true
, then the resource will be skipped during health checks by the health controller. The ManagedResource
conditions will not reflect the health condition of this resource anymore. The ResourcesProgressing
condition will also be set to False
.
Resource Class
By default, the gardener-resource-manager
controller watches for ManagedResource
s in all namespaces.
The .sourceClientConnection.namespace
field in the component configuration restricts the watch to ManagedResource
s in a single namespace only.
Note that this setting also affects all other controllers and webhooks since it’s a central configuration.
A ManagedResource
has an optional .spec.class
field that allows it to indicate that it belongs to a given class of resources.
The .controllers.resourceClass
field in the component configuration restricts the watch to ManagedResource
s with the given .spec.class
.
A default class is assumed if no class is specified.
Conditions
A ManagedResource
has a ManagedResourceStatus
, which has an array of Conditions. Conditions currently include:
Condition | Description |
---|---|
ResourcesApplied | True if all resources are applied to the target cluster |
ResourcesHealthy | True if all resources are present and healthy |
ResourcesProgressing | False if all resources have been fully rolled out |
ResourcesApplied
may be False
when:
- the resource
apiVersion
is not known to the target cluster - the resource spec is invalid (for example the label value does not match the required regex for it)
- …
ResourcesHealthy
may be False
when:
- the resource is not found
- the resource is a Deployment and the Deployment does not have the minimum availability.
- …
ResourcesProgressing
may be True
when:
- a
Deployment
,StatefulSet
orDaemonSet
has not been fully rolled out yet, i.e. not all replicas have been updated with the latest changes tospec.template
.
Each Kubernetes resources has different notion for being healthy. For example, a Deployment is considered healthy if the controller observed its current revision and if the number of updated replicas is equal to the number of replicas.
The following status.conditions
section describes a healthy ManagedResource
:
conditions:
- lastTransitionTime: "2022-05-03T10:55:39Z"
lastUpdateTime: "2022-05-03T10:55:39Z"
message: All resources are healthy.
reason: ResourcesHealthy
status: "True"
type: ResourcesHealthy
- lastTransitionTime: "2022-05-03T10:55:36Z"
lastUpdateTime: "2022-05-03T10:55:36Z"
message: All resources have been fully rolled out.
reason: ResourcesRolledOut
status: "False"
type: ResourcesProgressing
- lastTransitionTime: "2022-05-03T10:55:18Z"
lastUpdateTime: "2022-05-03T10:55:18Z"
message: All resources are applied.
reason: ApplySucceeded
status: "True"
type: ResourcesApplied
Ignoring Updates
In some cases, it is not desirable to update or re-apply some of the cluster components (for example, if customization is required or needs to be applied by the end-user). For these resources, the annotation “resources.gardener.cloud/ignore” needs to be set to “true” or a truthy value (Truthy values are “1”, “t”, “T”, “true”, “TRUE”, “True”) in the corresponding managed resource secrets. This can be done from the components that create the managed resource secrets, for example Gardener extensions or Gardener. Once this is done, the resource will be initially created and later ignored during reconciliation.
Preserving replicas
or resources
in Workload Resources
The objects which are part of the ManagedResource
can be annotated with:
resources.gardener.cloud/preserve-replicas=true
in case the.spec.replicas
field of workload resources likeDeployment
s,StatefulSet
s, etc., shall be preserved during updates.resources.gardener.cloud/preserve-resources=true
in case the.spec.containers[*].resources
fields of all containers of workload resources likeDeployment
s,StatefulSet
s, etc., shall be preserved during updates.
This can be useful if there are non-standard horizontal/vertical auto-scaling mechanisms in place. Standard mechanisms like
HorizontalPodAutoscaler
orVerticalPodAutoscaler
will be auto-recognized bygardener-resource-manager
, i.e., in such cases the annotations are not needed.
Origin
All the objects managed by the resource manager get a dedicated annotation
resources.gardener.cloud/origin
describing the ManagedResource
object that describes
this object. The default format is <namespace>/<objectname>
.
In multi-cluster scenarios (the ManagedResource
objects are maintained in a
cluster different from the one the described objects are managed), it might
be useful to include the cluster identity, as well.
This can be enforced by setting the .controllers.clusterID
field in the component configuration.
Here, several possibilities are supported:
- given a direct value: use this as id for the source cluster.
<cluster>
: read the cluster identity from acluster-identity
config map in thekube-system
namespace (attributecluster-identity
). This is automatically maintained in all clusters managed or involved in a gardener landscape.<default>
: try to read the cluster identity from the config map. If not found, no identity is used.- empty string: no cluster identity is used (completely cluster local scenarios).
By default, cluster id is not used. If cluster id is specified, the format is <cluster id>:<namespace>/<objectname>
.
In addition to the origin annotation, all objects managed by the resource manager get a dedicated label resources.gardener.cloud/managed-by
. This label can be used to describe these objects with a selector. By default it is set to “gardener”, but this can be overwritten by setting the .conrollers.managedResources.managedByLabelValue
field in the component configuration.
Garbage Collector For Immutable ConfigMap
s/Secret
s
In Kubernetes, workload resources (e.g., Pod
s) can mount ConfigMap
s or Secret
s or reference them via environment variables in containers.
Typically, when the content of such a ConfigMap
/Secret
gets changed, then the respective workload is usually not dynamically reloading the configuration, i.e., a restart is required.
The most commonly used approach is probably having the so-called checksum annotations in the pod template, which makes Kubernetes recreate the pod if the checksum changes.
However, it has the downside that old, still running versions of the workload might not be able to properly work with the already updated content in the ConfigMap
/Secret
, potentially causing application outages.
In order to protect users from such outages (and also to improve the performance of the cluster), the Kubernetes community provides the “immutable ConfigMap
s/Secret
s feature”.
Enabling immutability requires ConfigMap
s/Secret
s to have unique names.
Having unique names requires the client to delete ConfigMap
s/Secret
s no longer in use.
In order to provide a similarly lightweight experience for clients (compared to the well-established checksum annotation approach), the gardener-resource-manager
features an optional garbage collector controller (disabled by default).
The purpose of this controller is cleaning up such immutable ConfigMap
s/Secret
s if they are no longer in use.
How Does the Garbage Collector Work?
The following algorithm is implemented in the GC controller:
- List all
ConfigMap
s andSecret
s labeled withresources.gardener.cloud/garbage-collectable-reference=true
. - List all
Deployment
s,StatefulSet
s,DaemonSet
s,Job
s,CronJob
s,Pod
s and for each of them:- iterate over the
.metadata.annotations
and for each of them:- If the annotation key follows the
reference.resources.gardener.cloud/{configmap,secret}-<hash>
scheme and the value equals<name>
, then consider it as “in-use”.
- If the annotation key follows the
- iterate over the
- Delete all
ConfigMap
s andSecret
s not considered as “in-use”.
Consequently, clients need to:
Create immutable
ConfigMap
s/Secret
s with unique names (e.g., a checksum suffix based on the.data
).Label such
ConfigMap
s/Secret
s withresources.gardener.cloud/garbage-collectable-reference=true
.Annotate their workload resources with
reference.resources.gardener.cloud/{configmap,secret}-<hash>=<name>
for allConfigMap
s/Secret
s used by the containers of the respectivePod
s.⚠️ Add such annotations to
.metadata.annotations
, as well as to all templates of other resources (e.g.,.spec.template.metadata.annotations
inDeployment
s or.spec.jobTemplate.metadata.annotations
and.spec.jobTemplate.spec.template.metadata.annotations
forCronJob
s. This ensures that the GC controller does not unintentionally considerConfigMap
s/Secret
s as “not in use” just because there isn’t aPod
referencing them anymore (e.g., they could still be used by aDeployment
scaled down to0
).
ℹ️ For the last step, there is a helper function InjectAnnotations
in the pkg/controller/garbagecollector/references
, which you can use for your convenience.
Example:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: test-1234
namespace: default
labels:
resources.gardener.cloud/garbage-collectable-reference: "true"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: test-5678
namespace: default
labels:
resources.gardener.cloud/garbage-collectable-reference: "true"
---
apiVersion: v1
kind: Pod
metadata:
name: example
namespace: default
annotations:
reference.resources.gardener.cloud/configmap-82a3537f: test-5678
spec:
containers:
- name: nginx
image: nginx:1.14.2
terminationGracePeriodSeconds: 2
The GC controller would delete the ConfigMap/test-1234
because it is considered as not “in-use”.
ℹ️ If the GC controller is activated then the ManagedResource
controller will no longer delete ConfigMap
s/Secret
s having the above label.
How to Activate the Garbage Collector?
The GC controller can be activated by setting the .controllers.garbageCollector.enabled
field to true
in the component configuration.
TokenInvalidator Controller
The Kubernetes community is slowly transitioning from static ServiceAccount
token Secret
s to ServiceAccount
Token Volume Projection.
Typically, when you create a ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
then the serviceaccount-token
controller (part of kube-controller-manager
) auto-generates a Secret
with a static token:
apiVersion: v1
kind: Secret
metadata:
annotations:
kubernetes.io/service-account.name: default
kubernetes.io/service-account.uid: 86e98645-2e05-11e9-863a-b2d4d086dd5a)
name: default-token-ntxs9
type: kubernetes.io/service-account-token
data:
ca.crt: base64(cluster-ca-cert)
namespace: base64(namespace)
token: base64(static-jwt-token)
Unfortunately, when using ServiceAccount
Token Volume Projection in a Pod
, this static token is actually not used at all:
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
serviceAccountName: default
containers:
- image: nginx
name: nginx
volumeMounts:
- mountPath: /var/run/secrets/tokens
name: token
volumes:
- name: token
projected:
sources:
- serviceAccountToken:
path: token
expirationSeconds: 7200
While the Pod
is now using an expiring and auto-rotated token, the static token is still generated and valid.
As of Kubernetes v1.22, there is neither a way of preventing kube-controller-manager
to generate such static tokens, nor a way to proactively remove or invalidate them:
- https://github.com/kubernetes/kubernetes/issues/77599
- https://github.com/kubernetes/kubernetes/issues/77600
Disabling the serviceaccount-token
controller is an option, however, especially in the Gardener context it may either break end-users or it may not even be possible to control such settings.
Also, even if a future Kubernetes version supports native configuration of the above behaviour, Gardener still supports older versions which won’t get such features but need a solution as well.
This is where the TokenInvalidator comes into play:
Since it is not possible to prevent kube-controller-manager
from generating static ServiceAccount
Secret
s, the TokenInvalidator is, as its name suggests, just invalidating these tokens.
It considers all such Secret
s belonging to ServiceAccount
s with .automountServiceAccountToken=false
.
By default, all namespaces in the target cluster are watched, however, this can be configured by specifying the .targetClientConnection.namespace
field in the component configuration.
Note that this setting also affects all other controllers and webhooks since it’s a central configuration.
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-serviceaccount
automountServiceAccountToken: false
This will result in a static ServiceAccount
token secret whose token
value is invalid:
apiVersion: v1
kind: Secret
metadata:
annotations:
kubernetes.io/service-account.name: my-serviceaccount
kubernetes.io/service-account.uid: 86e98645-2e05-11e9-863a-b2d4d086dd5a
name: my-serviceaccount-token-ntxs9
type: kubernetes.io/service-account-token
data:
ca.crt: base64(cluster-ca-cert)
namespace: base64(namespace)
token: AAAA
Any attempt to regenerate the token or creating a new such secret will again make the component invalidating it.
You can opt-out of this behaviour for
ServiceAccount
s setting.automountServiceAccountToken=false
by labeling them withtoken-invalidator.resources.gardener.cloud/skip=true
.
In order to enable the TokenInvalidator you have to set both .controllers.tokenValidator.enabled=true
and .webhooks.tokenValidator.enabled=true
in the component configuration.
The below graphic shows an overview of the Token Invalidator for Service account secrets in the Shoot cluster.
TokenRequestor Controller
This controller provides the service to create and auto-renew tokens via the TokenRequest
API.
It provides a functionality similar to the kubelet’s Service Account Token Volume Projection. It was created to handle the special case of issuing tokens to pods that run in a different cluster than the API server they communicate with (hence, using the native token volume projection feature is not possible).
The controller differentiates between source cluster
and target cluster
.
The source cluster
hosts the gardener-resource-manager
pod. Secrets in this cluster are watched and modified by the controller.
The target cluster
can be configured to point to another cluster. The existence of ServiceAccounts are ensured and token requests are issued against the target.
When the gardener-resource-manager
is deployed next to the Shoot’s controlplane in the Seed, the source cluster
is the Seed while the target cluster
points to the Shoot.
Reconciliation Loop
This controller reconciles secrets in all namespaces in the source cluster with the label: resources.gardener.cloud/purpose: token-requestor
.
See this yaml file for an example of the secret.
The controller ensures a ServiceAccount
exists in the target cluster as specified in the annotations of the Secret
in the source cluster:
serviceaccount.resources.gardener.cloud/name: <sa-name>
serviceaccount.resources.gardener.cloud/namespace: <sa-namespace>
The requested tokens will act with the privileges which are assigned to this ServiceAccount
.
The controller will then request a token via the TokenRequest
API and populate it into the .data.token
field to the Secret
in the source cluster.
Alternatively, the client can provide a raw kubeconfig (in YAML or JSON format) via the Secret
’s .data.kubeconfig
field.
The controller will then populate the requested token in the kubeconfig for the user used in the .current-context
.
For example, if .data.kubeconfig
is
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: AAAA
server: some-server-url
name: shoot--foo--bar
contexts:
- context:
cluster: shoot--foo--bar
user: shoot--foo--bar-token
name: shoot--foo--bar
current-context: shoot--foo--bar
kind: Config
preferences: {}
users:
- name: shoot--foo--bar-token
user:
token: ""
then the .users[0].user.token
field of the kubeconfig will be updated accordingly.
The controller also adds an annotation to the Secret
to keep track when to renew the token before it expires.
By default, the tokens are issued to expire after 12 hours. The expiration time can be set with the following annotation:
serviceaccount.resources.gardener.cloud/token-expiration-duration: 6h
It automatically renews once 80% of the lifetime is reached, or after 24h
.
Optionally, the controller can also populate the token into a Secret
in the target cluster. This can be requested by annotating the Secret
in the source cluster with:
token-requestor.resources.gardener.cloud/target-secret-name: "foo"
token-requestor.resources.gardener.cloud/target-secret-namespace: "bar"
Overall, the TokenRequestor controller provides credentials with limited lifetime (JWT tokens) used by Shoot control plane components running in the Seed to talk to the Shoot API Server. Please see the graphic below:
Kubelet Server CertificateSigningRequest
Approver
Gardener configures the kubelets such that they request two certificates via the CertificateSigningRequest
API:
- client certificate for communicating with the
kube-apiserver
- server certificate for serving its HTTPS server
For client certificates, the kubernetes.io/kube-apiserver-client-kubelet
signer is used (see Certificate Signing Requests for more details).
The kube-controller-manager
’s csrapprover
controller is responsible for auto-approving such CertificateSigningRequest
s so that the respective certificates can be issued.
For server certificates, the kubernetes.io/kubelet-serving
signer is used.
Unfortunately, the kube-controller-manager
is not able to auto-approve such CertificateSigningRequest
s (see kubernetes/kubernetes#73356 for details).
That’s the motivation for having this controller as part of gardener-resource-manager
.
It watches CertificateSigningRequest
s with the kubernetes.io/kubelet-serving
signer and auto-approves them when all the following conditions are met:
- The
.spec.username
is prefixed withsystem:node:
. - There must be at least one DNS name or IP address as part of the certificate SANs.
- The common name in the CSR must match the
.spec.username
. - The organization in the CSR must only contain
system:nodes
. - There must be a
Node
object with the same name in the shoot cluster. - There must be exactly one
Machine
for the node in the seed cluster. - The DNS names part of the SANs must be equal to all
.status.addresses[]
of typeHostname
in theNode
. - The IP addresses part of the SANs must be equal to all
.status.addresses[]
of typeInternalIP
in theNode
.
If any one of these requirements is violated, the CertificateSigningRequest
will be denied.
Otherwise, once approved, the kube-controller-manager
’s csrsigner
controller will issue the requested certificate.
NetworkPolicy
Controller
This controller reconciles Service
s with a non-empty .spec.podSelector
.
It creates two NetworkPolicy
s for each port in the .spec.ports[]
list.
For example:
apiVersion: v1
kind: Service
metadata:
name: gardener-resource-manager
namespace: a
spec:
selector:
app: gardener-resource-manager
ports:
- name: server
port: 443
protocol: TCP
targetPort: 10250
leads to
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
annotations:
gardener.cloud/description: Allows ingress TCP traffic to port 10250 for pods
selected by the a/gardener-resource-manager service selector from pods running
in namespace a labeled with map[networking.resources.gardener.cloud/to-gardener-resource-manager-tcp-10250:allowed].
name: ingress-to-gardener-resource-manager-tcp-10250
namespace: a
spec:
ingress:
- from:
- podSelector:
matchLabels:
networking.resources.gardener.cloud/to-gardener-resource-manager-tcp-10250: allowed
ports:
- port: 10250
protocol: TCP
podSelector:
matchLabels:
app: gardener-resource-manager
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
annotations:
gardener.cloud/description: Allows egress TCP traffic to port 10250 from pods
running in namespace a labeled with map[networking.resources.gardener.cloud/to-gardener-resource-manager-tcp-10250:allowed]
to pods selected by the a/gardener-resource-manager service selector.
name: egress-to-gardener-resource-manager-tcp-10250
namespace: a
spec:
egress:
- to:
- podSelector:
matchLabels:
app: gardener-resource-manager
ports:
- port: 10250
protocol: TCP
podSelector:
matchLabels:
networking.resources.gardener.cloud/to-gardener-resource-manager-tcp-10250: allowed
policyTypes:
- Egress
A component that initiates the connection to gardener-resource-manager
’s tcp/10250
port can now be labeled with networking.resources.gardener.cloud/to-gardener-resource-manager-tcp-10250=allowed
.
That’s all this component needs to do - it does not need to create any NetworkPolicy
s itself.
Cross-Namespace Communication
Apart from this “simple” case where both communicating components run in the same namespace a
, there is also the cross-namespace communication case.
With above example, let’s say there are components running in another namespace b
, and they would like to initiate the communication with gardener-resource-manager
in a
.
To cover this scenario, the Service
can be annotated with networking.resources.gardener.cloud/namespace-selectors='[{"matchLabels":{"kubernetes.io/metadata.name":"b"}}]'
.
Note that you can specify multiple namespace selectors in this annotation which are OR-ed.
This will make the controller create additional NetworkPolicy
s as follows:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
annotations:
gardener.cloud/description: Allows ingress TCP traffic to port 10250 for pods selected
by the a/gardener-resource-manager service selector from pods running in namespace b
labeled with map[networking.resources.gardener.cloud/to-a-gardener-resource-manager-tcp-10250:allowed].
name: ingress-to-gardener-resource-manager-tcp-10250-from-b
namespace: a
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: b
podSelector:
matchLabels:
networking.resources.gardener.cloud/to-a-gardener-resource-manager-tcp-10250: allowed
ports:
- port: 10250
protocol: TCP
podSelector:
matchLabels:
app: gardener-resource-manager
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
annotations:
gardener.cloud/description: Allows egress TCP traffic to port 10250 from pods running in
namespace b labeled with map[networking.resources.gardener.cloud/to-a-gardener-resource-manager-tcp-10250:allowed]
to pods selected by the a/gardener-resource-manager service selector.
name: egress-to-a-gardener-resource-manager-tcp-10250
namespace: b
spec:
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: a
podSelector:
matchLabels:
app: gardener-resource-manager
ports:
- port: 10250
protocol: TCP
podSelector:
matchLabels:
networking.resources.gardener.cloud/to-a-gardener-resource-manager-tcp-10250: allowed
policyTypes:
- Egress
The components in namespace b
now need to be labeled with networking.resources.gardener.cloud/to-a-gardener-resource-manager-tcp-10250=allowed
, but that’s already it.
Obviously, this approach also works for namespace selectors different from
kubernetes.io/metadata.name
to cover scenarios where the namespace name is not known upfront or where multiple namespaces with a similar label are relevant. The controller creates two dedicated policies for each namespace matching the selectors.
Service
Targets In Multiple Namespaces
Finally, let’s say there is a Service
called example
which exists in different namespaces whose names are not static (e.g., foo-1
, foo-2
, etc.), and a component in namespace bar
wants to initiate connections with all of them.
The example
Service
s in these namespaces can now be annotated with networking.resources.gardener.cloud/namespace-selectors='[{"matchLabels":{"kubernetes.io/metadata.name":"bar"}}]'
.
As a consequence, the component in namespace bar
now needs to be labeled with networking.resources.gardener.cloud/to-foo-1-example-tcp-8080=allowed
, networking.resources.gardener.cloud/to-foo-2-example-tcp-8080=allowed
, etc.
This approach does not work in practice, however, since the namespace names are neither static nor known upfront.
To overcome this, it is possible to specify an alias for the concrete namespace in the pod label selector via the networking.resources.gardener.cloud/pod-label-selector-namespace-alias
annotation.
In above case, the example
Service
in the foo-*
namespaces could be annotated with networking.resources.gardener.cloud/pod-label-selector-namespace-alias=all-foos
.
This would modify the label selector in all NetworkPolicy
s related to cross-namespace communication, i.e. instead of networking.resources.gardener.cloud/to-foo-{1,2,...}-example-tcp-8080=allowed
, networking.resources.gardener.cloud/to-all-foos-example-tcp-8080=allowed
would be used.
Now the component in namespace bar
only needs this single label and is able to talk to all such Service
s in the different namespaces.
Real-world examples for this scenario are the
kube-apiserver
Service
(which exists in all shoot namespaces), or theistio-ingressgateway
Service
(which exists in allistio-ingress*
namespaces). In both cases, the names of the namespaces are not statically known and depend on user input.
Ingress From Everywhere
All above scenarios are about components initiating connections to some targets. However, some components also receive incoming traffic from sources outside the cluster. This traffic requires adequate ingress policies so that it can be allowed.
To cover this scenario, the Service
can be annotated with networking.resources.gardener.cloud/from-world-to-ports=[{"port":"10250","protocol":"TCP"}]
.
As a result, the controller creates the following NetworkPolicy
:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: ingress-to-gardener-resource-manager-from-world
namespace: a
spec:
ingress:
- from:
- namespaceSelector: {}
podSelector: {}
- ipBlock:
cidr: 0.0.0.0/0
ports:
- port: 10250
protocol: TCP
podSelector:
matchLabels:
app: gardener-resource-manager
policyTypes:
- Ingress
The respective pods don’t need any additional labels.
If the annotation’s value is empty ([]
) then all ports are allowed.
Webhooks
Mutating Webhooks
High Availability Config
This webhook is used to conveniently apply the configuration to make components deployed to seed or shoot clusters highly available. The details and scenarios are described in High Availability Of Deployed Components.
The webhook reacts on creation/update of Deployment
s, StatefulSet
s, HorizontalPodAutoscaler
s and HVPA
s in namespaces labeled with high-availability-config.resources.gardener.cloud/consider=true
.
The webhook performs the following actions:
The
.spec.replicas
(orspec.minReplicas
respectively) field is mutated based on thehigh-availability-config.resources.gardener.cloud/type
label of the resource and thehigh-availability-config.resources.gardener.cloud/failure-tolerance-type
annotation of the namespace:Failure Tolerance Type ➡️
/
⬇️ Component Type️ ️unset empty non-empty controller
2
1
2
server
2
2
2
- The replica count values can be overwritten by the
high-availability-config.resources.gardener.cloud/replicas
annotation. - It does NOT mutate the replicas when:
- the replicas are already set to
0
(hibernation case), or - when the resource is scaled horizontally by
HorizontalPodAutoscaler
orHvpa
, and the current replica count is higher than what was computed above.
- the replicas are already set to
- The replica count values can be overwritten by the
When the
high-availability-config.resources.gardener.cloud/zones
annotation is NOT empty and either thehigh-availability-config.resources.gardener.cloud/failure-tolerance-type
annotation is set or thehigh-availability-config.resources.gardener.cloud/zone-pinning
annotation is set totrue
, then it adds a node affinity to the pod template spec:spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/zone operator: In values: - <zone1> # - ...
This ensures that all pods are pinned to only nodes in exactly those concrete zones.
Topology Spread Constraints are added to the pod template spec when the
.spec.replicas
are greater than1
. When thehigh-availability-config.resources.gardener.cloud/zones
annotation …… contains only one zone, then the following is added:
spec: topologySpreadConstraints: - topologyKey: kubernetes.io/hostname maxSkew: 1 whenUnsatisfiable: ScheduleAnyway labelSelector: ...
This ensures that the (multiple) pods are scheduled across nodes on best-effort basis.
… contains at least two zones, then the following is added:
spec: topologySpreadConstraints: - topologyKey: kubernetes.io/hostname maxSkew: 1 whenUnsatisfiable: ScheduleAnyway labelSelector: ... - topologyKey: topology.kubernetes.io/zone maxSkew: 1 whenUnsatisfiable: DoNotSchedule labelSelector: ...
This enforces that the (multiple) pods are scheduled across zones. It circumvents a known limitation in Kubernetes for clusters < 1.26 (ref kubernetes/kubernetes#109364. In case the number of replicas is larger than twice the number of zones, then the
maxSkew=2
for the second spread constraints.
Independent on the number of zones, when the
high-availability-config.resources.gardener.cloud/failure-tolerance-type
annotation is set and NOT empty, then thewhenUnsatisfiable
is set toDoNotSchedule
for the constraint withtopologyKey=kubernetes.io/hostname
(which enforces the node-spread).
Auto-Mounting Projected ServiceAccount
Tokens
When this webhook is activated, then it automatically injects projected ServiceAccount
token volumes into Pod
s and all its containers if all of the following preconditions are fulfilled:
- The
Pod
is NOT labeled withprojected-token-mount.resources.gardener.cloud/skip=true
. - The
Pod
’s.spec.serviceAccountName
field is NOT empty and NOT set todefault
. - The
ServiceAccount
specified in thePod
’s.spec.serviceAccountName
sets.automountServiceAccountToken=false
. - The
Pod
’s.spec.volumes[]
DO NOT already contain a volume with a name prefixed withkube-api-access-
.
The projected volume will look as follows:
spec:
volumes:
- name: kube-api-access-gardener
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 43200
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
The
expirationSeconds
are defaulted to12h
and can be overwritten with the.webhooks.projectedTokenMount.expirationSeconds
field in the component configuration, or with theprojected-token-mount.resources.gardener.cloud/expiration-seconds
annotation on aPod
resource.
The volume will be mounted into all containers specified in the Pod
to the path /var/run/secrets/kubernetes.io/serviceaccount
.
This is the default location where client libraries expect to find the tokens and mimics the upstream ServiceAccount
admission plugin. See Managing Service Accounts for more information.
Overall, this webhook is used to inject projected service account tokens into pods running in the Shoot and the Seed cluster. Hence, it is served from the Seed GRM and each Shoot GRM. Please find an overview below for pods deployed in the Shoot cluster:
Pod Topology Spread Constraints
When this webhook is enabled, then it mimics the topologyKey feature for Topology Spread Constraints (TSC) on the label pod-template-hash
.
Concretely, when a pod is labelled with pod-template-hash
, the handler of this webhook extends any topology spread constraint in the pod:
metadata:
labels:
pod-template-hash: 123abc
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
pod-template-hash: 123abc # added by webhook
The procedure circumvents a known limitation with TSCs which leads to imbalanced deployments after rolling updates. Gardener enables this webhook to schedule pods of deployments across nodes and zones.
Please note that the gardener-resource-manager
itself as well as pods labelled with topology-spread-constraints.resources.gardener.cloud/skip
are excluded from any mutations.
System Components Webhook
If enabled, this webhook handles scheduling concerns for system components Pod
s (except those managed by DaemonSet
s).
The following tasks are performed by this webhook:
- Add
pod.spec.nodeSelector
as given in the webhook configuration. - Add
pod.spec.tolerations
as given in the webhook configuration. - Add
pod.spec.tolerations
for any existing nodes matching the node selector given in the webhook configuration. Known taints and tolerations used for taint based evictions are disregarded.
Gardener enables this webhook for kube-system
and kubernetes-dashboard
namespaces in shoot clusters, selecting Pod
s being labelled with resources.gardener.cloud/managed-by: gardener
.
It adds a configuration, so that Pod
s will get the worker.gardener.cloud/system-components: true
node selector (step 1) as well as tolerate any custom taint (step 2) that is added to system component worker nodes (shoot.spec.provider.workers[].systemComponents.allow: true
).
In addition, the webhook merges these tolerations with the ones required for at that time available system component Node
s in the cluster (step 3).
Both is required to ensure system component Pod
s can be scheduled or executed during an active shoot reconciliation that is happening due to any modifications to shoot.spec.provider.workers[].taints
, e.g. Pod
s must be scheduled while there are still Node
s not having the updated taint configuration.
You can opt-out of this behaviour for
Pod
s by labeling them withsystem-components-config.resources.gardener.cloud/skip=true
.
Validating Webhooks
Unconfirmed Deletion Prevention For Custom Resources And Definitions
As part of Gardener’s extensibility concepts, a lot of CustomResourceDefinition
s are deployed to the seed clusters that serve as extension points for provider-specific controllers.
For example, the Infrastructure
CRD triggers the provider extension to prepare the IaaS infrastructure of the underlying cloud provider for a to-be-created shoot cluster.
Consequently, these extension CRDs have a lot of power and control large portions of the end-user’s shoot cluster.
Accidental or undesired deletions of those resource can cause tremendous and hard-to-recover-from outages and should be prevented.
When this webhook is activated, it reacts for CustomResourceDefinition
s and most of the custom resources in the extensions.gardener.cloud/v1alpha1
API group.
It also reacts for the druid.gardener.cloud/v1alpha1.Etcd
resources.
The webhook prevents DELETE
requests for those CustomResourceDefinition
s labeled with gardener.cloud/deletion-protected=true
, and for all mentioned custom resources if they were not previously annotated with the confirmation.gardener.cloud/deletion=true
.
This prevents that undesired kubectl delete <...>
requests are accepted.
Extension Resource Validation
When this webhook is activated, it reacts for most of the custom resources in the extensions.gardener.cloud/v1alpha1
API group.
It also reacts for the druid.gardener.cloud/v1alpha1.Etcd
resources.
The webhook validates the resources specifications for CREATE
and UPDATE
requests.
1.2.13 - Scheduler
Gardener Scheduler
The Gardener Scheduler is in essence a controller that watches newly created shoots and assigns a seed cluster to them. Conceptually, the task of the Gardener Scheduler is very similar to the task of the Kubernetes Scheduler: finding a seed for a shoot instead of a node for a pod.
Either the scheduling strategy or the shoot cluster purpose hereby determines how the scheduler is operating. The following sections explain the configuration and flow in greater detail.
Why Is the Gardener Scheduler Needed?
1. Decoupling
Previously, an admission plugin in the Gardener API server conducted the scheduling decisions. This implies changes to the API server whenever adjustments of the scheduling are needed. Decoupling the API server and the scheduler comes with greater flexibility to develop these components independently from each other.
2. Extensibility
It should be possible to easily extend and tweak the scheduler in the future. Possibly, similar to the Kubernetes scheduler, hooks could be provided which influence the scheduling decisions. It should be also possible to completely replace the standard Gardener Scheduler with a custom implementation.
Algorithm Overview
The following sequence describes the steps involved to determine a seed candidate:
- Determine usable seeds with “usable” defined as follows:
- no
.metadata.deletionTimestamp
.spec.settings.scheduling.visible
istrue
- conditions
Bootstrapped
,GardenletReady
,BackupBucketsReady
(if available) aretrue
- no
- Filter seeds:
- matching
.spec.seedSelector
inCloudProfile
used by theShoot
- matching
.spec.seedSelector
inShoot
- having no network intersection with the
Shoot
’s networks (due to the VPN connectivity between seeds and shoots their networks must be disjoint) - whose taints (
.spec.taints
) are tolerated by theShoot
(.spec.tolerations
) - whose capacity for shoots would not be exceeded if the shoot is scheduled onto the seed, see Ensuring seeds capacity for shoots is not exceeded
- which have at least three zones in
.spec.provider.zones
if shoot requests a high available control plane with failure tolerance typezone
.
- matching
- Apply active strategy e.g., Minimal Distance strategy
- Choose least utilized seed, i.e., the one with the least number of shoot control planes, will be the winner and written to the
.spec.seedName
field of theShoot
.
Configuration
The Gardener Scheduler configuration has to be supplied on startup. It is a mandatory and also the only available flag. This yaml file holds an example scheduler configuration.
Most of the configuration options are the same as in the Gardener Controller Manager (leader election, client connection, …). However, the Gardener Scheduler on the other hand does not need a TLS configuration, because there are currently no webhooks configurable.
Strategies
The scheduling strategy is defined in the candidateDeterminationStrategy of the scheduler’s configuration and can have the possible values SameRegion
and MinimalDistance
.
The SameRegion
strategy is the default strategy.
Same Region strategy
The Gardener Scheduler reads the
spec.provider.type
and.spec.region
fields from theShoot
resource. It tries to find a seed that has the identical.spec.provider.type
and.spec.provider.region
fields set. If it cannot find a suitable seed, it adds an event to the shoot stating that it is unschedulable.Minimal Distance strategy
The Gardener Scheduler tries to find a valid seed with minimal distance to the shoot’s intended region. The distance is calculated based on the Levenshtein distance of the region. Therefore, the region name is split into a base name and an orientation. Possible orientations are
north
,south
,east
,west
andcentral
. The distance then is twice the Levenshtein distance of the region’s base name plus a correction value based on the orientation and the provider.If the orientations of shoot and seed candidate match, the correction value is 0, if they differ it is 2 and if either the seed’s or the shoot’s region does not have an orientation it is 1. If the provider differs, the correction value is additionally incremented by 2.
Because of this, a matching region with a matching provider is always prefered.
In order to put the scheduling decision into effect, the scheduler sends an update request for the Shoot
resource to
the API server. After validation, the Gardener Aggregated API server updates the shoot to have the spec.seedName
field set.
Subsequently, the Gardenlet picks up and starts to create the cluster on the specified seed.
- Special handling based on shoot cluster purpose
Every shoot cluster can have a purpose that describes what the cluster is used for, and also influences how the cluster is setup (see Shoot Cluster Purpose for more information).
In case the shoot has the testing
purpose, then the scheduler only reads the .spec.provider.type
from the Shoot
resource and tries to find a Seed
that has the identical .spec.provider.type
.
The region does not matter, i.e., testing
shoots may also be scheduled on a seed in a complete different region if it is better for balancing the whole Gardener system.
shoots/binding
Subresource
The shoots/binding
subresource is used to bind a Shoot
to a Seed
. On creation of a shoot cluster/s, the scheduler updates the binding automatically if an appropriate seed cluster is available.
Only an operator with the necessary RBAC can update this binding manually. This can be done by changing the .spec.seedName
of the shoot. However, if a different seed is already assigned to the shoot, this will trigger a control-plane migration. For required steps, please see Triggering the Migration.
spec.seedName
Field in the Shoot
Specification
Similarly to the .spec.nodeName
field in Pod
s, the Shoot
specification has an optional .spec.seedName
field. If this field is set on creation, the shoot will be scheduled to this seed. However, this field can only be set by users having RBAC for the shoots/binding
subresource. If this field is not set, the scheduler
will assign a suitable seed automatically and populate this field with the seed name.
seedSelector
Field in the Shoot
Specification
Similarly to the .spec.nodeSelector
field in Pod
s, the Shoot
specification has an optional .spec.seedSelector
field.
It allows the user to provide a label selector that must match the labels of the Seed
s in order to be scheduled to one of them.
The labels on the Seed
s are usually controlled by Gardener administrators/operators - end users cannot add arbitrary labels themselves.
If provided, the Gardener Scheduler will only consider as “suitable” those seeds whose labels match those provided in the .spec.seedSelector
of the Shoot
.
By default, only seeds with the same provider as the shoot are selected. By adding a providerTypes
field to the seedSelector
,
a dedicated set of possible providers (*
means all provider types) can be selected.
Ensuring a Seed’s Capacity for Shoots Is Not Exceeded
Seeds have a practical limit of how many shoots they can accommodate. Exceeding this limit is undesirable, as the system performance will be noticeably impacted. Therefore, the scheduler ensures that a seed’s capacity for shoots is not exceeded by taking into account a maximum number of shoots that can be scheduled onto a seed.
This mechanism works as follows:
- The
gardenlet
is configured with certain resources and their total capacity (and, for certain resources, the amount reserved for Gardener), see /example/20-componentconfig-gardenlet.yaml. Currently, the only such resource is the maximum number of shoots that can be scheduled onto a seed. - The
gardenlet
seed controller updates thecapacity
andallocatable
fields in the Seed status with the capacity of each resource and how much of it is actually available to be consumed by shoots. Theallocatable
value of a resource is equal tocapacity
minusreserved
. - When scheduling shoots, the scheduler filters out all candidate seeds whose allocatable capacity for shoots would be exceeded if the shoot is scheduled onto the seed.
Failure to Determine a Suitable Seed
In case the scheduler fails to find a suitable seed, the operation is being retried with exponential backoff.
Current Limitation / Future Plans
- Azure unfortunately has a geographically non-hierarchical naming pattern and does not start with the continent. This is the reason why we will exchange the implementation of the
MinimalDistance
strategy with a more suitable one in the future.
1.3 - Deployment
1.3.1 - Authentication Gardener Control Plane
Authentication of Gardener Control Plane Components Against the Garden Cluster
Note: This document refers to Gardener’s API server, admission controller, controller manager and scheduler components. Any reference to the term Gardener control plane component can be replaced with any of the mentioned above.
There are several authentication possibilities depending on whether or not the concept of Virtual Garden is used.
Virtual Garden is not used, i.e., the runtime
Garden cluster is also the target
Garden cluster.
Automounted Service Account Token
The easiest way to deploy a Gardener control plane component is to not provide a kubeconfig
at all. This way in-cluster configuration and an automounted service account token will be used. The drawback of this approach is that the automounted token will not be automatically rotated.
Service Account Token Volume Projection
Another solution is to use Service Account Token Volume Projection combined with a kubeconfig
referencing a token file (see the example below).
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: <CA-DATA>
server: https://default.kubernetes.svc.cluster.local
name: garden
contexts:
- context:
cluster: garden
user: garden
name: garden
current-context: garden
users:
- name: garden
user:
tokenFile: /var/run/secrets/projected/serviceaccount/token
This will allow for automatic rotation of the service account token by the kubelet
. The configuration can be achieved by setting both .Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true
and .Values.global.<GardenerControlPlaneComponent>.kubeconfig
in the respective chart’s values.yaml
file.
Virtual Garden is used, i.e., the runtime
Garden cluster is different from the target
Garden cluster.
Service Account
The easiest way to setup the authentication is to create a service account and the respective roles will be bound to this service account in the target
cluster. Then use the generated service account token and craft a kubeconfig
, which will be used by the workload in the runtime
cluster. This approach does not provide a solution for the rotation of the service account token. However, this setup can be achieved by setting .Values.global.deployment.virtualGarden.enabled: true
and following these steps:
- Deploy the
application
part of the charts in thetarget
cluster. - Get the service account token and craft the
kubeconfig
. - Set the crafted
kubeconfig
and deploy theruntime
part of the charts in theruntime
cluster.
Client Certificate
Another solution is to bind the roles in the target
cluster to a User
subject instead of a service account and use a client certificate for authentication. This approach does not provide a solution for the client certificate rotation. However, this setup can be achieved by setting both .Values.global.deployment.virtualGarden.enabled: true
and .Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name
, then following these steps:
- Generate a client certificate for the
target
cluster for the respective user. - Deploy the
application
part of the charts in thetarget
cluster. - Craft a
kubeconfig
using the already generated client certificate. - Set the crafted
kubeconfig
and deploy theruntime
part of the charts in theruntime
cluster.
Projected Service Account Token
This approach requires an already deployed and configured oidc-webhook-authenticator for the target
cluster. Also, the runtime
cluster should be registered as a trusted identity provider in the target
cluster. Then, projected service accounts tokens from the runtime
cluster can be used to authenticate against the target
cluster. The needed steps are as follows:
- Deploy OWA and establish the needed trust.
- Set
.Values.global.deployment.virtualGarden.enabled: true
and.Values.global.deployment.virtualGarden.<GardenerControlPlaneComponent>.user.name
.Note: username value will depend on the trust configuration, e.g.,
<prefix>:system:serviceaccount:<namespace>:<serviceaccount>
- Set
.Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.enabled: true
and.Values.global.<GardenerControlPlaneComponent>.serviceAccountTokenVolumeProjection.audience
.Note: audience value will depend on the trust configuration, e.g.,
<cliend-id-from-trust-config>
. - Craft a kubeconfig (see the example below).
- Deploy the
application
part of the charts in thetarget
cluster. - Deploy the
runtime
part of the charts in theruntime
cluster.
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: <CA-DATA>
server: https://virtual-garden.api
name: virtual-garden
contexts:
- context:
cluster: virtual-garden
user: virtual-garden
name: virtual-garden
current-context: virtual-garden
users:
- name: virtual-garden
user:
tokenFile: /var/run/secrets/projected/serviceaccount/token
1.3.2 - Configuring Logging
Configuring the Logging Stack via gardenlet Configurations
Enable the Logging
In order to install the Gardener logging stack, the logging.enabled
configuration option has to be enabled in the Gardenlet configuration:
logging:
enabled: true
From now on, each Seed is going to have a logging stack which will collect logs from all pods and some systemd services. Logs related to Shoots with testing
purpose are dropped in the fluent-bit
output plugin. Shoots with a purpose different than testing
have the same type of log aggregator (but different instance) as the Seed. The logs can be viewed in the Grafana in the garden
namespace for the Seed components and in the respective shoot control plane namespaces.
Enable Logs from the Shoot’s Node systemd Services
The logs from the systemd services on each node can be retrieved by enabling the logging.shootNodeLogging
option in the gardenlet configuration:
logging:
enabled: true
shootNodeLogging:
shootPurposes:
- "evaluation"
- "deployment"
Under the shootPurpose
section, just list all the shoot purposes for which the Shoot node logging feature will be enabled. Specifying the testing
purpose has no effect because this purpose prevents the logging stack installation.
Logs can be viewed in the operator Grafana!
The dedicated labels are unit
, syslog_identifier
, and nodename
in the Explore
menu.
Configuring the Log Processor
Under logging.fluentBit
there are three optional sections:
input
: This overwrites the input configuration of the fluent-bit log processor.output
: This overwrites the output configuration of the fluent-bit log processor.service
: This overwrites the service configuration of the fluent-bit log processor.
logging:
enabled: true
fluentBit:
output: |-
[Output]
...
input: |-
[Input]
...
service: |-
[Service]
...
Additional egress IPBlock for allow-fluentbit NetworkPolicy
The optional setting under logging.fluentBit.networkPolicy.additionalEgressIPBlocks
adds an additional egress IPBlock to allow-fluentbit
NetworkPolicy to forward logs to a central system.
logging:
enabled: true
fluentBit:
additionalEgressIpBlock:
- 123.123.123.123/32
Configure Central Logging
For central logging, the output configuration of the fluent-bit log processor can be overwritten (logging.fluentBit.output
) and the Loki instances deployments in the Garden and Shoot namespace can be enabled/disabled (logging.loki.enabled
), by default Loki is enabled.
logging:
enabled: true
fluentBit:
output: |-
[Output]
...
loki:
enabled: false
Configuring Central Loki Storage Capacity
By default, the central Loki has 100Gi
of storage capacity.
To overwrite the current central Loki storage capacity, the logging.loki.garden.storage
setting in the gardenlet’s component configuration should be altered.
If you need to increase it, you can do so without losing the current data by specifying a higher capacity. By doing so, the Loki’s PersistentVolume
capacity will be increased instead of deleting the current PV.
However, if you specify less capacity, then the PersistentVolume
will be deleted and with it the logs, too.
logging:
enabled: true
fluentBit:
output: |-
[Output]
...
loki:
garden:
storage: "200Gi"
1.3.3 - Deploy Gardenlet
Deploying Gardenlets
Gardenlets act as decentral “agents” to manage the shoot clusters of a seed cluster.
To support scaleability in an automated way, gardenlets are deployed automatically. However, you can still deploy gardenlets manually to be more flexible, for example, when the shoot clusters that need to be managed by Gardener are behind a firewall. The gardenlet only requires network connectivity from the gardenlet to the Garden cluster (not the other way round), so it can be used to register Kubernetes clusters with no public endpoint.
Procedure
First, an initial gardenlet needs to be deployed:
- Deploy it manually if you have special requirements. For more information, see Deploy a Gardenlet Manually.
- Let the Gardener installer deploy it automatically otherwise. For more information, see Automatic Deployment of Gardenlets.
To add additional seed clusters, it is recommended to use regular shoot clusters. You can do this by creating a
ManagedSeed
resource with agardenlet
section as described in Register Shoot as Seed.
1.3.4 - Deploy Gardenlet Automatically
Automatic Deployment of gardenlets
The gardenlet can automatically deploy itself into shoot clusters, and register a cluster as a seed cluster. These clusters are called “managed seeds” (aka “shooted seeds”). This procedure is the preferred way to add additional seed clusters, because shoot clusters already come with production-grade qualities that are also demanded for seed clusters.
Prerequisites
The only prerequisite is to register an initial cluster as a seed cluster that already has a gardenlet deployed in one of the following ways:
- The gardenlet was deployed as part of a Gardener installation using a setup tool (for example,
gardener/garden-setup
). - The gardenlet was deployed manually (for a step-by-step manual installation guide, see Deploy a Gardenlet Manually).
The initial cluster can be the garden cluster itself.
Self-Deployment of gardenlets in Additional Managed Seed Clusters
For a better scalability, you usually need more seed clusters that you can create, as follows:
- Use the initial cluster as the seed cluster for other managed seed clusters. It hosts the control planes of the other seed clusters.
- The gardenlet deployed in the initial cluster deploys itself automatically into the managed seed clusters.
The advantage of this approach is that there’s only one initial gardenlet installation required. Every other managed seed cluster has a gardenlet deployed automatically.
Related Links
1.3.5 - Deploy Gardenlet Manually
Deploy a gardenlet Manually
Manually deploying a gardenlet is required in the following cases:
The Kubernetes cluster to be registered as a seed cluster has no public endpoint, because it is behind a firewall. The gardenlet must then be deployed into the cluster itself.
The Kubernetes cluster to be registered as a seed cluster is managed externally (the Kubernetes cluster is not a shoot cluster, so Automatic Deployment of Gardenlets cannot be used).
The gardenlet runs outside of the Kubernetes cluster that should be registered as a seed cluster. (The gardenlet is not restricted to run in the seed cluster or to be deployed into a Kubernetes cluster at all).
Once you’ve deployed a gardenlet manually, for example, behind a firewall, you can deploy new gardenlets automatically. The manually deployed gardenlet is then used as a template for the new gardenlets. For more information, see Automatic Deployment of Gardenlets.
Prerequisites
Kubernetes Cluster that Should Be Registered as a Seed Cluster
Verify that the cluster has a supported Kubernetes version.
Determine the nodes, pods, and services CIDR of the cluster. You need to configure this information in the
Seed
configuration. Gardener uses this information to check that the shoot cluster isn’t created with overlapping CIDR ranges.Every seed cluster needs an Ingress controller which distributes external requests to internal components like Grafana and Prometheus. Gardener supports two approaches to achieve this:
a. Gardener managed Ingress controller and DNS records. For this, configure the following lines in your Seed resource:
spec:
dns:
provider:
type: aws-route53
secretRef:
name: ingress-secret
namespace: garden
ingress:
domain: ingress.my-seed.example.com
controller:
kind: nginx
providerConfig:
<some-optional-provider-specific-config-for-the-ingressController>
⚠ Please note that if you set .spec.ingress
, then .spec.dns.ingressDomain
must be nil
.
b. Self-managed DNS record and Ingress controller:
⚠️
There should exist a DNS record *.ingress.<SEED-CLUSTER-DOMAIN>
, where <SEED-CLUSTER-DOMAIN>
is the value of the .dns.ingressDomain
field of a Seed cluster resource (or the respective Gardenlet configuration).
This is how it could be done for the Nginx ingress controller
Deploy nginx into the kube-system
namespace in the Kubernetes cluster that should be registered as a Seed
.
Nginx will, on most cloud providers, create the service with type LoadBalancer
with an external IP.
NAME TYPE CLUSTER-IP EXTERNAL-IP
nginx-ingress-controller LoadBalancer 10.0.15.46 34.200.30.30
Create a wildcard A
record (e.g *.ingress.sweet-seed.Seed
cluster.
Please configure the ingress domain in the Seed
specification as follows:
spec:
dns:
ingressDomain: ingress.sweet-seed.<my-domain>
⚠ Please note that if you set .spec.dns.ingressDomain
, then .spec.ingress
must be nil
.
kubeconfig
for the Seed Cluster
The kubeconfig
is required to deploy the gardenlet Helm chart to the seed cluster.
The gardenlet requires certain privileges to be able to operate.
These privileges are described in RBAC resources in the gardenlet Helm chart (see charts/gardener/gardenlet/templates).
The Helm chart contains a service account gardenlet
that the gardenlet deployment uses by default to talk to the Seed API server.
If the gardenlet isn’t deployed in the seed cluster, the gardenlet can be configured to use a
kubeconfig
, which also requires the above-mentioned privileges, from a mounted directory. Thekubeconfig
is specified in theseedClientConnection.kubeconfig
section of the Gardenlet configuration. This configuration option isn’t used in the following, as this procedure only describes the recommended setup option where the gardenlet is running in the seed cluster itself.
Procedure Overview
Prepare the garden cluster:
Create a Bootstrap Token Secret in the kube-system
Namespace of the Garden Cluster
The gardenlet needs to talk to the Gardener API server residing in the garden cluster.
The gardenlet can be configured with an already existing garden cluster kubeconfig
in one of the following ways:
- By specifying
gardenClientConnection.kubeconfig
in the Gardenlet configuration. - By supplying the environment variable
GARDEN_KUBECONFIG
pointing to a mountedkubeconfig
file).
The preferred way, however, is to use the gardenlet’s ability to request a signed certificate for the garden cluster by leveraging Kubernetes Certificate Signing Requests. The gardenlet performs a TLS bootstrapping process that is similar to the Kubelet TLS Bootstrapping. Make sure that the API server of the garden cluster has bootstrap token authentication enabled.
The client credentials required for the gardenlet’s TLS bootstrapping process
need to be either token
or certificate
(OIDC isn’t supported) and have permissions
to create a Certificate Signing Request (CSR).
It’s recommended to use bootstrap tokens
due to their desirable security properties (such as a limited token lifetime).
Therefore, first create a bootstrap token secret for the garden cluster:
apiVersion: v1
kind: Secret
metadata:
# Name MUST be of form "bootstrap-token-<token id>"
name: bootstrap-token-07401b
namespace: kube-system
# Type MUST be 'bootstrap.kubernetes.io/token'
type: bootstrap.kubernetes.io/token
stringData:
# Human readable description. Optional.
description: "Token to be used by the gardenlet for Seed `sweet-seed`."
# Token ID and secret. Required.
token-id: 07401b # 6 characters
token-secret: f395accd246ae52d # 16 characters
# Expiration. Optional.
# expiration: 2017-03-10T03:22:11Z
# Allowed usages.
usage-bootstrap-authentication: "true"
usage-bootstrap-signing: "true"
When you later prepare the gardenlet Helm chart,
a kubeconfig
based on this token is shared with the gardenlet upon deployment.
Create RBAC Roles for the gardenlet to Allow Bootstrapping in the Garden Cluster
This step is only required if the gardenlet you deploy is the first gardenlet in the Gardener installation. Additionally, when using the control plane chart, the following resources are already contained in the Helm chart, that is, if you use it you can skip these steps as the needed RBAC roles already exist.
The gardenlet uses the configured bootstrap kubeconfig
in gardenClientConnection.bootstrapKubeconfig
to request a signed certificate for the user gardener.cloud:system:seed:<seed-name>
in the group gardener.cloud:system:seeds
.
Create a ClusterRole
and ClusterRoleBinding
that grant full admin permissions to authenticated gardenlets.
Create the following resources in the garden cluster:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: gardener.cloud:system:seeds
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: gardener.cloud:system:seeds
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: gardener.cloud:system:seeds
subjects:
- kind: Group
name: gardener.cloud:system:seeds
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: gardener.cloud:system:seed-bootstrapper
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests
verbs:
- create
- get
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/seedclient
verbs:
- create
---
# A kubelet/gardenlet authenticating using bootstrap tokens is authenticated as a user in the group system:bootstrappers
# Allows the Gardenlet to create a CSR
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: gardener.cloud:system:seed-bootstrapper
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: gardener.cloud:system:seed-bootstrapper
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
ℹ️ After bootstrapping, the gardenlet has full administrative access to the garden cluster. You might be interested to harden this and limit its permissions to only resources related to the seed cluster it is responsible for. Please take a look at Scoped API Access for Gardenlets.
Prepare the gardenlet Helm Chart
This section only describes the minimal configuration, using the global configuration values of the gardenlet Helm chart. For an overview over all values, see the configuration values. We refer to the global configuration values as gardenlet configuration in the following procedure.
Create a gardenlet configuration
gardenlet-values.yaml
based on this template.Create a bootstrap
kubeconfig
based on the bootstrap token created in the garden cluster.Replace the
<bootstrap-token>
withtoken-id.token-secret
(from our previous example:07401b.f395accd246ae52d
) from the bootstrap token secret.apiVersion: v1 kind: Config current-context: gardenlet-bootstrap@default clusters: - cluster: certificate-authority-data: <ca-of-garden-cluster> server: https://<endpoint-of-garden-cluster> name: default contexts: - context: cluster: default user: gardenlet-bootstrap name: gardenlet-bootstrap@default users: - name: gardenlet-bootstrap user: token: <bootstrap-token>
In the
gardenClientConnection.bootstrapKubeconfig
section of your gardenlet configuration, provide the bootstrapkubeconfig
together with a name and namespace to the gardenlet Helm chart.gardenClientConnection: bootstrapKubeconfig: name: gardenlet-kubeconfig-bootstrap namespace: garden kubeconfig: | <bootstrap-kubeconfig> # will be base64 encoded by helm
The bootstrap
kubeconfig
is stored in the specified secret.In the
gardenClientConnection.kubeconfigSecret
section of your gardenlet configuration, define a name and a namespace where the gardenlet stores the realkubeconfig
that it creates during the bootstrap process. If the secret doesn’t exist, the gardenlet creates it for you.gardenClientConnection: kubeconfigSecret: name: gardenlet-kubeconfig namespace: garden
Updating the Garden Cluster CA
The kubeconfig created by the gardenlet in step 4 will not be recreated as long as it exists, even if a new bootstrap kubeconfig is provided. To enable rotation of the garden cluster CA certificate, a new bundle can be provided via the gardenClientConnection.gardenClusterCACert
field. If the provided bundle differs from the one currently in the gardenlet’s kubeconfig secret then it will be updated. To remove the CA completely (e.g. when switching to a publicly trusted endpoint), this field can be set to either none
or null
.
Automatically Register a Shoot Cluster as a Seed Cluster
A seed cluster can either be registered by manually creating
the Seed
resource
or automatically by the gardenlet.
This functionality is useful for managed seed clusters,
as the gardenlet in the garden cluster deploys a copy of itself
into the cluster with automatic registration of the Seed
configured.
However, it can also be used to have a streamlined seed cluster registration process when manually deploying the gardenlet.
This procedure doesn’t describe all the possible configurations for the
Seed
resource. For more information, see:
Adjust the gardenlet Component Configuration
Supply the
Seed
resource in theseedConfig
section of your gardenlet configurationgardenlet-values.yaml
.Add the
seedConfig
to your gardenlet configurationgardenlet-values.yaml
. The fieldseedConfig.spec.provider.type
specifies the infrastructure provider type (for example,aws
) of the seed cluster. For all supported infrastructure providers, see Known Extension Implementations..... seedConfig: metadata: name: sweet-seed spec: dns: ingressDomain: ingress.sweet-seed.<my-domain> # see prerequisites networks: # see prerequisites nodes: 10.240.0.0/16 pods: 100.244.0.0/16 services: 100.32.0.0/13 shootDefaults: # optional: non-overlapping default CIDRs for shoot clusters of that Seed pods: 100.96.0.0/11 services: 100.64.0.0/13 provider: region: eu-west-1 type: <provider>
Optional: Enable HA Mode
You may consider running gardenlet
with multiple replicas, especially if the seed cluster is configured to host HA shoot control planes.
Therefore, the following Helm chart values define the degree of high availability you want to achieve for the gardenlet
deployment.
replicaCount: 2 # or more if a higher failure tolerance is required.
failureToleranceType: zone # One of `zone` or `node` - defines how replicas are spread.
Optional: Enable Backup and Restore
The seed cluster can be set up with backup and restore
for the main etcds
of shoot clusters.
Gardener uses etcd-backup-restore
that integrates with different storage providers
to store the shoot cluster’s main etcd
backups.
Make sure to obtain client credentials that have sufficient permissions with the chosen storage provider.
Create a secret in the garden cluster with client credentials for the storage provider. The format of the secret is cloud provider specific and can be found in the repository of the respective Gardener extension. For example, the secret for AWS S3 can be found in the AWS provider extension (30-etcd-backup-secret.yaml).
apiVersion: v1
kind: Secret
metadata:
name: sweet-seed-backup
namespace: garden
type: Opaque
data:
# client credentials format is provider specific
Configure the Seed
resource in the seedConfig
section of your gardenlet configuration to use backup and restore:
...
seedConfig:
metadata:
name: sweet-seed
spec:
backup:
provider: <provider>
secretRef:
name: sweet-seed-backup
namespace: garden
Deploy the gardenlet
The gardenlet doesn’t have to run in the same Kubernetes cluster as the seed cluster it’s registering and reconciling, but it is in most cases advantageous to use in-cluster communication to talk to the Seed API server. Running a gardenlet outside of the cluster is mostly used for local development.
The gardenlet-values.yaml
looks something like this
(with automatic Seed registration and backup for shoot clusters enabled):
<default config>
...
config:
gardenClientConnection:
...
bootstrapKubeconfig:
name: gardenlet-bootstrap-kubeconfig
namespace: garden
kubeconfig: |
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <dummy>
server: <my-garden-cluster-endpoint>
name: my-kubernetes-cluster
....
kubeconfigSecret:
name: gardenlet-kubeconfig
namespace: garden
...
<default config>
...
seedConfig:
metadata:
name: sweet-seed
spec:
dns:
ingressDomain: ingress.sweet-seed.<my-domain>
networks:
nodes: 10.240.0.0/16
pods: 100.244.0.0/16
services: 100.32.0.0/13
shootDefaults:
pods: 100.96.0.0/11
services: 100.64.0.0/13
provider:
region: eu-west-1
type: <provider>
backup:
provider: <provider>
secretRef:
name: sweet-seed-backup
namespace: garden
Deploy the gardenlet Helm chart to the Kubernetes cluster:
helm install gardenlet charts/gardener/gardenlet \
--namespace garden \
-f gardenlet-values.yaml \
--wait
This helm chart creates:
- A service account
gardenlet
that the gardenlet can use to talk to the Seed API server. - RBAC roles for the service account (full admin rights at the moment).
- The secret (
garden
/gardenlet-bootstrap-kubeconfig
) containing the bootstrapkubeconfig
. - The gardenlet deployment in the
garden
namespace.
Check that the gardenlet Is Successfully Deployed
Check that the gardenlets certificate bootstrap was successful.
Check if the secret
gardenlet-kubeconfig
in the namespacegarden
in the seed cluster is created and contains akubeconfig
with a valid certificate.Get the
kubeconfig
from the created secret.$ kubectl -n garden get secret gardenlet-kubeconfig -o json | jq -r .data.kubeconfig | base64 -d
Test against the garden cluster and verify it’s working.
Extract the
client-certificate-data
from the usergardenlet
.View the certificate:
$ openssl x509 -in ./gardenlet-cert -noout -text
Check that the certificate is valid for a year (that is the lifetime of new certificates).
Check that the bootstrap secret
gardenlet-bootstrap-kubeconfig
has been deleted from the seed cluster in namespacegarden
.Check that the seed cluster is registered and
READY
in the garden cluster.Check that the seed cluster
sweet-seed
exists and all conditions indicate that it’s available. If so, the Gardenlet is sending regular heartbeats and the seed bootstrapping was successful.Check that the conditions on the
Seed
resource look similar to the following:$ kubectl get seed sweet-seed -o json | jq .status.conditions [ { "lastTransitionTime": "2020-07-17T09:17:29Z", "lastUpdateTime": "2020-07-17T09:17:29Z", "message": "Gardenlet is posting ready status.", "reason": "GardenletReady", "status": "True", "type": "GardenletReady" }, { "lastTransitionTime": "2020-07-17T09:17:49Z", "lastUpdateTime": "2020-07-17T09:53:17Z", "message": "Seed cluster has been bootstrapped successfully.", "reason": "BootstrappingSucceeded", "status": "True", "type": "Bootstrapped" }, { "lastTransitionTime": "2020-07-17T09:17:49Z", "lastUpdateTime": "2020-07-17T09:53:17Z", "message": "Backup Buckets are available.", "reason": "BackupBucketsAvailable", "status": "True", "type": "BackupBucketsReady" } ]
Related Links
1.3.6 - Feature Gates
Feature Gates in Gardener
This page contains an overview of the various feature gates an administrator can specify on different Gardener components.
Overview
Feature gates are a set of key=value pairs that describe Gardener features. You can turn these features on or off using the component configuration file for a specific component.
Each Gardener component lets you enable or disable a set of feature gates that are relevant to that component. For example, this is the configuration of the gardenlet component.
The following tables are a summary of the feature gates that you can set on different Gardener components.
- The “Since” column contains the Gardener release when a feature is introduced or its release stage is changed.
- The “Until” column, if not empty, contains the last Gardener release in which you can still use a feature gate.
- If a feature is in the Alpha or Beta state, you can find the feature listed in the Alpha/Beta feature gate table.
- If a feature is stable you can find all stages for that feature listed in the Graduated/Deprecated feature gate table.
- The Graduated/Deprecated feature gate table also lists deprecated and withdrawn features.
Feature Gates for Alpha or Beta Features
Feature | Default | Stage | Since | Until |
---|---|---|---|---|
HVPA | false | Alpha | 0.31 | |
HVPAForShootedSeed | false | Alpha | 0.32 | |
ManagedIstio | false | Alpha | 1.5 | 1.18 |
ManagedIstio | true | Beta | 1.19 | |
ManagedIstio (deprecated) | true | Beta | 1.48 | |
APIServerSNI | false | Alpha | 1.7 | 1.18 |
APIServerSNI | true | Beta | 1.19 | |
APIServerSNI (deprecated) | true | Beta | 1.48 | |
SeedChange | false | Alpha | 1.12 | 1.52 |
SeedChange | true | Beta | 1.53 | |
CopyEtcdBackupsDuringControlPlaneMigration | false | Alpha | 1.37 | 1.52 |
CopyEtcdBackupsDuringControlPlaneMigration | true | Beta | 1.53 | |
ForceRestore | false | Alpha | 1.39 | |
HAControlPlanes | false | Alpha | 1.49 | |
DefaultSeccompProfile | false | Alpha | 1.54 | |
CoreDNSQueryRewriting | false | Alpha | 1.55 | |
IPv6SingleStack | false | Alpha | 1.63 |
Feature Gates for Graduated or Deprecated Features
Feature | Default | Stage | Since | Until |
---|---|---|---|---|
NodeLocalDNS | false | Alpha | 1.7 | |
NodeLocalDNS | Removed | 1.26 | ||
KonnectivityTunnel | false | Alpha | 1.6 | |
KonnectivityTunnel | Removed | 1.27 | ||
MountHostCADirectories | false | Alpha | 1.11 | 1.25 |
MountHostCADirectories | true | Beta | 1.26 | 1.27 |
MountHostCADirectories | true | GA | 1.27 | |
MountHostCADirectories | Removed | 1.30 | ||
DisallowKubeconfigRotationForShootInDeletion | false | Alpha | 1.28 | 1.31 |
DisallowKubeconfigRotationForShootInDeletion | true | Beta | 1.32 | 1.35 |
DisallowKubeconfigRotationForShootInDeletion | true | GA | 1.36 | |
DisallowKubeconfigRotationForShootInDeletion | Removed | 1.38 | ||
Logging | false | Alpha | 0.13 | 1.40 |
Logging | Removed | 1.41 | ||
AdminKubeconfigRequest | false | Alpha | 1.24 | 1.38 |
AdminKubeconfigRequest | true | Beta | 1.39 | 1.41 |
AdminKubeconfigRequest | true | GA | 1.42 | 1.49 |
AdminKubeconfigRequest | Removed | 1.50 | ||
UseDNSRecords | false | Alpha | 1.27 | 1.38 |
UseDNSRecords | true | Beta | 1.39 | 1.43 |
UseDNSRecords | true | GA | 1.44 | 1.49 |
UseDNSRecords | Removed | 1.50 | ||
CachedRuntimeClients | false | Alpha | 1.7 | 1.33 |
CachedRuntimeClients | true | Beta | 1.34 | 1.44 |
CachedRuntimeClients | true | GA | 1.45 | 1.49 |
CachedRuntimeClients | Removed | 1.50 | ||
DenyInvalidExtensionResources | false | Alpha | 1.31 | 1.41 |
DenyInvalidExtensionResources | true | Beta | 1.42 | 1.44 |
DenyInvalidExtensionResources | true | GA | 1.45 | 1.49 |
DenyInvalidExtensionResources | Removed | 1.50 | ||
RotateSSHKeypairOnMaintenance | false | Alpha | 1.28 | 1.44 |
RotateSSHKeypairOnMaintenance | true | Beta | 1.45 | 1.47 |
RotateSSHKeypairOnMaintenance (deprecated) | false | Beta | 1.48 | 1.50 |
RotateSSHKeypairOnMaintenance (deprecated) | Removed | 1.51 | ||
ShootMaxTokenExpirationOverwrite | false | Alpha | 1.43 | 1.44 |
ShootMaxTokenExpirationOverwrite | true | Beta | 1.45 | 1.47 |
ShootMaxTokenExpirationOverwrite | true | GA | 1.48 | 1.50 |
ShootMaxTokenExpirationOverwrite | Removed | 1.51 | ||
ShootMaxTokenExpirationValidation | false | Alpha | 1.43 | 1.45 |
ShootMaxTokenExpirationValidation | true | Beta | 1.46 | 1.47 |
ShootMaxTokenExpirationValidation | true | GA | 1.48 | 1.50 |
ShootMaxTokenExpirationValidation | Removed | 1.51 | ||
WorkerPoolKubernetesVersion | false | Alpha | 1.35 | 1.45 |
WorkerPoolKubernetesVersion | true | Beta | 1.46 | 1.49 |
WorkerPoolKubernetesVersion | true | GA | 1.50 | 1.51 |
WorkerPoolKubernetesVersion | Removed | 1.52 | ||
DisableDNSProviderManagement | false | Alpha | 1.41 | 1.49 |
DisableDNSProviderManagement | true | Beta | 1.50 | 1.51 |
DisableDNSProviderManagement | true | GA | 1.52 | 1.59 |
DisableDNSProviderManagement | Removed | 1.60 | ||
SecretBindingProviderValidation | false | Alpha | 1.38 | 1.50 |
SecretBindingProviderValidation | true | Beta | 1.51 | 1.52 |
SecretBindingProviderValidation | true | GA | 1.53 | 1.54 |
SecretBindingProviderValidation | Removed | 1.55 | ||
SeedKubeScheduler | false | Alpha | 1.15 | 1.54 |
SeedKubeScheduler | false | Deprecated | 1.55 | 1.60 |
SeedKubeScheduler | Removed | 1.61 | ||
ShootCARotation | false | Alpha | 1.42 | 1.50 |
ShootCARotation | true | Beta | 1.51 | 1.56 |
ShootCARotation | true | GA | 1.57 | 1.59 |
ShootCARotation | Removed | 1.60 | ||
ShootSARotation | false | Alpha | 1.48 | 1.50 |
ShootSARotation | true | Beta | 1.51 | 1.56 |
ShootSARotation | true | GA | 1.57 | 1.59 |
ShootSARotation | Removed | 1.60 | ||
ReversedVPN | false | Alpha | 1.22 | 1.41 |
ReversedVPN | true | Beta | 1.42 | 1.62 |
ReversedVPN | true | GA | 1.63 |
Using a Feature
A feature can be in Alpha, Beta or GA stage. An Alpha feature means:
- Disabled by default.
- Might be buggy. Enabling the feature may expose bugs.
- Support for feature may be dropped at any time without notice.
- The API may change in incompatible ways in a later software release without notice.
- Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.
A Beta feature means:
- Enabled by default.
- The feature is well tested. Enabling the feature is considered safe.
- Support for the overall feature will not be dropped, though details may change.
- The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
- Recommended for only non-critical uses because of potential for incompatible changes in subsequent releases.
Please do try Beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.
A General Availability (GA) feature is also referred to as a stable feature. It means:
- The feature is always enabled; you cannot disable it.
- The corresponding feature gate is no longer needed.
- Stable versions of features will appear in released software for many subsequent versions.
List of Feature Gates
Feature | Relevant Components | Description |
---|---|---|
HVPA | gardenlet , gardener-operator | Enables simultaneous horizontal and vertical scaling in garden or seed clusters. |
HVPAForShootedSeed | gardenlet | Enables simultaneous horizontal and vertical scaling in managed seed (aka “shooted seed”) clusters. |
ManagedIstio (deprecated) | gardenlet | Enables a Gardener-tailored Istio in each Seed cluster. Disable this feature if Istio is already installed in the cluster. Istio is not automatically removed if this feature is disabled. See the detailed documentation for more information. |
APIServerSNI (deprecated) | gardenlet | Enables only one LoadBalancer to be used for every Shoot cluster API server in a Seed. Enable this feature when ManagedIstio is enabled or Istio is manually deployed in the Seed cluster. See GEP-8 for more details. |
SeedChange | gardener-apiserver | Enables updating the spec.seedName field during shoot validation from a non-empty value in order to trigger shoot control plane migration. |
ReversedVPN | gardenlet | Reverses the connection setup of the VPN tunnel between the Seed and the Shoot cluster(s). It allows Seed and Shoot clusters to be in different networks with only direct access in one direction (Shoot -> Seed). In addition to that, it reduces the amount of load balancers required, i.e. no load balancers are required for the VPN tunnel anymore. It requires APIServerSNI and kubernetes version 1.18 or higher to work. Details can be found in GEP-14. |
CopyEtcdBackupsDuringControlPlaneMigration | gardenlet | Enables the copy of etcd backups from the object store of the source seed to the object store of the destination seed during control plane migration. |
SecretBindingProviderValidation | gardener-apiserver | Enables validations on Gardener API server that: - requires the provider type of a SecretBinding to be set (on SecretBinding creation) - requires the SecretBinding provider type to match the Shoot provider type (on Shoot creation) - enforces immutability on the provider type of a SecretBinding |
ForceRestore | gardenlet | Enables forcing the shoot’s restoration to the destination seed during control plane migration if the preparation for migration in the source seed is not finished after a certain grace period and is considered unlikely to succeed (falling back to the control plane migration “bad case” scenario). If you enable this feature gate, make sure to also enable CopyEtcdBackupsDuringControlPlaneMigration . |
HAControlPlanes | gardener-apiserver | HAControlPlanes allows shoot control planes to be run in high availability mode. |
DefaultSeccompProfile | gardenlet , gardener-operator | Enables the defaulting of the seccomp profile for Gardener managed workload in the garden or seed to RuntimeDefault . |
CoreDNSQueryRewriting | gardenlet | Enables automatic DNS query rewriting in shoot cluster’s CoreDNS to shortcut name resolution of fully qualified in-cluster and out-of-cluster names, which follow a user-defined pattern. Details can be found in DNS Search Path Optimization. |
IPv6SingleStack | gardener-apiserver | Allows creating shoot clusters with IPv6 single-stack networking (GEP-21). |
1.3.7 - gardenlet API Access
Scoped API Access for gardenlets
By default, gardenlet
s have administrative access in the garden cluster.
They are able to execute any API request on any object independent of whether the object is related to the seed cluster the gardenlet
is responsible for.
As RBAC is not powerful enough for fine-grained checks and for the sake of security, Gardener provides two optional but recommended configurations for your environments that scope the API access for gardenlet
s.
Similar to the Node
authorization mode in Kubernetes, Gardener features a SeedAuthorizer
plugin.
It is a special-purpose authorization plugin that specifically authorizes API requests made by the gardenlet
s.
Likewise, similar to the NodeRestriction
admission plugin in Kubernetes, Gardener features a SeedRestriction
plugin.
It is a special-purpose admission plugin that specifically limits the Kubernetes objects gardenlet
s can modify.
📚 You might be interested to look into the design proposal for scoped Kubelet API access from the Kubernetes community.
It can be translated to Gardener and Gardenlets with their Seed
and Shoot
resources.
Flow Diagram
The following diagram shows how the two plugins are included in the request flow of a gardenlet
.
When they are not enabled, then the kube-apiserver
is internally authorizing the request via RBAC before forwarding the request directly to the gardener-apiserver
, i.e., the gardener-admission-controller
would not be consulted (this is not entirely correct because it also serves other admission webhook handlers, but for simplicity reasons this document focuses on the API access scope only).
When enabling the plugins, there is one additional step for each before the gardener-apiserver
responds to the request.
Please note that the example shows a request to an object (Shoot
) residing in one of the API groups served by gardener-apiserver
.
However, the gardenlet
is also interacting with objects in API groups served by the kube-apiserver
(e.g., Secret
,ConfigMap
, etc.).
In this case, the consultation of the SeedRestriction
admission plugin is performed by the kube-apiserver
itself before it forwards the request to the gardener-apiserver
.
Today, the following rules are implemented:
Resource | Verbs | Path(s) | Description |
---|---|---|---|
BackupBucket | get , list , watch , create , update , patch , delete | BackupBucket -> Seed | Allow get , list , watch requests for all BackupBucket s. Allow only create , update , patch , delete requests for BackupBucket s assigned to the gardenlet ’s Seed . |
BackupEntry | get , list , watch , create , update , patch | BackupEntry -> Seed | Allow get , list , watch requests for all BackupEntry s. Allow only create , update , patch requests for BackupEntry s assigned to the gardenlet ’s Seed and referencing BackupBucket s assigned to the gardenlet ’s Seed . |
Bastion | get , list , watch , create , update , patch | Bastion -> Seed | Allow get , list , watch requests for all Bastion s. Allow only create , update , patch requests for Bastion s assigned to the gardenlet ’s Seed . |
CertificateSigningRequest | get , create | CertificateSigningRequest -> Seed | Allow only get , create requests for CertificateSigningRequest s related to the gardenlet ’s Seed . |
CloudProfile | get | CloudProfile -> Shoot -> Seed | Allow only get requests for CloudProfile s referenced by Shoot s that are assigned to the gardenlet ’s Seed . |
ClusterRoleBinding | create , get , update , patch , delete | ClusterRoleBinding -> ManagedSeed -> Shoot -> Seed | Allow create , get , update , patch requests for ManagedSeed s in the bootstrapping phase assigned to the gardenlet’s Seed s. Allow delete requests from gardenlets bootstrapped via ManagedSeed s. |
ConfigMap | get | ConfigMap -> Shoot -> Seed | Allow only get requests for ConfigMap s referenced by Shoot s that are assigned to the gardenlet ’s Seed . Allows reading the kube-system/cluster-identity ConfigMap . |
ControllerRegistration | get , list , watch | ControllerRegistration -> ControllerInstallation -> Seed | Allow get , list , watch requests for all ControllerRegistration s. |
ControllerDeployment | get | ControllerDeployment -> ControllerInstallation -> Seed | Allow get requests for ControllerDeployments s referenced by ControllerInstallation s assigned to the gardenlet ’s Seed . |
ControllerInstallation | get , list , watch , update , patch | ControllerInstallation -> Seed | Allow get , list , watch requests for all ControllerInstallation s. Allow only update , patch requests for ControllerInstallation s assigned to the gardenlet ’s Seed . |
Event | create , patch | none | Allow to create or patch all kinds of Event s. |
ExposureClass | get | ExposureClass -> Shoot -> Seed | Allow get requests for ExposureClass es referenced by Shoot s that are assigned to the gardenlet ’s Seed . Deny get requests to other ExposureClass es. |
Lease | create , get , watch , update | Lease -> Seed | Allow create , get , update , and delete requests for Lease s of the gardenlet ’s Seed . |
ManagedSeed | get , list , watch , update , patch | ManagedSeed -> Shoot -> Seed | Allow get , list , watch requests for all ManagedSeed s. Allow only update , patch requests for ManagedSeed s referencing a Shoot assigned to the gardenlet ’s Seed . |
Namespace | get | Namespace -> Shoot -> Seed | Allow get requests for Namespace s of Shoot s that are assigned to the gardenlet ’s Seed . Always allow get requests for the garden Namespace . |
Project | get | Project -> Namespace -> Shoot -> Seed | Allow get requests for Project s referenced by the Namespace of Shoot s that are assigned to the gardenlet ’s Seed . |
SecretBinding | get | SecretBinding -> Shoot -> Seed | Allow only get requests for SecretBinding s referenced by Shoot s that are assigned to the gardenlet ’s Seed . |
Secret | create , get , update , patch , delete (, list , watch ) | Secret -> Seed , Secret -> Shoot -> Seed , Secret -> SecretBinding -> Shoot -> Seed , BackupBucket -> Seed | Allow get , list , watch requests for all Secret s in the seed-<name> namespace. Allow only create , get , update , patch , delete requests for the Secret s related to resources assigned to the gardenlet ’s Seed s. |
Seed | get , list , watch , create , update , patch , delete | Seed | Allow get , list , watch requests for all Seed s. Allow only create , update , patch , delete requests for the gardenlet ’s Seed s. [1] |
ServiceAccount | create , get , update , patch , delete | ServiceAccount -> ManagedSeed -> Shoot -> Seed | Allow create , get , update , patch requests for ManagedSeed s in the bootstrapping phase assigned to the gardenlet ’s Seed s. Allow delete requests from gardenlets bootstrapped via ManagedSeed s. |
Shoot | get , list , watch , update , patch | Shoot -> Seed | Allow get , list , watch requests for all Shoot s. Allow only update , patch requests for Shoot s assigned to the gardenlet ’s Seed . |
ShootState | get , create , update , patch | ShootState -> Shoot -> Seed | Allow only get , create , update , patch requests for ShootState s belonging by Shoot s that are assigned to the gardenlet ’s Seed . |
[1] If you use
ManagedSeed
resources then thegardenlet
reconciling them (“parentgardenlet
”) may be allowed to submit certain requests for theSeed
resources resulting out of suchManagedSeed
reconciliations (even if the “parentgardenlet
” is not responsible for them):
ℹ️ It is allowed to delete the Seed
resources if the corresponding ManagedSeed
objects already have a deletionTimestamp
(this is secure as gardenlet
s themselves don’t have permissions for deleting ManagedSeed
s).
SeedAuthorizer
Authorization Webhook Enablement
The SeedAuthorizer
is implemented as a Kubernetes authorization webhook and part of the gardener-admission-controller
component running in the garden cluster.
🎛 In order to activate it, you have to follow these steps:
Set the following flags for the
kube-apiserver
of the garden cluster (i.e., thekube-apiserver
whose API is extended by Gardener):--authorization-mode=RBAC,Node,Webhook
(please note thatWebhook
should appear afterRBAC
in the list [1];Node
might not be needed if you use a virtual garden cluster)--authorization-webhook-config-file=<path-to-the-webhook-config-file>
--authorization-webhook-cache-authorized-ttl=0
--authorization-webhook-cache-unauthorized-ttl=0
The webhook config file (stored at
<path-to-the-webhook-config-file>
) should look as follows:apiVersion: v1 kind: Config clusters: - name: garden cluster: certificate-authority-data: base64(CA-CERT-OF-GARDENER-ADMISSION-CONTROLLER) server: https://gardener-admission-controller.garden/webhooks/auth/seed users: - name: kube-apiserver user: {} contexts: - name: auth-webhook context: cluster: garden user: kube-apiserver current-context: auth-webhook
When deploying the Gardener
controlplane
Helm chart, set.global.rbac.seedAuthorizer.enabled=true
. This will ensure that the RBAC resources granting global access for allgardenlet
s will be deployed.Delete the existing RBAC resources granting global access for all
gardenlet
s by running:kubectl delete \ clusterrole.rbac.authorization.k8s.io/gardener.cloud:system:seeds \ clusterrolebinding.rbac.authorization.k8s.io/gardener.cloud:system:seeds \ --ignore-not-found
Please note that you should activate the SeedRestriction
admission handler as well.
[1] The reason for the fact that
Webhook
authorization plugin should appear afterRBAC
is that thekube-apiserver
will be depending on thegardener-admission-controller
(serving the webhook). However, thegardener-admission-controller
can only start whengardener-apiserver
runs, butgardener-apiserver
itself can only start whenkube-apiserver
runs. IfWebhook
is beforeRBAC
, thengardener-apiserver
might not be able to start, leading to a deadlock.
Authorizer Decisions
As mentioned earlier, it’s the authorizer’s job to evaluate API requests and return one of the following decisions:
DecisionAllow
: The request is allowed, further configured authorizers won’t be consulted.DecisionDeny
: The request is denied, further configured authorizers won’t be consulted.DecisionNoOpinion
: A decision cannot be made, further configured authorizers will be consulted.
For backwards compatibility, no requests are denied at the moment, so that they are still deferred to a subsequent authorizer like RBAC. Though, this might change in the future.
First, the SeedAuthorizer
extracts the Seed
name from the API request. This requires a proper TLS certificate that the gardenlet
uses to contact the API server and is automatically given if TLS bootstrapping is used.
Concretely, the authorizer checks the certificate for name gardener.cloud:system:seed:<seed-name>
and group gardener.cloud:system:seeds
.
In cases where this information is missing e.g., when a custom Kubeconfig is used, the authorizer cannot make any decision. Thus, RBAC is still a considerable option to restrict the gardenlet
’s access permission if the above explained preconditions are not given.
With the Seed
name at hand, the authorizer checks for an existing path from the resource that a request is being made for to the Seed
belonging to the gardenlet
. Take a look at the Implementation Details section for more information.
Implementation Details
Internally, the SeedAuthorizer
uses a directed, acyclic graph data structure in order to efficiently respond to authorization requests for gardenlet
s:
- A vertex in this graph represents a Kubernetes resource with its kind, namespace, and name (e.g.,
Shoot:garden-my-project/my-shoot
). - An edge from vertex
u
to vertexv
in this graph exists when- (1)
v
is referred byu
andv
is aSeed
, or when - (2)
u
is referred byv
, or when - (3)
u
is strictly associated withv
.
- (1)
For example, a Shoot
refers to a Seed
, a CloudProfile
, a SecretBinding
, etc., so it has an outgoing edge to the Seed
(1) and incoming edges from the CloudProfile
and SecretBinding
vertices (2).
However, there might also be a ShootState
or a BackupEntry
resource strictly associated with this Shoot
, hence, it has incoming edges from these vertices (3).
In the above picture, the resources that are actively watched are shaded. Gardener resources are green, while Kubernetes resources are blue. It shows the dependencies between the resources and how the graph is built based on the above rules.
ℹ️ The above picture shows all resources that may be accessed by gardenlet
s, except for the Quota
resource which is only included for completeness.
Now, when a gardenlet
wants to access certain resources, then the SeedAuthorizer
uses a Depth-First traversal starting from the vertex representing the resource in question, e.g., from a Project
vertex.
If there is a path from the Project
vertex to the vertex representing the Seed
the gardenlet
is responsible for. then it allows the request.
Metrics
The SeedAuthorizer
registers the following metrics related to the mentioned graph implementation:
Metric | Description |
---|---|
gardener_admission_controller_seed_authorizer_graph_update_duration_seconds | Histogram of duration of resource dependency graph updates in seed authorizer, i.e., how long does it take to update the graph’s vertices/edges when a resource is created, changed, or deleted. |
gardener_admission_controller_seed_authorizer_graph_path_check_duration_seconds | Histogram of duration of checks whether a path exists in the resource dependency graph in seed authorizer. |
Debug Handler
When the .server.enableDebugHandlers
field in the gardener-admission-controller
’s component configuration is set to true
, then it serves a handler that can be used for debugging the resource dependency graph under /debug/resource-dependency-graph
.
🚨 Only use this setting for development purposes, as it enables unauthenticated users to view all data if they have access to the gardener-admission-controller
component.
The handler renders an HTML page displaying the current graph with a list of vertices and its associated incoming and outgoing edges to other vertices.
Depending on the size of the Gardener landscape (and consequently, the size of the graph), it might not be possible to render it in its entirety.
If there are more than 2000 vertices, then the default filtering will selected for kind=Seed
to prevent overloading the output.
Example output:
-------------------------------------------------------------------------------
|
| # Seed:my-seed
| <- (11)
| BackupBucket:73972fe2-3d7e-4f61-a406-b8f9e670e6b7
| BackupEntry:garden-my-project/shoot--dev--my-shoot--4656a460-1a69-4f00-9372-7452cbd38ee3
| ControllerInstallation:dns-external-mxt8m
| ControllerInstallation:extension-shoot-cert-service-4qw5j
| ControllerInstallation:networking-calico-bgrb2
| ControllerInstallation:os-gardenlinux-qvb5z
| ControllerInstallation:provider-gcp-w4mvf
| Secret:garden/backup
| Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------
|
| # Shoot:garden-my-project/my-shoot
| <- (5)
| CloudProfile:gcp
| Namespace:garden-my-project
| Secret:garden-my-project/my-dns-secret
| SecretBinding:garden-my-project/my-credentials
| ShootState:garden-my-project/my-shoot
| -> (1)
| Seed:my-seed
|
-------------------------------------------------------------------------------
|
| # ShootState:garden-my-project/my-shoot
| -> (1)
| Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------
... (etc., similarly for the other resources)
There are anchor links to easily jump from one resource to another, and the page provides means for filtering the results based on the kind
, namespace
, and/or name
.
Pitfalls
When there is a relevant update to an existing resource, i.e., when a reference to another resource is changed, then the corresponding vertex (along with all associated edges) is first deleted from the graph before it gets added again with the up-to-date edges.
However, this does only work for vertices belonging to resources that are only created in exactly one “watch handler”.
For example, the vertex for a SecretBinding
can either be created in the SecretBinding
handler itself or in the Shoot
handler.
In such cases, deleting the vertex before (re-)computing the edges might lead to race conditions and potentially renders the graph invalid.
Consequently, instead of deleting the vertex, only the edges the respective handler is responsible for are deleted.
If the vertex ends up with no remaining edges, then it also gets deleted automatically.
Afterwards, the vertex can either be added again or the updated edges can be created.
SeedRestriction
Admission Webhook Enablement
The SeedRestriction
is implemented as Kubernetes admission webhook and part of the gardener-admission-controller
component running in the garden cluster.
🎛 In order to activate it, you have to set .global.admission.seedRestriction.enabled=true
when using the Gardener controlplane
Helm chart.
This will add an additional webhook in the existing ValidatingWebhookConfiguration
of the gardener-admission-controller
which contains the configuration for the SeedRestriction
handler.
Please note that it should only be activated when the SeedAuthorizer
is active as well.
Admission Decisions
The admission’s purpose is to perform extended validation on requests which require the body of the object in question.
Additionally, it handles CREATE
requests of gardenlet
s (the above discussed resource dependency graph cannot be used in such cases because there won’t be any vertex/edge for non-existing resources).
Gardenlets are restricted to only create new resources which are somehow related to the seed clusters they are responsible for.
1.3.8 - Getting Started Locally
Deploying Gardener Locally
This document will walk you through deploying Gardener on your local machine. If you encounter difficulties, please open an issue so that we can make this process easier.
Overview
Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as both garden and seed cluster (please refer to the architecture overview) for simplicity.
Based on Skaffold, the container images for all required components will be built and deployed into the cluster (via their Helm charts).
Alternatives
When deploying Gardener on your local machine you might face several limitations:
- Your machine doesn’t have enough compute resources (see prerequisites) for hosting a second seed cluster or multiple shoot clusters.
- Testing Gardener’s IPv6 features requires a Linux machine and native IPv6 connectivity to the internet, but you’re on macOS or don’t have IPv6 connectivity in your office environment or via your home ISP.
In these cases, you might want to check out one of the following options that run the setup described in this guide elsewhere for circumventing these limitations:
- remote local setup: deploy on a remote pod for more compute resources
- dev box on Google Cloud: deploy on a Google Cloud machine for more compute resource and/or simple IPv4/IPv6 dual-stack networking
Prerequisites
- Make sure that you have followed the Local Setup guide up until the Get the sources step.
- Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least
8
CPUs and8Gi
memory; see here how to configure the resources for Docker for Mac).Please note that 8 CPU / 8Gi memory might not be enough for more than two
Shoot
clusters, i.e., you might need to increase these values if you want to run additionalShoot
s. If you plan on following the optional steps to create a second seed cluster, the required resources will be more - at least10
CPUs and18Gi
memory. Additionally, please configure at least120Gi
of disk size for the Docker daemon. Tip: You can clean up unused data withdocker system df
anddocker system prune -a
. - Make sure the
kind
docker network is using the CIDR172.18.0.0/16
.- If the network does not exist, it can be created with
docker network create kind --subnet 172.18.0.0/16
- If the network already exists, the CIDR can be checked with
docker network inspect kind | jq '.[].IPAM.Config[].Subnet'
. If it is not172.18.0.0/16
, delete the network withdocker network rm kind
and create it with the command above.
- If the network does not exist, it can be created with
Setting Up the KinD Cluster (Garden and Seed)
make kind-up
If you want to setup an IPv6 KinD cluster, use
make kind-up IPFAMILY=ipv6
instead.
This command sets up a new KinD cluster named gardener-local
and stores the kubeconfig in the ./example/gardener-local/kind/local/kubeconfig
file.
It might be helpful to copy this file to
$HOME/.kube/config
, since you will need to target this KinD cluster multiple times. Alternatively, make sure to set yourKUBECONFIG
environment variable to./example/gardener-local/kind/local/kubeconfig
for all future steps viaexport KUBECONFIG=example/gardener-local/kind/local/kubeconfig
.
All of the following steps assume that you are using this kubeconfig.
Additionally, this command also deploys a local container registry to the cluster, as well as a few registry mirrors, that are set up as a pull-through cache for all upstream registries Gardener uses by default.
This is done to speed up image pulls across local clusters.
The local registry can be accessed as localhost:5001
for pushing and pulling.
The storage directories of the registries are mounted to the host machine under dev/local-registry
.
With this, mirrored images don’t have to be pulled again after recreating the cluster.
The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy
support (the default kindnet
CNI doesn’t provide NetworkPolicy
support).
Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.
Outgoing IPv6 Single-Stack Networking (optional)
If you want to test IPv6-related features, we need to configure NAT for outgoing traffic from the kind network to the internet.
After make kind-up IPFAMILY=ipv6
, check the network created by kind:
$ docker network inspect kind | jq '.[].IPAM.Config[].Subnet'
"172.18.0.0/16"
"fc00:f853:ccd:e793::/64"
Determine which device is used for outgoing internet traffic by looking at the default route:
$ ip route show default
default via 192.168.195.1 dev enp3s0 proto dhcp src 192.168.195.34 metric 100
Configure NAT for traffic from the kind cluster to the internet using the IPv6 range and the network device from the previous two steps:
ip6tables -t nat -A POSTROUTING -o enp3s0 -s fc00:f853:ccd:e793::/64 -j MASQUERADE
Setting Up Gardener
make gardener-up
This will first build the base image (which might take a bit if you do it for the first time). Afterwards, the Gardener resources will be deployed into the cluster.
Creating a Shoot
Cluster
You can wait for the Seed
to be ready by running:
kubectl wait --for=condition=gardenletready --for=condition=extensionsready --for=condition=bootstrapped seed local --timeout=5m
Alternatively, you can run kubectl get seed local
and wait for the STATUS
to indicate readiness:
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
local Ready local local 4m42s vX.Y.Z-dev v1.21.1
In order to create a first shoot cluster, just run:
kubectl apply -f example/provider-local/shoot.yaml
You can wait for the Shoot
to be ready by running:
kubectl wait --for=condition=apiserveravailable --for=condition=controlplanehealthy --for=condition=observabilitycomponentshealthy --for=condition=everynodeready --for=condition=systemcomponentshealthy shoot local -n garden-local --timeout=10m
Alternatively, you can run kubectl -n garden-local get shoot local
and wait for the LAST OPERATION
to reach 100%
:
NAME CLOUDPROFILE PROVIDER REGION K8S VERSION HIBERNATION LAST OPERATION STATUS AGE
local local local local 1.21.0 Awake Create Processing (43%) healthy 94s
(Optional): You could also execute a simple e2e test (creating and deleting a shoot) by running:
make test-e2e-local-simple KUBECONFIG="$PWD/example/gardener-local/kind/local/kubeconfig"
Accessing the Shoot
Cluster
⚠️ Please note that in this setup, shoot clusters are not accessible by default when you download the kubeconfig and try to communicate with them.
The reason is that your host most probably cannot resolve the DNS names of the clusters since provider-local
extension runs inside the KinD cluster (for more details, see DNSRecord).
Hence, if you want to access the shoot cluster, you have to run the following command which will extend your /etc/hosts
file with the required information to make the DNS names resolvable:
cat <<EOF | sudo tee -a /etc/hosts
# Manually created to access local Gardener shoot clusters.
# TODO: Remove this again when the shoot cluster access is no longer required.
127.0.0.1 api.local.local.external.local.gardener.cloud
127.0.0.1 api.local.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-managedseed.garden.external.local.gardener.cloud
127.0.0.1 api.e2e-managedseed.garden.internal.local.gardener.cloud
127.0.0.1 api.e2e-hibernated.local.external.local.gardener.cloud
127.0.0.1 api.e2e-hibernated.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-unpriv.local.external.local.gardener.cloud
127.0.0.1 api.e2e-unpriv.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-wake-up.local.external.local.gardener.cloud
127.0.0.1 api.e2e-wake-up.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-migrate.local.external.local.gardener.cloud
127.0.0.1 api.e2e-migrate.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-rotate.local.external.local.gardener.cloud
127.0.0.1 api.e2e-rotate.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-default.local.external.local.gardener.cloud
127.0.0.1 api.e2e-default.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-update-node.local.external.local.gardener.cloud
127.0.0.1 api.e2e-update-node.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-update-zone.local.external.local.gardener.cloud
127.0.0.1 api.e2e-update-zone.local.internal.local.gardener.cloud
127.0.0.1 api.e2e-upgrade.local.external.local.gardener.cloud
127.0.0.1 api.e2e-upgrade.local.internal.local.gardener.cloud
EOF
Now you can access it by running:
kubectl -n garden-local get secret local.kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > /tmp/kubeconfig-shoot-local.yaml
kubectl --kubeconfig=/tmp/kubeconfig-shoot-local.yaml get nodes
(Optional): Setting Up a Second Seed Cluster
There are cases where you would want to create a second cluster seed in your local setup. For example, if you want to test the control plane migration feature. The following steps describe how to do that.
make kind2-up
This command sets up a new KinD cluster named gardener-local2
and stores its kubeconfig in the ./example/gardener-local/kind/local2/kubeconfig
file.
In order to deploy required resources in the KinD cluster that you just created, run:
make gardenlet-kind2-up
The following steps assume that you are using the kubeconfig that points to the gardener-local
cluster (first KinD cluster): export KUBECONFIG=example/gardener-local/kind/local/kubeconfig
.
You can wait for the local2
Seed
to be ready by running:
kubectl wait --for=condition=gardenletready --for=condition=extensionsready --for=condition=bootstrapped seed local2 --timeout=5m
Alternatively, you can run kubectl get seed local2
and wait for the STATUS
to indicate readiness:
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
local2 Ready local local 4m42s vX.Y.Z-dev v1.21.1
If you want to perform control plane migration, you can follow the steps outlined in Control Plane Migration to migrate the shoot cluster to the second seed you just created.
Deleting the Shoot
Cluster
./hack/usage/delete shoot local garden-local
(Optional): Tear Down the Second Seed Cluster
make kind2-down
Tear Down the Gardener Environment
make kind-down
Remote Local Setup
Just like Prow is executing the KinD based integration tests in a K8s pod, it is possible to interactively run this KinD based Gardener development environment, aka “local setup”, in a “remote” K8s pod.
k apply -f docs/deployment/content/remote-local-setup.yaml
k exec -it deployment/remote-local-setup -- sh
tmux -u a
Caveats
Please refer to the TMUX documentation for working effectively inside the remote-local-setup pod.
To access Grafana, Prometheus or other components in a browser, two port forwards are needed:
The port forward from the laptop to the pod:
k port-forward deployment/remote-local-setup 3000
The port forward in the remote-local-setup pod to the respective component:
k port-forward -n shoot--local--local deployment/grafana-operators 3000
Related Links
1.3.9 - Getting Started Locally With Extensions
Deploying Gardener Locally and Enabling Provider-Extensions
This document will walk you through deploying Gardener on your local machine and bootstrapping your own seed clusters on an existing Kubernetes cluster. It is supposed to run your local Gardener developments on a real infrastructure. For running Gardener only entirely local, please check the getting started locally documentation. If you encounter difficulties, please open an issue so that we can make this process easier.
Overview
Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as garden cluster. Any Kubernetes cluster could be used as seed clusters in order to support provider extensions (please refer to the architecture overview). This guide is tested for using Kubernetes clusters provided by Gardener, AWS, Azure, and GCP as seed so far.
Based on Skaffold, the container images for all required components will be built and deployed into the clusters (via their Helm charts).
Prerequisites
- Make sure that you have prepared your setup and checked out Gardener sources as described in the Local Setup guide.
- Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least
8
CPUs and8Gi
memory; see the Docker documentation for how to configure the resources for Docker for Mac).Additionally, please configure at least
120Gi
of disk size for the Docker daemon. Tip: You can clean up unused data withdocker system df
anddocker system prune -a
. - Make sure that you have access to a Kubernetes cluster you can use as a seed cluster in this setup.
- The seed cluster requires at least 16 CPUs in total to run one shoot cluster
- You could use any Kubernetes cluster for your seed cluster. However, using a Gardener shoot cluster for your seed simplifies some configuration steps.
- When bootstrapping
gardenlet
to the cluster, your new seed will have the same provider type as the shoot cluster you use - an AWS shoot will become an AWS seed, a GCP shoot will become a GCP seed, etc. (only relevant when using a Gardener shoot as seed).
Provide Infrastructure Credentials and Configuration
As this setup is running on a real infrastructure, you have to provide credentials for DNS, the infrastructure, and the kubeconfig for the Gardener cluster you want to use as seed.
There are
.gitignore
entries for all files and directories which include credentials. Nevertheless, please double check and make sure that credentials are not commited.
DNS
Gardener control plane requires DNS for default and internal domains. Thus, you have to configure a valid DNS provider for your setup.
Please maintain your DNS provider configuration and credentials at ./example/provider-extensions/garden/controlplane/values.yaml
.
You can find a template for the file at ./example/provider-extensions/garden/controlplane/values.yaml.tmpl
.
Infrastructure
Infrastructure secrets and the corresponding secret bindings should be maintained at:
./example/provider-extensions/garden/project/credentials/infrastructure-secrets.yaml
./example/provider-extensions/garden/project/credentials/secretbindings.yaml
There are templates with .tmpl
suffixes for the files in the same folder.
Seed Cluster Preparation
The kubeconfig
of your Kubernetes cluster you would like to use as seed should be placed at ./example/provider-extensions/seed/kubeconfig
.
Additionally, please maintain the configuration of your seed in ./example/provider-extensions/gardenlet/values.yaml
. It is automatically copied from values.yaml.tmpl
in the same directory when you run make gardener-extensions-up
for the first time. It also includes explanations of the properties you should set.
Using a Gardener cluster as seed simplifies the process, because some configuration options can be taken from shoot-info
and creating DNS entries and TLS certificates is automated.
However, you can use different Kubernetes clusters for your seed too and configure these things manually. Please configure the options of ./example/provider-extensions/gardenlet/values.yaml
upfront. For configuring DNS and TLS certificates, make gardener-extensions-up
, which is explained later, will pause and tell you what to do.
External Controllers
You might plan to deploy and register external controllers for networking, operating system, providers, etc. Please put ControllerDeployment
s and ControllerRegistration
s into the S./example/provider-extensions/garden/controllerregistrations
directory. The whole content of this folder will be applied to your KinD cluster.
CloudProfile
s
There are no demo CloudProfiles
yet. Thus, please copy CloudProfiles
from another landscape to the ./example/provider-extensions/garden/cloudprofiles
directory or create your own CloudProfiles
based on the gardener examples. Please check the GitHub repository of your desired provider-extension. Most of them include example CloudProfile
s. All files you place in this folder will be applied to your KinD cluster.
Setting Up the KinD Cluster
make kind-extensions-up
This command sets up a new KinD cluster named gardener-local
and stores the kubeconfig in the ./example/gardener-local/kind/extensions/kubeconfig
file.
It might be helpful to copy this file to
$HOME/.kube/config
, since you will need to target this KinD cluster multiple times. Alternatively, make sure to set yourKUBECONFIG
environment variable to./example/gardener-local/kind/extensions/kubeconfig
for all future steps viaexport KUBECONFIG=$PWD/example/gardener-local/kind/extensions/kubeconfig
.
All of the following steps assume that you are using this kubeconfig.
Additionally, this command deploys a local container registry to the cluster as well as a few registry mirrors that are set up as a pull-through cache for all upstream registries Gardener uses by default.
This is done to speed up image pulls across local clusters.
The local registry can be accessed as localhost:5001
for pushing and pulling.
The storage directories of the registries are mounted to your machine under dev/local-registry
.
With this, mirrored images don’t have to be pulled again after recreating the cluster.
The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy
support (the default kindnet
CNI doesn’t provide NetworkPolicy
support).
Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.
Setting Up Gardener (Garden on KinD, Seed on Gardener Cluster)
make gardener-extensions-up
This will first prepare the basic configuration of your KinD and Gardener clusters.
Afterwards, the images for the Garden cluster are built and deployed into the KinD cluster.
Finally, the images for the Seed cluster are built, pushed to a container registry on the Seed, and the gardenlet
is started.
Pause and Unpause the KinD Cluster
The KinD cluster can be paused by stopping and keeping its docker container. This can be done by running:
make kind-extensions-down
When you run make kind-extensions-up
again, you will start the docker container with your previous Gardener configuration again.
This provides the option to switch off your local KinD cluster fast without leaving orphaned infrastructure elements behind.
Creating a Shoot
Cluster
You can wait for the Seed
to be ready by running:
kubectl wait --for=condition=gardenletready seed provider-extensions --timeout=5m
make kind-extensions-up
already includes such a check. However, it might be useful when you wake up your Seed
from hibernation or unpause you KinD cluster.
Alternatively, you can run kubectl get seed provider-extensions
and wait for the STATUS
to indicate readiness:
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
provider-extensions Ready gcp europe-west1 111m v1.61.0-dev v1.24.7
In order to create a first shoot cluster, please create your own Shoot
definition and apply it to your KinD cluster. gardener-scheduler
includes candidateDeterminationStrategy: MinimalDistance
configuration so you are able to run schedule Shoot
s of different providers on your Seed
.
You can wait for your Shoot
s to be ready by running kubectl -n garden-local get shoots
and wait for the LAST OPERATION
to reach 100%
. The output depends on your Shoot
definition. This is an example output:
NAME CLOUDPROFILE PROVIDER REGION K8S VERSION HIBERNATION LAST OPERATION STATUS AGE
aws aws aws eu-west-1 1.24.3 Awake Create Processing (43%) healthy 84s
aws-arm64 aws aws eu-west-1 1.24.3 Awake Create Processing (43%) healthy 65s
azure az azure westeurope 1.24.2 Awake Create Processing (43%) healthy 57s
gcp gcp gcp europe-west1 1.24.3 Awake Create Processing (43%) healthy 94s
Accessing the Shoot
Cluster
Your shoot clusters will have a public DNS entries for their API servers, so that they could be reached via the Internet via kubectl
after you have created their kubeconfig
.
We encourage you to use the adminkubeconfig subresource for accessing your shoot cluster. You can find an example how to use it in Accessing Shoot Clusters.
Deleting the Shoot
Clusters
Before tearing down your environment, you have to delete your shoot clusters. This is highly recommended because otherwise you would leave orphaned items on your infrastructure accounts.
./hack/usage/delete shoot <your-shoot> garden-local
Tear Down the Gardener Environment
Before you delete your local KinD cluster, you should shut down your Shoots
and Seed
in a clean way to avoid orphaned infrastructure elements in your projects.
Please ensure that your KinD and Seed clusters are online (not paused or hibernated) and run:
make gardener-extensions-down
This will delete all Shoots
first (this could take a couple of minutes), then uninstall gardenlet
from the Seed and the gardener components from the KinD. Finally, the additional components like container registry, etc., are deleted from both clusters.
When this is done, you can securely delete your local KinD cluster by running:
make kind-extensions-clean
1.3.10 - Image Vector
Image Vector
The Gardenlet is deploying several different container images into the seed and the shoot clusters. The image repositories and tags are defined in a central image vector file. Obviously, the image versions defined there must fit together with the deployment manifests (e.g., some command-line flags do only exist in certain versions).
Example
images:
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.4"
version: "1.20.x"
architectures:
- amd64
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.5"
version: ">= 1.21"
architectures:
- amd64
...
That means that the Gardenlet will use the pause-container
with tag 3.4
for all seed/shoot clusters with Kubernetes version 1.20.x
, and tag 3.5
for all clusters with Kubernetes >= 1.21
.
Image Vector Architecture
images:
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.5"
architectures:
- amd64
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.5"
architectures:
- arm64
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: registry.k8s.io/pause
tag: "3.5"
architectures:
- amd64
- arm64
...
architectures
is an optional field of image. It is a list of strings specifying CPU architecture of machines on which this image can be used. The valid options for the architectures field are as follows:
amd64
: This specifies that the image can run only on machines having CPU architectureamd64
.arm64
: This specifies that the image can run only on machines having CPU architecturearm64
.
If an image doesn’t specify any architectures, then by default it is considered to support both amd64
and arm64
architectures.
Overwrite Image Vector
In some environments it is not possible to use these “pre-defined” images that come with a Gardener release.
A prominent example for that is Alicloud in China, which does not allow access to Google’s GCR.
In these cases, you might want to overwrite certain images, e.g., point the pause-container
to a different registry.
⚠️ If you specify an image that does not fit to the resource manifest, then the seed/shoot reconciliation might fail.
In order to overwrite the images, you must provide a similar file to gardenlet:
images:
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: my-custom-image-registry/pause
tag: "3.4"
version: "1.20.x"
- name: pause-container
sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
repository: my-custom-image-registry/pause
tag: "3.5"
version: ">= 1.21"
...
During deployment of the gardenlet, create a ConfigMap
containing the above content and mount it as a volume into the gardenlet pod.
Next, specify the environment variable IMAGEVECTOR_OVERWRITE
, whose value must be the path to the file you just mounted:
apiVersion: v1
kind: ConfigMap
metadata:
name: gardenlet-images-overwrite
namespace: garden
data:
images_overwrite.yaml: |
images:
- ...
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: gardenlet
namespace: garden
spec:
template:
...
spec:
containers:
- name: gardenlet
env:
- name: IMAGEVECTOR_OVERWRITE
value: /charts-overwrite/images_overwrite.yaml
volumeMounts:
- name: gardenlet-images-overwrite
mountPath: /charts-overwrite
...
volumes:
- name: gardenlet-images-overwrite
configMap:
name: gardenlet-images-overwrite
...
Image Vectors for Dependent Components
The gardenlet is deploying a lot of different components that might deploy other images themselves. These components might use an image vector as well. Operators might want to customize the image locations for these transitive images as well, hence, they might need to specify an image vector overwrite for the components directly deployed by Gardener.
It is possible to specify the IMAGEVECTOR_OVERWRITE_COMPONENTS
environment variable to the gardenlet that points to a file with the following content:
components:
- name: etcd-druid
imageVectorOverwrite: |
images:
- name: etcd
tag: v1.2.3
repository: etcd/etcd
...
The gardenlet will, if supported by the directly deployed component (etcd-druid
in this example), inject the given imageVectorOverwrite
into the Deployment
manifest.
The respective component is responsible for using the overwritten images instead of its defaults.
1.3.11 - Migration V0 To V1
Migration from Gardener v0
to v1
Please refer to the document for older Gardener versions.
1.3.12 - Secret Binding Provider Controller
SecretBinding Provider Controller
This page describes the process on how to enable the SecretBinding provider controller.
Overview
With Gardener v1.38.0, the SecretBinding resource now contains a new optional field .provider.type
(details about the motivation can be found in https://github.com/gardener/gardener/issues/4888). To make the process of setting the new field automated and afterwards to enforce validation on the new field in backwards compatible manner, Gardener features the SecretBinding provider controller and a feature gate - SecretBindingProviderValidation
.
Process
A Gardener landscape operator can follow the following steps:
Enable the SecretBinding provider controller of Gardener Controller Manager.
The SecretBinding provider controller is responsible for populating the
.provider.type
field of a SecretBinding based on its current usage by Shoot resources. For example, if a Shootcrazy-botany
with.provider.type=aws
is using a SecretBindingmy-secret-binding
, then the SecretBinding provider controller will take care to set the.provider.type
field of the SecretBinding to the same provider type (aws
). To enable the SecretBinding provider controller, set thecontroller.secretBindingProvider.concurentSyncs
field in the ControllerManagerConfiguration (e.g set it to5
). Although that it is not recommended, the API allows Shoots from different provider types to reference the same SecretBinding (assuming that the backing Secret contains data for both of the provider types). To preserve the backwards compatibility for such SecretBindings, the provider controller will maintain the multiple provider types in the field (it will join them with the separator,
- for exampleaws,gcp
).Disable the SecretBinding provider controller and enable the
SecretBindingProviderValidation
feature gate of Gardener API server.The
SecretBindingProviderValidation
feature gate of Gardener API server enables a set of validations for the SecretBinding provider field. It forbids creating a Shoot that has a different provider type from the referenced SecretBinding’s one. It also enforces immutability on the field. After making sure that SecretBinding provider controller is enabled and it populated the.provider.type
field of a majority of the SecretBindings on a Gardener landscape (the SecretBindings that are unused will have their provider type unset), a Gardener landscape operator has to disable the SecretBinding provider controller and to enable theSecretBindingProviderValidation
feature gate of Gardener API server. To disable the SecretBinding provider controller, set thecontroller.secretBindingProvider.concurentSyncs
field in the ControllerManagerConfiguration to0
.
Implementation History
- Gardener v1.38: The SecretBinding resource has a new optional field
.provider.type
. The SecretBinding provider controller is disabled by default. TheSecretBindingProviderValidation
feature gate of Gardener API server is disabled by default. - Gardener v1.42: The SecretBinding provider controller is enabled by default.
- Gardener v1.51: The
SecretBindingProviderValidation
feature gate of Gardener API server is enabled by default and the SecretBinding provider controller is disabled by default. - Gardener v1.53: The
SecretBindingProviderValidation
feature gate of Gardener API server is unconditionally enabled (can no longer be disabled). - Gardener v1.55: The
SecretBindingProviderValidation
feature gate of Gardener API server and the SecretBinding provider controller are removed.
1.3.13 - Setup Gardener
Deploying Gardener into a Kubernetes Cluster
Similarly to Kubernetes, Gardener consists out of control plane components (Gardener API server, Gardener controller manager, Gardener scheduler), and an agent component (gardenlet). The control plane is deployed in the so-called garden cluster, while the agent is installed into every seed cluster. Please note that it is possible to use the garden cluster as seed cluster by simply deploying the gardenlet into it.
We are providing Helm charts in order to manage the various resources of the components. Please always make sure that you use the Helm chart version that matches the Gardener version you want to deploy.
Deploying the Gardener Control Plane (API Server, Admission Controller, Controller Manager, Scheduler)
The configuration values depict the various options to configure the different components. Please consult Gardener Configuration and Usage for component specific configurations and Authentication of Gardener Control Plane Components Against the Garden Cluster for authentication related specifics.
Also, note that all resources and deployments need to be created in the garden
namespace (not overrideable).
If you enable the Gardener admission controller as part of you setup, please make sure the garden
namespace is labelled with app: gardener
.
Otherwise, the backing service account for the admission controller Pod might not be created successfully.
No action is necessary if you deploy the garden
namespace with the Gardener control plane Helm chart.
After preparing your values in a separate controlplane-values.yaml
file (values.yaml can be used as starting point), you can run the following command against your garden cluster:
helm install charts/gardener/controlplane \
--namespace garden \
--name gardener-controlplane \
-f controlplane-values.yaml \
--wait
Deploying Gardener Extensions
Gardener is an extensible system that does not contain the logic for provider-specific things like DNS management, cloud infrastructures, network plugins, operating system configs, and many more.
You have to install extension controllers for these parts. Please consult the documentation regarding extensions to get more information.
Deploying the Gardener Agent (gardenlet)
Please refer to Deploying Gardenlets on how to deploy a gardenlet.
1.3.14 - Version Skew Policy
Version Skew Policy
This document describes the maximum version skew supported between various Gardener components.
Supported Gardener Versions
Gardener versions are expressed as x.y.z
, where x
is the major version, y
is the minor version, and z
is the patch version, following Semantic Versioning terminology.
The Gardener project maintains release branches for the three most recent minor releases.
Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility. Patch releases are cut from those branches at a regular cadence, plus additional urgent releases when required.
For more information, see the Releases document.
Supported Version Skew
Technically, we follow the same policy as the Kubernetes project.
However, given that our release cadence is much more frequent compared to Kubernetes (every 14d
vs. every 120d
), in many cases it is possible to skip a version.
Still, to be on the safe side, it is highly recommended to follow the described policy.
gardener-apiserver
In multi-instance setups of Gardener, the newest and oldest gardener-apiserver
instances must be within one minor version.
Example:
- newest
gardener-apiserver
is at 1.37 - other
gardener-apiserver
instances are supported at 1.37 and v1.36
gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet
gardener-controller-manager
, gardener-scheduler
, gardener-admission-controller
, and gardenlet
must not be newer than the gardener-apiserver
instances they communicate with.
They are expected to match the gardener-apiserver
minor version, but may be up to one minor version older (to allow live upgrades).
Example:
gardener-apiserver
is at v1.37gardener-controller-manager
,gardener-scheduler
,gardener-admission-controller
, andgardenlet
are supported at 1.37 and v1.36
Supported Component Upgrade Order
The supported version skew between components has implications on the order in which components must be upgraded. This section describes the order in which components must be upgraded to transition an existing Gardener installation from version 1.37 to version 1.38.
gardener-apiserver
Prerequisites:
- In a single-instance setup, the existing
gardener-apiserver
instance is 1.37. - In a multi-instance setup, all
gardener-apiserver
instances are at 1.37 or 1.38 (this ensures maximum skew of 1 minor version between the oldest and newestgardener-apiserver
instance). - The
gardener-controller-manager
,gardener-scheduler
,gardener-admission-controller
, andgardenlet
instances that communicate with thisgardener-apiserver
are at version 1.37 (this ensures they are not newer than the existing API server version and are within 1 minor version of the new API server version).
Actions:
- Upgrade
gardener-apiserver
to 1.38.
gardener-controller-manager, gardener-scheduler, gardener-admission-controller, gardenlet
Prerequisites:
- The
gardener-apiserver
instances these components communicate with are at 1.38 (in multi-instance setups in which these components can communicate with anygardener-apiserver
instance in the cluster, allgardener-apiserver
instances must be upgraded before upgrading these components)
Actions:
- Upgrade
gardener-controller-manager
,gardener-scheduler
,gardener-admission-controller
, andgardenlet
to 1.38
Supported Kubernetes Versions
Please refer to Supported Kubernetes Versions.
1.4 - Development
1.4.1 - Changing the API
Changing the API
This document describes the steps that need to be performed when changing the API. It provides guidance for API changes to both (Gardener system in general or component configurations).
Generally, as Gardener is a Kubernetes-native extension, it follows the same API conventions and guidelines like Kubernetes itself. The Kubernetes API Conventions as well as Changing the API topics already provide a good overview and general explanation of the basic concepts behind it. We are following the same approaches.
Gardener API
The Gardener API is defined in the pkg/apis/{core,extensions,settings}
directories and is the main point of interaction with the system.
It must be ensured that the API is always backwards-compatible.
Changing the API
Checklist when changing the API:
- Modify the field(s) in the respective Golang files of all external versions and the internal version.
- Make sure new fields are being added as “optional” fields, i.e., they are of pointer types, they have the
// +optional
comment, and they have theomitempty
JSON tag. - Make sure that the existing field numbers in the protobuf tags are not changed.
- Make sure new fields are being added as “optional” fields, i.e., they are of pointer types, they have the
- If necessary, implement/adapt the conversion logic defined in the versioned APIs (e.g.,
pkg/apis/core/v1beta1/conversions*.go
). - If necessary, implement/adapt defaulting logic defined in the versioned APIs (e.g.,
pkg/apis/core/v1beta1/defaults*.go
). - Run the code generation:
make generate
- If necessary, implement/adapt validation logic defined in the internal API (e.g.,
pkg/apis/core/validation/validation*.go
). - If necessary, adapt the exemplary YAML manifests of the Gardener resources defined in
example/*.yaml
. - In most cases, it makes sense to add/adapt the documentation for administrators/operators and/or end-users in the
docs
folder to provide information on purpose and usage of the added/changed fields. - When opening the pull request, always add a release note so that end-users are becoming aware of the changes.
Removing a Field
If fields shall be removed permanently from the API, then a proper deprecation period must be adhered to so that end-users have enough time to adapt their clients.
Once the deprecation period is over, the field should be dropped from the API in a two-step process, i.e., in two release cycles. In the first step, all the usages in the code base should be dropped. In the second step, the field should be dropped from API. We need to follow this two-step process cause there can be the case where gardener-apiserver
is upgraded to a new version in which the field has been removed but other controllers are still on the old version of Gardener. This can lead to nil
pointer exceptions or other unexpected behaviour.
The steps for removing a field from the code base is:
The field in the external version(s) has to be commented out with appropriate doc string that the protobuf number of the corresponding field is reserved. Example:
- SeedTemplate *gardencorev1beta1.SeedTemplate `json:"seedTemplate,omitempty" protobuf:"bytes,2,opt,name=seedTemplate"` + // SeedTemplate is tombstoned to show why 2 is reserved protobuf tag. + // SeedTemplate *gardencorev1beta1.SeedTemplate `json:"seedTemplate,omitempty" protobuf:"bytes,2,opt,name=seedTemplate"`
The reasoning behind this is to prevent the same protobuf number being used by a new field. Introducing a new field with the same protobuf number would be a breaking change for clients still using the old protobuf definitions that have the old field for the given protobuf number. The field in the internal version can be removed.
A unit test has to be added to make sure that a new field does not reuse the already reserved protobuf tag.
Example of field removal can be found in the Remove seedTemplate
field from ManagedSeed API PR.
Component Configuration APIs
Most Gardener components have a component configuration that follows similar principles to the Gardener API.
Those component configurations are defined in pkg/{controllermanager,gardenlet,scheduler},pkg/apis/config
.
Hence, the above checklist also applies for changes to those APIs.
However, since these APIs are only used internally and only during the deployment of Gardener, the guidelines with respect to changes and backwards-compatibility are slightly relaxed.
If necessary, it is allowed to remove fields without a proper deprecation period if the release note uses the breaking operator
keywords.
In addition to the above checklist:
- If necessary, then adapt the Helm chart of Gardener defined in
charts/gardener
. Adapt thevalues.yaml
file as well as the manifest templates.
1.4.2 - Component Checklist
Checklist For Adding New Components
Adding new components that run in the garden, seed, or shoot cluster is theoretically quite simple - we just need a Deployment
(or other similar workload resource), the respective container image, and maybe a bit of configuration.
In practice, however, there are a couple of things to keep in mind in order to make the deployment production-ready.
This document provides a checklist for them that you can walk through.
General
Avoid usage of Helm charts (example)
Nowadays, we use Golang components instead of Helm charts for deploying components to a cluster. Please find a typical structure of such components in the provided metrics_server.go file (configuration values are typically managed in a
Values
structure). There are a few exceptions (e.g., Istio) still using charts, however the default should be using a Golang-based implementation. For the exceptional cases, use Golang’s embed package to embed the Helm chart directory (example 1, example 2).Choose the proper deployment way (example 1 (direct application w/ client), example 2 (using
ManagedResource
), example 3 (mixed scenario))For historic reasons, resources related to shoot control plane components are applied directly with the client. All other resources (seed or shoot system components) are deployed via
gardener-resource-manager
’s Resource controller (ManagedResource
s) since it performs health checks out-of-the-box and has a lot of other features (see its documentation for more information). Components that can run as both seed system component or shoot control plane component (e.g., VPA orkube-state-metrics
) can make use of these utility functions.Do not hard-code container image references (example 1, example 2, example 3)
We define all image references centrally in the
charts/images.yaml
file. Hence, the image references must not be hard-coded in the pod template spec but read from this so-called image vector instead.Use unique
ConfigMap
s/Secret
s (example 1, example 2)Unique
ConfigMap
s/Secret
s are immutable for modification and have a unique name. This has a couple of benefits, e.g. thekubelet
doesn’t watch these resources, and it is always clear which resource contains which data since it cannot be changed. As a consequence, unique/immutableConfigMap
s/Secret
are superior to checksum annotations on the pod templates. Stale/unusedConfigMap
s/Secret
s are garbage-collected bygardener-resource-manager
’s GarbageCollector. There are utility functions (see examples above) for using uniqueConfigMap
s/Secret
s in Golang components. It is essential to inject the annotations into the workload resource to make the garbage-collection work.
Note that someConfigMap
s/Secret
s should not be unique (e.g., those containing monitoring or logging configuration). The reason is that the old revision stays in the cluster even if unused until the garbage-collector acts. During this time, they would be wrongly aggregated to the full configuration.Manage certificates/secrets via secrets manager (example)
You should use the secrets manager for the management of any kind of credentials. This makes sure that credentials rotation works out-of-the-box without you requiring to think about it. Generally, do not use client certificates (see the Security section).
Consider hibernation when calculating replica count (example)
Shoot clusters can be hibernated meaning that all control plane components in the shoot namespace in the seed cluster are scaled down to zero and all worker nodes are terminated. If your component runs in the seed cluster then you have to consider this case and provide the proper replica count. There is a utility function available (see example).
Ensure task dependencies are as precise as possible in shoot flows (example 1, example 2)
Only define the minimum of needed dependency tasks in the shoot reconciliation/deletion flows.
Handle shoot system components
Shoot system components deployed by
gardener-resource-manager
are labelled withresource.gardener.cloud/managed-by: gardener
. This makes Gardener adding required label selectors and tolerations so that non-DaemonSet
managedPod
s will exclusively run on selected nodes (for more information, see System Components Webhook).DaemonSet
s on the other hand, should generally tolerate anyNoSchedule
orNoExecute
taints so that they can run on anyNode
, regardless of user added taints.
Security
Use a dedicated
ServiceAccount
and disable auto-mount (example)Components that need to talk to the API server of their runtime cluster must always use a dedicated
ServiceAccount
(do not usedefault
), withautomountServiceAccountToken
set tofalse
. This makesgardener-resource-manager
’s TokenInvalidator invalidate the static token secret and itsProjectedTokenMount
webhook inject a projected token automatically.Use shoot access tokens instead of a client certificates (example)
For components that need to talk to a target cluster different from their runtime cluster (e.g., running in seed cluster but talking to shoot) the
gardener-resource-manager
’s TokenRequestor should be used to manage a so-called “shoot access token”.Define RBAC roles with minimal privileges (example)
The component’s
ServiceAccount
(if it exists) should have as little privileges as possible. Consequently, please define proper RBAC roles for it. This might include a combination ofClusterRole
s andRole
s. Please do not provide elevated privileges due to laziness (e.g., because there is already aClusterRole
that can be extended vs. creating aRole
only when access to a single namespace is needed).Use
NetworkPolicy
s to restrict network traffic (example)You should restrict both ingress and egress traffic to/from your component as much as possible to ensure that it only gets access to/from other components if really needed. Gardener provides a few default policies for typical usage scenarios. For more information, see Seed Network Policies and Shoot Network Policies.
Do not run components in privileged mode (example 1, example 2)
Avoid running components with
privileged=true
. Instead, define the needed Linux capabilities.Choose the proper Seccomp profile (example 1, example 2)
The Seccomp profile will be defaulted by
gardener-resource-manager
’s SeccompProfile webhook which works well for the majority of components. However, in some special cases you might need to overwrite it.Define
PodSecurityPolicy
s (example)PodSecurityPolicy
s are deprecated, however Gardener still supports shoot clusters with older Kubernetes versions (ref). To make sure that such clusters can run with.spec.kubernetes.allowPrivilegedContainers=false
, you have to define properPodSecurityPolicy
s. For more information, see Pod Security.
High Availability / Stability
Specify the component type label for high availability (example)
To support high-availability deployments,
gardener-resource-manager
s HighAvailabilityConfig webhook injects the proper specification like replica or topology spread constraints. You only need to specify the type label. For more information, see High Availability Of Deployed Components.Define a
PodDisruptionBudget
(example)Closely related to high availability but also to stability in general: The definition of a
PodDisruptionBudget
withmaxUnavailable=1
should be provided by default.Choose the right
PriorityClass
(example)Each cluster runs many components with different priorities. Gardener provides a set of default
PriorityClass
es. For more information, see Priority Classes.Consider defining liveness and readiness probes (example)
To ensure smooth rolling update behaviour, consider the definition of liveness and/or readiness probes.
Scalability
Provide resource requirements (example)
All components should have resource requirements. Generally, they should always request CPU and memory, while only memory shall be limited (no CPU limits!).
Define a
VerticalPodAutoscaler
(example)We typically perform vertical auto-scaling via the VPA managed by the Kubernetes community. Each component should have a respective
VerticalPodAutoscaler
with “min allowed” resources, “auto update mode”, and “requests only”-mode. VPA is always enabled in garden or seed clusters, while it is optional for shoot clusters.Define a
HorizontalPodAutoscaler
if needed (example)If your component is capable of scaling horizontally, you should consider defining a
HorizontalPodAutoscaler
.
Observability / Operations Productivity
Provide monitoring scrape config and alerting rules (example 1, example 2)
Components should provide scrape configuration and alerting rules for Prometheus/Alertmanager if appropriate. This should be done inside a dedicated
monitoring.go
file. Extensions should follow the guidelines described in Extensions Monitoring Integration.Provide logging parsers and filters (example 1, example 2)
Components should provide parsers and filters for fluent-bit, if appropriate. This should be done inside a dedicated
logging.go
file. Extensions should follow the guidelines described in Fluent-bit log parsers and filters.Set the
revisionHistoryLimit
to2
forDeployment
s (example)In order to allow easy inspection of two
ReplicaSet
s to quickly find the changes that lead to a rolling update, the revision history limit should be set to2
.Define health checks (example 1, example 2)
gardenlet
’s care controllers regularly check the health status of system or control plane components. You need to enhance the lists of components to check if your component related to the seed system or shoot control plane (shoot system components are automatically checked via their respectiveManagedResource
conditions), see examples above.Configure automatic restarts in shoot maintenance time window (example 1, example 2)
Gardener offers to restart components during the maintenance time window. For more information, see Restart Control Plane Controllers and Restart Some Core Addons. You can consider adding the needed label to your control plane component to get this automatic restart (probably not needed for most components).
1.4.3 - Dependencies
Dependency Management
We are using go modules for depedency management.
In order to add a new package dependency to the project, you can perform go get <PACKAGE>@<VERSION>
or edit the go.mod
file and append the package along with the version you want to use.
Updating Dependencies
The Makefile
contains a rule called revendor
which performs go mod tidy
and go mod vendor
:
go mod tidy
makes surego.mod
matches the source code in the module. It adds any missing modules necessary to build the current module’s packages and dependencies, and it removes unused modules that don’t provide any relevant packages.go mod vendor
resets the main module’s vendor directory to include all packages needed to build and test all the main module’s packages. It does not include test code for vendored packages.
make revendor
The dependencies are installed into the vendor
folder, which should be added to the VCS.
⚠️ Make sure that you test the code after you have updated the dependencies!
Exported Packages
This repository contains several packages that could be considered “exported packages”, in a sense that they are supposed to be reused in other Go projects. For example:
- Gardener’s API packages:
pkg/apis
- Library for building Gardener extensions:
extensions
- Gardener’s Test Framework:
test/framework
There are a few more folders in this repository (non-Go sources) that are reused across projects in the Gardener organization:
- GitHub templates:
.github
- Concourse / cc-utils related helpers:
hack/.ci
- Development, build and testing helpers:
hack
These packages feature a dummy doc.go
file to allow other Go projects to pull them in as go mod dependencies.
These packages are explicitly not supposed to be used in other projects (consider them as “non-exported”):
- API validation packages:
pkg/apis/*/*/validation
- Operation package (main Gardener business logic regarding
Seed
andShoot
clusters):pkg/operation
- Third party code:
third_party
Currently, we don’t have a mechanism yet for selectively syncing out these exported packages into dedicated repositories like kube’s staging mechanism (publishing-bot).
Import Restrictions
We want to make sure that other projects can depend on this repository’s “exported” packages without pulling in the entire repository (including “non-exported” packages) or a high number of other unwanted dependencies. Hence, we have to be careful when adding new imports or references between our packages.
ℹ️ General rule of thumb: the mentioned “exported” packages should be as self-contained as possible and depend on as few other packages in the repository and other projects as possible.
In order to support that rule and automatically check compliance with that goal, we leverage import-boss.
The tool checks all imports of the given packages (including transitive imports) against rules defined in .import-restrictions
files in each directory.
An import is allowed if it matches at least one allowed prefix and does not match any forbidden prefixes.
Note:
''
(the empty string) is a prefix of everything. For more details, see the import-boss topic.
import-boss
is executed on every pull request and blocks the PR if it doesn’t comply with the defined import restrictions.
You can also run it locally using make check
.
Import restrictions should be changed in the following situations:
- We spot a new pattern of imports across our packages that was not restricted before but makes it more difficult for other projects to depend on our “exported” packages. In that case, the imports should be further restricted to disallow such problematic imports, and the code/package structure should be reworked to comply with the newly given restrictions.
- We want to share code between packages, but existing import restrictions prevent us from doing so. In that case, please consider what additional dependencies it will pull in, when loosening existing restrictions. Also consider possible alternatives, like code restructurings or extracting shared code into dedicated packages for minimal impact on dependent projects.
1.4.4 - Getting Started Locally
Developing Gardener Locally
This document will walk you through running Gardener on your local machine for development purposes. If you encounter difficulties, please open an issue so that we can make this process easier.
Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as both garden and seed cluster (please refer to the architecture overview) for simplicity.
The Gardener components, however, will be run as regular processes on your machine (hence, no container images are being built).
Alternatives
When developing Gardener on your local machine you might face several limitations:
- Your machine doesn’t have enough compute resources (see prerequisites) for hosting a second seed cluster or multiple shoot clusters.
- Developing Gardener’s IPv6 features requires a Linux machine and native IPv6 connectivity to the internet, but you’re on macOS or don’t have IPv6 connectivity in your office environment or via your home ISP.
In these cases, you might want to check out one of the following options that run the setup described in this guide elsewhere for circumventing these limitations:
- remote local setup: develop on a remote pod for more compute resources
- dev box on Google Cloud: develop on a Google Cloud machine for more compute resource and/or simple IPv4/IPv6 dual-stack networking
Prerequisites
Make sure that you have followed the Local Setup guide up until the Get the sources step.
Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least
4
CPUs and4Gi
memory; see here how to configure the resources for Docker for Mac).Please note that 4 CPU / 4Gi memory might not be enough for more than one
Shoot
cluster, i.e., you might need to increase these values if you want to run additionalShoot
s. If you plan on following the optional steps to create a second seed cluster, the required resources will be more - at least10
CPUs and16Gi
memory.Additionally, please configure at least
120Gi
of disk size for the Docker daemon.Tip: With
docker system df
anddocker system prune -a
you can cleanup unused data.Make sure the
kind
docker network is using the CIDR172.18.0.0/16
.- If the network does not exist, it can be created with
docker network create kind --subnet 172.18.0.0/16
- If the network already exists, the CIDR can be checked with
docker network inspect kind | jq '.[].IPAM.Config[].Subnet'
. If it is not172.18.0.0/16
, delete the network withdocker network rm kind
and create it with the command above.
- If the network does not exist, it can be created with
Make sure that you increase the maximum number of open files on your host:
On Mac, run
sudo launchctl limit maxfiles 65536 200000
On Linux, extend the
/etc/security/limits.conf
file with* hard nofile 97816 * soft nofile 97816
and reload the terminal.
Setting Up the KinD Cluster (Garden and Seed)
make kind-up KIND_ENV=local
If you want to setup an IPv6 KinD cluster, use
make kind-up IPFAMILY=ipv6
instead.
This command sets up a new KinD cluster named gardener-local
and stores the kubeconfig in the ./example/gardener-local/kind/local/kubeconfig
file.
It might be helpful to copy this file to
$HOME/.kube/config
since you will need to target this KinD cluster multiple times. Alternatively, make sure to set yourKUBECONFIG
environment variable to./example/gardener-local/kind/local/kubeconfig
for all future steps viaexport KUBECONFIG=example/gardener-local/kind/local/kubeconfig
.
All following steps assume that you are using this kubeconfig.
Additionally, this command also deploys a local container registry to the cluster as well as a few registry mirrors, that are set up as a pull-through cache for all upstream registries Gardener uses by default.
This is done to speed up image pulls across local clusters.
The local registry can be accessed as localhost:5001
for pushing and pulling.
The storage directories of the registries are mounted to the host machine under dev/local-registry
.
With this, mirrored images don’t have to be pulled again after recreating the cluster.
The command also deploys a default calico installation as the cluster’s CNI implementation with NetworkPolicy
support (the default kindnet
CNI doesn’t provide NetworkPolicy
support).
Furthermore, it deploys the metrics-server in order to support HPA and VPA on the seed cluster.
Outgoing IPv6 Single-Stack Networking (optional)
If you want to test IPv6-related features, we need to configure NAT for outgoing traffic from the kind network to the internet.
After make kind-up IPFAMILY=ipv6
, check the network created by kind:
$ docker network inspect kind | jq '.[].IPAM.Config[].Subnet'
"172.18.0.0/16"
"fc00:f853:ccd:e793::/64"
Determine which device is used for outgoing internet traffic by looking at the default route:
$ ip route show default
default via 192.168.195.1 dev enp3s0 proto dhcp src 192.168.195.34 metric 100
Configure NAT for traffic from the kind cluster to the internet using the IPv6 range and the network device from the previous two steps:
ip6tables -t nat -A POSTROUTING -o enp3s0 -s fc00:f853:ccd:e793::/64 -j MASQUERADE
Setting Up Gardener
In a terminal pane, run:
make dev-setup # preparing the environment (without webhooks for now)
kubectl wait --for=condition=ready pod -l run=etcd -n garden --timeout 2m # wait for etcd to be ready
make start-apiserver # starting gardener-apiserver
In a new terminal pane, run:
kubectl wait --for=condition=available apiservice v1beta1.core.gardener.cloud # wait for gardener-apiserver to be ready
make start-admission-controller # starting gardener-admission-controller
In a new terminal pane, run:
make dev-setup DEV_SETUP_WITH_WEBHOOKS=true # preparing the environment with webhooks
make start-controller-manager # starting gardener-controller-manager
(Optional): In a new terminal pane, run:
make start-scheduler # starting gardener-scheduler
In a new terminal pane, run:
make register-local-env # registering the local environment (CloudProfile, Seed, etc.)
make start-gardenlet SEED_NAME=local # starting gardenlet
In a new terminal pane, run:
make start-extension-provider-local # starting gardener-extension-provider-local
ℹ️ The provider-local
is started with elevated privileges since it needs to manipulate your /etc/hosts
file to enable you accessing the created shoot clusters from your local machine, see this for more details.
Creating a Shoot
Cluster
You can wait for the Seed
to become ready by running:
kubectl wait --for=condition=gardenletready --for=condition=extensionsready --for=condition=bootstrapped seed local --timeout=5m
Alternatively, you can run kubectl get seed local
and wait for the STATUS
to indicate readiness:
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
local Ready local local 4m42s vX.Y.Z-dev v1.21.1
In order to create a first shoot cluster, just run:
kubectl apply -f example/provider-local/shoot.yaml
You can wait for the Shoot
to be ready by running:
kubectl wait --for=condition=apiserveravailable --for=condition=controlplanehealthy --for=condition=observabilitycomponentshealthy --for=condition=everynodeready --for=condition=systemcomponentshealthy shoot local -n garden-local --timeout=10m
Alternatively, you can run kubectl -n garden-local get shoot local
and wait for the LAST OPERATION
to reach 100%
:
NAME CLOUDPROFILE PROVIDER REGION K8S VERSION HIBERNATION LAST OPERATION STATUS AGE
local local local local 1.21.0 Awake Create Processing (43%) healthy 94s
(Optional): You could also execute a simple e2e test (creating and deleting a shoot) by running:
make test-e2e-local-simple KUBECONFIG="$PWD/example/gardener-local/kind/local/kubeconfig"
When the shoot got successfully created you can access it as follows:
kubectl -n garden-local get secret local.kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > /tmp/kubeconfig-shoot-local.yaml
kubectl --kubeconfig=/tmp/kubeconfig-shoot-local.yaml get nodes
(Optional): Setting Up a Second Seed Cluster
There are cases where you would want to create a second seed cluster in your local setup. For example, if you want to test the control plane migration feature. The following steps describe how to do that.
Add a new IP address on your loopback device which will be necessary for the new KinD cluster that you will create. On Mac, the default loopback device is lo0
.
sudo ip addr add 127.0.0.2 dev lo0 # adding 127.0.0.2 ip to the loopback interface
Next, setup the second KinD cluster:
make kind2-up KIND_ENV=local
This command sets up a new KinD cluster named gardener-local2
and stores its kubeconfig in the ./example/gardener-local/kind/local2/kubeconfig
file. You will need this file when starting the provider-local
extension controller for the second seed cluster.
make register-kind2-env # registering the local2 seed
make start-gardenlet SEED_NAME=local2 # starting gardenlet for the local2 seed
In a new terminal pane, run:
export KUBECONFIG=./example/gardener-local/kind/local2/kubeconfig # setting KUBECONFIG to point to second kind cluster
make start-extension-provider-local \
WEBHOOK_SERVER_PORT=9444 \
WEBHOOK_CERT_DIR=/tmp/gardener-extension-provider-local2 \
SERVICE_HOST_IP=127.0.0.2 \
METRICS_BIND_ADDRESS=:8082 \
HEALTH_BIND_ADDRESS=:8083 # starting gardener-extension-provider-local
If you want to perform a control plane migration you can follow the steps outlined in the Control Plane Migration topic to migrate the shoot cluster to the second seed you just created.
Deleting the Shoot
Cluster
./hack/usage/delete shoot local garden-local
(Optional): Tear Down the Second Seed Cluster
make tear-down-kind2-env
make kind2-down
Tear Down the Gardener Environment
make tear-down-local-env
make kind-down
Remote Local Setup
Just like Prow is executing the KinD based integration tests in a K8s pod, it is possible to interactively run this KinD based Gardener development environment aka “local setup” in a “remote” K8s pod.
k apply -f docs/development/content/remote-local-setup.yaml
k exec -it deployment/remote-local-setup -- sh
tmux -u a
Caveats
Please refer to the TMUX documentation for working effectively inside the remote-local-setup pod.
To access Grafana, Prometheus, or other components in a browser, two port forwards are needed:
The port forward from the laptop to the pod:
k port-forward deployment/remote-local-setup 3000
The port forward in the remote-local-setup pod to the respective component:
k port-forward -n shoot--local--local deployment/grafana-operators 3000
Related Links
1.4.5 - High Availability
High Availability of Deployed Components
gardenlet
s and extension controllers are deploying components via Deployment
s, StatefulSet
s, etc., as part of the shoot control plane, or the seed or shoot system components.
Some of the above component deployments must be further tuned to improve fault tolerance / resilience of the service. This document outlines what needs to be done to achieve this goal.
Please be forwarded to the Convenient Application Of These Rules section, if you want to take a shortcut to the list of actions that require developers’ attention.
Seed Clusters
The worker nodes of seed clusters can be deployed to one or multiple availability zones.
The Seed
specification allows you to provide the information which zones are available:
spec:
provider:
region: europe-1
zones:
- europe-1a
- europe-1b
- europe-1c
Independent of the number of zones, seed system components like the gardenlet
or the extension controllers themselves, or others like etcd-druid
, dependency-watchdog
, etc., should always be running with multiple replicas.
Concretely, all seed system components should respect the following conventions:
Replica Counts
Component Type < 3
Zones>= 3
ZonesComment Observability (Monitoring, Logging) 1 1 Downtimes accepted due to cost reasons Controllers 2 2 / (Webhook) Servers 2 2 / Apart from the above, there might be special cases where these rules do not apply, for example:
istio-ingressgateway
is scaled horizontally, hence the above numbers are the minimum values.nginx-ingress-controller
in the seed cluster is used to advertise all shoot observability endpoints, so due to performance reasons it runs with2
replicas at all times. In the future, this component might disappear in favor of theistio-ingressgateway
anyways.
Topology Spread Constraints
When the component has
>= 2
replicas …… then it should also have a
topologySpreadConstraint
, ensuring the replicas are spread over the nodes:spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway matchLabels: ...
Hence, the node spread is done on best-effort basis only.
… and the seed cluster has
>= 2
zones, then the component should also have a secondtopologySpreadConstraint
, ensuring the replicas are spread over the zones:spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule matchLabels: ...
According to these conventions, even seed clusters with only one availability zone try to be highly available “as good as possible” by spreading the replicas across multiple nodes. Hence, while such seed clusters obviously cannot handle zone outages, they can at least handle node failures.
Shoot Clusters
The Shoot
specification allows configuring “high availability” as well as the failure tolerance type for the control plane components, see Highly Available Shoot Control Plane for details.
Regarding the seed cluster selection, the only constraint is that shoot clusters with failure tolerance type zone
are only allowed to run on seed clusters with at least three zones.
All other shoot clusters (non-HA or those with failure tolerance type node
) can run on seed clusters with any number of zones.
Control Plane Components
All control plane components should respect the following conventions:
Replica Counts
Component Type w/o HA w/ HA ( node
)w/ HA ( zone
)Comment Observability (Monitoring, Logging) 1 1 1 Downtimes accepted due to cost reasons Controllers 1 2 2 / (Webhook) Servers 2 2 2 / Apart from the above, there might be special cases where these rules do not apply, for example:
etcd
is a server, though the most critical component of a cluster requiring a quorum to survive failures. Hence, it should have3
replicas even when the failure tolerance isnode
only.kube-apiserver
is scaled horizontally, hence the above numbers are the minimum values (even when the shoot cluster is not HA, there might be multiple replicas).
Topology Spread Constraints
When the component has
>= 2
replicas …… then it should also have a
topologySpreadConstraint
ensuring the replicas are spread over the nodes:spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway matchLabels: ...
Hence, the node spread is done on best-effort basis only.
However, if the shoot cluster has defined a failure tolerance type, the
whenUnsafisfiable
field should be set toDoNotSchedule
.… and the failure tolerance type of the shoot cluster is
zone
, then the component should also have a secondtopologySpreadConstraint
ensuring the replicas are spread over the zones:spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule matchLabels: ...
Node Affinity
The
gardenlet
annotates the shoot namespace in the seed cluster with thehigh-availability-config.resources.gardener.cloud/zones
annotation.- If the shoot cluster is non-HA or has failure tolerance type
node
, then the value will be always exactly one zone (e.g.,high-availability-config.resources.gardener.cloud/zones=europe-1b
). - If the shoot cluster has failure tolerance type
zone
, then the value will always contain exactly three zones (e.g.,high-availability-config.resources.gardener.cloud/zones=europe-1a,europe-1b,europe-1c
).
For backwards-compatibility, this annotation might contain multiple zones for shoot clusters created before
gardener/gardener@v1.60
and not having failure tolerance typezone
. This is because their volumes might already exist in multiple zones, hence pinning them to only one zone would not work.Hence, in case this annotation is present, the components should have the following node affinity:
spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/zone operator: In values: - europe-1a # - ...
This is to ensure all pods are running in the same (set of) availability zone(s) such that cross-zone network traffic is avoided as much as possible (such traffic is typically charged by the underlying infrastructure provider).
- If the shoot cluster is non-HA or has failure tolerance type
System Components
The availability of system components is independent of the control plane since they run on the shoot worker nodes while the control plane components run on the seed worker nodes (for more information, see the Kubernetes architecture overview).
Hence, it only depends on the number of availability zones configured in the shoot worker pools via .spec.provider.workers[].zones
.
Concretely, the highest number of zones of a worker pool with systemComponents.allow=true
is considered.
All system components should respect the following conventions:
Replica Counts
Component Type 1
or2
Zones>= 3
ZonesControllers 2 2 (Webhook) Servers 2 2 Apart from the above, there might be special cases where these rules do not apply, for example:
coredns
is scaled horizontally (today), hence the above numbers are the minimum values (possibly, scaling these components vertically may be more appropriate, but that’s unrelated to the HA subject matter).- Optional addons like
nginx-ingress
orkubernetes-dashboard
are only provided on best-effort basis for evaluation purposes, hence they run with1
replica at all times.
Topology Spread Constraints
When the component has
>= 2
replicas …… then it should also have a
topologySpreadConstraint
ensuring the replicas are spread over the nodes:spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway matchLabels: ...
Hence, the node spread is done on best-effort basis only.
… and the cluster has
>= 2
zones, then the component should also have a secondtopologySpreadConstraint
ensuring the replicas are spread over the zones:spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule matchLabels: ...
Convenient Application of These Rules
According to above scenarios and conventions, the replicas
, topologySpreadConstraints
or affinity
settings of the deployed components might need to be adapted.
In order to apply those conveniently and easily for developers, Gardener installs a mutating webhook into both seed and shoot clusters which reacts on Deployment
s and StatefulSet
s deployed to namespaces with the high-availability-config.resources.gardener.cloud/consider=true
label set.
The following actions have to be taken by developers:
Check if
components
are prepared to run concurrently with multiple replicas, e.g. controllers usually use leader election to achieve this.All components should be generally equipped with
PodDisruptionBudget
s with.spec.maxUnavailable=1
:
spec:
maxUnavailable: 1
selector:
matchLabels: ...
- Add the label
high-availability-config.resources.gardener.cloud/type
todeployment
s orstatefulset
s, as well as optionally involvedhorizontalpodautoscaler
s orHVPA
s where the following two values are possible:
controller
server
Type server
is also preferred if a component is a controller and (webhook) server at the same time.
You can read more about the webhook’s internals in High Availability Config.
gardenlet
Internals
Make sure you have read the above document about the webhook internals before continuing reading this section.
Seed
Controller
The gardenlet
performs the following changes on all namespaces running seed system components:
- adds the label
high-availability-config.resources.gardener.cloud/consider=true
. - adds the annotation
high-availability-config.resources.gardener.cloud/zones=<zones>
, where<zones>
is the list provided in.spec.provider.zones[]
in theSeed
specification.
Note that neither the high-availability-config.resources.gardener.cloud/failure-tolerance-type
, nor the high-availability-config.resources.gardener.cloud/zone-pinning
annotations are set, hence the node affinity would never be touched by the webhook.
The only exception to this rule are the istio ingress gateway namespaces. This includes the default istio ingress gateway when SNI is enabled, as well as analogous namespaces for exposure classes and zone-specific istio ingress gateways. Those namespaces
will additionally be annotated with high-availability-config.resources.gardener.cloud/zone-pinning
set to true
, resulting in the node affinities and the topology spread constraints being set. The replicas are not touched, as the istio ingress gateways
are scaled by a horizontal autoscaler instance.
Shoot
Controller
Control Plane
The gardenlet
performs the following changes on the namespace running the shoot control plane components:
- adds the label
high-availability-config.resources.gardener.cloud/consider=true
. This makes the webhook mutate the replica count and the topology spread constraints. - adds the annotation
high-availability-config.resources.gardener.cloud/failure-tolerance-type
with value equal to.spec.controlPlane.highAvailability.failureTolerance.type
(or""
, if.spec.controlPlane.highAvailability=nil
). This makes the webhook mutate the node affinity according to the specified zone(s). - adds the annotation
high-availability-config.resources.gardener.cloud/zones=<zones>
, where<zones>
is a …- … random zone chosen from the
.spec.provider.zones[]
list in theSeed
specification (always only one zone (even if there are multiple available in the seed cluster)) in case theShoot
has no HA setting (i.e.,spec.controlPlane.highAvailability=nil
) or when theShoot
has HA setting with failure tolerance typenode
. - … list of three randomly chosen zones from the
.spec.provider.zones[]
list in theSeed
specification in case theShoot
has HA setting with failure tolerance typezone
.
- … random zone chosen from the
System Components
The gardenlet
performs the following changes on all namespaces running shoot system components:
- adds the label
high-availability-config.resources.gardener.cloud/consider=true
. This makes the webhook mutate the replica count and the topology spread constraints. - adds the annotation
high-availability-config.resources.gardener.cloud/zones=<zones>
where<zones>
is the merged list of zones provided in.zones[]
withsystemComponents.allow=true
for all worker pools in.spec.provider.workers[]
in theShoot
specification.
Note that neither the high-availability-config.resources.gardener.cloud/failure-tolerance-type
, nor the high-availability-config.resources.gardener.cloud/zone-pinning
annotations are set, hence the node affinity would never be touched by the webhook.
1.4.6 - Kubernetes Clients
Kubernetes Clients in Gardener
This document aims at providing a general developer guideline on different aspects of using Kubernetes clients in a large-scale distributed system and project like Gardener. The points included here are not meant to be consulted as absolute rules, but rather as general rules of thumb that allow developers to get a better feeling about certain gotchas and caveats. It should be updated with lessons learned from maintaining the project and running Gardener in production.
Prerequisites:
Please familiarize yourself with the following basic Kubernetes API concepts first, if you’re new to Kubernetes. A good understanding of these basics will help you better comprehend the following document.
- Kubernetes API Concepts (including terminology, watch basics, etc.)
- Extending the Kubernetes API (including Custom Resources and aggregation layer / extension API servers)
- Extend the Kubernetes API with CustomResourceDefinitions
- Working with Kubernetes Objects
- Sample Controller (the diagram helps to build an understanding of an controller’s basic structure)
Client Types: Client-Go, Generated, Controller-Runtime
For historical reasons, you will find different kinds of Kubernetes clients in Gardener:
Client-Go Clients
client-go is the default/official client for talking to the Kubernetes API in Golang.
It features the so called “client sets” for all built-in Kubernetes API groups and versions (e.g. v1
(aka core/v1
), apps/v1
, etc.).
client-go clients are generated from the built-in API types using client-gen and are composed of interfaces for every known API GroupVersionKind.
A typical client-go usage looks like this:
var (
ctx context.Context
c kubernetes.Interface // "k8s.io/client-go/kubernetes"
deployment *appsv1.Deployment // "k8s.io/api/apps/v1"
)
updatedDeployment, err := c.AppsV1().Deployments("default").Update(ctx, deployment, metav1.UpdateOptions{})
Important characteristics of client-go clients:
- clients are specific to a given API GroupVersionKind, i.e., clients are hard-coded to corresponding API-paths (don’t need to use the discovery API to map GVK to a REST endpoint path).
- client’s don’t modify the passed in-memory object (e.g.
deployment
in the above example). Instead, they return a new in-memory object.
This means that controllers have to continue working with the new in-memory object or overwrite the shared object to not lose any state updates.
Generated Client Sets for Gardener APIs
Gardener’s APIs extend the Kubernetes API by registering an extension API server (in the garden cluster) and CustomResourceDefinition
s (on Seed clusters), meaning that the Kubernetes API will expose additional REST endpoints to manage Gardener resources in addition to the built-in API resources.
In order to talk to these extended APIs in our controllers and components, client-gen is used to generate client-go-style clients to pkg/client/{core,extensions,seedmanagement,...}
.
Usage of these clients is equivalent to client-go
clients, and the same characteristics apply. For example:
var (
ctx context.Context
c gardencoreclientset.Interface // "github.com/gardener/gardener/pkg/client/core/clientset/versioned"
shoot *gardencorev1beta1.Shoot // "github.com/gardener/gardener/pkg/apis/core/v1beta1"
)
updatedShoot, err := c.CoreV1beta1().Shoots("garden-my-project").Update(ctx, shoot, metav1.UpdateOptions{})
Controller-Runtime Clients
controller-runtime is a Kubernetes community project (kubebuilder subproject) for building controllers and operators for custom resources. Therefore, it features a generic client that follows a different approach and does not rely on generated client sets. Instead, the client can be used for managing any Kubernetes resources (built-in or custom) homogeneously. For example:
var (
ctx context.Context
c client.Client // "sigs.k8s.io/controller-runtime/pkg/client"
deployment *appsv1.Deployment // "k8s.io/api/apps/v1"
shoot *gardencorev1beta1.Shoot // "github.com/gardener/gardener/pkg/apis/core/v1beta1"
)
err := c.Update(ctx, deployment)
// or
err = c.Update(ctx, shoot)
A brief introduction to controller-runtime and its basic constructs can be found at the official Go documentation.
Important characteristics of controller-runtime clients:
- The client functions take a generic
client.Object
orclient.ObjectList
value. These interfaces are implemented by all Golang types, that represent Kubernetes API objects or lists respectively which can be interacted with via usual API requests. [1] - The client first consults a
runtime.Scheme
(configured during client creation) for recognizing the object’sGroupVersionKind
(this happens on the client-side only).
Aruntime.Scheme
is basically a registry for Golang API types, defaulting and conversion functions. Schemes are usually provided perGroupVersion
(see this example forapps/v1
) and can be combined to one single scheme for further usage (example). In controller-runtime clients, schemes are used only for mapping a typed API object to itsGroupVersionKind
. - It then consults a
meta.RESTMapper
(also configured during client creation) for mapping theGroupVersionKind
to aRESTMapping
, which contains theGroupVersionResource
andScope
(namespaced or cluster-scoped). From these values, the client can unambiguously determine the REST endpoint path of the corresponding API resource. For instance:appsv1.DeploymentList
is available at/apis/apps/v1/deployments
or/apis/apps/v1/namespaces/<namespace>/deployments
respectively.- There are different
RESTMapper
implementations, but generally they are talking to the API server’s discovery API for retrievingRESTMappings
for all API resources known to the API server (either built-in, registered via API extension orCustomResourceDefinition
s). - The default implementation of a controller-runtime (which Gardener uses as well) is the dynamic
RESTMapper
. It caches discovery results (i.e.RESTMappings
) in-memory and only re-discovers resources from the API server when a client tries to use an unknownGroupVersionKind
, i.e., when it encounters aNo{Kind,Resource}MatchError
.
- There are different
- The client writes back results from the API server into the passed in-memory object.
- This means that controllers don’t have to worry about copying back the results and should just continue to work on the given in-memory object.
- This is a nice and flexible pattern, and helper functions should try to follow it wherever applicable. Meaning, if possible accept an object param, pass it down to clients and keep working on the same in-memory object instead of creating a new one in your helper function.
- The benefit is that you don’t lose updates to the API object and always have the last-known state in memory. Therefore, you don’t have to read it again, e.g., for getting the current
resourceVersion
when working with optimistic locking, and thus minimize the chances for running into conflicts. - However, controllers must not use the same in-memory object concurrently in multiple goroutines. For example, decoding results from the API server in multiple goroutines into the same maps (e.g., labels, annotations) will cause panics because of “concurrent map writes”. Also, reading from an in-memory API object in one goroutine while decoding into it in another goroutine will yield non-atomic reads, meaning data might be corrupt and represent a non-valid/non-existing API object.
- Therefore, if you need to use the same in-memory object in multiple goroutines concurrently (e.g., shared state), remember to leverage proper synchronization techniques like channels, mutexes,
atomic.Value
and/or copy the object prior to use. The average controller however, will not need to share in-memory API objects between goroutines, and it’s typically an indicator that the controller’s design should be improved.
- The client decoder erases the object’s
TypeMeta
(apiVersion
andkind
fields) after retrieval from the API server, see kubernetes/kubernetes#80609, kubernetes-sigs/controller-runtime#1517. Unstructured and metadata-only requests objects are an exception to this because the containedTypeMeta
is the only way to identify the object’s type. Because of this behavior,obj.GetObjectKind().GroupVersionKind()
is likely to return an emptyGroupVersionKind
. I.e., you must not rely onTypeMeta
being set orGetObjectKind()
to return something usable.
If you need to identify an object’sGroupVersionKind
, use a scheme and itsObjectKinds
function instead (or the helper functionapiutil.GVKForObject
). This is not specific to controller-runtime clients and applies to client-go clients as well.
[1] Other lower level, config or internal API types (e.g., such as AdmissionReview
) don’t implement client.Object
. However, you also can’t interact with such objects via the Kubernetes API and thus also not via a client, so this can be disregarded at this point.
Metadata-Only Clients
Additionally, controller-runtime clients can be used to easily retrieve metadata-only objects or lists.
This is useful for efficiently checking if at least one object of a given kind exists, or retrieving metadata of an object, if one is not interested in the rest (e.g., spec/status).
The Accept
header sent to the API server then contains application/json;as=PartialObjectMetadataList;g=meta.k8s.io;v=v1
, which makes the API server only return metadata of the retrieved object(s).
This saves network traffic and CPU/memory load on the API server and client side.
If the client fully lists all objects of a given kind including their spec/status, the resulting list can be quite large and easily exceed the controllers available memory.
That’s why it’s important to carefully check if a full list is actually needed, or if metadata-only list can be used instead.
For example:
var (
ctx context.Context
c client.Client // "sigs.k8s.io/controller-runtime/pkg/client"
shootList = &metav1.PartialObjectMetadataList{} // "k8s.io/apimachinery/pkg/apis/meta/v1"
)
shootList.SetGroupVersionKind(gardencorev1beta1.SchemeGroupVersion.WithKind("ShootList"))
if err := c.List(ctx, shootList, client.InNamespace("garden-my-project"), client.Limit(1)); err != nil {
return err
}
if len(shootList.Items) > 0 {
// project has at least one shoot
} else {
// project doesn't have any shoots
}
Gardener’s Client Collection, ClientMaps
The Gardener codebase has a collection of clients (kubernetes.Interface
), which can return all the above mentioned client types.
Additionally, it contains helpers for rendering and applying helm charts (ChartRender
, ChartApplier
) and retrieving the API server’s version (Version
).
Client sets are managed by so called ClientMap
s, which are a form of registry for all client set for a given type of cluster, i.e., Garden, Seed and Shoot.
ClientMaps manage the whole lifecycle of clients: they take care of creating them if they don’t exist already, running their caches, refreshing their cached server version and invalidating them when they are no longer needed.
var (
ctx context.Context
cm clientmap.ClientMap // "github.com/gardener/gardener/pkg/client/kubernetes/clientmap"
shoot *gardencorev1beta1.Shoot
)
cs, err := cm.GetClient(ctx, keys.ForShoot(shoot)) // kubernetes.Interface
if err != nil {
return err
}
c := cs.Client() // client.Client
The client collection mainly exist for historical reasons (there used to be a lot of code using the client-go style clients). However, Gardener is in the process of moving more towards controller-runtime and only using their clients, as they provide many benefits and are much easier to use. Also, gardener/gardener#4251 aims at refactoring our controller and admission components to native controller-runtime components.
⚠️ Please always prefer controller-runtime clients over other clients when writing new code or refactoring existing code.
Cache Types: Informers, Listers, Controller-Runtime Caches
Similar to the different types of client(set)s, there are also different kinds of Kubernetes client caches.
However, all of them are based on the same concept: Informer
s.
An Informer
is a watch-based cache implementation, meaning it opens watch connections to the API server and continuously updates cached objects based on the received watch events (ADDED
, MODIFIED
, DELETED
).
Informer
s offer to add indices to the cache for efficient object lookup (e.g., by name or labels) and to add EventHandler
s for the watch events.
The latter is used by controllers to fill queues with objects that should be reconciled on watch events.
Informers are used in and created via several higher-level constructs:
SharedInformerFactories, Listers
The generated clients (built-in as well as extended) feature a SharedInformerFactory
for every API group, which can be used to create and retrieve Informers
for all GroupVersionKinds.
Similarly, it can be used to retrieve Listers
that allow getting and listing objects from the Informer
’s cache.
However, both of these constructs are only used for historical reasons, and we are in the process of migrating away from them in favor of cached controller-runtime clients (see gardener/gardener#2414, gardener/gardener#2822). Thus, they are described only briefly here.
Important characteristics of Listers:
- Objects read from Informers and Listers can always be slightly out-out-date (i.e., stale) because the client has to first observe changes to API objects via watch events (which can intermittently lag behind by a second or even more).
- Thus, don’t make any decisions based on data read from Listers if the consequences of deciding wrongfully based on stale state might be catastrophic (e.g. leaking infrastructure resources). In such cases, read directly from the API server via a client instead.
- Objects retrieved from Informers or Listers are pointers to the cached objects, so they must not be modified without copying them first, otherwise the objects in the cache are also modified.
Controller-Runtime Caches
controller-runtime features a cache implementation that can be used equivalently as their clients. In fact, it implements a subset of the client.Client
interface containing the Get
and List
functions.
Under the hood, a cache.Cache
dynamically creates Informers
(i.e., opens watches) for every object GroupVersionKind that is being retrieved from it.
Note that the underlying Informers of a controller-runtime cache (cache.Cache
) and the ones of a SharedInformerFactory
(client-go) are not related in any way.
Both create Informers
and watch objects on the API server individually.
This means that if you read the same object from different cache implementations, you may receive different versions of the object because the watch connections of the individual Informers are not synced.
⚠️ Because of this, controllers/reconcilers should get the object from the same cache in the reconcile loop, where the
EventHandler
was also added to set up the controller. For example, if aSharedInformerFactory
is used for setting up the controller then read the object in the reconciler from theLister
instead of from a cached controller-runtime client.
By default, the client.Client
created by a controller-runtime Manager
is a DelegatingClient
. It delegates Get
and List
calls to a Cache
, and all other calls to a client that talks directly to the API server. Exceptions are requests with *unstructured.Unstructured
objects and object kinds that were configured to be excluded from the cache in the DelegatingClient
.
ℹ️
kubernetes.Interface.Client()
returns aDelegatingClient
that uses the cache returned fromkubernetes.Interface.Cache()
under the hood. This means that allClient()
usages need to be ready for cached clients and should be able to cater with stale cache reads.
Important characteristics of cached controller-runtime clients:
- Like for Listers, objects read from a controller-runtime cache can always be slightly out of date. Hence, don’t base any important decisions on data read from the cache (see above).
- In contrast to Listers, controller-runtime caches fill the passed in-memory object with the state of the object in the cache (i.e., they perform something like a “deep copy into”). This means that objects read from a controller-runtime cache can safely be modified without unintended side effects.
- Reading from a controller-runtime cache or a cached controller-runtime client implicitly starts a watch for the given object kind under the hood. This has important consequences:
- Reading a given object kind from the cache for the first time can take up to a few seconds depending on size and amount of objects as well as API server latency. This is because the cache has to do a full list operation and wait for an initial watch sync before returning results.
- ⚠️ Controllers need appropriate RBAC permissions for the object kinds they retrieve via cached clients (i.e.,
list
andwatch
). - ⚠️ By default, watches started by a controller-runtime cache are cluster-scoped, meaning it watches and caches objects across all namespaces. Thus, be careful which objects to read from the cache as it might significantly increase the controller’s memory footprint.
- There is no interaction with the cache on writing calls (
Create
,Update
,Patch
andDelete
), see below.
Uncached objects, filtered caches, APIReader
s:
In order to allow more granular control over which object kinds should be cached and which calls should bypass the cache, controller-runtime offers a few mechanisms to further tweak the client/cache behavior:
- When creating a
DelegatingClient
, certain object kinds can be configured to always be read directly from the API instead of from the cache. Note that this does not prevent starting a new Informer when retrieving them directly from the cache. - Watches can be restricted to a given (set of) namespace(s) by using
cache.MultiNamespacedCacheBuilder
or settingcache.Options.Namespace
. - Watches can be filtered (e.g., by label) per object kind by configuring
cache.Options.SelectorsByObject
on creation of the cache. - Retrieving metadata-only objects or lists from a cache results in a metadata-only watch/cache for that object kind.
- The
APIReader
can be used to always talk directly to the API server for a givenGet
orList
call (use with care and only as a last resort!).
To Cache or Not to Cache
Although watch-based caches are an important factor for the immense scalability of Kubernetes, it definitely comes at a price (mainly in terms of memory consumption). Thus, developers need to be careful when introducing new API calls and caching new object kinds. Here are some general guidelines on choosing whether to read from a cache or not:
- Always try to use the cache wherever possible and make your controller able to tolerate stale reads.
- Leverage optimistic locking: use deterministic naming for objects you create (this is what the
Deployment
controller does [2]). - Leverage optimistic locking / concurrency control of the API server: send updates/patches with the last-known
resourceVersion
from the cache (see below). This will make the request fail, if there were concurrent updates to the object (conflict error), which indicates that we have operated on stale data and might have made wrong decisions. In this case, let the controller handle the error with exponential backoff. This will make the controller eventually consistent. - Track the actions you took, e.g., when creating objects with
generateName
(this is what theReplicaSet
controller does [3]). The actions can be tracked in memory and repeated if the expected watch events don’t occur after a given amount of time. - Always try to write controllers with the assumption that data will only be eventually correct and can be slightly out of date (even if read directly from the API server!).
- If there is already some other code that needs a cache (e.g., a controller watch), reuse it instead of doing extra direct reads.
- Don’t read an object again if you just sent a write request. Write requests (
Create
,Update
,Patch
andDelete
) don’t interact with the cache. Hence, use the current state that the API server returned (filled into the passed in-memory object), which is basically a “free direct read” instead of reading the object again from a cache, because this will probably set back the object to an olderresourceVersion
.
- Leverage optimistic locking: use deterministic naming for objects you create (this is what the
- If you are concerned about the impact of the resulting cache, try to minimize that by using filtered or metadata-only watches.
- If watching and caching an object type is not feasible, for example because there will be a lot of updates, and you are only interested in the object every ~5m, or because it will blow up the controllers memory footprint, fallback to a direct read. This can either be done by disabling caching the object type generally or doing a single request via an
APIReader
. In any case, please bear in mind that every direct API call results in a quorum read from etcd, which can be costly in a heavily-utilized cluster and impose significant scalability limits. Thus, always try to minimize the impact of direct calls by filtering results by namespace or labels, limiting the number of results and/or using metadata-only calls.
[2] The Deployment
controller uses the pattern <deployment-name>-<podtemplate-hash>
for naming ReplicaSets
. This means, the name of a ReplicaSet
it tries to create/update/delete at any given time is deterministically calculated based on the Deployment
object. By this, it is insusceptible to stale reads from its ReplicaSets
cache.
[3] In simple terms, the ReplicaSet
controller tracks its CREATE pod
actions as follows: when creating new Pods
, it increases a counter of expected ADDED
watch events for the corresponding ReplicaSet
. As soon as such events arrive, it decreases the counter accordingly. It only creates new Pods
for a given ReplicaSet
once all expected events occurred (counter is back to zero) or a timeout has occurred. This way, it prevents creating more Pods
than desired because of stale cache reads and makes the controller eventually consistent.
Conflicts, Concurrency Control, and Optimistic Locking
Every Kubernetes API object contains the metadata.resourceVersion
field, which identifies an object’s version in the backing data store, i.e., etcd. Every write to an object in etcd results in a newer resourceVersion
.
This field is mainly used for concurrency control on the API server in an optimistic locking fashion, but also for efficient resumption of interrupted watch connections.
Optimistic locking in the Kubernetes API sense means that when a client wants to update an API object, then it includes the object’s resourceVersion
in the request to indicate the object’s version the modifications are based on.
If the resourceVersion
in etcd has not changed in the meantime, the update request is accepted by the API server and the updated object is written to etcd.
If the resourceVersion
sent by the client does not match the one of the object stored in etcd, there were concurrent modifications to the object. Consequently, the request is rejected with a conflict error (status code 409
, API reason Conflict
), for example:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "Operation cannot be fulfilled on configmaps \"foo\": the object has been modified; please apply your changes to the latest version and try again",
"reason": "Conflict",
"details": {
"name": "foo",
"kind": "configmaps"
},
"code": 409
}
This concurrency control is an important mechanism in Kubernetes as there are typically multiple clients acting on API objects at the same time (humans, different controllers, etc.). If a client receives a conflict error, it should read the object’s latest version from the API server, make the modifications based on the newest changes, and retry the update. The reasoning behind this is that a client might choose to make different decisions based on the concurrent changes made by other actors compared to the outdated version that it operated on.
Important points about concurrency control and conflicts:
- The
resourceVersion
field carries a string value and clients must not assume numeric values (the type and structure of versions depend on the backing data store). This means clients may compareresourceVersion
values to detect whether objects were changed. But they must not compareresourceVersion
s to figure out which one is newer/older, i.e., no greater/less-than comparisons are allowed. - By default, update calls (e.g. via client-go and controller-runtime clients) use optimistic locking as the passed in-memory usually object contains the latest
resourceVersion
known to the controller, which is then also sent to the API server. - API servers can also choose to accept update calls without optimistic locking (i.e., without a
resourceVersion
in the object’s metadata) for any given resource. However, sending update requests without optimistic locking is strongly discouraged, as doing so overwrites the entire object, discarding any concurrent changes made to it. - On the other side, patch requests can always be executed either with or without optimistic locking, by (not) including the
resourceVersion
in the patched object’s metadata. Sending patch requests without optimistic locking might be safe and even desirable as a patch typically updates only a specific section of the object. However, there are also situations where patching without optimistic locking is not safe (see below).
Don’t Retry on Conflict
Similar to how a human would typically handle a conflict error, there are helper functions implementing RetryOnConflict
-semantics, i.e., try an update call, then re-read the object if a conflict occurs, apply the modification again and retry the update.
However, controllers should generally not use RetryOnConflict
-semantics. Instead, controllers should abort their current reconciliation run and let the queue handle the conflict error with exponential backoff.
The reasoning behind this is that a conflict error indicates that the controller has operated on stale data and might have made wrong decisions earlier on in the reconciliation.
When using a helper function that implements RetryOnConflict
-semantics, the controller doesn’t check which fields were changed and doesn’t revise its previous decisions accordingly.
Instead, retrying on conflict basically just ignores any conflict error and blindly applies the modification.
To properly solve the conflict situation, controllers should immediately return with the error from the update call. This will cause retries with exponential backoff so that the cache has a chance to observe the latest changes to the object. In a later run, the controller will then make correct decisions based on the newest version of the object, not run into conflict errors, and will then be able to successfully reconcile the object. This way, the controller becomes eventually consistent.
The other way to solve the situation is to modify objects without optimistic locking in order to avoid running into a conflict in the first place (only if this is safe). This can be a preferable solution for controllers with long-running reconciliations (which is actually an anti-pattern but quite unavoidable in some of Gardener’s controllers). Aborting the entire reconciliation run is rather undesirable in such cases, as it will add a lot of unnecessary waiting time for end users and overhead in terms of compute and network usage.
However, in any case, retrying on conflict is probably not the right option to solve the situation (there are some correct use cases for it, though, they are very rare). Hence, don’t retry on conflict.
To Lock or Not to Lock
As explained before, conflicts are actually important and prevent clients from doing wrongful concurrent updates. This means that conflicts are not something we generally want to avoid or ignore. However, in many cases controllers are exclusive owners of the fields they want to update and thus it might be safe to run without optimistic locking.
For example, the gardenlet is the exclusive owner of the spec
section of the Extension resources it creates on behalf of a Shoot (e.g., the Infrastructure
resource for creating VPC, etc.). Meaning, it knows the exact desired state and no other actor is supposed to update the Infrastructure’s spec
fields.
When the gardenlet now updates the Infrastructures spec
section as part of the Shoot reconciliation, it can simply issue a PATCH
request that only updates the spec
and runs without optimistic locking.
If another controller concurrently updated the object in the meantime (e.g., the status
section), the resourceVersion
got changed, which would cause a conflict error if running with optimistic locking.
However, concurrent status
updates would not change the gardenlet’s mind on the desired spec
of the Infrastructure resource as it is determined only by looking at the Shoot’s specification.
If the spec
section was changed concurrently, it’s still fine to overwrite it because the gardenlet should reconcile the spec
back to its desired state.
Generally speaking, if a controller is the exclusive owner of a given set of fields and they are independent of concurrent changes to other fields in that object, it can patch these fields without optimistic locking. This might ignore concurrent changes to other fields or blindly overwrite changes to the same fields, but this is fine if the mentioned conditions apply. Obviously, this applies only to patch requests that modify only a specific set of fields but not to update requests that replace the entire object.
In such cases, it’s even desirable to run without optimistic locking as it will be more performant and save retries. If certain requests are made with high frequency and have a good chance of causing conflicts, retries because of optimistic locking can cause a lot of additional network traffic in a large-scale Gardener installation.
Updates, Patches, Server-Side Apply
There are different ways of modifying Kubernetes API objects. The following snippet demonstrates how to do a given modification with the most frequently used options using a controller-runtime client:
var (
ctx context.Context
c client.Client
shoot *gardencorev1beta1.Shoot
)
// update
shoot.Spec.Kubernetes.Version = "1.22"
err := c.Update(ctx, shoot)
// json merge patch
patch := client.MergeFrom(shoot.DeepCopy())
shoot.Spec.Kubernetes.Version = "1.22"
err = c.Patch(ctx, shoot, patch)
// strategic merge patch
patch = client.StrategicMergeFrom(shoot.DeepCopy())
shoot.Spec.Kubernetes.Version = "1.22"
err = c.Patch(ctx, shoot, patch)
Important characteristics of the shown request types:
- Update requests always send the entire object to the API server and update all fields accordingly. By default, optimistic locking is used (
resourceVersion
is included). - Both patch types run without optimistic locking by default. However, it can be enabled explicitly if needed:
// json merge patch + optimistic locking patch := client.MergeFromWithOptions(shoot.DeepCopy(), client.MergeFromWithOptimisticLock{}) // ... // strategic merge patch + optimistic locking patch = client.StrategicMergeFrom(shoot.DeepCopy(), client.MergeFromWithOptimisticLock{}) // ...
- Patch requests only contain the changes made to the in-memory object between the copy passed to
client.*MergeFrom
and the object passed toClient.Patch()
. The diff is calculated on the client-side based on the in-memory objects only. This means that if in the meantime some fields were changed on the API server to a different value than the one on the client-side, the fields will not be changed back as long as they are not changed on the client-side as well (there will be no diff in memory). - Thus, if you want to ensure a given state using patch requests, always read the object first before patching it, as there will be no diff otherwise, meaning the patch will be empty. For more information, see gardener/gardener#4057 and the comments in gardener/gardener#4027.
- Also, always send updates and patch requests even if your controller hasn’t made any changes to the current state on the API server. I.e., don’t make any optimization for preventing empty patches or no-op updates. There might be mutating webhooks in the system that will modify the object and that rely on update/patch requests being sent (even if they are no-op). Gardener’s extension concept makes heavy use of mutating webhooks, so it’s important to keep this in mind.
- JSON merge patches always replace lists as a whole and don’t merge them. Keep this in mind when operating on lists with merge patch requests. If the controller is the exclusive owner of the entire list, it’s safe to run without optimistic locking. Though, if you want to prevent overwriting concurrent changes to the list or its items made by other actors (e.g., additions/removals to the
metadata.finalizers
list), enable optimistic locking. - Strategic merge patches are able to make more granular modifications to lists and their elements without replacing the entire list. It uses Golang struct tags of the API types to determine which and how lists should be merged. See Update API Objects in Place Using kubectl patch or the strategic merge patch documentation for more in-depth explanations and comparison with JSON merge patches.
With this, controllers might be able to issue patch requests for individual list items without optimistic locking, even if they are not exclusive owners of the entire list. Remember to check the
patchStrategy
andpatchMergeKey
struct tags of the fields you want to modify before blindly adding patch requests without optimistic locking. - Strategic merge patches are only supported by built-in Kubernetes resources and custom resources served by Extension API servers. Strategic merge patches are not supported by custom resources defined by
CustomResourceDefinition
s (see this comparison). In that case, fallback to JSON merge patches. - Server-side Apply is yet another mechanism to modify API objects, which is supported by all API resources (in newer Kubernetes versions). However, it has a few problems and more caveats preventing us from using it in Gardener at the time of writing. See gardener/gardener#4122 for more details.
Generally speaking, patches are often the better option compared to update requests because they can save network traffic, encoding/decoding effort, and avoid conflicts under the presented conditions. If choosing a patch type, consider which type is supported by the resource you’re modifying and what will happen in case of a conflict. Consider whether your modification is safe to run without optimistic locking. However, there is no simple rule of thumb on which patch type to choose.
On Helper Functions
Here is a note on some helper functions, that should be avoided and why:
controllerutil.CreateOrUpdate
does a basic get, mutate and create or update call chain, which is often used in controllers. We should avoid using this helper function in Gardener, because it is likely to cause conflicts for cached clients and doesn’t send no-op requests if nothing was changed, which can cause problems because of the heavy use of webhooks in Gardener extensions (see above).
That’s why usage of this function was completely replaced in gardener/gardener#4227 and similar PRs.
controllerutil.CreateOrPatch
is similar to CreateOrUpdate
but does a patch request instead of an update request. It has the same drawback as CreateOrUpdate
regarding no-op updates.
Also, controllers can’t use optimistic locking or strategic merge patches when using CreateOrPatch
.
Another reason for avoiding use of this function is that it also implicitly patches the status section if it was changed, which is confusing for others reading the code. To accomplish this, the func does some back and forth conversion, comparison and checks, which are unnecessary in most of our cases and simply wasted CPU cycles and complexity we want to avoid.
There were some Try{Update,UpdateStatus,Patch,PatchStatus}
helper functions in Gardener that were already removed by gardener/gardener#4378 but are still used in some extension code at the time of writing.
The reason for eliminating these functions is that they implement RetryOnConflict
-semantics. Meaning, they first get the object, mutate it, then try to update and retry if a conflict error occurs.
As explained above, retrying on conflict is a controller anti-pattern and should be avoided in almost every situation.
The other problem with these functions is that they read the object first from the API server (always do a direct call), although in most cases we already have a recent version of the object at hand. So, using this function generally does unnecessary API calls and therefore causes unwanted compute and network load.
For the reasons explained above, there are similar helper functions that accomplish similar things but address the mentioned drawbacks: controllerutils.{GetAndCreateOrMergePatch,GetAndCreateOrStrategicMergePatch}
.
These can be safely used as replacements for the aforementioned helper funcs.
If they are not fitting for your use case, for example because you need to use optimistic locking, just do the appropriate calls in the controller directly.
Related Links
- Kubernetes Client usage in Gardener (Community Meeting talk, 2020-06-26)
These resources are only partially related to the topics covered in this doc, but might still be interesting for developer seeking a deeper understanding of Kubernetes API machinery, architecture and foundational concepts.
1.4.7 - Local Setup
Overview
Conceptually, all Gardener components are designed to run as a Pod inside a Kubernetes cluster. The Gardener API server extends the Kubernetes API via the user-aggregated API server concepts. However, if you want to develop it, you may want to work locally with the Gardener without building a Docker image and deploying it to a cluster each and every time. That means that the Gardener runs outside a Kubernetes cluster which requires providing a Kubeconfig in your local filesystem and point the Gardener to it when starting it (see below).
Further details can be found in
This guide is split into three main parts:
- Preparing your setup by installing all dependencies and tools
- Building and starting Gardener components locally
- Using your local Gardener setup to create a Shoot
Preparing the Setup
[macOS only] Installing homebrew
The copy-paste instructions in this guide are designed for macOS and use the package manager Homebrew.
On macOS run
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Installing git
We use git
as VCS which you need to install. On macOS run
brew install git
For other OS, please check the Git installation documentation.
Installing Go
Install the latest version of Go. On macOS run
brew install go
For other OS, please check Go installation documentation.
Installing kubectl
Install kubectl
. Please make sure that the version of kubectl
is at least v1.20.x
. On macOS run
brew install kubernetes-cli
For other OS, please check the kubectl installation documentation.
Installing Docker
You need to have docker installed and running. On macOS run
brew install --cask docker
For other OS please check the docker installation documentation.
Installing iproute2
iproute2
provides a collection of utilities for network administration and configuration. On macOS run
brew install iproute2mac
Installing jq
brew install jq
Installing GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel, used by the code generation scripts (make generate
). On macOS run
brew install parallel
[macOS only] Install GNU Core Utilities
When running on macOS, install the GNU core utilities and friends:
brew install coreutils gnu-sed gnu-tar grep
This will create symbolic links for the GNU utilities with g
prefix in /usr/local/bin
, e.g., gsed
or gbase64
. To allow using them without the g
prefix please put /usr/local/opt/coreutils/libexec/gnubin
etc., at the beginning of your PATH
environment variable, e.g., export PATH=/usr/local/opt/coreutils/libexec/gnubin:$PATH
(brew
will print out instructions for each installed formula).
export PATH=/usr/local/opt/coreutils/libexec/gnubin:$PATH
export PATH=/usr/local/opt/gnu-sed/libexec/gnubin:$PATH
export PATH=/usr/local/opt/gnu-tar/libexec/gnubin:$PATH
export PATH=/usr/local/opt/grep/libexec/gnubin:$PATH
[Windows Only] WSL2
Apart from Linux distributions and macOS, the local gardener setup can also run on the Windows Subsystem for Linux 2.
While WSL1, plain docker for Windows and various Linux distributions and local Kubernetes environments may be supported, this setup was verified with:
- WSL2
- Docker Desktop WSL2 Engine
- Ubuntu 18.04 LTS on WSL2
- Nodeless local garden (see below)
The Gardener repository and all the above-mentioned tools (git, golang, kubectl, …) should be installed in your WSL2 distro, according to the distribution-specific Linux installation instructions.
Start Gardener Locally
Get the Sources
Clone the repository from GitHub into your $GOPATH
.
mkdir -p $(go env GOPATH)/src/github.com/gardener
cd $(go env GOPATH)/src/github.com/gardener
git clone git@github.com:gardener/gardener.git
cd gardener
Note: Gardener is using Go modules and cloning the repository into
$GOPATH
is not a hard requirement. However it is still recommended to clone into$GOPATH
becausek8s.io/code-generator
does not work yet outside of$GOPATH
- kubernetes/kubernetes#86753.
Start the Gardener
ℹ️ In the following guide, you have to define the configuration (CloudProfile
s, SecretBinding
s, Seed
s, etc.) manually for the infrastructure environment you want to develop against.
Additionally, you have to register the respective Gardener extensions manually.
If you are rather looking for a quick start guide to develop entirely locally on your machine (no real cloud provider or infrastructure involved), then you should rather follow this guide.
Start a Local Kubernetes Cluster
For the development of Gardener you need a Kubernetes API server on which you can register Gardener’s own Extension API Server as APIService
. This cluster doesn’t need any worker nodes to run pods, though, therefore, you can use the “nodeless Garden cluster setup” residing in hack/local-garden
. This will start all minimally required components of a Kubernetes cluster (etcd
, kube-apiserver
, kube-controller-manager
)
and an etcd
Instance for the gardener-apiserver
as Docker containers. This is the easiest way to get your
Gardener development setup up and running.
Using the nodeless cluster setup
Use the provided Makefile rules to start your local Garden:
make local-garden-up
[...]
Starting gardener-dev kube-etcd cluster..!
Starting gardener-dev kube-apiserver..!
Starting gardener-dev kube-controller-manager..!
Starting gardener-dev gardener-etcd cluster..!
namespace/garden created
clusterrole.rbac.authorization.k8s.io/gardener.cloud:admin created
clusterrolebinding.rbac.authorization.k8s.io/front-proxy-client created
[...]
ℹ️ [Optional] If you want to develop the SeedAuthorization
feature then you have to run make ACTIVATE_SEEDAUTHORIZER=true local-garden-up
. However, please note that this forces you to start the gardener-admission-controller
via make start-admission-controller
.
To tear down the local Garden cluster and remove the Docker containers, simply run:
make local-garden-down
Alternative: Using a local Kubernetes cluster
Instead of starting a Kubernetes API server and etcd as docker containers, you can also opt for running a local Kubernetes cluster, provided by e.g. minikube, kind or docker desktop.
Note: Gardener requires self-contained kubeconfig files because of a security issue. You can configure your minikube to create self-contained kubeconfig files via:
minikube config set embed-certs true
or when starting the local cluster
minikube start --embed-certs
Alternative: Using a remote Kubernetes cluster
For some testing scenarios, you may want to use a remote cluster instead of a local one as your Garden cluster.
To do this, you can use the “remote Garden cluster setup” residing in hack/remote-garden
. This will start an etcd
instance for the gardener-apiserver
as a Docker container, and open tunnels for accessing local gardener components from the remote cluster.
To avoid mistakes, the remote cluster must have a garden
namespace labeled with gardener.cloud/purpose=remote-garden
.
You must create the garden
namespace and label it manually before running make remote-garden-up
as described below.
Use the provided Makefile
rules to bootstrap your remote Garden:
export KUBECONFIG=<path to kubeconfig>
make remote-garden-up
[...]
# Start gardener etcd used to store gardener resources (e.g., seeds, shoots)
Starting gardener-dev-remote gardener-etcd cluster!
[...]
# Open tunnels for accessing local gardener components from the remote cluster
[...]
To close the tunnels and remove the locally-running Docker containers, run:
make remote-garden-down
ℹ️ [Optional] If you want to use the remote Garden cluster setup with the SeedAuthorization
feature, you have to adapt the kube-apiserver
process of your remote Garden cluster. To do this, perform the following steps after running make remote-garden-up
:
Create an authorization webhook configuration file using the IP of the
garden/quic-server
pod running in your remote Garden cluster and port 10444 that tunnels to your locally runninggardener-admission-controller
process.apiVersion: v1 kind: Config current-context: seedauthorizer clusters: - name: gardener-admission-controller cluster: insecure-skip-tls-verify: true server: https://<quic-server-pod-ip>:10444/webhooks/auth/seed users: - name: kube-apiserver user: {} contexts: - name: seedauthorizer context: cluster: gardener-admission-controller user: kube-apiserver
Change or add the following command line parameters to your
kube-apiserver
process:--authorization-mode=<...>,Webhook
--authorization-webhook-config-file=<path to config file>
--authorization-webhook-cache-authorized-ttl=0
--authorization-webhook-cache-unauthorized-ttl=0
Delete the cluster role and rolebinding
gardener.cloud:system:seeds
from your remote Garden cluster.
If your remote Garden cluster is a Gardener shoot, and you can access the seed on which this shoot is scheduled, you can automate the above steps by running the enable-seed-authorizer
script and passing the kubeconfig of the seed cluster and the shoot namespace as parameters:
hack/local-development/remote-garden/enable-seed-authorizer <seed kubeconfig> <namespace>
Note: This script is not working anymore, as the
ReversedVPN
feature can’t be disabled. The annotationalpha.featuregates.shoot.gardener.cloud/reversed-vpn
onShoot
s is no longer respected.
To prevent Gardener from reconciling the shoot and overwriting your changes, add the annotation shoot.gardener.cloud/ignore: 'true'
to the remote Garden shoot. Note that this annotation takes effect only if it is enabled via the constollers.shoot.respectSyncPeriodOverwrite: true
option in the gardenlet
configuration.
To disable the seed authorizer again, run the same script with -d
as a third parameter:
hack/local-development/remote-garden/enable-seed-authorizer <seed kubeconfig> <namespace> -d
If the seed authorizer is enabled, you also have to start the gardener-admission-controller
via make start-admission-controller
.
⚠️ In the remote garden setup all Gardener components run with administrative permissions, i.e., there is no fine-grained access control via RBAC (as opposed to productive installations of Gardener).
Prepare the Gardener
Now, that you have started your local cluster, we can go ahead and register the Gardener API Server.
Just point your KUBECONFIG
environment variable to the cluster you created in the previous step and run:
make dev-setup
[...]
namespace/garden created
namespace/garden-dev created
deployment.apps/etcd created
service/etcd created
service/gardener-apiserver created
service/gardener-admission-controller created
endpoints/gardener-apiserver created
endpoints/gardener-admission-controller created
apiservice.apiregistration.k8s.io/v1alpha1.core.gardener.cloud created
apiservice.apiregistration.k8s.io/v1beta1.core.gardener.cloud created
apiservice.apiregistration.k8s.io/v1alpha1.seedmanagement.gardener.cloud created
apiservice.apiregistration.k8s.io/v1alpha1.settings.gardener.cloud created
ℹ️ [Optional] If you want to enable logging, in the gardenlet configuration add:
logging:
enabled: true
The Gardener exposes the API servers of Shoot clusters via Kubernetes services of type LoadBalancer
.
In order to establish stable endpoints (robust against changes of the load balancer address), it creates DNS records pointing to these load balancer addresses. They are used internally and by all cluster components to communicate.
You need to have control over a domain (or subdomain) for which these records will be created.
Please provide an internal domain secret (see this for an example) which contains credentials with the proper privileges. Further information can be found in Gardener Configuration and Usage.
kubectl apply -f example/10-secret-internal-domain-unmanaged.yaml
secret/internal-domain-unmanaged created
Run the Gardener
Next, run the Gardener API Server, the Gardener Controller Manager (optionally), the Gardener Scheduler (optionally), and the gardenlet in different terminal windows/panes using rules in the Makefile
.
make start-apiserver
[...]
I0306 15:23:51.044421 74536 plugins.go:84] Registered admission plugin "ResourceReferenceManager"
I0306 15:23:51.044523 74536 plugins.go:84] Registered admission plugin "DeletionConfirmation"
[...]
I0306 15:23:51.626836 74536 secure_serving.go:116] Serving securely on [::]:8443
[...]
(Optional) Now you are ready to launch the Gardener Controller Manager.
make start-controller-manager
time="2019-03-06T15:24:17+02:00" level=info msg="Starting Gardener controller manager..."
time="2019-03-06T15:24:17+02:00" level=info msg="Feature Gates: "
time="2019-03-06T15:24:17+02:00" level=info msg="Starting HTTP server on 0.0.0.0:2718"
time="2019-03-06T15:24:17+02:00" level=info msg="Acquired leadership, starting controllers."
time="2019-03-06T15:24:18+02:00" level=info msg="Starting HTTPS server on 0.0.0.0:2719"
time="2019-03-06T15:24:18+02:00" level=info msg="Found internal domain secret internal-domain-unmanaged for domain nip.io."
time="2019-03-06T15:24:18+02:00" level=info msg="Successfully bootstrapped the Garden cluster."
time="2019-03-06T15:24:18+02:00" level=info msg="Gardener controller manager (version 1.0.0-dev) initialized."
time="2019-03-06T15:24:18+02:00" level=info msg="ControllerRegistration controller initialized."
time="2019-03-06T15:24:18+02:00" level=info msg="SecretBinding controller initialized."
time="2019-03-06T15:24:18+02:00" level=info msg="Project controller initialized."
time="2019-03-06T15:24:18+02:00" level=info msg="Quota controller initialized."
time="2019-03-06T15:24:18+02:00" level=info msg="CloudProfile controller initialized."
[...]
(Optional) Now you are ready to launch the Gardener Scheduler.
make start-scheduler
time="2019-05-02T16:31:50+02:00" level=info msg="Starting Gardener scheduler ..."
time="2019-05-02T16:31:50+02:00" level=info msg="Starting HTTP server on 0.0.0.0:10251"
time="2019-05-02T16:31:50+02:00" level=info msg="Acquired leadership, starting scheduler."
time="2019-05-02T16:31:50+02:00" level=info msg="Gardener scheduler initialized (with Strategy: SameRegion)"
time="2019-05-02T16:31:50+02:00" level=info msg="Scheduler controller initialized."
[...]
The Gardener should now be ready to operate on Shoot resources. You can use
kubectl get shoots
No resources found.
to operate against your local running Gardener API Server.
Note: It may take several seconds until the Gardener API server has been started and is available.
No resources found
is the expected result of our initial development setup.
Create a Shoot
The steps below describe the general process of creating a Shoot. Have in mind that the steps do not provide full example manifests. The reader needs to check the provider documentation and adapt the manifests accordingly.
1. Copy the Example Manifests
The next steps require modifications of the example manifests. These modifications are part of local setup and should not be git push
-ed. To do not interfere with git, let’s copy the example manifests to dev/
which is ignored by git.
cp example/*.yaml dev/
2. Create a Project
Every Shoot is associated with a Project. Check the corresponding example manifests dev/00-namespace-garden-dev.yaml
and dev/05-project-dev.yaml
. Adapt them and create them.
kubectl apply -f dev/00-namespace-garden-dev.yaml
kubectl apply -f dev/05-project-dev.yaml
Make sure that the Project is successfully reconciled:
$ kubectl get project dev
NAME NAMESPACE STATUS OWNER CREATOR AGE
dev garden-dev Ready john.doe@example.com kubernetes-admin 6s
3. Create a CloudProfile
The CloudProfile
resource is provider specific and describes the underlying cloud provider (available machine types, regions, machine images, etc.). Check the corresponding example manifest dev/30-cloudprofile.yaml
. Check also the documentation and example manifests of the provider extension. Adapt dev/30-cloudprofile.yaml
and apply it.
kubectl apply -f dev/30-cloudprofile.yaml
4. Install Necessary Gardener Extensions
The Known Extension Implementations section contains a list of available extension implementations. You need to create a ControllerRegistration and ControllerDeployment for:
- at least one infrastructure provider
- a DNS provider (if the DNS for the Seed is not disabled)
- at least one operating system extension
- at least one network plugin extension
As a convention, the example ControllerRegistration manifest (containing also the necessary ControllerDeployment) for an extension is located under example/controller-registration.yaml
in the corresponding repository (for example for AWS the ControllerRegistration can be found here). An example creation for provider-aws (make sure to replace <version>
with the newest released version tag):
kubectl apply -f https://raw.githubusercontent.com/gardener/gardener-extension-provider-aws/<version>/example/controller-registration.yaml
Instead of updating extensions manually you can use Gardener Extensions Manager to install and update extension controllers. This is especially useful if you want to keep and maintain your development setup for a longer time. Also, please refer to Registering Extension Controllers for further information about how extensions are registered in case you want to use other versions than the latest releases.
5. Register a Seed
Shoot controlplanes run in seed clusters, so we need to create our first Seed now.
Check the corresponding example manifest dev/40-secret-seed.yaml
and dev/50-seed.yaml
. Update dev/40-secret-seed.yaml
with base64 encoded kubeconfig of the cluster that will be used as Seed (the scope of the permissions should be identical to the kubeconfig that the gardenlet creates during bootstrapping - for now, cluster-admin
privileges are recommended).
kubectl apply -f dev/40-secret-seed.yaml
Adapt dev/50-seed.yaml
- adjust .spec.secretRef
to refer the newly created Secret, adjust .spec.provider
with the Seed cluster provider and revise the other fields.
kubectl apply -f dev/50-seed.yaml
6. Start the gardenlet
Once the Seed is created, start the gardenlet to reconcile it. The make start-gardenlet
command will automatically configure the local gardenlet process to use the Seed and its kubeconfig. If you have multiple Seeds, you have to specify which to use by setting the SEED_NAME
environment variable like in make start-gardenlet SEED_NAME=my-first-seed
.
make start-gardenlet
time="2019-11-06T15:24:17+02:00" level=info msg="Starting Gardenlet..."
time="2019-11-06T15:24:17+02:00" level=info msg="Feature Gates: HVPA=true, Logging=true"
time="2019-11-06T15:24:17+02:00" level=info msg="Acquired leadership, starting controllers."
time="2019-11-06T15:24:18+02:00" level=info msg="Found internal domain secret internal-domain-unmanaged for domain nip.io."
time="2019-11-06T15:24:18+02:00" level=info msg="Gardenlet (version 1.0.0-dev) initialized."
time="2019-11-06T15:24:18+02:00" level=info msg="ControllerInstallation controller initialized."
time="2019-11-06T15:24:18+02:00" level=info msg="Shoot controller initialized."
time="2019-11-06T15:24:18+02:00" level=info msg="Seed controller initialized."
[...]
The gardenlet will now reconcile the Seed. Check the progess from time to time until it’s Ready
:
kubectl get seed
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
seed-aws Ready aws eu-west-1 4m v1.61.0-dev v1.24.8
7. Create a Shoot
A Shoot requires a SecretBinding. The SecretBinding refers to a Secret that contains the cloud provider credentials. The Secret data keys are provider specific and you need to check the documentation of the provider to find out which data keys are expected (for example for AWS the related documentation can be found at Provider Secret Data). Adapt dev/70-secret-provider.yaml
and dev/80-secretbinding.yaml
and apply them.
kubectl apply -f dev/70-secret-provider.yaml
kubectl apply -f dev/80-secretbinding.yaml
After the SecretBinding creation, you are ready to proceed with the Shoot creation. You need to check the documentation of the provider to find out the expected configuration (for example for AWS the related documentation and example Shoot manifest can be found at Using the AWS provider extension with Gardener as end-user). Adapt dev/90-shoot.yaml
and apply it.
To make sure that a specific Seed cluster will be chosen or to skip the scheduling (the sheduling requires Gardener Scheduler to be running), specify the .spec.seedName
field (see here).
kubectl apply -f dev/90-shoot.yaml
Watch the progress of the operation and make sure that the Shoot will be successfully created.
watch kubectl get shoot --all-namespaces
1.4.8 - Log Parsers
How to Create Log Parser for Container into fluent-bit
If our log message is parsed correctly, it has to be showed in Grafana like this:
{"log":"OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io","pid":"1","severity":"INFO","source":"controller.go:107"}
Otherwise it will looks like this:
{
"log":"{
\"level\":\"info\",\"ts\":\"2020-06-01T11:23:26.679Z\",\"logger\":\"gardener-resource-manager.health-reconciler\",\"msg\":\"Finished ManagedResource health checks\",\"object\":\"garden/provider-aws-dsm9r\"
}\n"
}
}
Create a Custom Parser
First of all, we need to know how the log for the specific container looks like (for example, lets take a log from the
alertmanager
:level=info ts=2019-01-28T12:33:49.362015626Z caller=main.go:175 build_context="(go=go1.11.2, user=root@4ecc17c53d26, date=20181109-15:40:48)
)We can see that this log contains 4 subfields(severity=info, timestamp=2019-01-28T12:33:49.362015626Z, source=main.go:175 and the actual message). So we have to write a regex which matches this log in 4 groups(We can use https://regex101.com/ like helping tool). So, for this purpose our regex looks like this:
^level=(?<severity>\w+)\s+ts=(?<time>\d{4}-\d{2}-\d{2}[Tt].*[zZ])\s+caller=(?<source>[^\s]*+)\s+(?<log>.*)
- Now we have to create correct time format for the timestamp (We can use this site for this purpose: http://ruby-doc.org/stdlib-2.4.1/libdoc/time/rdoc/Time.html#method-c-strptime). So our timestamp matches correctly the following format:
%Y-%m-%dT%H:%M:%S.%L
- It’s time to apply our new regex into fluent-bit configuration. Go to
fluent-bit-configmap.yaml
and create new filter using the following template:
[FILTER]
Name parser
Match kubernetes.<< pod-name >>*<< container-name >>*
Key_Name log
Parser << parser-name >>
Reserve_Data True
EXAMPLE
[FILTER]
Name parser
Match kubernetes.alertmanager*alertmanager*
Key_Name log
Parser alermanagerParser
Reserve_Data True
- Now lets check if there already exists parser with such a regex and time format that we need. If it doesn’t, create one:
[PARSER]
Name << parser-name >>
Format regex
Regex << regex >>
Time_Key time
Time_Format << time-format >>
EXAMPLE
[PARSER]
Name alermanagerParser
Format regex
Regex ^level=(?<severity>\w+)\s+ts=(?<time>\d{4}-\d{2}-\d{2}[Tt].*[zZ])\s+caller=(?<source>[^\s]*+)\s+(?<log>.*)
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Follow your development setup to validate that the parsers are working correctly.
1.4.9 - Logging
Logging in Gardener Components
This document aims at providing a general developer guideline on different aspects of logging practices and conventions used in the Gardener codebase. It contains mostly Gardener-specific points, and references other existing and commonly accepted logging guidelines for general advice. Developers and reviewers should consult this guide when writing, refactoring, and reviewing Gardener code. If parts are unclear or new learnings arise, this guide should be adapted accordingly.
Logging Libraries / Implementations
Historically, Gardener components have been using logrus.
There is a global logrus logger (logger.Logger
) that is initialized by components on startup and used across the codebase.
In most places, it is used as a printf
-style logger and only in some instances we make use of logrus’ structured logging functionality.
In the process of migrating our components to native controller-runtime components (see gardener/gardener#4251), we also want to make use of controller-runtime’s built-in mechanisms for streamlined logging. controller-runtime uses logr, a simple structured logging interface, for library-internal logging and logging in controllers.
logr itself is only an interface and doesn’t provide an implementation out of the box. Instead, it needs to be backed by a logging implementation like zapr. Code that uses the logr interface is thereby not tied to a specific logging implementation and makes the implementation easily exchangeable. controller-runtime already provides a set of helpers for constructing zapr loggers, i.e., logr loggers backed by zap, which is a popular logging library in the go community. Hence, we are migrating our component logging from logrus to logr (backed by zap) as part of gardener/gardener#4251.
⚠️
logger.Logger
(logrus logger) is deprecated in Gardener and shall not be used in new code – use logr loggers when writing new code! (also see Migration from logrus to logr)ℹ️ Don’t use zap loggers directly, always use the logr interface in order to avoid tight coupling to a specific logging implementation.
gardener-apiserver differs from the other components as it is based on the apiserver library and therefore uses klog – just like kube-apiserver. As gardener-apiserver writes (almost) no logs in our coding (outside the apiserver library), there is currently no plan for switching the logging implementation. Hence, the following sections focus on logging in the controller and admission components only.
logcheck
Tool
To ensure a smooth migration to logr and make logging in Gardener components more consistent, the logcheck
tool was added.
It enforces (parts of) this guideline and detects programmer-level errors early on in order to prevent bugs.
Please check out the tool’s documentation for a detailed description.
Structured Logging
Similar to efforts in the Kubernetes project, we want to migrate our component logs to structured logging. As motivated above, we will use the logr interface instead of klog though.
You can read more about the motivation behind structured logging in logr’s background and FAQ (also see this blog post by Dave Cheney). Also, make sure to check out controller-runtime’s logging guideline with specifics for projects using the library. The following sections will focus on the most important takeaways from those guidelines and give general instructions on how to apply them to Gardener and its controller-runtime components.
Note: Some parts in this guideline differ slightly from controller-runtime’s document.
TL;DR of Structured Logging
❌ Stop using printf
-style logging:
var logger *logrus.Logger
logger.Infof("Scaling deployment %s/%s to %d replicas", deployment.Namespace, deployment.Name, replicaCount)
✅ Instead, write static log messages and enrich them with additional structured information in form of key-value pairs:
var logger logr.Logger
logger.Info("Scaling deployment", "deployment", client.ObjectKeyFromObject(deployment), "replicas", replicaCount)
Log Configuration
Gardener components can be configured to either log in json
(default) or text
format:
json
format is supposed to be used in production, while text
format might be nicer for development.
# json
{"level":"info","ts":"2021-12-16T08:32:21.059+0100","msg":"Hello botanist","garden":"eden"}
# text
2021-12-16T08:32:21.059+0100 INFO Hello botanist {"garden": "eden"}
Components can be set to one of the following log levels (with increasing verbosity): error
, info
(default), debug
.
Log Levels
logr uses V-levels (numbered log levels), higher V-level means higher verbosity.
V-levels are relative (in contrast to klog
’s absolute V-levels), i.e., V(1)
creates a logger, that is one level more verbose than its parent logger.
In Gardener components, the mentioned log levels in the component config (error
, info
, debug
) map to the zap levels with the same names (see here).
Hence, our loggers follow the same mapping from numerical logr levels to named zap levels like described in zapr, i.e.:
- component config specifies
debug
➡️ bothV(0)
andV(1)
are enabled - component config specifies
info
➡️V(0)
is enabled,V(1)
will not be shown - component config specifies
error
➡️ neitherV(0)
norV(1)
will be shown Error()
logs will always be shown
This mapping applies to the components’ root loggers (the ones that are not “derived” from any other logger; constructed on component startup).
If you derive a new logger with e.g. V(1)
, the mapping will shift by one. For example, V(0)
will then log at zap’s debug
level.
There is no warning
level (see Dave Cheney’s post).
If there is an error condition (e.g., unexpected error received from a called function), the error should either be handled or logged at error
if it is neither handled nor returned.
If you have an error
value at hand that doesn’t represent an actual error condition, but you still want to log it as an informational message, log it at info
level with key err
.
We might consider to make use of a broader range of log levels in the future when introducing more logs and common command line flags for our components (comparable to --v
of Kubernetes components).
For now, we stick to the mentioned two log levels like controller-runtime: info (V(0)
) and debug (V(1)
).
Logging in Controllers
Named Loggers
Controllers should use named loggers that include their name, e.g.:
controllerLogger := rootLogger.WithName("controller").WithName("shoot")
controllerLogger.Info("Deploying kube-apiserver")
results in
2021-12-16T09:27:56.550+0100 INFO controller.shoot Deploying kube-apiserver
Logger names are hierarchical. You can make use of it, where controllers are composed of multiple “subcontrollers”, e.g., controller.shoot.hibernation
or controller.shoot.maintenance
.
Using the global logger logf.Log
directly is discouraged and should be rather exceptional because it makes correlating logs with code harder.
Preferably, all parts of the code should use some named logger.
Reconciler Loggers
In your Reconcile
function, retrieve a logger from the given context.Context
.
It inherits from the controller’s logger (i.e., is already named) and is preconfigured with name
and namespace
values for the reconciliation request:
func (r *reconciler) Reconcile(ctx context.Context, request reconcile.Request) (reconcile.Result, error) {
log := logf.FromContext(ctx)
log.Info("Reconciling Shoot")
// ...
return reconcile.Result{}, nil
}
results in
2021-12-16T09:35:59.099+0100 INFO controller.shoot Reconciling Shoot {"name": "sunflower", "namespace": "garden-greenhouse"}
The logger is injected by controller-runtime’s Controller
implementation. The logger returned by logf.FromContext
is never nil
. If the context doesn’t carry a logger, it falls back to the global logger (logf.Log
), which might discard logs if not configured, but is also never nil
.
⚠️ Make sure that you don’t overwrite the
name
ornamespace
value keys for such loggers, otherwise you will lose information about the reconciled object.
The controller implementation (controller-runtime) itself takes care of logging the error returned by reconcilers. Hence, don’t log an error that you are returning. Generally, functions should not return an error, if they already logged it, because that means the error is already handled and not an error anymore. See Dave Cheney’s post for more on this.
Messages
- Log messages should be static. Don’t put variable content in there, i.e., no
fmt.Sprintf
or string concatenation (+
). Use key-value pairs instead. - Log messages should be capitalized. Note: This contrasts with error messages, that should not be capitalized. However, both should not end with a punctuation mark.
Keys and Values
Use
WithValues
instead of repeatedly adding key-value pairs for multiple log statements.WithValues
creates a new logger from the parent, that carries the given key-value pairs. E.g., use it when acting on one object in multiple steps and logging something for each step:log := parentLog.WithValues("infrastructure", client.ObjectKeyFromObject(infrastrucutre)) // ... log.Info("Creating Infrastructure") // ... log.Info("Waiting for Infrastructure to be reconciled") // ...
Note:
WithValues
bypasses controller-runtime’s special zap encoder that nicely encodesObjectKey
/NamespacedName
andruntime.Object
values, see kubernetes-sigs/controller-runtime#1290. Thus, the end result might look different depending on the value and itsStringer
implementation.
Use lowerCamelCase for keys. Don’t put spaces in keys, as it will make log processing with simple tools like
jq
harder.Keys should be constant, human-readable, consistent across the codebase and naturally match parts of the log message, see logr guideline.
When logging object keys (name and namespace), use the object’s type as the log key and a
client.ObjectKey
/types.NamespacedName
value as value, e.g.:var deployment *appsv1.Deployment log.Info("Creating Deployment", "deployment", client.ObjectKeyFromObject(deployment))
which results in
{"level":"info","ts":"2021-12-16T08:32:21.059+0100","msg":"Creating Deployment","deployment":{"name": "bar", "namespace": "foo"}}
Earlier, we often used
kutil.ObjectName()
for logging object keys, which encodes them into a flat string likefoo/bar
. However, this flat string cannot be processed so easily by logging stacks (orjq
) like a structured log. Hence, the use ofkutil.ObjectName()
for logging object keys is discouraged. Existing usages should be refactored to useclient.ObjectKeyFromObject()
instead.There are cases where you don’t have the full object key or the object itself at hand, e.g., if an object references another object (in the same namespace) by name (think
secretRef
or similar). In such a cases, either construct the full object key including the implied namespace or log the object name under a key ending inName
, e.g.:var ( // object to reconcile shoot *gardencorev1beta1.Shoot // retrieved via logf.FromContext, preconfigured by controller with namespace and name of reconciliation request log logr.Logger ) // option a: full object key, manually constructed log.Info("Shoot uses SecretBinding", "secretBinding", client.ObjectKey{Namespace: shoot.Namespace, Name: shoot.Spec.SecretBindingName}) // option b: only name under respective *Name log key log.Info("Shoot uses SecretBinding", "secretBindingName", shoot.Spec.SecretBindingName)
Both options result in well-structured logs, that are easy to interpret and process:
{"level":"info","ts":"2022-01-18T18:00:56.672+0100","msg":"Shoot uses SecretBinding","name":"my-shoot","namespace":"garden-project","secretBinding":{"namespace":"garden-project","name":"aws"}} {"level":"info","ts":"2022-01-18T18:00:56.673+0100","msg":"Shoot uses SecretBinding","name":"my-shoot","namespace":"garden-project","secretBindingName":"aws"}
When handling generic
client.Object
values (e.g. in helper funcs), useobject
as key.When adding timestamps to key-value pairs, use
time.Time
values. By this, they will be encoded in the same format as the log entry’s timestamp.
Don’t usemetav1.Time
values, as they will be encoded in a different format by theirStringer
implementation. Pass<someTimestamp>.Time
to loggers in case you have ametav1.Time
value at hand.Same applies to durations. Use
time.Duration
values instead of*metav1.Duration
. Durations can be handled specially by zap just like timestamps.Event recorders not only create
Event
objects but also log them. However, both Gardener’s manually instantiated event recorders and the ones that controller-runtime provides log todebug
level and use generic formats, that are not very easy to interpret or process (no structured logs). Hence, don’t use event recorders as replacements for well-structured logs. If a controller records an event for a completed action or important information, it should probably log it as well, e.g.:log.Info("Creating ManagedSeed", "replica", r.GetObjectKey()) a.recorder.Eventf(managedSeedSet, corev1.EventTypeNormal, EventCreatingManagedSeed, "Creating ManagedSeed %s", r.GetFullName())
Logging in Test Code
If the tested production code requires a logger, you can pass
logr.Discard()
orlogf.NullLogger{}
in your test, which simply discards all logs.logf.Log
is safe to use in tests and will not cause a nil pointer deref, even if it’s not initialized vialogf.SetLogger
. It is initially set to aNullLogger
by default, which means all logs are discarded, unlesslogf.SetLogger
is called in the first 30 seconds of execution.Pass
zap.WriteTo(GinkgoWriter)
in tests where you want to see the logs on test failure but not on success, for example:logf.SetLogger(logger.MustNewZapLogger(logger.DebugLevel, logger.FormatJSON, zap.WriteTo(GinkgoWriter))) log := logf.Log.WithName("test")
1.4.10 - Monitoring Stack
Extending the Monitoring Stack
This document provides instructions to extend the Shoot cluster monitoring stack by integrating new scrape targets, alerts and dashboards.
Please ensure that you have understood the basic principles of Prometheus and its ecosystem before you continue.
‼️ The purpose of the monitoring stack is to observe the behaviour of the control plane and the system components deployed by Gardener onto the worker nodes. Monitoring of custom workloads running in the cluster is out of scope.
Overview
Each Shoot cluster comes with its own monitoring stack. The following components are deployed into the seed and shoot:
- Seed
- Prometheus
- Grafana
- blackbox-exporter
- kube-state-metrics (Seed metrics)
- kube-state-metrics (Shoot metrics)
- Alertmanager (Optional)
- Shoot
In each Seed cluster there is a Prometheus in the garden
namespace responsible for collecting metrics from the Seed kubelets and cAdvisors. These metrics are provided to each Shoot Prometheus via federation.
The alerts for all Shoot clusters hosted on a Seed are routed to a central Alertmanger running in the garden
namespace of the Seed. The purpose of this central alertmanager is to forward all important alerts to the operators of the Gardener setup.
The Alertmanager in the Shoot namespace on the Seed is only responsible for forwarding alerts from its Shoot cluster to a cluster owner/cluster alert receiver via email. The Alertmanager is optional and the conditions for a deployment are already described in Alerting.
Adding New Monitoring Targets
After exploring the metrics which your component provides or adding new metrics, you should be aware which metrics are required to write the needed alerts and dashboards.
Prometheus prefers a pull based metrics collection approach and therefore the targets to observe need to be defined upfront. The targets are defined in charts/seed-monitoring/charts/prometheus/templates/config.yaml
.
New scrape jobs can be added in the section scrape_configs
. Detailed information how to configure scrape jobs and how to use the kubernetes service discovery are available in the Prometheus documentation.
The job_name
of a scrape job should be the name of the component e.g. kube-apiserver
or vpn
. The collection interval should be the default of 30s
. You do not need to specify this in the configuration.
Please do not ingest all metrics which are provided by a component. Rather, collect only those metrics which are needed to define the alerts and dashboards (i.e. whitelist). This can be achieved by adding the following metric_relabel_configs
statement to your scrape jobs (replace exampleComponent
with component name).
- job_name: example-component
...
metric_relabel_configs:
{{ include "prometheus.keep-metrics.metric-relabel-config" .Values.allowedMetrics.exampleComponent | indent 6 }}
The whitelist for the metrics of your job can be maintained in charts/seed-monitoring/charts/prometheus/values.yaml
in section allowedMetrics.exampleComponent
(replace exampleComponent
with component name). Check the following example:
allowedMetrics:
...
exampleComponent:
* metrics_name_1
* metrics_name_2
...
Adding Alerts
The alert definitons are located in charts/seed-monitoring/charts/prometheus/rules
. There are two approaches for adding new alerts.
- Adding additional alerts for a component which already has a set of alerts. In this case you have to extend the existing rule file for the component.
- Adding alerts for a new component. In this case a new rule file with name scheme
example-component.rules.yaml
needs to be added. - Add the new alert to
alertInhibitionGraph.dot
, add any required inhibition flows and render the new graph. To render the graph, run:
dot -Tpng ./content/alertInhibitionGraph.dot -o ./content/alertInhibitionGraph.png
- Create a test for the new alert. See
Alert Tests
.
Example alert:
groups:
* name: example.rules
rules:
* alert: ExampleAlert
expr: absent(up{job="exampleJob"} == 1)
for: 20m
labels:
service: example
severity: critical # How severe is the alert? (blocker|critical|info|warning)
type: shoot # For which topology is the alert relevant? (seed|shoot)
visibility: all # Who should receive the alerts? (all|operator|owner)
annotations:
description: A longer description of the example alert that should also explain the impact of the alert.
summary: Short summary of an example alert.
If the deployment of component is optional then the alert definitions needs to be added to charts/seed-monitoring/charts/prometheus/optional-rules
instead. Furthermore the alerts for component need to be activatable in charts/seed-monitoring/charts/prometheus/values.yaml
via rules.optional.example-component.enabled
. The default should be true
.
Basic instruction how to define alert rules can be found in the Prometheus documentation.
Routing Tree
The Alertmanager is grouping incoming alerts based on labels into buckets. Each bucket has its own configuration like alert receivers, initial delaying duration or resending frequency, etc. You can find more information about Alertmanager routing in the Prometheus/Alertmanager documentation. The routing trees for the Alertmanagers deployed by Gardener are depicted below.
Central Seed Alertmanager
∟ main route (all alerts for all shoots on the seed will enter)
∟ group by project and shoot name
∟ group by visibility "all" and "operator"
∟ group by severity "blocker", "critical", and "info" → route to Garden operators
∟ group by severity "warning" (dropped)
∟ group by visibility "owner" (dropped)
Shoot Alertmanager
∟ main route (only alerts for one Shoot will enter)
∟ group by visibility "all" and "owner"
∟ group by severity "blocker", "critical", and "info" → route to cluster alert receiver
∟ group by severity "warning" (dropped, will change soon → route to cluster alert receiver)
∟ group by visibility "operator" (dropped)
Alert Inhibition
All alerts related to components running on the Shoot workers are inhibited in case of an issue with the vpn connection, because those components can’t be scraped anymore and Prometheus will fire alerts in consequence. The components running on the workers are probably healthy and the alerts are presumably false positives. The inhibition flow is shown in the figure below. If you add a new alert, make sure to add it to the diagram.
Alert Attributes
Each alert rule definition has to contain the following annotations:
- summary: A short description of the issue.
- description: A detailed explanation of the issue with hints to the possible root causes and the impact assessment of the issue.
In addtion, each alert must contain the following labels:
- type
shoot
: Components running on the Shoot worker nodes in thekube-system
namespace.seed
: Components running on the Seed in the Shoot namespace as part of/next to the control plane.
- service
- Name of the component (in lowercase) e.g.
kube-apiserver
,alertmanager
orvpn
.
- Name of the component (in lowercase) e.g.
- severity
blocker
: All issues which make the cluster entirely unusable, e.g.KubeAPIServerDown
orKubeSchedulerDown
critical
: All issues which affect single functionalities/components but do not affect the cluster in its core functionality e.g.VPNDown
orKubeletDown
.info
: All issues that do not affect the cluster or its core functionality, but if this component is down we cannot determine if a blocker alert is firing. (i.e. A component with an info level severity is a dependency for a component with a blocker severity)warning
: No current existing issue, rather a hint for situations which could lead to real issue in the close future e.g.HighLatencyApiServerToWorkers
orApiServerResponseSlow
.
Alert Tests
To test the Prometheus alerts:
make test-prometheus
If you want to add alert tests:
Create a new file in
rules-tests
in the form<alert-group-name>.rules.test.yaml
or if the alerts are for an existing component with existing tests, simply add the tests to the appropriate files.Make sure that newly added tests succeed. See above.
Adding Grafana Dashboards
The dashboard definition files are located in charts/seed-monitoring/charts/grafana/dashboards
. Every dashboard needs its own file.
If you are adding a new component dashboard please also update the overview dashboard by adding a chart for its current up/down status and with a drill down option to the component dashboard.
Dashboard Structure
The dashboards should be structured in the following way. The assignment of the component dashboards to the categories should be handled via dashboard tags.
- Kubernetes control plane components (Tag:
control-plane
)- All components which are part of the Kubernetes control plane e. g. Kube API Server, Kube Controller Manager, Kube Scheduler and Cloud Controller Manager
- ETCD + Backup/Restore
- Kubernetes Addon Manager
- Node/Machine components (Tag:
node/machine
)- All metrics which are related to the behaviour/control of the Kubernetes nodes and kubelets
- Machine-Controller-Manager + Cluster Autoscaler
- Networking components (Tag:
network
)- CoreDNS, KubeProxy, Calico, VPN, Nginx Ingress
- Addon components (Tag:
addon
)- Cert Broker
- Monitoring components (Tag:
monitoring
) - Logging components (Tag:
logging
)
Mandatory Charts for Component Dashboards
For each new component, its corresponding dashboard should contain the following charts in the first row, before adding custom charts for the component in the subsequent rows.
- Pod up/down status
up{job="example-component"}
- Pod/containers cpu utilization
- Pod/containers memorty consumption
- Pod/containers network i/o
That information is provided by the cAdvisor metrics. These metrics are already integrated. Please check the other dashboards for detailed information on how to query.
Chart Requirements
Each chart needs to contain:
- a meaningful name
- a detailed description (for non trivial charts)
- appropriate x/y axis descriptions
- appropriate scaling levels for the x/y axis
- proper units for the x/y axis
Dashboard Parameters
The following parameters should be added to all dashboards to ensure a homogeneous experience across all dashboards.
Dashboards have to:
- contain a title which refers to the component name(s)
- contain a timezone statement which should be the browser time
- contain tags which express where the component is running (
seed
orshoot
) and to which category the component belong (see dashboard structure) - contain a version statement with a value of 1
- be immutable
Example dashboard configuration:
{
"title": "example-component",
"timezone": "utc",
"tags": [
"seed",
"control-plane"
],
"version": 1,
"editable": "false"
}
Furthermore, all dashboards should contain the following time options:
{
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"30s",
"1m",
"5m"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"10d"
]
}
}
1.4.11 - New Cloud Provider
Adding Cloud Providers
This document provides an overview of how to integrate a new cloud provider into Gardener. Each component that requires integration has a detailed description of how to integrate it and the steps required.
Cloud Components
Gardener is composed of 2 or more Kubernetes clusters:
- Shoot: These are the end-user clusters, the regular Kubernetes clusters you have seen. They provide places for your workloads to run.
- Seed: This is the “management” cluster. It manages the control planes of shoots by running them as native Kubernetes workloads.
These two clusters can run in the same cloud provider, but they do not need to. For example, you could run your Seed in AWS, while having one shoot in Azure, two in Google, two in Alicloud, and three in Equinix Metal.
The Seed cluster deploys and manages the Shoot clusters. Importantly, for this discussion, the etcd
data store backing each Shoot runs as workloads inside the Seed. Thus, to use the above example, the clusters in Azure, Google, Alicloud and Equinix Metal will have their worker nodes and master nodes running in those clouds, but the etcd
clusters backing them will run as separate deployments in the Seed Kubernetes cluster on AWS.
This distinction becomes important when preparing the integration to a new cloud provider.
Gardener Cloud Integration
Gardener and its related components integrate with cloud providers at the following key lifecycle elements:
- Create/destroy/get/list machines for the Shoot.
- Create/destroy/get/list infrastructure components for the Shoot, e.g. VPCs, subnets, routes, etc.
- Backup/restore etcd for the Seed via writing files to and reading them from object storage.
Thus, the integrations you need for your cloud provider depend on whether you want to deploy Shoot clusters to the provider, Seed or both.
- Shoot Only: machine lifecycle management, infrastructure
- Seed: etcd backup/restore
Gardener API
In addition to the requirements to integrate with the cloud provider, you also need to enable the core Gardener app to receive, validate, and process requests to use that cloud provider.
- Expose the cloud provider to the consumers of the Gardener API, so it can be told to use that cloud provider as an option.
- Validate that API as requests come in.
- Write cloud provider specific implementation (called “provider extension”).
Cloud Provider API Requirements
In order for a cloud provider to integrate with Gardener, the provider must have an API to perform machine lifecycle events, specifically:
- Create a machine
- Destroy a machine
- Get information about a machine and its state
- List machines
In addition, if the Seed is to run on the given provider, it also must have an API to save files to block storage and retrieve them, for etcd backup/restore.
The current integration with cloud providers is to add their API calls to Gardener and the Machine Controller Manager. As both Gardener and the Machine Controller Manager are written in go, the cloud provider should have a go SDK. However, if it has an API that is wrappable in go, e.g. a REST API, then you can use that to integrate.
The Gardener team is working on bringing cloud provider integrations out-of-tree, making them pluggable, which should simplify the process and make it possible to use other SDKs.
Summary
To add a new cloud provider, you need some or all of the following. Each repository contains instructions on how to extend it to a new cloud provider.
Type | Purpose | Location | Documentation |
---|---|---|---|
Seed or Shoot | Machine Lifecycle | machine-controller-manager | MCM new cloud provider |
Seed only | etcd backup/restore | etcd-backup-restore | In process |
All | Extension implementation | gardener | Extension controller |
1.4.12 - New Kubernetes Version
Adding Support For a New Kubernetes Version
This document describes the steps needed to perform in order to confidently add support for a new Kubernetes minor version.
⚠️ Typically, once a minor Kubernetes version
vX.Y
is supported by Gardener, then all patch versionsvX.Y.Z
are also automatically supported without any required action. This is because patch versions do not introduce any new feature or API changes, so there is nothing that needs to be adapted ingardener/gardener
code.
The Kubernetes community release a new minor version roughly every 4 months. Please refer to the official documentation about their release cycles for any additional information.
Shortly before a new release, an “umbrella” issue should be opened which is used to collect the required adaptations and to track the work items.
For example, #5102 can be used as a template for the issue description.
As you can see, the task of supporting a new Kubernetes version also includes the provider extensions maintained in the gardener
GitHub organization and is not restricted to gardener/gardener
only.
Generally, the work items can be split into two groups: The first group contains Kubernetes release-independent tasks, the second group contains tasks specific to the changes in the given Kubernetes release.
ℹ️ Upgrading the
k8s.io/*
andsigs.k8s.io/controller-runtime
Golang dependencies is typically tracked and worked on separately (see e.g. #4772 or #5282).
Deriving Release-Specific Tasks
Most new minor Kubernetes releases incorporate API changes, deprecations, or new features.
The community announces them via their change logs.
In order to derive the release-specific tasks, the respective change log for the new version vX.Y
has to be read and understood (for example, the changelog for v1.24
).
As already mentioned, typical changes to watch out for are:
- API version promotions or deprecations
- Feature gate promotions or deprecations
- CLI flag changes for Kubernetes components
- New default values in resources
- New available fields in resources
- New features potentially relevant for the Gardener system
- Changes of labels or annotations Gardener relies on
- …
Obviously, this requires a certain experience and understanding of the Gardener project so that all “relevant changes” can be identified.
While reading the change log, add the tasks (along with the respective PR in kubernetes/kubernetes
to the umbrella issue).
ℹ️ Some of the changes might be specific to certain cloud providers. Pay attention to those as well and add related tasks to the issue.
List Of Release-Independent Tasks
The following paragraphs describe recurring tasks that need to be performed for each new release.
Make Sure a New hyperkube
Image Is Released
The gardener/hyperkube
repository is used to release container images consisting of the kubectl
and kubelet
binaries.
There is a CI/CD job that runs periodically and releases a new hyperkube
image when there is a new Kubernetes release. Before proceeding with the next steps, make sure that a new hyperkube
image is released for the corresponding new Kubernetes minor version. Make sure that container image is present in GCR.
Adapting Gardener
- Allow instantiation of a Kubernetes client for the new minor version and update the
README.md
:- See this example commit.
- Maintain the Kubernetes feature gates used for validation of
Shoot
resources:- The feature gates are maintained in this file.
- To maintain this list for new Kubernetes versions, run
hack/compare-k8s-feature-gates.sh <old-version> <new-version>
(e.g.hack/compare-k8s-feature-gates.sh v1.22 v1.23
). - It will present 3 lists of feature gates: those added and those removed in
<new-version>
compared to<old-version>
and feature gates that got locked to default in<new-version>
. - Add all added feature gates to the map with
<new-version>
asAddedInVersion
and noRemovedInVersion
. - For any removed feature gates, add
<new-version>
asRemovedInVersion
to the already existing feature gate in the map. - For feature gates locked to default, add
<new-version>
asLockedToDefaultInVersion
to the already existing feature gate in the map. - See this example commit.
- Maintain the Kubernetes
kube-apiserver
admission plugins used for validation ofShoot
resources:- The admission plugins are maintained in this file.
- To maintain this list for new Kubernetes versions, run
hack/compare-k8s-admission-plugins.sh <old-version> <new-version>
(e.g.hack/compare-k8s-admission-plugins.sh 1.24 1.25
). - It will present 2 lists of admission plugins: those added and those removed in
<new-version>
compared to<old-version>
. - Add all added admission plugins to the
admissionPluginsVersionRanges
map with<new-version>
asAddedInVersion
and noRemovedInVersion
. - For any removed admission plugins, add
<new-version>
asRemovedInVersion
to the already existing admission plugin in the map. - Flag any admission plugins that are required (plugins that must not be disabled in the
Shoot
spec) by setting theRequired
boolean variable to true for the admission plugin in the map. - Flag any admission plugins that are forbidden by setting the
Forbidden
boolean variable to true for the admission plugin in the map.
- Maintain the
ServiceAccount
names for the controllers part ofkube-controller-manager
:- The names are maintained in this file.
- To maintain this list for new Kubernetes versions, run
hack/compare-k8s-controllers.sh <old-version> <new-version>
(e.g.hack/compare-k8s-controllers.sh 1.22 1.23
). - It will present 2 lists of controllers: those added and those removed in
<new-version>
compared to<old-version>
. - Double check whether such
ServiceAccount
indeed appears in thekube-system
namespace when creating a cluster with<new-version>
. Note that it sometimes might be hidden behind a default-off feature gate. You can create a local cluster with the new version using the local provider. - If it appears, add all added controllers to the list based on the Kubernetes version (example).
- For any removed controllers, add them only to the Kubernetes version if it is low enough.
- Bump the used Kubernetes version for local
Shoot
and local e2e test.- See this example commit.
Filing the Pull Request
Work on all the tasks you have collected and validate them using the local provider. Execute the e2e tests and if everything looks good, then go ahead and file the PR (