그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그
8 minute read
Using the OpenStack provider extension with Gardener as end-user
core.gardener.cloud/v1beta1.Shoot resource declares a few fields that are meant to contain provider-specific configuration.
In this document we are describing how this configuration looks like for OpenStack and provide an example
Shoot manifest with minimal configuration that you can use to create an OpenStack cluster (modulo the landscape-specific information like cloud profile names, secret binding names, etc.).
Provider Secret Data
Every shoot cluster references a
SecretBinding which itself references a
Secret, and this
Secret contains the provider credentials of your OpenStack tenant.
Secret must look as follows:
apiVersion: v1 kind: Secret metadata: name: core-openstack namespace: garden-dev type: Opaque data: domainName: base64(domain-name) tenantName: base64(tenant-name) # either use username/password username: base64(user-name) password: base64(password) # or application credentials #applicationCredentialID: base64(app-credential-id) #applicationCredentialName: base64(app-credential-name) # optional #applicationCredentialSecret: base64(app-credential-secret)
Please look up https://docs.openstack.org/keystone/pike/admin/identity-concepts.html as well.
For authentication with username/password see Keystone username/password
Alternatively, for authentication with application credentials see Keystone Application Credentials.
⚠️ Depending on your API usage it can be problematic to reuse the same provider credentials for different Shoot clusters due to rate limits. Please consider spreading your Shoots over multiple credentials from different tenants if you are hitting those limits.
The infrastructure configuration mainly describes how the network layout looks like in order to create the shoot worker nodes in a later step, thus, prepares everything relevant to create VMs, load balancers, volumes, etc.
InfrastructureConfig for the OpenStack extension looks as follows:
apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1 kind: InfrastructureConfig floatingPoolName: MY-FLOATING-POOL # floatingPoolSubnetName: my-floating-pool-subnet-name networks: # id: 12345678-abcd-efef-08af-0123456789ab # router: # id: 1234 workers: 10.250.0.0/19 # shareNetwork: # enabled: true
floatingPoolName is the name of the floating pool you want to use for your shoot.
If you don’t know which floating pools are available look it up in the respective
floatingPoolSubnetName you can explicitly define to which subnet in the floating pool network (defined via
floatingPoolName) the router should be attached to.
networks.id is an optional field. If it is given, you can specify the uuid of an existing private Neutron network (created manually, by other tooling, …) that should be reused. A new subnet for the Shoot will be created in it.
networks.id is given and calico shoot clusters are created without a network overlay within one network make sure that the pod CIDR specified in
shoot.spec.networking.pods is not overlapping with any other pod CIDR used in that network.
Overlapping pod CIDRs will lead to disfunctional shoot clusters.
networks.router section describes whether you want to create the shoot cluster in an already existing router or whether to create a new one:
networks.router.idis given then you have to specify the router id of the existing router that was created by other means (manually, other tooling, …). If you want to get a fresh router for the shoot then just omit the
In any case, the shoot cluster will be created in a new subnet.
networks.workers section describes the CIDR for a subnet that is used for all shoot worker nodes, i.e., VMs which later run your applications.
You can freely choose these CIDRs and it is your responsibility to properly design the network layout to suit your needs.
Apart from the router and the worker subnet the OpenStack extension will also create a network, router interfaces, security groups, and a key pair.
networks.shareNetwork.enabled field controls the creation of a share network. This is only needed if shared
file system storage (like NFS) should be used. Note, that in this case, the
ControlPlaneConfig needs additional configuration, too.
The control plane configuration mainly contains values for the OpenStack-specific control plane components.
Today, the only component deployed by the OpenStack extension is the
ControlPlaneConfig for the OpenStack extension looks as follows:
apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1 kind: ControlPlaneConfig loadBalancerProvider: haproxy loadBalancerClasses: - name: lbclass-1 purpose: default floatingNetworkID: fips-1-id floatingSubnetName: internet-* - name: lbclass-2 floatingNetworkID: fips-1-id floatingSubnetTags: internal,private - name: lbclass-3 purpose: private subnetID: internal-id cloudControllerManager: featureGates: CustomResourceValidation: true #storage: # csiManila: # enabled: true
loadBalancerProvider is the provider name you want to use for load balancers in your shoot.
If you don’t know which types are available look it up in the respective
loadBalancerClasses field contains an optional list of load balancer classes which will be available in the cluster. Each entry can have the following fields:
nameto select the load balancer class via the kubernetes service annotations
- The configuration of the
defaultload balancer class will be used as default for all other kubernetes loadbalancer services without a class annotation
- The configuration of the
privateload balancer class will be also set to the global loadbalancer configuration of the cluster, but will be overridden by the
- The configuration of the
floatingNetworkIDcan be specified to receive an ip from an floating/external network, additionally the subnet in this network can be selected via
floatingSubnetNamecan be either a full subnet name or a regex/glob to match subnet name
floatingSubnetTagsa comma seperated list of subnet tags
floatingSubnetIDthe id of a specific subnet
subnetIDcan be specified by to receive an ip from an internal subnet (will not have an effect in combination with floating/external network configuration)
cloudControllerManager.featureGates contains a map of explicitly enabled or disabled feature gates.
For production usage it’s not recommended to use this field at all as you can enable alpha features or disable beta/stable features, potentially impacting the cluster stability.
If you don’t want to configure anything for the
cloudControllerManager simply omit the key in the YAML specification.
storage.csiManila.enabled field is used to enable the deployment of the CSI Manila driver to support NFS persistent volumes.
In this case, please ensure to set
networks.shareNetwork.enabled=true in the
Additionally, if CSI Manila driver is enabled, for each availability zone a NFS
StorageClass will be created on the shoot
Each worker group in a shoot may contain provider-specific configurations and options. These are contained in the
providerConfig section of a worker group and can be configured using a
An example of a
WorkerConfig looks as follows:
apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1 kind: WorkerConfig serverGroup: policy: soft-anti-affinity # nodeTemplate: # (to be specified only if the node capacity would be different from cloudprofile info during runtime) # capacity: # cpu: 2 # gpu: 0 # memory: 50Gi # machineLabels: # - name: my-label # value: foo # - name: my-rolling-label # value: bar # triggerRollingOnUpdate: true # means any change of the machine label value will trigger rolling of all machines of the worker pool
When you specify the
serverGroup section in your worker group configuration, a new server group will be created with the configured policy for each worker group that enabled this setting and all machines managed by this worker group will be assigned as members of the created server group.
For users to have access to the server group feature, it must be enabled on the
CloudProfile by your operator.
Existing clusters can take advantage of this feature by updating the server group configuration of their respective worker groups. Worker groups that are already configured with server groups can update their setting to change the policy used, or remove it altogether at any time.
Users must be aware that any change to the server group settings will result in a rolling deployment of new nodes for the affected worker group.
Please note the following restrictions when deploying workers with server groups:
serverGroupsection is optional, but if it is included in the worker configuration, it must contain a valid policy value.
- The available
policyvalues that can be used, are defined in the provider specific section of
CloudProfileby your operator.
- Certain policy values may induce further constraints. Using the
affinitypolicy is only allowed when the worker group utilizes a single zone.
machineLabels section in the worker group configuration allows to specify additional machine labels. These labels are added to the machine
instances only, but not to the node object. Additionally, they have an optional
triggerRollingOnUpdate field. If it is set to
true, changing the label value
will trigger a rolling of all machines of this worker pool.
Node templates allow users to override the capacity of the nodes as defined by the server flavor specified in the
machineTypes. This is useful for certain dynamic scenarios as it allows users to customize cluster-autoscaler’s behavior for these workergroup with their provided values.
Shoot manifest (one availability zone)
Please find below an example
Shoot manifest for one availability zone:
apiVersion: core.gardener.cloud/v1beta1 kind: Shoot metadata: name: johndoe-openstack namespace: garden-dev spec: cloudProfileName: openstack region: europe-1 secretBindingName: core-openstack provider: type: openstack infrastructureConfig: apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1 kind: InfrastructureConfig floatingPoolName: MY-FLOATING-POOL networks: workers: 10.250.0.0/19 controlPlaneConfig: apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1 kind: ControlPlaneConfig loadBalancerProvider: haproxy workers: - name: worker-xoluy machine: type: medium_4_8 minimum: 2 maximum: 2 zones: - europe-1a networking: nodes: 10.250.0.0/16 type: calico kubernetes: version: 1.24.3 maintenance: autoUpdate: kubernetesVersion: true machineImageVersion: true addons: kubernetesDashboard: enabled: true nginxIngress: enabled: true
CSI volume provisioners
Every OpenStack shoot cluster will be deployed with the OpenStack Cinder CSI driver.
It is compatible with the legacy in-tree volume provisioner that was deprecated by the Kubernetes community and will be removed in future versions of Kubernetes.
End-users might want to update their custom
StorageClasses to the new
Kubernetes Versions per Worker Pool
This extension supports
WorkerPoolKubernetesVersion feature gate, i.e., having worker pools with overridden Kubernetes versions since
Shoot CA Certificate and
ServiceAccount Signing Key Rotation
This extension supports
ShootSARotation feature gates since