This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Operating System Extensions

Gardener extension controllers for the supported operating systems

1 - CoreOS/FlatCar OS

Gardener extension controller for the CoreOS/FlatCar Container Linux operating system

Gardener Extension for CoreOS Container Linux

Project Gardener implements the automated management and operation of Kubernetes clusters as a service. Its main principle is to leverage Kubernetes concepts for all of its tasks.

Recently, most of the vendor specific logic has been developed in-tree. However, the project has grown to a size where it is very hard to extend, maintain, and test. With GEP-1 we have proposed how the architecture can be changed in a way to support external controllers that contain their very own vendor specifics. This way, we can keep Gardener core clean and independent.

This controller operates on the OperatingSystemConfig resource in the extensions.gardener.cloud/v1alpha1 API group. It supports CoreOS Container Linux and Flatcar Container Linux (“a friendly fork of CoreOS Container Linux”).

The controller manages those objects that are requesting CoreOS Container Linux configuration (.spec.type=coreos) or Flatcar Container Linux configuration (.spec.type=flatcar):

---
apiVersion: extensions.gardener.cloud/v1alpha1
kind: OperatingSystemConfig
metadata:
  name: pool-01-original
  namespace: default
spec:
  type: coreos
  units:
    ...
  files:
    ...

Please find a concrete example in the example folder.

After reconciliation the resulting data will be stored in a secret within the same namespace (as the config itself might contain confidential data). The name of the secret will be written into the resource’s .status field:

...
status:
  ...
  cloudConfig:
    secretRef:
      name: osc-result-pool-01-original
      namespace: default
  command: /usr/bin/coreos-cloudinit -from-file=<path>
  units:
  - docker-monitor.service
  - kubelet-monitor.service
  - kubelet.service

The secret has one data key cloud_config that stores the generation.

An example for a ControllerRegistration resource that can be used to register this controller to Gardener can be found here.

Please find more information regarding the extensibility concepts and a detailed proposal here.

How to start using or developing this extension controller locally

You can run the controller locally on your machine by executing make start. Please make sure to have the kubeconfig to the cluster you want to connect to ready in the ./dev/kubeconfig file.

Static code checks and tests can be executed by running make verify. We are using Go modules for Golang package dependency management and Ginkgo/Gomega for testing.

Feedback and Support

Feedback and contributions are always welcome. Please report bugs or suggestions as GitHub issues or join our Slack channel #gardener (please invite yourself to the Kubernetes workspace here).

Learn more!

Please find further resources about out project here:

1.1 - Usage

Using the CoreOS extension with Gardener as end-user

The core.gardener.cloud/v1beta1.Shoot resource declares a few fields that must be considered when this OS extension is used.

In this document we describe how this configuration looks like and under which circumstances your attention may be required.

AWS VPC settings for CoreOS workers

Gardener allows you to create CoreOS based worker nodes by:

Using a Gardener managed VPC
Reusing a VPC that already exists (VPC id specified in InfrastructureConfig]

If the second option applies to your use-case please make sure that your VPC has enabled DNS Support. Otherwise CoreOS based nodes aren’t able to join or operate in your cluster properly.

DNS settings (required):

enableDnsHostnames: true (necessary for collecting node metrics)
enableDnsSupport: true

2 - Garden Linux OS

Gardener extension controller for the Garden Linux operating system

Gardener Extension for Garden Linux OS

This controller operates on the OperatingSystemConfig resource in the extensions.gardener.cloud/v1alpha1 API group.

It manages those objects that are requesting…

Garden Linux OS configuration (.spec.type=gardenlinux):

---
apiVersion: extensions.gardener.cloud/v1alpha1
kind: OperatingSystemConfig
metadata:
  name: pool-01-original
  namespace: default
spec:
  type: gardenlinux
  units:
    ...
  files:
    ...

Please find a concrete example in the example folder.

MemoryOne on Garden Linux configuration (spec.type=memoryone-gardenlinux):

---
apiVersion: extensions.gardener.cloud/v1alpha1
kind: OperatingSystemConfig
metadata:
  name: pool-01-original
  namespace: default
spec:
  type: memoryone-gardenlinux
  units:
    ...
  files:
    ...
  providerConfig:
    apiVersion: memoryone-gardenlinux.os.extensions.gardener.cloud/v1alpha1
    kind: OperatingSystemConfiguration
    memoryTopology: "2"
    systemMemory: "6x"

Please find a concrete example in the example folder.

After reconciliation the resulting data will be stored in a secret within the same namespace (as the config itself might contain confidential data). The name of the secret will be written into the resource’s .status field:

...
status:
  ...
  cloudConfig:
    secretRef:
      name: osc-result-pool-01-original
      namespace: default
  command: /usr/bin/env bash <path>
  units:
  - docker-monitor.service
  - kubelet-monitor.service
  - kubelet.service

The secret has one data key cloud_config that stores the generation.

An example for a ControllerRegistration resource that can be used to register this controller to Gardener can be found here.

Please find more information regarding the extensibility concepts and a detailed proposal here.

How to start using or developing this extension controller locally

You can run the controller locally on your machine by executing make start. Please make sure to have the kubeconfig to the cluster you want to connect to ready in the ./dev/kubeconfig file. Static code checks and tests can be executed by running make verify. We are using Go modules for Golang package dependency management and Ginkgo/Gomega for testing.

Feedback and Support

Feedback and contributions are always welcome. Please report bugs or suggestions as GitHub issues or join our Slack channel #gardener (please invite yourself to the Kubernetes workspace here).

Learn more!

Please find further resources about out project here:

3 - SUSE CHost OS

Gardener extension controller for the SUSE Container Host operating system (CHost)

Gardener Extension for SUSE CHost

This controller operates on the OperatingSystemConfig resource in the extensions.gardener.cloud/v1alpha1 API group. It manages those objects that are requesting SUSE Container Host configuration, i.e. suse-chost type:

---
apiVersion: extensions.gardener.cloud/v1alpha1
kind: OperatingSystemConfig
metadata:
  name: pool-01-original
  namespace: default
spec:
  type: suse-chost
  units:
    ...
  files:
    ...

Please find a concrete example in the example folder.

It is also capable of supporting the vSMP MemoryOne operating system with the memoryone-chost type. Please find more information here.

After reconciliation the resulting data will be stored in a secret within the same namespace (as the config itself might contain confidential data). The name of the secret will be written into the resource’s .status field:

...
status:
  ...
  cloudConfig:
    secretRef:
      name: osc-result-pool-01-original
      namespace: default
  command: /usr/bin/env bash <path>
  units:
  - docker-monitor.service
  - kubelet-monitor.service
  - kubelet.service

The secret has one data key cloud_config that stores the generation.

An example for a ControllerRegistration resource that can be used to register this controller to Gardener can be found here.

Please find more information regarding the extensibility concepts and a detailed proposal here.

How to start using or developing this extension controller locally

You can run the controller locally on your machine by executing make start. Please make sure to have the kubeconfig to the cluster you want to connect to ready in the ./dev/kubeconfig file. Static code checks and tests can be executed by running make verify. We are using Go modules for Golang package dependency management and Ginkgo/Gomega for testing.

Feedback and Support

Feedback and contributions are always welcome. Please report bugs or suggestions as GitHub issues or join our Slack channel #gardener (please invite yourself to the Kubernetes workspace here).

Learn more!

Please find further resources about out project here:

3.1 - Usage

Using the SuSE CHost extension with Gardener as end-user

The core.gardener.cloud/v1beta1.Shoot resource declares a few fields that must be considered when this OS extension is used.

In this document we describe how this configuration looks like and under which circumstances your attention may be required.

AWS VPC settings for SuSE CHost workers

Gardener allows you to create SuSE CHost based worker nodes by:

Using a Gardener managed VPC
Reusing a VPC that already exists (VPC id specified in InfrastructureConfig]

If the second option applies to your use-case please make sure that your VPC has enabled DNS Support. Otherwise SuSE CHost based nodes aren’t able to join or operate in your cluster properly.

DNS settings (required):

enableDnsHostnames: true
enableDnsSupport: true

Support for vSMP MemoryOne

This extension controller is also capable of generating user-data for the vSMP MemoryOne operating system in conjunction with SuSE CHost. It reacts on the memoryone-chost extension type. Additionally, it allows certain customizations with the following configuration:

apiVersion: memoryone-chost.os.extensions.gardener.cloud/v1alpha1
kind: OperatingSystemConfiguration
memoryTopology: "3"
systemMemory: "7x"

The memoryTopology field controls the mem_topology setting. If it’s not provided then it will default to 2.
The systemMemory field controls the system_memory setting. If it’s not provided then it defaults to 6x.

Please note that it was only e2e-tested on AWS. Additionally, you need a snapshot ID of a SuSE CHost/CHost volume (see below how to create it).

An exemplary worker pool configuration inside a Shoot resource using for the vSMP MemoryOne operating system would look as follows:

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: vsmp-memoryone
  namespace: garden-foo
spec:
  ...
  workers:
  - name: cpu-worker3
    minimum: 1
    maximum: 1
    maxSurge: 1
    maxUnavailable: 0
    machine:
      image:
        name: memoryone-chost
        version: 9.5.195
        providerConfig:
          apiVersion: memoryone-chost.os.extensions.gardener.cloud/v1alpha1
          kind: OperatingSystemConfiguration
          memoryTopology: "2"
          systemMemory: "6x"
      type: c5d.metal
    volume:
      size: 20Gi
      type: gp2
    dataVolumes:
    - name: chost
      size: 50Gi
      type: gp2
    providerConfig:
      apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
      kind: WorkerConfig
      dataVolumes:
      - name: chost
        snapshotID: snap-123456
    zones:
    - eu-central-1b

Please note that vSMP MemoryOne only works for EC2 bare-metal instance types such as M5d, R5, C5, C5d, etc. - please consult the EC2 instance types overview page and the documentation of vSMP MemoryOne to find out whether the instance type in question is eligible.

Generating an AWS snapshot ID for the CHost/CHost operating system

The following script will help to generate the snapshot ID on AWS. It runs in the region that is selected in your $HOME/.aws/config file. Consequently, if you want to generate the snapshot in multiple regions, you have to run in multiple times after configuring the respective region using aws configure.

ami="ami-1234" #Replace the ami with the intended one.
name=`aws ec2 describe-images --image-ids $ami  --query="Images[].Name" --output=text`
cur=`aws ec2 describe-snapshots --filter="Name=description,Values=snap-$name" --query="Snapshots[].Description" --output=text`
if [ -n "$cur" ]; then
  echo "AMI $name exists as snapshot $cur"
  continue
fi
echo "AMI $name ... creating private snapshot"
inst=`aws ec2 run-instances --instance-type t3.nano --image-id $ami --query 'Instances[0].InstanceId' --output=text --subnet-id subnet-1234 --tag-specifications 'ResourceType=instance,Tags=[{Key=scalemp-test,Value=scalemp-test}]'` #Replace the subnet-id with the intended one.
aws ec2 wait instance-running --instance-ids $inst
vol=`aws ec2 describe-instances --instance-ids $inst --query "Reservations[].Instances[].BlockDeviceMappings[0].Ebs.VolumeId" --output=text`
snap=`aws ec2 create-snapshot --description "snap-$name" --volume-id $vol --query='SnapshotId' --tag-specifications "ResourceType=snapshot,Tags=[{Key=Name,Value=\"$name\"}]" --output=text`
aws ec2 wait snapshot-completed --snapshot-ids $snap
aws ec2 terminate-instances --instance-id $inst > /dev/null
echo $snap

4 - Ubuntu OS

Gardener extension controller for the Ubuntu operating system

Gardener Extension for Ubuntu OS

This controller operates on the OperatingSystemConfig resource in the extensions.gardener.cloud/v1alpha1 API group. It manages those objects that are requesting Ubuntu OS configuration (.spec.type=ubuntu). An experimental support for Ubuntu Pro is added (.spec.type=ubuntu-pro):

---
apiVersion: extensions.gardener.cloud/v1alpha1
kind: OperatingSystemConfig
metadata:
  name: pool-01-original
  namespace: default
spec:
  type: ubuntu
  units:
    ...
  files:
    ...

Please find a concrete example in the example folder.

After reconciliation the resulting data will be stored in a secret within the same namespace (as the config itself might contain confidential data). The name of the secret will be written into the resource’s .status field:

...
status:
  ...
  cloudConfig:
    secretRef:
      name: osc-result-pool-01-original
      namespace: default
  command: /usr/bin/env bash <path>
  units:
  - docker-monitor.service
  - kubelet-monitor.service
  - kubelet.service

The secret has one data key cloud_config that stores the generation.

An example for a ControllerRegistration resource that can be used to register this controller to Gardener can be found here.

Please find more information regarding the extensibility concepts and a detailed proposal here.

How to start using or developing this extension controller locally

You can run the controller locally on your machine by executing make start. Please make sure to have the kubeconfig to the cluster you want to connect to ready in the ./dev/kubeconfig file. Static code checks and tests can be executed by running make verify. We are using Go modules for Golang package dependency management and Ginkgo/Gomega for testing.

Feedback and Support

Feedback and contributions are always welcome. Please report bugs or suggestions as GitHub issues or join our Slack channel #gardener (please invite yourself to the Kubernetes workspace here).

Learn more!

Please find further resources about out project here:

4.1 - Usage

Using the Ubuntu extension with Gardener as end-user

The core.gardener.cloud/v1beta1.Shoot resource declares a few fields that must be considered when this OS extension is used.

In this document we describe how this configuration looks like and under which circumstances your attention may be required.

AWS VPC settings for Ubuntu workers

Gardener allows you to create Ubuntu based worker nodes by:

Using a Gardener managed VPC
Reusing a VPC that already exists (VPC id specified in InfrastructureConfig]

If the second option applies to your use-case please make sure that your VPC has enabled DNS Support. Otherwise Ubuntu based nodes aren’t able to join or operate in your cluster properly.

DNS settings (required):

enableDnsHostnames: true
enableDnsSupport: true