This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Lakom service

A k8s admission controller verifying pods are using signed images (cosign signatures) and a gardener extension to install it for shoots and seeds.

Gardener Extension for lakom services

REUSE status

Project Gardener implements the automated management and operation of Kubernetes clusters as a service. Its main principle is to leverage Kubernetes concepts for all of its tasks.

Recently, most of the vendor specific logic has been developed in-tree. However, the project has grown to a size where it is very hard to extend, maintain, and test. With GEP-1 we have proposed how the architecture can be changed in a way to support external controllers that contain their very own vendor specifics. This way, we can keep Gardener core clean and independent.

This controller implements Gardener’s extension contract for the shoot-lakom-service extension.

An example for a ControllerRegistration resource that can be used to register this controller to Gardener can be found here.

Please find more information regarding the extensibility concepts and a detailed proposal here.

Lakom Admission Controller

Lakom is kubernetes admission controller which purpose is to implement cosign image signature verification against public cosign key. It also takes care to resolve image tags to sha256 digests. It also caches all OCI artifacts to reduce the load toward the OCI registry.

Extension Resources

Example extension resource:

apiVersion: extensions.gardener.cloud/v1alpha1
kind: Extension
metadata:
  name: extension-shoot-lakom-service
  namespace: shoot--project--abc
spec:
  type: shoot-lakom-service

When an extension resource is reconciled, the extension controller will create an instance of lakom admission controller. These resources are placed inside the shoot namespace on the seed. Also, the controller takes care about generating necessary RBAC resources for the seed as well as for the shoot.

Please note, this extension controller relies on the Gardener-Resource-Manager to deploy k8s resources to seed and shoot clusters.

How to start using or developing this extension controller locally

The Lakom admission controller can be configured with make dev-setup and started with make start-lakom. You can run the lakom extension controller locally on your machine by executing make start.

If you’d like to develop Lakom using a local cluster such as KinD, make sure your KUBECONFIG environment variable is targeting the local Garden cluster. Add 127.0.0.1 garden.local.gardener.cloud to your /etc/hosts. You can then run:

make extension-up

This will trigger a skaffold deployment that builds the images, pushes them to the registry and installs the helm charts from /charts.

We are using Go modules for Golang package dependency management and Ginkgo/Gomega for testing.

Feedback and Support

Feedback and contributions are always welcome. Please report bugs or suggestions as GitHub issues or join our Slack channel #gardener (please invite yourself to the Kubernetes workspace here).

Learn more

Please find further resources about out project here:

1 - Deployment

Gardener Lakom Service for Shoots

Introduction

Gardener allows Shoot clusters to use Lakom admission controller for cosign image signing verification. To support this the Gardener must be installed with the shoot-lakom-service extension.

Configuration

To generally enable the Lakom service for shoot objects the shoot-lakom-service extension must be registered by providing an appropriate extension registration in the garden cluster.

Here it is possible to decide whether the extension should be always available for all shoots or whether the extension must be separately enabled per shoot.

If the extension should be used for all shoots the globallyEnabled flag should be set to true.

spec:
  resources:
    - kind: Extension
      type: shoot-lakom-service
      globallyEnabled: true

Shoot Feature Gate

If the shoot Lakom service is not globally enabled by default (depends on the extension registration on the garden cluster), it can be enabled per shoot. To enable the service for a shoot, the shoot manifest must explicitly add the shoot-lakom-service extension.

...
spec:
  extensions:
    - type: shoot-lakom-service
...

If the shoot Lakom service is globally enabled by default, it can be disabled per shoot. To disable the service for a shoot, the shoot manifest must explicitly state it.

...
spec:
  extensions:
    - type: shoot-lakom-service
      disabled: true
...

2 - Lakom

Introduction

Lakom is kubernetes admission controller which purpose is to implement cosign image signature verification with public cosign key. It also takes care to resolve image tags to sha256 digests. A built-in cache mechanism can be enabled to reduce the load toward the OCI registry.

Flags

Lakom admission controller is configurable via command line flags. The trusted cosign public keys and the associated algorithms associated with them are set viq configuration file provided with the flag --lakom-config-path.

Flag NameDescriptionDefault Value
--bind-addressAddress to bind to“0.0.0.0”
--cache-refresh-intervalRefresh interval for the cached objects30s
--cache-ttlTTL for the cached objects. Set to 0, if cache has to be disabled10m0s
--contention-profilingEnable lock contention profiling, if profiling is enabledfalse
--health-bind-addressBind address for the health server“:8081”
-h, --helphelp for lakom
--insecure-allow-insecure-registriesIf set, communication via HTTP with registries will be allowed.false
--insecure-allow-untrusted-imagesIf set, the webhook will just return warning for the images without trusted signatures.false
--kubeconfigPaths to a kubeconfig. Only required if out-of-cluster.
--lakom-config-pathPath to file with lakom configuration containing cosign public keys used to verify the image signatures
--metrics-bind-addressBind address for the metrics server“:8080”
--portWebhook server port9443
--profilingEnable profiling via web interface host:port/debug/pprof/false
--tls-cert-dirDirectory with server TLS certificate and key (must contain a tls.crt and tls.key file
--use-only-image-pull-secretsIf set, only the credentials from the image pull secrets of the pod are used to access the OCI registry. Otherwise, the node identity and docker config are also used.false
--versionprints version information and quits; –version=vX.Y.Z… sets the reported version

Lakom Cosign Public Keys Configuration File

Lakom cosign public keys configuration file should be YAML or JSON formatted. It can set multiple trusted keys, as each key must be given a name. The supported types of public keys are RSA, ECDSA and Ed25519. The RSA keys can be additionally configured with a signature verification algorithm specifying the scheme and hash function used during signature verification. As of now ECDSA and Ed25519 keys cannot be configured with specific algorithm.

publicKeys:
- name: example-public-key
  algorithm: RSASSA-PSS-SHA256
  key: |-
    -----BEGIN PUBLIC KEY-----
    MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAPeQXbIWMMXYV+9+j9b4jXTflnpfwn4E
    GMrmqYVhm0sclXb2FPP5aV/NFH6SZdHDZcT8LCNsNgxzxV4N+UE/JIsCAwEAAQ==
    -----END PUBLIC KEY-----    

Here:

  • name is logical human friendly name of the key.
  • algorithm is the algorithm that has to be used to verify the signature, see Supported RSA Signature Verification Algorithms for the list of supported algorithms.
  • key is the cryptographic public key that will be used for image signature validation.

Supported RSA Signature Verification Algorithms

  • RSASSA-PKCS1-v1_5-SHA256: uses RSASSA-PKCS1-v1_5 scheme with SHA256 hash func
  • RSASSA-PKCS1-v1_5-SHA384: uses RSASSA-PKCS1-v1_5 scheme with SHA384 hash func
  • RSASSA-PKCS1-v1_5-SHA512: uses RSASSA-PKCS1-v1_5 scheme with SHA512 hash func
  • RSASSA-PSS-SHA256: uses RSASSA-PSS scheme with SHA256 hash func
  • RSASSA-PSS-SHA384: uses RSASSA-PSS scheme with SHA384 hash func
  • RSASSA-PSS-SHA512: uses RSASSA-PSS scheme with SHA512 hash func

Supported Resources for Verification

By default, Lakom validates only Pod resources in the clusters that it covers. However, it also has the capabilities to validate the following Gardener specific resources:

  • controllerdeployments.core.gardener.cloud/v1
  • gardenlets.seedmanagement.gardener.cloud/v1alpha1
  • extensions.operator.gardener.cloud/v1alpha1
message-outline

Important

When deploying Lakom via the helm chart in /charts/lakom, the admissionConfig.rules key can be fully customized to include any of the listed resources above. Make sure that they are registered with the same group & versions as the ones listed above. Any difference will cause Lakom to skip validation and approve the request, making it a security risk.

3 - Oci Registries

OCI Registries

This document aims to introduce the reader to the general architecture of OCI Registries, their history and jargon that might be encountered.

History of Container Registries

Most readers might be familiar with the container registry that was introduced by Docker. It aims to provide a way for developers to store and distribute container images that can be used for running applications in containers.

In the beginning, the registry was meant to only host images in a way that the different layers of the image could be downloaded in parallel, while being assembled on the client side.

In June 2015, multiple companies agreed to standardize the container image format, including its distribution and runtime. This was the birth of the Open Container Initiative (OCI).

Understanding OCI Registries

The most important document that pertains to OCI Registries in the case of Lakom is the OCI distribution spec. It outlines the way that artifacts shall be stored and fetched from a registry that adheres to the OCI standard.

OCI Registries are meant to store content. While in the beginning the main problem that they were trying to solve was image storage, during the evolution of the image registry concept, a decision was made to make it more general for storage of any form of objects, sometimes called artifacts.

An artifact is an abstract idea of multiple components that together describe how an object is built along with storing metadata about the object.

To implement this, every OCI registry has to expose an API for creating 3 types of objects:

  • Blobs
  • Manifests
  • Tags

Blobs

A blob is just a set of bytes. Nothing more. The important part about blobs is that they are content addressable. Content Addressable Storage (CAS) is a method of storing data in such a way that the address of the data is derived from the data itself. This means that if you store the same blob twice, it will be stored only once, and the address of the blob will be the same. This is achieved by using the digest of the blob as the address.

This is the actual mechanism for allowing images to be optimal in terms of storage. Since most images begin by depending on the same base image, this means that it can be stored only once and referenced by multiple images.

Manifests

A manifest is just a JSON document. It is stored separately from the blobs. But similarly to the blobs, it is also content addressable. The manifest gets formatted to a canonical form and then hashed. The hash becomes the address of the manifest.

Where manifests become powerful is when they are used to reference to other manifests or blobs. These references are called OCI Content Descriptors. For example, the Image Manifest that most readers would be familiar with contains a list of layers that are used to build the image. Each layer is a reference to a blob.

An additional reference outside the layer references is used for storing additional configuration. More info can be found in the official OCI Image Manifest spec.

The term artifact is just a generalization over the Image Manifest idea. The Image Manifest has a specific format that can be found in the aforementioned OCI Image Manifest spec. The important field is the .config.mediaType field. Based on the official guidelines, it can be used for defining types other than an image. More info can be found here.

Combining all of these ideas, we can see that artifacts can be represented by 1 specific object - a manifest. That manifest, based on the media type in the config, can be interpreted by different programs for their specific use case. Eg. a Helm chart. Helm charts specifically have a mediaType of application/vnd.cncf.helm.config.v1+json. Layers, too, have a media type. The media type of a layer when the artifact is a helm chart is one of:

  • application/vnd.cncf.helm.config.v1+json
  • application/vnd.cncf.helm.chart.content.v1.tar+gzip
  • application/vnd.cncf.helm.chart.provenance.v1.prov

More info specifically on Helm charts as OCI Artifacts can be found here: Helm OCI MediaTypes.

Tags

While manifests and blobs being content addressable is nice, it becomes hard to address them in a human-readable format. Tags allow us to add a reference to a given manifest that differs from the digest of the object.

Normally, whenever we push an image to a registry, if we don’t give it an explicit tag, the registry would most likely give it the tag latest. This, actually is not mandatory based on the distribution spec. Manifests don’t require that they have a tag.

Multiple tags can point to the same manifest.

Cosign

Cosign is generally a part of the bigger project called sigstore. Sigstore is an “open-source project for improving software supply chain security”. It aims for developers to have a hassle-free way to sign their artifacts while leveraging other components that add additional layers of security to the signature process. Eg. Rekor and Fulcio.

Cosign is a CLI tool aiming to tie all the components of sigstore together. In our specific case, we use it only as a format for creating signatures and then verifying them. In theory, we wouldn’t need to use Cosign for this, but it uses a popular format for the signatures and the libraries help us save work.

In the context of Lakom, what interests us is how Cosign signs OCI artifacts and how it stores them. The signing process in a nutshell is the following – Sign(sha256(SimpleSigningPayload(sha256(Image Manifest)))), where:

  • Image Manifest is what it says. Just the manifest of the image (artifact).
  • sha256 is the hashing algorithm used.
  • SimpleSigningPayload is the interesting part here. Instead of just signing the hash of the manifest itself, the hash itself is packed into a JSON doc that is called the SimpleSigningPayload. Example payload:
{
    "critical": {
           "identity": {
               "docker-reference": "testing/manifest"
           },
           "image": {
               "Docker-manifest-digest": "sha256:20be...fe55"
           },
           "type": "cosign container image signature"
    },
    "optional": {
           "creator": "atomic",
           "timestamp": 1458239713
    }
}

We can see that the hash of the manifest is stored in the Docker-manifest-digest. Now this object is the one that gets signed using our private key. The signed version of this objest is the one that gets stored and gets used for verification.

Storage of the signature is done via an image manifest that gets uploaded in the same repo as the image, but tagged with sha256-<container-image-digest>.sig. Thus, the signature discovery mechanism is to just fetch the image digest that is tagged with the aforementioned tag.

More info on this whole process can be found here.

Lakom relies on the cosign libraries to automate this whole process. When validating an artifact, if we know its digest, we can fetch the signature and verify it. If the signature is valid, we can be sure that the artifact has not been tampered with.

4 - Shoot Extension

Introduction

This extension implements cosign image verification. It is strictly limited only to the kubernetes system components deployed by Gardener and other Gardener Extensions in the kube-system namespace of a shoot cluster.

Shoot Feature Gate

In most of the Gardener setups the shoot-lakom-service extension is enabled globally and thus can be configured per shoot cluster. Please adapt the shoot specification by the configuration shown below to disable the extension individually.

kind: Shoot
...
spec:
  resources:
  - name: lakom-ref
    resourceRef:
      apiVersion: v1
      kind: Secret
      name: lakom-secret
  extensions:
  - type: shoot-lakom-service
    disabled: true
    providerConfig:
      apiVersion: lakom.extensions.gardener.cloud/v1alpha1
      kind: LakomConfig
      scope: KubeSystem
      trustedKeysResourceName: lakom-ref
...

Scope

The scope field instruct lakom which pods to validate. The possible values are:

  • KubeSystem Lakom will validate all pods in the kube-system namespace.
  • KubeSystemManagedByGardener Lakom will validate all pods in the kube-system namespace that are annotated with “managed-by/gardener”. This is the default value.
  • Cluster Lakom will validate all pods in all namespaces.

TrustedKeysResourceName

Lakom, by default, tries to verify only workloads that belong to Gardener. Because of this, the only public keys that it uses to do its job are the ones for the Gardener workload.

If you’d like to use Lakom as a tool for verifying your own workload, you’ll need to add your own public keys to the ones that Lakom is already using. This can be achieved using Gardener referenced resources. More information about the keys and their format can be found here.

Simply:

  1. Create a secret in your project namespace that contains a field keys with your keys as a value. Example keys:
- name: example-client-key1
  algorithm: RSASSA-PSS-SHA256
  key: |-
    -----BEGIN PUBLIC KEY-----
    MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAPeQXbIWMMXYV+9+j9b4jXTflnpfwn4E
    GMrmqYVhm0sclXb2FPP5aV/NFH6SZdHDZcT8LCNsNgxzxV4N+UE/JIsCAwEAAQ==
    -----END PUBLIC KEY-----
- name: example-client-key2
  algorithm: RSASSA-PSS-SHA256
  key: |-
    -----BEGIN PUBLIC KEY-----
    MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAPeQXbIWMMXYV+9+j9b4jXTflnpfwn4E
    GMrmqYVhm0sclXb2FPP5aV/NFH6SZdHDZcT8LCNsNgxzxV4N+UE/JIsCAwEAAQ==
    -----END PUBLIC KEY-----
  1. Add a reference to your secret via the resources field in the shoot spec as shown above.
  2. Add the name of your referenece in trustedKeysResourceName in the provider config as shown above.

Now, whenever Lakom tries to verify a Pod, it will make sure to use your public keys as well.