Lakom service
A k8s admission controller verifying pods are using signed images (cosign signatures) and a gardener extension to install it for shoots and seeds.

Project Gardener implements the automated management and operation of Kubernetes clusters as a service.
Its main principle is to leverage Kubernetes concepts for all of its tasks.
Recently, most of the vendor specific logic has been developed in-tree.
However, the project has grown to a size where it is very hard to extend, maintain, and test.
With GEP-1 we have proposed how the architecture can be changed in a way to support external controllers that contain their very own vendor specifics.
This way, we can keep Gardener core clean and independent.
This controller implements Gardener’s extension contract for the shoot-lakom-service
extension.
An example for a ControllerRegistration
resource that can be used to register this controller to Gardener can be found here.
Please find more information regarding the extensibility concepts and a detailed proposal here.
Lakom Admission Controller
Lakom is kubernetes admission controller which purpose is to implement cosign image signature verification against public cosign key. It also takes care to resolve image tags to sha256 digests. It also caches all OCI artifacts to reduce the load toward the OCI registry.
Extension Resources
Example extension resource:
apiVersion: extensions.gardener.cloud/v1alpha1
kind: Extension
metadata:
name: extension-shoot-lakom-service
namespace: shoot--project--abc
spec:
type: shoot-lakom-service
When an extension resource is reconciled, the extension controller will create an instance of lakom
admission controller. These resources are placed inside the shoot namespace on the seed. Also, the controller takes care about generating necessary RBAC
resources for the seed as well as for the shoot.
Please note, this extension controller relies on the Gardener-Resource-Manager to deploy k8s resources to seed and shoot clusters.
How to start using or developing this extension controller locally
The Lakom
admission controller can be configured with make dev-setup
and started with make start-lakom
.
You can run the lakom extension controller locally on your machine by executing make start
.
If you’d like to develop Lakom using a local cluster such as KinD, make sure your KUBECONFIG environment variable is targeting the local Garden cluster.
Add 127.0.0.1 garden.local.gardener.cloud
to your /etc/hosts
. You can then run:
This will trigger a skaffold deployment that builds the images, pushes them to the registry and installs the helm charts from /charts
.
We are using Go modules for Golang package dependency management and Ginkgo/Gomega for testing.
Feedback and Support
Feedback and contributions are always welcome. Please report bugs or suggestions as GitHub issues or join our Slack channel #gardener (please invite yourself to the Kubernetes workspace here).
Learn more
Please find further resources about out project here:
1 - Deployment
Gardener Lakom Service for Shoots
Introduction
Gardener allows Shoot clusters to use Lakom
admission controller for cosign image signing verification. To support this the Gardener must be installed with the shoot-lakom-service
extension.
Configuration
To generally enable the Lakom service for shoot objects the shoot-lakom-service
extension must be registered by providing an appropriate extension registration in the garden cluster.
Here it is possible to decide whether the extension should be always available for all shoots or whether the extension must be separately enabled per shoot.
If the extension should be used for all shoots the globallyEnabled
flag should be set to true
.
spec:
resources:
- kind: Extension
type: shoot-lakom-service
globallyEnabled: true
Shoot Feature Gate
If the shoot Lakom service is not globally enabled by default (depends on the extension registration on the garden cluster), it can be enabled per shoot. To enable the service for a shoot, the shoot manifest must explicitly add the shoot-lakom-service
extension.
...
spec:
extensions:
- type: shoot-lakom-service
...
If the shoot Lakom service is globally enabled by default, it can be disabled per shoot. To disable the service for a shoot, the shoot manifest must explicitly state it.
...
spec:
extensions:
- type: shoot-lakom-service
disabled: true
...
2 - Lakom
Introduction
Lakom is kubernetes admission controller which purpose is to implement
cosign image signature verification with
public cosign key. It also takes care to resolve image tags to sha256 digests. A
built-in cache mechanism can be enabled to reduce the load toward the OCI
registry.
Flags
Lakom admission controller is configurable via command line flags. The trusted
cosign public keys and the associated algorithms associated with them are set
viq configuration file provided with the flag --lakom-config-path
.
Flag Name | Description | Default Value |
---|
--bind-address | Address to bind to | “0.0.0.0” |
--cache-refresh-interval | Refresh interval for the cached objects | 30s |
--cache-ttl | TTL for the cached objects. Set to 0, if cache has to be disabled | 10m0s |
--contention-profiling | Enable lock contention profiling, if profiling is enabled | false |
--health-bind-address | Bind address for the health server | “:8081” |
-h , --help | help for lakom | |
--insecure-allow-insecure-registries | If set, communication via HTTP with registries will be allowed. | false |
--insecure-allow-untrusted-images | If set, the webhook will just return warning for the images without trusted signatures. | false |
--kubeconfig | Paths to a kubeconfig. Only required if out-of-cluster. | |
--lakom-config-path | Path to file with lakom configuration containing cosign public keys used to verify the image signatures | |
--metrics-bind-address | Bind address for the metrics server | “:8080” |
--port | Webhook server port | 9443 |
--profiling | Enable profiling via web interface host:port/debug/pprof/ | false |
--tls-cert-dir | Directory with server TLS certificate and key (must contain a tls.crt and tls.key file | |
--use-only-image-pull-secrets | If set, only the credentials from the image pull secrets of the pod are used to access the OCI registry. Otherwise, the node identity and docker config are also used. | false |
--version | prints version information and quits; –version=vX.Y.Z… sets the reported version | |
Lakom Cosign Public Keys Configuration File
Lakom cosign public keys configuration file should be YAML or JSON formatted. It
can set multiple trusted keys, as each key must be given a name. The supported
types of public keys are RSA
, ECDSA
and Ed25519
. The RSA
keys can be
additionally configured with a signature verification algorithm specifying the
scheme and hash function used during signature verification. As of now ECDSA
and Ed25519
keys cannot be configured with specific algorithm.
publicKeys:
- name: example-public-key
algorithm: RSASSA-PSS-SHA256
key: |-
-----BEGIN PUBLIC KEY-----
MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAPeQXbIWMMXYV+9+j9b4jXTflnpfwn4E
GMrmqYVhm0sclXb2FPP5aV/NFH6SZdHDZcT8LCNsNgxzxV4N+UE/JIsCAwEAAQ==
-----END PUBLIC KEY-----
Here:
name
is logical human friendly name of the key.algorithm
is the algorithm that has to be used to verify the signature, see Supported RSA Signature Verification Algorithms for the list of supported algorithms.key
is the cryptographic public key that will be used for image signature validation.
Supported RSA Signature Verification Algorithms
RSASSA-PKCS1-v1_5-SHA256
: uses RSASSA-PKCS1-v1_5
scheme with SHA256
hash funcRSASSA-PKCS1-v1_5-SHA384
: uses RSASSA-PKCS1-v1_5
scheme with SHA384
hash funcRSASSA-PKCS1-v1_5-SHA512
: uses RSASSA-PKCS1-v1_5
scheme with SHA512
hash funcRSASSA-PSS-SHA256
: uses RSASSA-PSS
scheme with SHA256
hash funcRSASSA-PSS-SHA384
: uses RSASSA-PSS
scheme with SHA384
hash funcRSASSA-PSS-SHA512
: uses RSASSA-PSS
scheme with SHA512
hash func
Supported Resources for Verification
By default, Lakom validates only Pod
resources in the clusters that it covers.
However, it also has the capabilities to validate the following Gardener specific resources:
controllerdeployments.core.gardener.cloud/v1
gardenlets.seedmanagement.gardener.cloud/v1alpha1
extensions.operator.gardener.cloud/v1alpha1
When deploying Lakom via the helm chart in /charts/lakom
, the admissionConfig.rules
key
can be fully customized to include any of the listed resources above.
Make sure that they are registered with the same group & versions as the ones listed above.
Any difference will cause Lakom to skip validation and approve the request,
making it a security risk.
3 - Oci Registries
OCI Registries
This document aims to introduce the reader to the general architecture of OCI
Registries, their history and jargon that might be encountered.
History of Container Registries
Most readers might be familiar with the container registry that was introduced
by Docker. It aims to provide a way for developers to store and distribute
container images that can be used for running applications in containers.
In the beginning, the registry was meant to only host images in a way that the
different layers of the image could be downloaded in parallel, while being
assembled on the client side.
In June 2015, multiple companies agreed to standardize the container image format,
including its distribution and runtime. This was the birth of the Open Container
Initiative (OCI).
Understanding OCI Registries
The most important document that pertains to OCI Registries in the case of Lakom is the OCI
distribution spec.
It outlines the way that artifacts shall be stored and fetched from a registry
that adheres to the OCI standard.
OCI Registries are meant to store content. While in the beginning the main
problem that they were trying to solve was image storage, during the evolution
of the image registry concept, a decision was made to make it more general
for storage of any form of objects, sometimes called artifacts.
An artifact is an abstract idea of multiple components that together describe
how an object is built along with storing metadata about the object.
To implement this, every OCI registry has to expose an API for creating
3 types of objects:
Blobs
A blob is just a set of bytes. Nothing more. The important part about blobs
is that they are content addressable. Content Addressable Storage (CAS)
is a method of storing data in such a way that the address of the data is derived
from the data itself. This means that if you store the same blob twice, it will
be stored only once, and the address of the blob will be the same. This is achieved
by using the digest of the blob as the address.
This is the actual mechanism for allowing images to be optimal in terms of storage.
Since most images begin by depending on the same base image, this means that
it can be stored only once and referenced by multiple images.
Manifests
A manifest is just a JSON document. It is stored separately from the blobs.
But similarly to the blobs, it is also content addressable. The manifest gets
formatted to a canonical form and then hashed. The hash becomes the address of
the manifest.
Where manifests become powerful is when they are used to reference to other
manifests or blobs. These references are called OCI Content Descriptors.
For example, the Image Manifest that most readers would be familiar with contains
a list of layers that are used to build the image. Each layer is a reference to
a blob.
An additional reference outside the layer references is used for storing
additional configuration. More info can be found in the official OCI Image Manifest spec.
The term artifact is just a generalization over the Image Manifest idea.
The Image Manifest has a specific format that can be found in the aforementioned OCI Image Manifest spec.
The important field is the .config.mediaType
field. Based on the official guidelines,
it can be used for defining types other than an image. More info can be found here.
Combining all of these ideas, we can see that artifacts can be represented by 1
specific object - a manifest. That manifest, based on the media type in the config, can be
interpreted by different programs for their specific use case. Eg. a Helm chart.
Helm charts specifically have a mediaType of application/vnd.cncf.helm.config.v1+json
.
Layers, too, have a media type. The media type of a layer when the artifact
is a helm chart is one of:
- application/vnd.cncf.helm.config.v1+json
- application/vnd.cncf.helm.chart.content.v1.tar+gzip
- application/vnd.cncf.helm.chart.provenance.v1.prov
More info specifically on Helm charts as OCI Artifacts can be found here: Helm OCI MediaTypes.
While manifests and blobs being content addressable is nice, it becomes hard
to address them in a human-readable format. Tags allow us to add a reference
to a given manifest that differs from the digest of the object.
Normally, whenever we push an image to a registry, if we don’t give it an
explicit tag, the registry would most likely give it the tag latest
. This,
actually is not mandatory based on the distribution spec. Manifests don’t
require that they have a tag.
Multiple tags can point to the same manifest.
Cosign
Cosign is generally a part of the bigger project called sigstore.
Sigstore is an “open-source project for improving software supply chain security”.
It aims for developers to have a hassle-free way to sign their artifacts while
leveraging other components that add additional layers of security to the
signature process. Eg. Rekor and Fulcio.
Cosign is a CLI tool aiming to tie all the components of sigstore together.
In our specific case, we use it only as a format for creating signatures and
then verifying them. In theory, we wouldn’t need to use Cosign for this, but
it uses a popular format for the signatures and the libraries help us
save work.
In the context of Lakom, what interests us is how Cosign signs OCI artifacts
and how it stores them.
The signing process in a nutshell is the following – Sign(sha256(SimpleSigningPayload(sha256(Image Manifest))))
, where:
- Image Manifest is what it says. Just the manifest of the image (artifact).
- sha256 is the hashing algorithm used.
- SimpleSigningPayload is the interesting part here. Instead of just signing the
hash of the manifest itself, the hash itself is packed into a JSON doc that
is called the
SimpleSigningPayload
. Example payload:
{
"critical": {
"identity": {
"docker-reference": "testing/manifest"
},
"image": {
"Docker-manifest-digest": "sha256:20be...fe55"
},
"type": "cosign container image signature"
},
"optional": {
"creator": "atomic",
"timestamp": 1458239713
}
}
We can see that the hash of the manifest is stored in the Docker-manifest-digest
.
Now this object is the one that gets signed using our private key. The signed
version of this objest is the one that gets stored and gets used for
verification.
Storage of the signature is done via an image manifest that gets uploaded
in the same repo as the image, but tagged with sha256-<container-image-digest>.sig
.
Thus, the signature discovery mechanism is to just fetch the image digest
that is tagged with the aforementioned tag.
More info on this whole process can be found here.
Lakom relies on the cosign libraries to automate this whole process. When
validating an artifact, if we know its digest, we can fetch the signature and
verify it. If the signature is valid, we can be sure that the artifact has not
been tampered with.
4 - Shoot Extension
Introduction
This extension implements cosign image verification. It is strictly limited only to the kubernetes system components deployed by Gardener and other Gardener Extensions in the kube-system
namespace of a shoot cluster.
Shoot Feature Gate
In most of the Gardener setups the shoot-lakom-service
extension is enabled globally and thus can be configured per shoot cluster. Please adapt the shoot specification by the configuration shown below to disable the extension individually.
kind: Shoot
...
spec:
resources:
- name: lakom-ref
resourceRef:
apiVersion: v1
kind: Secret
name: lakom-secret
extensions:
- type: shoot-lakom-service
disabled: true
providerConfig:
apiVersion: lakom.extensions.gardener.cloud/v1alpha1
kind: LakomConfig
scope: KubeSystem
trustedKeysResourceName: lakom-ref
...
Scope
The scope
field instruct lakom which pods to validate. The possible values are:
KubeSystem
Lakom will validate all pods in the kube-system
namespace.KubeSystemManagedByGardener
Lakom will validate all pods in the kube-system
namespace that are annotated with “managed-by/gardener”. This is the default value.Cluster
Lakom will validate all pods in all namespaces.
TrustedKeysResourceName
Lakom, by default, tries to verify only workloads that belong to Gardener. Because of this, the only public keys that it uses to do its job are the ones for the Gardener workload.
If you’d like to use Lakom as a tool for verifying your own workload, you’ll need to add your own public keys to the ones that Lakom is already using. This can be achieved using Gardener referenced resources. More information about the keys and their format can be found here.
Simply:
- Create a secret in your project namespace that contains a field
keys
with your keys as a value. Example keys:
- name: example-client-key1
algorithm: RSASSA-PSS-SHA256
key: |-
-----BEGIN PUBLIC KEY-----
MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAPeQXbIWMMXYV+9+j9b4jXTflnpfwn4E
GMrmqYVhm0sclXb2FPP5aV/NFH6SZdHDZcT8LCNsNgxzxV4N+UE/JIsCAwEAAQ==
-----END PUBLIC KEY-----
- name: example-client-key2
algorithm: RSASSA-PSS-SHA256
key: |-
-----BEGIN PUBLIC KEY-----
MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAPeQXbIWMMXYV+9+j9b4jXTflnpfwn4E
GMrmqYVhm0sclXb2FPP5aV/NFH6SZdHDZcT8LCNsNgxzxV4N+UE/JIsCAwEAAQ==
-----END PUBLIC KEY-----
- Add a reference to your secret via the
resources
field in the shoot spec as shown above. - Add the name of your referenece in
trustedKeysResourceName
in the provider config as shown above.
Now, whenever Lakom tries to verify a Pod, it will make sure to use your public keys as well.