DNS services

apiVersion: extensions.gardener.cloud/v1alpha1
kind: Extension
metadata:
  name: "extension-dns-service"
  namespace: shoot--project--abc
spec:
  type: shoot-dns-service

1 - Deployment

Deployment of the shoot DNS service extension

Disclaimer: This document is NOT a step by step deployment guide for the shoot DNS service extension and only contains some configuration specifics regarding the deployment of different components via the helm charts residing in the shoot DNS service extension repository.

gardener-extension-admission-shoot-dns-service

Authentication against the Garden cluster

There are several authentication possibilities depending on whether or not the concept of Virtual Garden is used.

Virtual Garden is not used, i.e., the runtime Garden cluster is also the target Garden cluster.

Automounted Service Account Token

The easiest way to deploy the gardener-extension-admission-shoot-dns-service component will be to not provide kubeconfig at all. This way in-cluster configuration and an automounted service account token will be used. The drawback of this approach is that the automounted token will not be automatically rotated.

Service Account Token Volume Projection

Another solution will be to use Service Account Token Volume Projection combined with a kubeconfig referencing a token file (see example below).

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <CA-DATA>
    server: https://default.kubernetes.svc.cluster.local
  name: garden
contexts:
- context:
    cluster: garden
    user: garden
  name: garden
current-context: garden
users:
- name: garden
  user:
    tokenFile: /var/run/secrets/projected/serviceaccount/token

This will allow for automatic rotation of the service account token by the kubelet. The configuration can be achieved by setting both .Values.global.serviceAccountTokenVolumeProjection.enabled: true and .Values.global.kubeconfig in the respective chart’s values.yaml file.

Virtual Garden is used, i.e., the runtime Garden cluster is different from the target Garden cluster.

Service Account

The easiest way to setup the authentication will be to create a service account and the respective roles will be bound to this service account in the target cluster. Then use the generated service account token and craft a kubeconfig which will be used by the workload in the runtime cluster. This approach does not provide a solution for the rotation of the service account token. However, this setup can be achieved by setting .Values.global.virtualGarden.enabled: true and following these steps:

1. Deploy the application part of the charts in the target cluster.
2. Get the service account token and craft the kubeconfig.
3. Set the crafted kubeconfig and deploy the runtime part of the charts in the runtime cluster.

Client Certificate

Another solution will be to bind the roles in the target cluster to a User subject instead of a service account and use a client certificate for authentication. This approach does not provide a solution for the client certificate rotation. However, this setup can be achieved by setting both .Values.global.virtualGarden.enabled: true and .Values.global.virtualGarden.user.name, then following these steps:

1. Generate a client certificate for the target cluster for the respective user.
2. Deploy the application part of the charts in the target cluster.
3. Craft a kubeconfig using the already generated client certificate.
4. Set the crafted kubeconfig and deploy the runtime part of the charts in the runtime cluster.

Projected Service Account Token

This approach requires an already deployed and configured oidc-webhook-authenticator for the target cluster. Also the runtime cluster should be registered as a trusted identity provider in the target cluster. Then projected service accounts tokens from the runtime cluster can be used to authenticate against the target cluster. The needed steps are as follows:

1. Deploy OWA and establish the needed trust.
2. Set .Values.global.virtualGarden.enabled: true and .Values.global.virtualGarden.user.name. Note: username value will depend on the trust configuration, e.g., <prefix>:system:serviceaccount:<namespace>:<serviceaccount>
3. Set .Values.global.serviceAccountTokenVolumeProjection.enabled: true and .Values.global.serviceAccountTokenVolumeProjection.audience. Note: audience value will depend on the trust configuration, e.g., <cliend-id-from-trust-config>.
4. Craft a kubeconfig (see example below).
5. Deploy the application part of the charts in the target cluster.
6. Deploy the runtime part of the charts in the runtime cluster.

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <CA-DATA>
    server: https://virtual-garden.api
  name: virtual-garden
contexts:
- context:
    cluster: virtual-garden
    user: virtual-garden
  name: virtual-garden
current-context: virtual-garden
users:
- name: virtual-garden
  user:
    tokenFile: /var/run/secrets/projected/serviceaccount/token

2 - DNS Names

Request DNS Names in Shoot Clusters

Introduction

Within a shoot cluster, it is possible to request DNS records via the following resource types:

• Ingress
• Service
• DNSEntry

It is necessary that the Gardener installation your shoot cluster runs in is equipped with a shoot-dns-service extension. This extension uses the seed’s dns management infrastructure to maintain DNS names for shoot clusters. Please ask your Gardener operator if the extension is available in your environment.

Shoot Feature Gate

In some Gardener setups the shoot-dns-service extension is not enabled globally and thus must be configured per shoot cluster. Please adapt the shoot specification by the configuration shown below to activate the extension individually.

kind: Shoot
...
spec:
  extensions:
    - type: shoot-dns-service
...

DNS providers, domain scope

Gardener can only manage DNS records on your behalf if you have proper DNS providers in place. Please consult this page for more information.

Request DNS records via Service/Ingress resources

To request a DNS name for an Ingress or Service object in the shoot cluster it must be annotated with the DNS class garden and an annotation denoting the desired DNS names.

Example for an annotated Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    dns.gardener.cloud/dnsnames: '*' # collects domains names from .spec.rules[].host
    dns.gardener.cloud/class: garden
    # If you are delegating the certificate management to Gardener, uncomment the following line
    #cert.gardener.cloud/purpose: managed
  name: test-ingress
  namespace: default
spec:
  rules:
    - host: test.ingress.my-dns-domain.com
      http:
        paths:
          - backend:
              service:
                name: my-service
                port:
                  number: 9000
            path: /
            pathType: Prefix
  tls:
    - hosts:
        - test.ingress.my-dns-domain.com
      secretName: my-cert-secret-name

For a Service (it must have the type LoadBalancer) this looks like this:

apiVersion: v1
kind: Service
metadata:
  annotations:
    dns.gardener.cloud/class: garden
    dns.gardener.cloud/dnsnames: my.subdomain.for.some.domain.cloud
  name: my-service
  namespace: default
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  type: LoadBalancer

The dnsnames annotation accepts a comma-separated list of DNS names, if multiple names are required.

For an Ingress, the dns names are already declared in the specification. Nevertheless the dnsnames annotation must be present. Here a subset of the dns names of the ingress can be specified. If DNS names for all names are desired, the value all can be used.

If one of the accepted dns names is a direct subname of the shoot’s ingress domain, this is already handled by the standard wildcard entry for the ingress domain. Therefore this name should be excluded from the dnsnames list in the annotation. If only this dns name is configured in the ingress, no explicit dns entry is required, and the dns annotations should be omitted at all.

More examples can be found here

Request DNS records via DNSEntry resources

apiVersion: dns.gardener.cloud/v1alpha1
kind: DNSEntry
metadata:
  annotations:
    dns.gardener.cloud/class: garden
  name: dns
  namespace: default
spec:
  dnsName: "my.subdomain.for.shootsomain.cloud"
  ttl: 600
  # txt records, either text or targets must be specified
# text:
# - foo-bar
  targets:
  # target records (CNAME or A records)
  - 8.8.8.8

DNS record events

The DNS controller publishes Kubernetes events for the resource which requested the DNS record (Ingress, Service, DNSEntry). These events reveal more information about the DNS requests being processed and are especially useful to check any kind of misconfiguration, e.g. requests for a domain you don’t own.

Events for a successfully created DNS record:

$ kubectl -n default describe service my-service

Events:
  Type    Reason          Age                From                    Message
  ----    ------          ----               ----                    -------
  Normal  dns-annotation  19s                dns-controller-manager  my.subdomain.for.shootsomain.cloud: dns entry is pending
  Normal  dns-annotation  19s (x3 over 19s)  dns-controller-manager  my.subdomain.for.shootsomain.cloud: dns entry pending: waiting for dns reconciliation
  Normal  dns-annotation  9s (x3 over 10s)   dns-controller-manager  my.subdomain.for.shootsomain.cloud: dns entry active

Please note, events vanish after their retention period (usually 1h).

DNSEntry status

DNSEntry resources offer a .status sub-resource which can be used to check the current state of the object.

Status of a erroneous DNSEntry.

  status:
    message: No responsible provider found
    observedGeneration: 3
    provider: remote
    state: Error

3 - DNS Providers

DNS Providers

Introduction

Gardener can manage DNS records on your behalf, so that you can request them via different resource types (see here) within the shoot cluster. The domains for which you are permitted to request records, are however restricted and depend on the DNS provider configuration.

Shoot provider

By default, every shoot cluster is equipped with a default provider. It is the very same provider that manages the shoot cluster’s kube-apiserver public DNS record (DNS address in your Kubeconfig).

kind: Shoot
...
dns:
  domain: shoot.project.default-domain.gardener.cloud

You are permitted to request any sub-domain of .dns.domain that is not already taken (e.g. api.shoot.project.default-domain.gardener.cloud, *.ingress.shoot.project.default-domain.gardener.cloud) with this provider.

Additional providers

If you need to request DNS records for domains not managed by the default provider, additional providers can be configured in the shoot specification. Alternatively, if it is enabled, it can be added as DNSProvider resources to the shoot cluster.

Additional providers in the shoot specification

To add a providers in the shoot spec, you need set them in the spec.dns.providers list.

For example:

kind: Shoot
...
spec:
  dns:
    domain: shoot.project.default-domain.gardener.cloud
    providers:
    - secretName: my-aws-account
      type: aws-route53
    - secretName: my-gcp-account
      type: google-clouddns

Please consult the API-Reference to get a complete list of supported fields and configuration options.

Referenced secrets should exist in the project namespace in the Garden cluster and must comply with the provider specific credentials format. The External-DNS-Management project provides corresponding examples (20-secret-<provider-name>-credentials.yaml) for known providers.

Additional providers as resources in the shoot cluster

If it is not enabled globally, you have to enable the feature in the shoot manifest:

Kind: Shoot
...
spec:
  extensions:
    - type: shoot-dns-service
      providerConfig:
        apiVersion: service.dns.extensions.gardener.cloud/v1alpha1
        kind: DNSConfig
        dnsProviderReplication:
          enabled: true
...

To add a provider directly in the shoot cluster, provide a DNSProvider in any namespace together with Secret containing the credentials.

For example if the domain is hosted with AWS Route 53 (provider type aws-route53):

apiVersion: dns.gardener.cloud/v1alpha1
kind: DNSProvider
metadata:
  annotations:
    dns.gardener.cloud/class: garden
  name: my-own-domain
  namespace: my-namespace
spec:
  type: aws-route53
  secretRef:
    name: my-own-domain-credentials
  domains:
    include:
    - my.own.domain.com
---
apiVersion: v1
kind: Secret
metadata:
  name: my-own-domain-credentials
  namespace: my-namespace
type: Opaque
data:
  # replace '...' with values encoded as base64
  AWS_ACCESS_KEY_ID: ...
  AWS_SECRET_ACCESS_KEY: ...

The External-DNS-Management project provides examples with more details for DNSProviders (30-provider-<provider-name>.yaml) and credential Secrets (20-secret-<provider-name>.yaml) at https://github.com/gardener/external-dns-management//examples for all supported provider types.

4 - Setup

Gardener DNS Management for Shoots

Introduction

Gardener allows Shoot clusters to request DNS names for Ingresses and Services out of the box. To support this the gardener must be installed with the shoot-dns-service extension. This extension uses the seed’s dns management infrastructure to maintain DNS names for shoot clusters. So, far only the external DNS domain of a shoot (already used for the kubernetes api server and ingress DNS names) can be used for managed DNS names.

Configuration

A general description for configuring the DNS management of the gardener can be found here.

To generally enable the DNS management for shoot objects the shoot-dns-service extension must be registered by providing an appropriate extension registration in the garden cluster.

Here it is possible to decide whether the extension should be always available for all shoots or whether the extension must be separately enabled per shoot.

If the extension should be used for all shoots the registration must set the globallyEnabled flag to true.

spec:
  resources:
    - kind: Extension
      type: shoot-dns-service
      globallyEnabled: true

Providing Base Domains usable for a Shoot

So, far only the external DNS domain of a shoot already used for the kubernetes api server and ingress DNS names can be used for managed DNS names. This is either the shoot domain as subdomain of the default domain configured for the gardener installation, or a dedicated domain with dedicated access credentials configured for a dedicated shoot via the shoot manifest.

Alternatively, you can specify DNSProviders and its credentials Secret directly in the shoot, if this feature is enabled. By default, DNSProvider replication is disabled, but it can be enabled globally in the ControllerDeployment or for a shoot cluster in the shoot manifest (details see further below).

apiVersion: core.gardener.cloud/v1beta1
kind: ControllerDeployment
metadata:
  name: extension-shoot-dns-service
type: helm
providerConfig:
  chart: ...
  values:
    image:
      ...
    dnsProviderReplication:
      enabled: true

See example files (20-* and 30-*) for details for the various provider types.

Shoot Feature Gate

If the shoot DNS feature is not globally enabled by default (depends on the extension registration on the garden cluster), it must be enabled per shoot.

To enable the feature for a shoot, the shoot manifest must explicitly add the shoot-dns-service extension.

...
spec:
  extensions:
    - type: shoot-dns-service
...

Enable/disable DNS provider replication for a shoot

The DNSProvider` replication feature enablement can be overwritten in the shoot manifest, e.g.

Kind: Shoot
...
spec:
  extensions:
    - type: shoot-dns-service
      providerConfig:
        apiVersion: service.dns.extensions.gardener.cloud/v1alpha1
        kind: DNSConfig
        dnsProviderReplication:
          enabled: true
...

DNSActivation for DNSOwner

To support migration of the control plane of shoots, the DNSOwner created and used for all DNSEntries created by the Shoot-DNS-Service can optionally be activated and deactivated by a DNS record. If the DNSActivation feature is enabled, the DNSOwner will be only be active if the value of the owner DNS record managed by Gardener matches the cluster identity of the seed hosting the control plane. This feature must only be enabled if the Gardener feature gate UseDNSRecords is enabled for all seeds.

By default this feature is enabled and can be disabled in the controller deployment:

apiVersion: core.gardener.cloud/v1beta1
kind: ControllerDeployment
metadata:
  name: extension-shoot-dns-service
type: helm
providerConfig:
  chart: ...
  values:
    image:
      ...
    ownerDnsActivation:
      enabled: false