This is the multi-page printable view of this section. Click here to print.
Networking
1 - Enable IPv4/IPv6 (dual-stack) Ingress on AWS
Using IPv4/IPv6 (dual-stack) Ingress in an IPv4 single-stack cluster
Motivation
IPv6 adoption is continuously growing, already overtaking IPv4 in certain regions, e.g. India, or scenarios, e.g. mobile. Even though most IPv6 installations deploy means to reach IPv4, it might still be beneficial to expose services natively via IPv4 and IPv6 instead of just relying on IPv4.
Disadvantages of full IPv4/IPv6 (dual-stack) Deployments
Enabling full IPv4/IPv6 (dual-stack) support in a kubernetes cluster is a major endeavor. It requires a lot of changes and restarts of all pods so that all pods get addresses for both IP families. A side-effect of dual-stack networking is that failures may be hidden as network traffic may take the other protocol to reach the target. For this reason and also due to reduced operational complexity, service teams might lean towards staying in a single-stack environment as much as possible. Luckily, this is possible with Gardener and IPv4/IPv6 (dual-stack) ingress on AWS.
Simplifying IPv4/IPv6 (dual-stack) Ingress with Protocol Translation on AWS
Fortunately, the network load balancer on AWS supports automatic protocol translation, i.e. it can expose both IPv4 and IPv6 endpoints while communicating with just one protocol to the backends. Under the hood, automatic protocol translation takes place. Client IP address preservation can be achieved by using proxy protocol.
This approach enables users to expose IPv4 workload to IPv6-only clients without having to change the workload/service. Without requiring invasive changes, it allows a fairly simple first step into the IPv6 world for services just requiring ingress (incoming) communication.
Necessary Shoot Cluster Configuration Changes for IPv4/IPv6 (dual-stack) Ingress
To be able to utilize IPv4/IPv6 (dual-stack) Ingress in an IPv4 shoot cluster, the cluster needs to meet two preconditions:
dualStack.enabled
needs to be set totrue
to configure VPC/subnet for IPv6 and add a routing rule for IPv6. (This does not add IPv6 addresses to kubernetes nodes.)loadBalancerController.enabled
needs to be set totrue
as well to use the load balancer controller, which supports dual-stack ingress.
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
...
spec:
provider:
type: aws
infrastructureConfig:
apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
kind: InfrastructureConfig
dualStack:
enabled: true
controlPlaneConfig:
apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
kind: ControlPlaneConfig
loadBalancerController:
enabled: true
...
When infrastructureConfig.networks.vpc.id
is set to the ID of an existing VPC, please make sure that your VPC has an Amazon-provided IPv6 CIDR block added.
After adapting the shoot specification and reconciling the cluster, dual-stack load balancers can be created using kubernetes services objects.
Creating an IPv4/IPv6 (dual-stack) Ingress
With the preconditions set, creating an IPv4/IPv6 load balancer is as easy as annotating a service with the correct annotations:
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-ip-address-type: dualstack
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
service.beta.kubernetes.io/aws-load-balancer-type: external
name: ...
namespace: ...
spec:
...
type: LoadBalancer
In case the client IP address should be preserved, the following annotation can be used to enable proxy protocol. (The pod receiving the traffic needs to be configured for proxy protocol as well.)
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
Please note that changing an existing Service
to dual-stack may cause the creation of a new load balancer without
deletion of the old AWS load balancer resource. While this helps in a seamless migration by not cutting existing
connections it may lead to wasted/forgotten resources. Therefore, the (manual) cleanup needs to be taken into account
when migrating an existing Service
instance.
For more details see AWS Load Balancer Documentation - Network Load Balancer.
DNS Considerations to Prevent Downtime During a Dual-Stack Migration
In case the migration of an existing service is desired, please check if there are DNS entries directly linked to the corresponding load balancer. The migrated load balancer will have a new domain name immediately, which will not be ready in the beginning. Therefore, a direct migration of the domain name entries is not desired as it may cause a short downtime, i.e. domain name entries without backing IP addresses.
If there are DNS entries directly linked to the corresponding load balancer and they are managed by the
shoot-dns-service, you can identify this via
annotations with the prefix dns.gardener.cloud/
. Those annotations can be linked to a Service
, Ingress
or
Gateway
resources. Alternatively, they may also use DNSEntry
or DNSAnnotation
resources.
For a seamless migration without downtime use the following three step approach:
- Temporarily prevent direct DNS updates
- Migrate the load balancer and wait until it is operational
- Allow DNS updates again
To prevent direct updates of the DNS entries when the load balancer is migrated add the annotation
dns.gardener.cloud/ignore: 'true'
to all affected resources next to the other dns.gardener.cloud/...
annotations
before starting the migration. For example, in case of a Service
ensure that the service looks like the following:
kind: Service
metadata:
annotations:
dns.gardener.cloud/ignore: 'true'
dns.gardener.cloud/class: garden
dns.gardener.cloud/dnsnames: '...'
...
Next, migrate the load balancer to be dual-stack enabled by adding/changing the corresponding annotations.
You have multiple options how to check that the load balancer has been provisioned successfully. It might be useful
to peek into status.loadBalancer.ingress
of the corresponding Service
to identify the load balancer:
- Check in the AWS console for the corresponding load balancer provisioning state
- Perform domain name lookups with
nslookup
/dig
to check whether the name resolves to an IP address. - Call your workload via the new load balancer, e.g. using
curl --resolve <my-domain-name>:<port>:<IP-address> https://<my-domain-name>:<port>
, which allows you to call your service with the “correct” domain name without using actual name resolution. - Wait a fixed period of time as load balancer creation is usually finished within 15 minutes
Once the load balancer has been provisioned, you can remove the annotation dns.gardener.cloud/ignore: 'true'
again
from the affected resources. It may take some additional time until the domain name change finally propagates
(up to one hour).
2 - Manage Certificates with Gardener
Manage certificates with Gardener for public domain
Introduction
Dealing with applications on Kubernetes which offer a secure service endpoints (e.g. HTTPS) also require you to enable a secured communication via SSL/TLS. With the certificate extension enabled, Gardener can manage commonly trusted X.509 certificate for your application endpoint. From initially requesting certificate, it also handeles their renewal in time using the free Let’s Encrypt API.
There are two senarios with which you can use the certificate extension
- You want to use a certificate for a subdomain the shoot’s default DNS (see
.spec.dns.domain
of your shoot resource, e.g.short.ingress.shoot.project.default-domain.gardener.cloud
). If this is your case, please see Manage certificates with Gardener for default domain - You want to use a certificate for a custom domain. If this is your case, please keep reading this article.
Prerequisites
Before you start this guide there are a few requirements you need to fulfill:
- You have an existing shoot cluster
- Your custom domain is under a public top level domain (e.g.
.com
) - Your custom zone is resolvable with a public resolver via the internet (e.g.
8.8.8.8
) - You have a custom DNS provider configured and working (see “DNS Providers”)
As part of the Let’s Encrypt ACME challenge validation process, Gardener sets a DNS TXT entry and Let’s Encrypt checks if it can both resolve and authenticate it. Therefore, it’s important that your DNS-entries are publicly resolvable. You can check this by querying e.g. Googles public DNS server and if it returns an entry your DNS is publicly visible:
# returns the A record for cert-example.example.com using Googles DNS server (8.8.8.8)
dig cert-example.example.com @8.8.8.8 A
DNS provider
In order to issue certificates for a custom domain you need to specify a DNS provider which is permitted to create DNS records for subdomains of your requested domain in the certificate. For example, if you request a certificate for host.example.com
your DNS provider must be capable of managing subdomains of host.example.com
.
DNS providers are normally specified in the shoot manifest. To learn more on how to configure one, please see the DNS provider documentation.
Issue a certificate
Every X.509 certificate is represented by a Kubernetes custom resource certificate.cert.gardener.cloud
in your cluster. A Certificate
resource may be used to initiate a new certificate request as well as to manage its lifecycle. Gardener’s certificate service regularly checks the expiration timestamp of Certificates, triggers a renewal process if necessary and replaces the existing X.509 certificate with a new one.
Your application should be able to reload replaced certificates in a timely manner to avoid service disruptions.
Certificates can be requested via 3 resources type
- Ingress
- Service (type LoadBalancer)
- Gateways (both Istio gateways and from the Gateway API)
- Certificate (Gardener CRD)
If either of the first 2 are used, a corresponding Certificate
resource will be created automatically.
Using an Ingress Resource
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: amazing-ingress
annotations:
cert.gardener.cloud/purpose: managed
# Optional but recommended, this is going to create the DNS entry at the same time
dns.gardener.cloud/class: garden
dns.gardener.cloud/ttl: "600"
#cert.gardener.cloud/commonname: "*.example.com" # optional, if not specified the first name from spec.tls[].hosts is used as common name
#cert.gardener.cloud/dnsnames: "" # optional, if not specified the names from spec.tls[].hosts are used
#cert.gardener.cloud/follow-cname: "true" # optional, same as spec.followCNAME in certificates
#cert.gardener.cloud/secret-labels: "key1=value1,key2=value2" # optional labels for the certificate secret
#cert.gardener.cloud/issuer: custom-issuer # optional to specify custom issuer (use namespace/name for shoot issuers)
#cert.gardener.cloud/preferred-chain: "chain name" # optional to specify preferred-chain (value is the Subject Common Name of the root issuer)
#cert.gardener.cloud/private-key-algorithm: ECDSA # optional to specify algorithm for private key, allowed values are 'RSA' or 'ECDSA'
#cert.gardener.cloud/private-key-size: "384" # optional to specify size of private key, allowed values for RSA are "2048", "3072", "4096" and for ECDSA "256" and "384"
spec:
tls:
- hosts:
# Must not exceed 64 characters.
- amazing.example.com
# Certificate and private key reside in this secret.
secretName: tls-secret
rules:
- host: amazing.example.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: amazing-svc
port:
number: 8080
Replace the hosts
and rules[].host
value again with your own domain and adjust the remaining Ingress attributes in accordance with your deployment (e.g. the above is for an istio
Ingress controller and forwards traffic to a service1
on port 80).
Using a Service of type LoadBalancer
apiVersion: v1
kind: Service
metadata:
annotations:
cert.gardener.cloud/secretname: tls-secret
dns.gardener.cloud/dnsnames: example.example.com
dns.gardener.cloud/class: garden
# Optional
dns.gardener.cloud/ttl: "600"
cert.gardener.cloud/commonname: "*.example.example.com"
cert.gardener.cloud/dnsnames: ""
#cert.gardener.cloud/follow-cname: "true" # optional, same as spec.followCNAME in certificates
#cert.gardener.cloud/secret-labels: "key1=value1,key2=value2" # optional labels for the certificate secret
#cert.gardener.cloud/issuer: custom-issuer # optional to specify custom issuer (use namespace/name for shoot issuers)
#cert.gardener.cloud/preferred-chain: "chain name" # optional to specify preferred-chain (value is the Subject Common Name of the root issuer)
#cert.gardener.cloud/private-key-algorithm: ECDSA # optional to specify algorithm for private key, allowed values are 'RSA' or 'ECDSA'
#cert.gardener.cloud/private-key-size: "384" # optional to specify size of private key, allowed values for RSA are "2048", "3072", "4096" and for ECDSA "256" and "384"
name: test-service
namespace: default
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
type: LoadBalancer
Using a Gateway resource
Please see Istio Gateways or Gateway API for details.
Using the custom Certificate resource
apiVersion: cert.gardener.cloud/v1alpha1
kind: Certificate
metadata:
name: cert-example
namespace: default
spec:
commonName: amazing.example.com
secretRef:
name: tls-secret
namespace: default
# Optionnal if using the default issuer
issuerRef:
name: garden
# If delegated domain for DNS01 challenge should be used. This has only an effect if a CNAME record is set for
# '_acme-challenge.amazing.example.com'.
# For example: If a CNAME record exists '_acme-challenge.amazing.example.com' => '_acme-challenge.writable.domain.com',
# the DNS challenge will be written to '_acme-challenge.writable.domain.com'.
#followCNAME: true
# optionally set labels for the secret
#secretLabels:
# key1: value1
# key2: value2
# Optionally specify the preferred certificate chain: if the CA offers multiple certificate chains, prefer the chain with an issuer matching this Subject Common Name. If no match, the default offered chain will be used.
#preferredChain: "ISRG Root X1"
# Optionally specify algorithm and key size for private key. Allowed algorithms: "RSA" (allowed sizes: 2048, 3072, 4096) and "ECDSA" (allowed sizes: 256, 384)
# If not specified, RSA with 2048 is used.
#privateKey:
# algorithm: ECDSA
# size: 384
Supported attributes
Here is a list of all supported annotations regarding the certificate extension:
Path | Annotation | Value | Required | Description |
---|---|---|---|---|
N/A | cert.gardener.cloud/purpose: | managed | Yes when using annotations | Flag for Gardener that this specific Ingress or Service requires a certificate |
spec.commonName | cert.gardener.cloud/commonname: | E.g. “*.demo.example.com” or “special.example.com” | Certificate and Ingress : No Service: Yes, if DNS names unset | Specifies for which domain the certificate request will be created. If not specified, the names from spec.tls[].hosts are used. This entry must comply with the 64 character limit. |
spec.dnsNames | cert.gardener.cloud/dnsnames: | E.g. “special.example.com” | Certificate and Ingress : No Service: Yes, if common name unset | Additional domains the certificate should be valid for (Subject Alternative Name). If not specified, the names from spec.tls[].hosts are used. Entries in this list can be longer than 64 characters. |
spec.secretRef.name | cert.gardener.cloud/secretname: | any-name | Yes for certificate and Service | Specifies the secret which contains the certificate/key pair. If the secret is not available yet, it’ll be created automatically as soon as the certificate has been issued. |
spec.issuerRef.name | cert.gardener.cloud/issuer: | E.g. gardener | No | Specifies the issuer you want to use. Only necessary if you request certificates for custom domains. |
N/A | cert.gardener.cloud/revoked: | true otherwise always false | No | Use only to revoke a certificate, see reference for more details |
spec.followCNAME | cert.gardener.cloud/follow-cname | E.g. true | No | Specifies that the usage of a delegated domain for DNS challenges is allowed. Details see Follow CNAME. |
spec.preferredChain | cert.gardener.cloud/preferred-chain | E.g. ISRG Root X1 | No | Specifies the Common Name of the issuer for selecting the certificate chain. Details see Preferred Chain. |
spec.secretLabels | cert.gardener.cloud/secret-labels | for annotation use e.g. key1=value1,key2=value2 | No | Specifies labels for the certificate secret. |
spec.privateKey.algorithm | cert.gardener.cloud/private-key-algorithm | RSA , ECDSA | No | Specifies algorithm for private key generation. The default value is depending on configuration of the extension (default of the default is RSA ). You may request a new certificate without privateKey settings to find out the concrete defaults in your Gardener. |
spec.privateKey.size | cert.gardener.cloud/private-key-size | "256" , "384" , "2048" , "3072" , "4096" | No | Specifies size for private key generation. Allowed values for RSA are 2048 , 3072 , and 4096 . For ECDSA allowed values are 256 and 384 . The default values are depending on the configuration of the extension (defaults of the default values are 3072 for RSA and 384 for ECDSA respectively). |
Request a wildcard certificate
In order to avoid the creation of multiples certificates for every single endpoints, you may want to create a wildcard certificate for your shoot’s default cluster.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: amazing-ingress
annotations:
cert.gardener.cloud/purpose: managed
cert.gardener.cloud/commonName: "*.example.com"
spec:
tls:
- hosts:
- amazing.example.com
secretName: tls-secret
rules:
- host: amazing.example.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: amazing-svc
port:
number: 8080
Please note that this can also be achived by directly adding an annotation to a Service type LoadBalancer. You could also create a Certificate object with a wildcard domain.
Using a custom Issuer
Most Gardener deployment with the certification extension enabled have a preconfigured garden
issuer. It is also usually configured to use Let’s Encrypt as the certificate provider.
If you need a custom issuer for a specific cluster, please see Using a custom Issuer
Quotas
For security reasons there may be a default quota on the certificate requests per day set globally in the controller registration of the shoot-cert-service.
The default quota only applies if there is no explicit quota defined for the issuer itself with the field
requestsPerDayQuota
, e.g.:
kind: Shoot
...
spec:
extensions:
- type: shoot-cert-service
providerConfig:
apiVersion: service.cert.extensions.gardener.cloud/v1alpha1
kind: CertConfig
issuers:
- email: your-email@example.com
name: custom-issuer # issuer name must be specified in every custom issuer request, must not be "garden"
server: 'https://acme-v02.api.letsencrypt.org/directory'
requestsPerDayQuota: 10
DNS Propagation
As stated before, cert-manager uses the ACME challenge protocol to authenticate that you are the DNS owner for the domain’s certificate you are requesting.
This works by creating a DNS TXT record in your DNS provider under _acme-challenge.example.example.com
containing a token to compare with. The TXT record is only applied during the domain validation.
Typically, the record is propagated within a few minutes. But if the record is not visible to the ACME server for any reasons, the certificate request is retried again after several minutes.
This means you may have to wait up to one hour after the propagation problem has been resolved before the certificate request is retried. Take a look in the events with kubectl describe ingress example
for troubleshooting.
Character Restrictions
Due to restriction of the common name to 64 characters, you may to leave the common name unset in such cases.
For example, the following request is invalid:
apiVersion: cert.gardener.cloud/v1alpha1
kind: Certificate
metadata:
name: cert-invalid
namespace: default
spec:
commonName: morethan64characters.ingress.shoot.project.default-domain.gardener.cloud
But it is valid to request a certificate for this domain if you have left the common name unset:
apiVersion: cert.gardener.cloud/v1alpha1
kind: Certificate
metadata:
name: cert-example
namespace: default
spec:
dnsNames:
- morethan64characters.ingress.shoot.project.default-domain.gardener.cloud
References
3 - Manage Certificates with Gardener for Default Domain
Manage certificates with Gardener for default domain
Introduction
Dealing with applications on Kubernetes which offer a secure service endpoints (e.g. HTTPS) also require you to enable a secured communication via SSL/TLS. With the certificate extension enabled, Gardener can manage commonly trusted X.509 certificate for your application endpoint. From initially requesting certificate, it also handeles their renewal in time using the free Let’s Encrypt API.
There are two senarios with which you can use the certificate extension
- You want to use a certificate for a subdomain the shoot’s default DNS (see
.spec.dns.domain
of your shoot resource, e.g.short.ingress.shoot.project.default-domain.gardener.cloud
). If this is your case, please keep reading this article. - You want to use a certificate for a custom domain. If this is your case, please see Manage certificates with Gardener for public domain
Prerequisites
Before you start this guide there are a few requirements you need to fulfill:
- You have an existing shoot cluster
Since you are using the default DNS name, all DNS configuration should already be done and ready.
Issue a certificate
Every X.509 certificate is represented by a Kubernetes custom resource certificate.cert.gardener.cloud
in your cluster. A Certificate
resource may be used to initiate a new certificate request as well as to manage its lifecycle. Gardener’s certificate service regularly checks the expiration timestamp of Certificates, triggers a renewal process if necessary and replaces the existing X.509 certificate with a new one.
Your application should be able to reload replaced certificates in a timely manner to avoid service disruptions.
Certificates can be requested via 3 resources type
- Ingress
- Service (type LoadBalancer)
- certificate (Gardener CRD)
If either of the first 2 are used, a corresponding Certificate
resource will automatically be created.
Using an ingress Resource
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: amazing-ingress
annotations:
cert.gardener.cloud/purpose: managed
#cert.gardener.cloud/issuer: custom-issuer # optional to specify custom issuer (use namespace/name for shoot issuers)
#cert.gardener.cloud/follow-cname: "true" # optional, same as spec.followCNAME in certificates
#cert.gardener.cloud/secret-labels: "key1=value1,key2=value2" # optional labels for the certificate secret
#cert.gardener.cloud/preferred-chain: "chain name" # optional to specify preferred-chain (value is the Subject Common Name of the root issuer)
#cert.gardener.cloud/private-key-algorithm: ECDSA # optional to specify algorithm for private key, allowed values are 'RSA' or 'ECDSA'
#cert.gardener.cloud/private-key-size: "384" # optional to specify size of private key, allowed values for RSA are "2048", "3072", "4096" and for ECDSA "256" and "384"spec:
tls:
- hosts:
# Must not exceed 64 characters.
- short.ingress.shoot.project.default-domain.gardener.cloud
# Certificate and private key reside in this secret.
secretName: tls-secret
rules:
- host: short.ingress.shoot.project.default-domain.gardener.cloud
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: amazing-svc
port:
number: 8080
Using a service type LoadBalancer
apiVersion: v1
kind: Service
metadata:
annotations:
cert.gardener.cloud/purpose: managed
# Certificate and private key reside in this secret.
cert.gardener.cloud/secretname: tls-secret
# You may add more domains separated by commas (e.g. "service.shoot.project.default-domain.gardener.cloud, amazing.shoot.project.default-domain.gardener.cloud")
dns.gardener.cloud/dnsnames: "service.shoot.project.default-domain.gardener.cloud"
dns.gardener.cloud/ttl: "600"
#cert.gardener.cloud/issuer: custom-issuer # optional to specify custom issuer (use namespace/name for shoot issuers)
#cert.gardener.cloud/follow-cname: "true" # optional, same as spec.followCNAME in certificates
#cert.gardener.cloud/secret-labels: "key1=value1,key2=value2" # optional labels for the certificate secret
#cert.gardener.cloud/preferred-chain: "chain name" # optional to specify preferred-chain (value is the Subject Common Name of the root issuer)
#cert.gardener.cloud/private-key-algorithm: ECDSA # optional to specify algorithm for private key, allowed values are 'RSA' or 'ECDSA'
#cert.gardener.cloud/private-key-size: "384" # optional to specify size of private key, allowed values for RSA are "2048", "3072", "4096" and for ECDSA "256" and "384" name: test-service
namespace: default
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
type: LoadBalancer
Using the custom Certificate resource
apiVersion: cert.gardener.cloud/v1alpha1
kind: Certificate
metadata:
name: cert-example
namespace: default
spec:
commonName: short.ingress.shoot.project.default-domain.gardener.cloud
secretRef:
name: tls-secret
namespace: default
# Optionnal if using the default issuer
issuerRef:
name: garden
If you’re interested in the current progress of your request, you’re advised to consult the description, more specifically the status
attribute in case the issuance failed.
Request a wildcard certificate
In order to avoid the creation of multiples certificates for every single endpoints, you may want to create a wildcard certificate for your shoot’s default cluster.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: amazing-ingress
annotations:
cert.gardener.cloud/purpose: managed
cert.gardener.cloud/commonName: "*.ingress.shoot.project.default-domain.gardener.cloud"
spec:
tls:
- hosts:
- amazing.ingress.shoot.project.default-domain.gardener.cloud
secretName: tls-secret
rules:
- host: amazing.ingress.shoot.project.default-domain.gardener.cloud
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: amazing-svc
port:
number: 8080
Please note that this can also be achived by directly adding an annotation to a Service type LoadBalancer. You could also create a Certificate object with a wildcard domain.
More information
For more information and more examples about using the certificate extension, please see Manage certificates with Gardener for public domain
4 - Managing DNS with Gardener
Request DNS Names in Shoot Clusters
Introduction
Within a shoot cluster, it is possible to request DNS records via the following resource types:
It is necessary that the Gardener installation your shoot cluster runs in is equipped with a shoot-dns-service
extension. This extension uses the seed’s dns management infrastructure to maintain DNS names for shoot clusters. Please ask your Gardener operator if the extension is available in your environment.
Shoot Feature Gate
In some Gardener setups the shoot-dns-service
extension is not enabled globally and thus must be configured per shoot cluster. Please adapt the shoot specification by the configuration shown below to activate the extension individually.
kind: Shoot
...
spec:
extensions:
- type: shoot-dns-service
...
Before you start
You should :
- Have created a shoot cluster
- Have created and correctly configured a DNS Provider (Please consult this page for more information)
- Have a basic understanding of DNS (see link under References)
There are 2 types of DNS that you can use within Kubernetes :
- internal (usually managed by coreDNS)
- external (managed by a public DNS provider).
This page, and the extension, exclusively works for external DNS handling.
Gardener allows 2 way of managing your external DNS:
- Manually, which means you are in charge of creating / maintaining your Kubernetes related DNS entries
- Via the Gardener DNS extension
Gardener DNS extension
The managed external DNS records feature of the Gardener clusters makes all this easier. You do not need DNS service provider specific knowledge, and in fact you do not need to leave your cluster at all to achieve that. You simply annotate the Ingress / Service that needs its DNS records managed and it will be automatically created / managed by Gardener.
Managed external DNS records are supported with the following DNS provider types:
- aws-route53
- azure-dns
- azure-private-dns
- google-clouddns
- openstack-designate
- alicloud-dns
- cloudflare-dns
Request DNS records for Ingress resources
To request a DNS name for Ingress
, Service
or Gateway
(Istio or Gateway API) objects in the shoot cluster it must be annotated with the DNS class garden
and an annotation denoting the desired DNS names.
Example for an annotated Ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: amazing-ingress
annotations:
# Let Gardener manage external DNS records for this Ingress.
dns.gardener.cloud/dnsnames: special.example.com # Use "*" to collects domains names from .spec.rules[].host
dns.gardener.cloud/ttl: "600"
dns.gardener.cloud/class: garden
# If you are delegating the certificate management to Gardener, uncomment the following line
#cert.gardener.cloud/purpose: managed
spec:
rules:
- host: special.example.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: amazing-svc
port:
number: 8080
# Uncomment the following part if you are delegating the certificate management to Gardener
#tls:
# - hosts:
# - special.example.com
# secretName: my-cert-secret-name
For an Ingress, the DNS names are already declared in the specification. Nevertheless the dnsnames annotation must be present. Here a subset of the DNS names of the ingress can be specified. If DNS names for all names are desired, the value all
can be used.
Keep in mind that ingress resources are ignored unless an ingress controller is set up. Gardener does not provide an ingress controller by default. For more details, see Ingress Controllers and Service in the Kubernetes documentation.
Request DNS records for service type LoadBalancer
Example for an annotated Service (it must have the type LoadBalancer
) resource:
apiVersion: v1
kind: Service
metadata:
name: amazing-svc
annotations:
# Let Gardener manage external DNS records for this Service.
dns.gardener.cloud/dnsnames: special.example.com
dns.gardener.cloud/ttl: "600"
dns.gardener.cloud/class: garden
spec:
selector:
app: amazing-app
ports:
- protocol: TCP
port: 80
targetPort: 8080
type: LoadBalancer
Request DNS records for Gateway resources
Please see Istio Gateways or Gateway API for details.
Creating a DNSEntry resource explicitly
It is also possible to create a DNS entry via the Kubernetes resource called DNSEntry
:
apiVersion: dns.gardener.cloud/v1alpha1
kind: DNSEntry
metadata:
annotations:
# Let Gardener manage this DNS entry.
dns.gardener.cloud/class: garden
name: special-dnsentry
namespace: default
spec:
dnsName: special.example.com
ttl: 600
targets:
- 1.2.3.4
If one of the accepted DNS names is a direct subname of the shoot’s ingress domain, this is already handled by the standard wildcard entry for the ingress domain. Therefore this name should be excluded from the dnsnames list in the annotation. If only this DNS name is configured in the ingress, no explicit DNS entry is required, and the DNS annotations should be omitted at all.
You can check the status of the DNSEntry
with
$ kubectl get dnsentry
NAME DNS TYPE PROVIDER STATUS AGE
mydnsentry special.example.com aws-route53 default/aws Ready 24s
As soon as the status of the entry is Ready
, the provider has accepted the new DNS record. Depending on the provider and your DNS settings and cache, it may take up to 24 hours for the new entry to be propagated over all internet.
More examples can be found here
Request DNS records for Service/Ingress resources using a DNSAnnotation resource
In rare cases it may not be possible to add annotations to a Service
or Ingress
resource object.
E.g.: the helm chart used to deploy the resource may not be adaptable for some reasons or some automation is used, which always restores the original content of the resource object by dropping any additional annotations.
In these cases, it is recommended to use an additional DNSAnnotation
resource in order to have more flexibility that DNSentry resources
. The DNSAnnotation
resource makes the DNS shoot service behave as if annotations have been added to the referenced resource.
For the Ingress example shown above, you can create a DNSAnnotation
resource alternatively to provide the annotations.
apiVersion: dns.gardener.cloud/v1alpha1
kind: DNSAnnotation
metadata:
annotations:
dns.gardener.cloud/class: garden
name: test-ingress-annotation
namespace: default
spec:
resourceRef:
kind: Ingress
apiVersion: networking.k8s.io/v1
name: test-ingress
namespace: default
annotations:
dns.gardener.cloud/dnsnames: '*'
dns.gardener.cloud/class: garden
Note that the DNSAnnotation resource itself needs the dns.gardener.cloud/class=garden
annotation. This also only works for annotations known to the DNS shoot service (see Accepted External DNS Records Annotations).
For more details, see also DNSAnnotation objects
Accepted External DNS Records Annotations
Here are all of the accepted annotation related to the DNS extension:
Annotation | Description |
---|---|
dns.gardener.cloud/dnsnames | Mandatory for service and ingress resources, accepts a comma-separated list of DNS names if multiple names are required. For ingress you can use the special value '*' . In this case, the DNS names are collected from .spec.rules[].host . |
dns.gardener.cloud/class | Mandatory, in the context of the shoot-dns-service it must always be set to garden . |
dns.gardener.cloud/ttl | Recommended, overrides the default Time-To-Live of the DNS record. |
dns.gardener.cloud/cname-lookup-interval | Only relevant if multiple domain name targets are specified. It specifies the lookup interval for CNAMEs to map them to IP addresses (in seconds) |
dns.gardener.cloud/realms | Internal, for restricting provider access for shoot DNS entries. Typcially not set by users of the shoot-dns-service. |
dns.gardener.cloud/ip-stack | Only relevant for provider type aws-route53 if target is an AWS load balancer domain name. Can be set for service, ingress and DNSEntry resources. It specify which DNS records with alias targets are created instead of the usual CNAME records. If the annotation is not set (or has the value ipv4 ), only an A record is created. With value dual-stack , both A and AAAA records are created. With value ipv6 only an AAAA record is created. |
service.beta.kubernetes.io/aws-load-balancer-ip-address-type=dualstack | For services, behaves similar to dns.gardener.cloud/ip-stack=dual-stack . |
loadbalancer.openstack.org/load-balancer-address | Internal, for services only: support for PROXY protocol on Openstack (which needs a hostname as ingress). Typcially not set by users of the shoot-dns-service. |
If one of the accepted DNS names is a direct subdomain of the shoot’s ingress domain, this is already handled by the standard wildcard entry for the ingress domain. Therefore, this name should be excluded from the dnsnames list in the annotation. If only this DNS name is configured in the ingress, no explicit DNS entry is required, and the DNS annotations should be omitted at all.
Troubleshooting
General DNS tools
To check the DNS resolution, use the nslookup
or dig
command.
$ nslookup special.your-domain.com
or with dig
$ dig +short special.example.com
Depending on your network settings, you may get a successful response faster using a public DNS server (e.g. 8.8.8.8, 8.8.4.4, or 1.1.1.1)
dig @8.8.8.8 +short special.example.com
DNS record events
The DNS controller publishes Kubernetes events for the resource which requested the DNS record (Ingress, Service, DNSEntry). These events reveal more information about the DNS requests being processed and are especially useful to check any kind of misconfiguration, e.g. requests for a domain you don’t own.
Events for a successfully created DNS record:
$ kubectl describe service my-service
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal dns-annotation 19s dns-controller-manager special.example.com: dns entry is pending
Normal dns-annotation 19s (x3 over 19s) dns-controller-manager special.example.com: dns entry pending: waiting for dns reconciliation
Normal dns-annotation 9s (x3 over 10s) dns-controller-manager special.example.com: dns entry active
Please note, events vanish after their retention period (usually 1h
).
DNSEntry status
DNSEntry
resources offer a .status
sub-resource which can be used to check the current state of the object.
Status of a erroneous DNSEntry
.
status:
message: No responsible provider found
observedGeneration: 3
provider: remote
state: Error