SSL certificates on Kubernetes

Kubernetes carries certificates in a kubernetes.io/tls Secret with exactly two keys: tls.crt and tls.key. The fullchain trap is here too — tls.crt must hold the leaf followed by every intermediate, because the ingress controller sends that file verbatim.

kubectl create secret tls example-com-tls \
  --cert=fullchain.pem --key=example.com.key -n web

The Secret is namespaced. An Ingress in web cannot reference a Secret in default, and the failure is silent — the controller falls back to its own self-signed certificate and every visitor sees a warning.

Let cert-manager do it instead

Hand-managed Secrets have to be replaced every 200 days, and soon every 47. cert-manager runs the ACME loop inside the cluster and rewrites the Secret in place. Install it, then declare one issuer for the whole cluster:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

From there the shortest path is the ingress-shim: annotate the Ingress and cert-manager creates the Certificate, orders it, and fills the Secret named in tls.secretName.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  namespace: web
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  ingressClassName: nginx
  tls:
    - hosts: [example.com, www.example.com]
      secretName: example-com-tls
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: example
                port:
                  number: 80

A wildcard cannot use HTTP-01 at all — it needs a DNS-01 solver with credentials for your DNS provider, and a Certificate resource written out rather than the annotation shortcut.

When it does not issue

cert-manager fails in a chain of resources, and the useful error is always at the far end of it. Walk it in order rather than guessing:

kubectl describe certificate example-com-tls -n web
kubectl describe certificaterequest -n web
kubectl describe order -n web
kubectl describe challenge -n web

HTTP-01 needs port 80 open to the ingress from the public internet. A cluster reachable only on 443 never completes the challenge.
Point at staging while you iterate. Let's Encrypt rate limits are per account and a CrashLoopBackOff can burn a week's budget in an afternoon — https://acme-staging-v02.api.letsencrypt.org/directory costs nothing, and you can run a staging order from here first.
DNS must already resolve to the ingress. cert-manager will retry a failing challenge for a long time without ever saying the record is missing.
Check the Secret, not the Certificate. A Ready Certificate with an empty or stale Secret means the controller wrote somewhere else — usually the wrong namespace.

Verify it from the outside

A Ready=True condition says cert-manager is happy, not that visitors are. The ingress controller has to reload and pick the Secret up, and it may be terminating TLS for several hosts from one process:

kubectl get secret example-com-tls -n web -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -dates

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -dates

If those two disagree, the controller is still serving the old certificate. Or use the Domain SSL Checker, which performs the same handshake from this server and flags an incomplete chain explicitly.

SSL certificates on Kubernetes

Let cert-manager do it instead

When it does not issue

Verify it from the outside

Other platforms