Let’s Encrypt Test Plan

Intent

Validate the package’s optional Let’s Encrypt support in a disposable Big Bang environment using staging, not production. This plan covers package-specific issuer, mesh, policy, and evidence checks, not general cluster/bootstrap setup. ACME requires public DNS/reachability or provider credentials and is intentionally excluded from the default package pipeline.

What CI Covers

The unit suites check default-disabled and upgrade-only issuer behavior, staging/production/custom rendering, validation, and generated networking. The runtime checks verify admission and self-signed issuance, not public ACME, renewal, or external trust distribution. These checks are not evidence of a successful Let’s Encrypt Order in a particular environment.

What Manual ACME Testing Proves

Keep the normal Istio, NetworkPolicy, private-registry, and Kyverno controls enabled. Prove that the staging issuer and Certificate become Ready, the related Order and Challenge reach valid, and solver resources have the necessary network/mesh/provider access without admission rejection. Use a delegated test zone and least-privilege provider credentials.

Recommended Big Bang Test Path

Prefer DNS-01 for repeatable development validation: it does not require a public route into a k3d load balancer. HTTP-01 needs the verified route described below. Neither path creates a fixed cert-manager development subdomain.

Environment prerequisites

Use the umbrella’s development environment tooling or an existing disposable Big Bang cluster, and its deployment guidance for Flux, registry credentials, and bootstrap. Select those references at the Big Bang release under test rather than copying bootstrap commands here.

Confirm kubeconfig, namespace, issuer scope, FQDN, ACME directory, and credential retention before mutation. Use a pushed package ref that Flux can read and keep Registry1 credentials in the environment’s protected/encrypted configuration. Retain all package Iron Bank images, including the temporary ACME solver, to exercise the real registry path. Do not commit provider tokens or account keys.

Install Standard Big Bang and cert-manager

Use the package’s standard or ambient umbrella selection. Select a feature source using the umbrella’s package branch test workflow. The canonical path is packages.certManager with the v1 discriminator, not packages.cert-manager. Do not set a branch and tag together; clear the inherited tag when selecting a branch. In the legacy contract use certManager instead.

Wait for Flux reconciliation and verify the release, deployments, and webhook admission before the issuer upgrade. The selections enable self-signed management with createOnInstall: false; follow the staged procedure. Keep Istio, Kyverno Policies, monitoring, and private-registry handling active. Use only the narrowly scoped policy exceptions required by the chosen development environment, not a blanket cert-manager namespace exemption.

Create the DNS Credential Secret

Create the credential separately after the namespace exists. The illustrative Cloudflare configuration below references cloudflare-api-token with key api-token in cert-manager for a namespaced issuer. Use the provider’s requirements and restrict access to the delegated zone. For other providers or cluster-scoped issuers, verify Secret placement and identity permissions before proceeding.

Enable the Staging DNS-01 Issuer

Apply an uncommitted staging override after the webhook is ready. Replace the example email/zone and provider configuration; this file contains only Secret references, not credentials:

packageConfiguration:
  version: v1
packages:
  certManager:
    values:
      bb-common:
        routes:
          outbound:
            cloudflare-api:
              enabled: true
              hosts:
                - api.cloudflare.com
      issuers:
        letsEncrypt:
          enabled: true
          environment: staging
          email: platform-team@example.mil
          solvers:
            - dns01:
                cloudflare:
                  apiTokenSecretRef:
                    name: cloudflare-api-token
                    key: api-token
              selector:
                dnsZones:
                  - acme-test.example.mil

Use this override last in the environment’s values ordering and reconcile the umbrella upgrade. The chart registers the ACME directory in sidecar mode and adds controller egress when enabled; provider API registration and suitable CIDRs remain environment-specific. Follow network prerequisites. Confirm the intended kubeconfig and wait for the default namespaced staging issuer before the request. These commands inspect/wait for existing resources; they do not install or enable the issuer:

kubectl config current-context
kubectl -n cert-manager wait --for=condition=Ready --timeout=300s \
  issuer/cert-manager-letsencrypt-staging
kubectl -n cert-manager get issuer cert-manager-letsencrypt-staging \
  -o jsonpath='{.spec.acme.server}{"\n"}'

Expect https://acme-staging-v02.api.letsencrypt.org/directory, not production. Adjust the name and namespace for the configured release. For an intentionally cluster-scoped issuer, wait on clusterissuer/<actual-name> instead.

Request and Verify the Certificate

From the package root, run the script against a controlled, non-wildcard name in the delegated zone:

RUN_ID=acme-doc-check \
ACME_TEST_FQDN=cert-manager.acme-test.example.mil \
ACME_ISSUER_NAME=cert-manager-letsencrypt-staging \
ACME_ISSUER_KIND=Issuer \
NAMESPACE=cert-manager \
bash chart/tests/scripts/cert_manager.sh

The script also exercises self-signed issuance and the negative missing-issuer case. For a configured packaged self-signed issuer, set REQUIRE_BUILTIN_ISSUER=true to require it, and set BUILTIN_ISSUER_NAME/BUILTIN_ISSUER_KIND to its identity. For custom releases also set INSTANCE_NAME, WEBHOOK_SERVICE, and WEBHOOK_CA_SECRET as needed; Helm’s test wiring supplies them automatically.

ACME_EXPECTED_SERVER defaults to Let’s Encrypt staging and the script verifies the selected issuer’s ACME directory. Set it explicitly only for an approved custom test server, not production. Use a separate DNS-01 test for wildcard names.

The script creates run-scoped resources and deletes its test Certificates, Secrets, and temporary self-signed issuer on exit, not the packaged issuers or ACME account Secret. Set RUN_ID for a recognizable run and capture evidence while it executes; use a separately managed retained Certificate if durable evidence or renewal testing is needed.

Capture the test Certificate, its CertificateRequest, related Order/Challenge, events, controller/webhook logs, and Kyverno reports. Use the Certificate’s cert-manager.io/test-run label and resource ownership to isolate the run; related resources need not all have the same label. Redact credentials and keys. Investigate admission, DNS propagation, provider access, image pulls, and mesh egress before requesting narrow policy changes. Do not disable Kyverno.

For the script invocation above, use another terminal with the same kubeconfig and run ID while the script executes. These read-only commands inspect its ACME Certificate and show owner names to help locate related resources:

RUN_ID=acme-doc-check
kubectl -n cert-manager get certificates -l "cert-manager.io/test-run=$RUN_ID"
kubectl -n cert-manager describe certificate "test-acme-cert-$RUN_ID"
kubectl -n cert-manager get certificaterequests,orders,challenges \
  -o custom-columns='KIND:.kind,NAME:.metadata.name,OWNER:.metadata.ownerReferences[*].name'

Follow the Certificate -> CertificateRequest -> Order -> Challenge owner chain; do not assume every resource inherited the test label. The owner listing can include other namespace resources: select only those belonging to this run and do not publish unrelated certificate names or DNS metadata. For a failed run, inspect the exact related objects and recent controller errors:

REQUEST_NAME="replace-with-this-run-certificaterequest-name"
ORDER_NAME="replace-with-this-run-order-name"
CHALLENGE_NAME="replace-with-this-run-challenge-name"
kubectl -n cert-manager describe issuer cert-manager-letsencrypt-staging
kubectl -n cert-manager describe certificaterequest "$REQUEST_NAME"
kubectl -n cert-manager describe order "$ORDER_NAME"
kubectl -n cert-manager describe challenge "$CHALLENGE_NAME"
kubectl -n cert-manager logs deployment/cert-manager -c upstream-controller --since=10m --tail=200

Adjust the Deployment/container for a custom release and inspect admission events and Kyverno reports for the exact failing resource. These commands do not read Secret data; still redact sensitive metadata and logs before sharing them. Use a unique lowercase alphanumeric/dash run ID for each concurrent run and keep it consistent between the script and inspection terminal. The script normalizes and truncates IDs to 20 characters and may clean its resources before inspection.

HTTP-01 With Big Bang Istio

Public DNS must resolve the exact requested FQDN to a reachable HTTP listener on TCP 80. Workstation tunnels, /etc/hosts, private MetalLB addresses, and sshuttle are not public Let’s Encrypt reachability.

An Istio networking.istio.io/Gateway and a Gateway API Gateway are different APIs. cert-manager’s automatic solvers create Kubernetes Ingress or Gateway API HTTPRoute resources, not routes on an arbitrary existing Istio Gateway. Choose and verify one path:

• An explicitly configured Kubernetes Ingress controller with an appropriate IngressClass. Do not assume ingressClassName: istio makes Istio reconcile it.
• A public Gateway API Gateway with a port 80 listener and allowedRoutes permitting the challenge namespace, plus cert-manager’s version-specific Gateway support.
• Environment-owned automation connecting the temporary solver Service to a verified public route.

Keep /.well-known/acme-challenge/ reachable through load balancers, firewalls, and gateway routing. Namespace owners must provide solver ingress on TCP 8089 and review mesh enrollment/authorization and Kyverno compatibility. The ingress solver’s opt-out metadata does not apply automatically to Gateway API solvers. Use the matching Ingress solver example or Gateway API solver example, staged upgrade, and script. Evidence must include the generated route and successful public challenge response as well as Certificate, Order, and Challenge status.

Production and Ingress Certificates

A test certificate in cert-manager does not replace the gateway’s TLS Secret. Separately design Secret placement/distribution, gateway references, renewal ownership, and trust before production use. Development ingress certificates are not the ACME test certificate and do not satisfy its challenge. After staging, review production DNS names, account-key retention, rate limits, and gateway rollout separately; do not switch this test to production for browser trust.

Cleanup

The script cleans its run-scoped test resources. Remove any separately retained test Certificate/Secret and provider credential only after confirming exact names and shared use. Disable the test issuer in desired state before deliberately discarding its staging account key, or reconciliation can create another account. Follow account-key retention before namespace deletion. Use the matching umbrella tooling to remove provisioned infrastructure; local k3d cluster delete alone may leave cloud resources behind.

Acceptance Criteria

• Staging, scoped credentials, and normal registry/mesh/network/Kyverno controls.
• Issuer and Certificate Ready=True, related Order and Challenge valid.
• Evidence of provider/solver reachability without admission or image-pull failures.
• Redacted Certificate/Request/Order/Challenge/event/policy evidence and deliberate cleanup/account retention. Renewal and external trust need separate evidence.

Pipeline Decision

Keep external ACME acceptance testing separate from deterministic package CI. Only run it in an explicitly provisioned environment with DNS ownership, credential handling, and cleanup agreed beforehand.