Skip to content

Deploy Standardized ESO ClusterSecretStore for Hashicorp Vault

This section demonstrates what is needed for deployment. Users can evaluate and adjust for their own environment. Vault provides multiple authentication methods, the two most relevant for External Secrets are:

  • Token Authentication: Static token to authenticate into Vault.
  • Kubernetes Authentication: ESO authenticates to Vault using a Kubernetes service account JWT.

There are multiple types of authentication methods: token, approle, kubernetes, etc. Here we use the token method. See the External Secrets Vault provider docs for full details.

Prerequisites

Network and Istio Requirements

When deploying in a Big Bang environment with hardened Istio and NetworkPolicies, ESO needs network-level and service mesh-level access to Vault.

Handled automatically by the ESO chart:

  • Egress NetworkPolicy: The ESO chart creates allow-egress-vault when networkPolicies.enabled: true, allowing ESO pods to reach the vault namespace on port 8200.
  • Ingress NetworkPolicy (vault-side): The ESO chart defaults the pod label vault-ingress: "true" on ESO pods. The BB Vault chart includes a NetworkPolicy (allow-ingress-to-vault-port-8200-from-custom-app-ingress) that allows ingress on port 8200 from any pod with this label.

Required in your Vault configuration:

The BB Vault chart deploys a deny-all Istio AuthorizationPolicy by default. You must add an ALLOW policy for the external-secrets namespace. Add the following to your Vault Big Bang override values:

addons:
  vault:
    values:
      istio:
        hardened:
          customAuthorizationPolicies:
            - name: allow-external-secrets
              enabled: true
              spec:
                action: ALLOW
                rules:
                  - from:
                      - source:
                          namespaces:
                            - external-secrets
                    to:
                      - operation:
                          ports:
                            - "8200"

Authentication Method: Token

Create a Secret containing your Vault Token

This is needed for the token auth. Example below.

kubectl create secret generic vault-token -n external-secrets --from-literal=token='<VAULT_TOKEN>'

Big Bang Override Values

addons:
  externalSecrets:
    enabled: true
    values:
      networkPolicies:
        enabled: true
      clusterSecretStoreConfiguration:
        enabled: true
        clusterSecretStoreList:
          - name: vault-secrets
            source:
              provider: vault
              server: "https://vault.example.com:8200"
              path: kv
              version: v2
              auth:
                type: token
                secretName: vault-token
                namespace: external-secrets
                key: token

Authentication Method: Kubernetes

Vault’s Kubernetes authentication allows ESO to log in using a Kubernetes ServiceAccount. Vault verifies the SA’s JWT with the Kubernetes API and issues a Vault token mapped to a Vault role.

Prerequisites: - A Vault server accessible from the cluster - Kubernetes auth path enabled and configured in Vault - A Vault role bound to the ESO ServiceAccount

Configure Vault

Enable Kubernetes Auth in Vault:

vault auth enable kubernetes

Use the /config endpoint to configure Vault to talk to Kubernetes. Use kubectl cluster-info to validate the Kubernetes host address and TCP port.

vault write auth/kubernetes/config \
    token_reviewer_jwt="<your reviewer service account JWT>" \
    kubernetes_host=https://192.168.99.100:<your TCP port or blank for 443> \
    kubernetes_ca_cert=@ca.crt

Create a named role:

vault write auth/kubernetes/role/eso-role \
    bound_service_account_names=external-secrets \
    bound_service_account_namespaces=external-secrets \
    policies=default \
    ttl=1h

Big Bang Override Values

addons:
  externalSecrets:
    enabled: true
    values:
      networkPolicies:
        enabled: true
      clusterSecretStoreConfiguration:
        enabled: true
        clusterSecretStoreList:
          - name: vault-secrets
            source:
              provider: vault
              server: "https://vault.example.com:8200"
              path: kv
              version: v2
              auth:
                type: kubernetes
                mountPath: "kubernetes"
                role: "eso-role"
                serviceAccountRef:
                  name: "external-secrets"
                  namespace: "external-secrets"

Fetching Secrets with ExternalSecret

Once the ClusterSecretStore is configured, create ExternalSecret resources to fetch data from Vault. The name field must match the ClusterSecretStore name defined above.

Using individual keys (data)

addons:
  externalSecrets:
    enabled: true
    values:
      externalSecretsConfiguration:
        enabled: true
        secretList:
          - name: vault-secrets
            namespace: external-secrets
            secrets:
              targetName: my-app-secret
              targetPolicy: "Owner"
              data:
                - secretKey: username
                  remoteRef:
                    key: my-app/config
                    property: username
                - secretKey: password
                  remoteRef:
                    key: my-app/config
                    property: password

Using all keys from a path (dataFrom)

addons:
  externalSecrets:
    enabled: true
    values:
      externalSecretsConfiguration:
        enabled: true
        secretList:
          - name: vault-secrets
            namespace: external-secrets
            secrets:
              targetName: my-app-secret
              targetPolicy: "Owner"
              dataFrom:
                - extract:
                    key: my-app/config

Troubleshooting

ClusterSecretStore shows InvalidProviderConfig with 503

Symptom: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused

Causes and fixes:

  1. Missing Istio AuthorizationPolicy on Vault: The Vault namespace has a deny-all AuthorizationPolicy. Add the customAuthorizationPolicies shown in the Prerequisites section to your Vault override values.

  2. Missing vault-ingress pod label: Verify ESO pods have the vault-ingress: "true" label. This is defaulted by the chart but can be overridden: 

    kubectl get pods -n external-secrets -l app.kubernetes.io/name=external-secrets -o jsonpath='{.items[0].metadata.labels.vault-ingress}'
    If missing, ensure your values include: 
    upstream:
  podLabels:
    vault-ingress: "true"

  3. NetworkPolicies not enabled: Ensure networkPolicies.enabled: true is set in your ESO values so the egress policy to Vault is created.

Invalid vault credentials with non-printable characters

If you encounter invalid vault credentials: configured Vault token contains non-printable characters and cannot be used, trim the token before creating the secret:

TOKEN_CLEAN="$(tr -d '\r\n' < /path/to/token)"
kubectl create secret generic vault-token -n external-secrets --from-literal=token="$TOKEN_CLEAN"