Skip to content

Vault With Prometheus📜


This guide creates a Role called prometheus-monitoring and binds the Role to the monitoring-monitoring-kube-prometheus service account in the monitoring namespace


Vault Initialization & Kubernetes Authentication Method📜

The Kubernetes Authentication Method can be enabled/configured using Vault’s web interface or by the vault cli.

See HashiCorp Vault Kubernetes Auth Method for more details

  • Exec into a vault pod & login via the CLI (see below) and run the following commands:
vault operator init

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
  token_reviewer_jwt="$(cat /var/run/secrets/" \
  kubernetes_ca_cert=@/var/run/secrets/ \
  • To get the root token for the vault deployment if using the BigBang developer autoInit job:
kubectl get secret -n vault vault-token -o go-template='{{.data.key | base64decode}}'

For each of these steps we will be running commands against vault from the CLI so you should be exec’d in and logged in like so:

kubectl exec -it pod/vault-vault-0 -n vault -- /bin/bash

vault login

Vault ACL Policy for Metrics📜

Vault exposes Prometheus metrics at the /sys/metrics url.

Since Prometheus needs read ability, we create the policy via the CLI by exec-ing into a vault pod:

vault policy write prometheus-monitoring - << EOF
    path "/sys/metrics" {
      capabilities = ["read"]

Then attach the policy to the existing monitoring-monitoring-kube-prometheus ServiceAccount used by the Prometheus pod:

vault write auth/kubernetes/role/prometheus \
      bound_service_account_names=monitoring-monitoring-kube-prometheus \
      bound_service_account_namespaces=monitoring \
      policies="default,prometheus-monitoring" \

Prometheus Configuration📜

Vault Sidecar Injection📜

Using a k8s MutatingWebhookConfiguration (vault-vault-agent-injector-cfg), Vault will add a sidecar to annotated pods.

See HashiCorp Vault Agent Annotations for details

The Big Bang prometheus pod is annotated with the following values when both monitoring & vault are enabled within BigBang: "true" "true" "true" "prometheus"

With these settings, the Vault sidecar will mount the Vault token assigned to the kubernetes/prometheus role inside the pod at /vault/secrets/token .

Mounting the token inside the pod allows the Prometheus configuration to pass the token when it scrapes data from Vault.

Metrics Endpoint Configuration📜

Prometheus is configured using an additionalScrapeConfig passthrough value to scrape the metrics from Vault. We define the endpoint and location of the token file to use for authentication.


    - job_name: vault
      metrics_path: /v1/sys/metrics
        format: ['prometheus']
      scheme: https
        credentials_file: /vault/secrets/token
      - targets: []

Known issues📜

These only apply to development and testing environments which are using the Big Bang default settings

  1. Permission denied trying to re-authenticate after upgrade

Vault Reference Documentation📜

Last update: 2023-05-11 by Rob Ferguson