Skip to content

How to upgrade the Tempo Package chart📜

  1. Navigate to the upstream chart repo and folder and find the tag that corresponds with the new chart version for this update

  2. Checkout the renovate/ironbank branch

  3. From the root of the repo run kpt pkg update chart@<tag> --strategy alpha-git-patch, where tag is found in step 1 (Tempo ref: tempo-<tag>)

    • Run a KPT package update

      kpt pkg update chart@tempo-<tag> --strategy alpha-git-patch

    • Restore all BigBang added templates and tests:

      git checkout chart/templates/bigbang/
      git checkout chart/tests/
      git checkout chart/templates/tests

    • Follow the Modifications made to upstream section of this document for a list of changes per file to be aware of, for how Big Bang differs from upstream.
  4. Modify the version in Chart.yaml and append -bb.0 to the chart version from upstream. See Update main chart section of this document.

  5. Update dependencies and binaries using helm dependency update ./chart

    • If needed, log into registry1
      helm registry login -u ${registry1.username}
      helm registry logout

    Pull assets and commit the binaries as well as the Chart.lock file that was generated.

    helm dependency update ./chart
    Then log out.
    helm registry logout

  6. Update adding an entry for the new version and noting all changes in a list (at minimum should include - Updated <chart or dependency> to x.x.x).

  7. Generate the updates by following the guide in gluon.

  8. Push up your changes, add upgrade notices if applicable, validate that CI passes.

    • If there are any failures, follow the information in the pipeline to make the necessary updates.

    • Add the debug label to the MR for more detailed information.

    • Reach out to the CODEOWNERS if needed.

  9. Follow the Testing new Tempo Version section of this document for manual testing.

Update main chart📜


  • Update tempo version and appVersion
  • Ensure Big Bang version suffix is appended to chart version
    version: $VERSION-bb.0
  • Ensure gluon dependencies and annotations are present and up to date
      - name: gluon
        version: $GLUON_VERSION
        repository: oci://
    annotations: |
        - Tempo: $TEMPO_VERSION
        - Tempo Query: $TEMPO_VERSION |
        - name: tempo
        - name: tempo-query

Modifications made to upstream📜


  • line 14, update tempo.repository to pull hardened images from registry1

      # -- Docker image repository

  • line 29, ensure tempo.resources requests and limits are set

          cpu: 500m
          memory: 4Gi
          cpu: 500m
          memory: 4Gi

  • line 46, ensure tempo.ingester values are set

        trace_idle_period: 10s
        max_block_bytes: 1_000_000
        max_block_duration: 5m

  • line 54, ensure tempo.retention is set to 336h

      retention: 336h # 2 weeks retention

  • line 97, ensure tempo.receivers contains values for zipkin


  • line 106, ensure tempo.securityContext is set

           - ALL

  • line 165, update tempoQuery.repository to pull hardened images from registry1

      # -- Docker image repository

  • line 180, ensure tempoQuery.resources requests and limits are set

      # -- Resource for query container
          cpu: 300m
          memory: 256Mi
          cpu: 300m
          memory: 256Mi

  • line 181, ensure tempoQuery.enabled is true

Note: this upstream commit disabled tempo-query by default in the chart. Evidently this is because tempo-query was always meant as a shim between Tempo and Grafana, but it hasn’t been necessary since 7.5.0, as Grafana is capable of querying Tempo directly now.

Currently, Big Bang uses tempo-query for Cypress testing and users may expect a basic web interface for Tempo without Grafana (Tempo has non natively, only a HTTP API). This may be changed in an upcoming release, but we will keep utilizing tempo-query for the benefits of the interface.

  enabled: true

  • line 199, ensure tempoQuery.securityContext is set

           - ALL

  • line 209, ensure securityContext for containers is set

    # -- securityContext for container
      fsGroup: 1001
      runAsGroup: 1001
      runAsNonRoot: true
      runAsUser: 1001

  • line 223, ensure serviceAccount.imagePullSecrets contains private-registry pull secret for IronBank images

      # -- Image pull secrets for the service account
        - name: private-registry

  • line 245, ensure persistence is enabled and size is increased to 15Gi

      enabled: true
      # storageClassName: local-path
        - ReadWriteOnce
      size: 15Gi

  • line 253, ensure podAnnotations includes istio inbound ports

    podAnnotations: "16687,16686,3100"

  • line 262, ensure serviceAccount.automountServiceAccountToken is set to false This helps maintain our NSA hardening guide-compliance

      automountServiceAccountToken: false

  • EOF, add default hostname and addditional Big Bang values


Added protocols to each port name (i.e. tcp, http, etc)

  • line 35, ensure name is http-tempo-prom-metrics
  • line 39, ensure name is http-jaeger-metrics
  • line 42, ensure name is http-tempo-query-jaeger-ui
  • line 46, ensure name is udp-tempo-jaeger-thrift-compact
  • line 50, ensure name is udp-tempo-jaeger-thrift-binary
  • line 54, ensure name is http-tempo-jaeger-thrift-http
  • line 62, ensure name is tcp-tempo-zipkin
  • line 66, ensure name is tcp-tempo-otlp-legacy
  • line 70, ensure name is http-tempo-otlp-http-legacy
  • line 78, ensure name is http-tempo-otlp-http
  • line 82, ensure name is tcp-tempo-opencensus


Modified ports to match naming convention with http- prefix

  • line 26, ensure port is http-tempo-prom-metrics
  • line 40, ensure port is http-jaeger-metrics


  • line 79-83, add in envFrom section to the tempo container
    {{- if and .Values.objectStorage.access_key_id .Values.objectStorage.secret_access_key }}
    - secretRef:
        name: tempo-object-storage
    {{- end }}


  • Add Big Bang network Policies as applicable
  • Add VirtualService for tempo-query UI access
  • Add openTelemetry collector deployment/configurations


  • Add cypress testing configuration and tests
  • Add scripts for testing

Testing new Tempo Version📜

NOTE: For these testing steps it is good to do them on both a clean install and an upgrade. For clean install, point Loki to your branch. For an upgrade do an install with Loki pointing to the latest tag, then perform a helm upgrade with Loki pointing to your branch.

You will want to install with: - Tempo, monitoring and Istio packages enabled - Jaeger disabled



  interval: 1m
    cleanupOnFail: false

  enabled: false

  enabled: false

  enabled: true

  enabled: true

  enabled: true

  enabled: false

  enabled: false

  enabled: true
    tag: null
    branch: "renovate/ironbank"

  enabled: false

  • Visit
  • Ensure Services are listed and traces are being rendered
  • Check the logs for the tempo pod and container and ensure traceIDs are getting sent over from the istio mesh
  • Visit and login with default credentials
  • Search for Data Sources -> click Tempo -> click Save & Test datasource at the bottom

When in doubt with any testing or upgrade steps, reach out to the CODEOWNERS for assistance.

Last update: 2024-02-14 by Blane Staskiewicz