Skip to content

How to upgrade the Tempo Package chart💣

Check the upstream release notes

Upgrade💣

Find the tempo helm release version in the grafana helm charts repo that corresponds with the app version identified by Rennovate

Run a KPT package update

kpt pkg update chart@tempo-${chart.version} --strategy alpha-git-patch

Restore all BigBang added templates and tests:

git checkout chart/templates/bigbang/
git checkout chart/tests/
git checkout chart/templates/tests

Update binaries💣

If needed, log into registry1

helm registry login https://registry1.dso.mil -u ${registry1.username}
helm registry logout https://registry1.dso.mil

Pull assets and commit the binaries as well as the Chart.lock file that was generated.

export HELM_EXPERIMENTAL_OCI=1
helm dependency update ./chart

Update main chart💣

chart/Chart.yaml

  • Update tempo version and appVersion
  • Ensure Big Bang version suffix is appended to chart version
version: $VERSION-bb.0
  • Ensure gluon dependencies and annotations are present and up to date
dependencies:
  - name: gluon
    version: $GLUON_VERSION
    repository: oci://registry1.dso.mil/bigbang
annotations:
  bigbang.dev/applicationVersions: |
    - Tempo: $TEMPO_VERSION
    - Tempo Query: $TEMPO_VERSION
  helm.sh/images: |
    - name: tempo
      image: registry1.dso.mil/ironbank/opensource/grafana/tempo:$TEMPO_VERSION
    - name: tempo-query
      image: registry1.dso.mil/ironbank/opensource/grafana/tempo-query:$TEMPO_VERSION

Modifications made to upstream💣

chart/values.yaml

  • line 14, update tempo.repository to pull hardened images from registry1
  # -- Docker image repository
  repository: registry1.dso.mil/ironbank/opensource/grafana/tempo
  • line 29, ensure tempo.resources requests and limits are set
  resources:
    limits:
      cpu: 500m
      memory: 4Gi
    requests:
      cpu: 500m
      memory: 4Gi
  • line 46, ensure tempo.ingester values are set
  ingester:
    trace_idle_period: 10s
    max_block_bytes: 1_000_000
    max_block_duration: 5m
  • line 54, ensure tempo.retention is set to 336h
  retention: 336h # 2 weeks retention
  • line 97, ensure tempo.receivers contains values for zipkin
    zipkin:
      endpoint: 0.0.0.0:9411
  • line 106, ensure tempo.securityContext is set
  securityContext:
     capabilities:
       drop:
       - ALL
  • line 165, update tempoQuery.repository to pull hardened images from registry1
  # -- Docker image repository
  repository: registry1.dso.mil/ironbank/opensource/grafana/tempo
  • line 180, ensure tempoQuery.resources requests and limits are set
  # -- Resource for query container
  resources:
    limits:
      cpu: 300m
      memory: 256Mi
    requests:
      cpu: 300m
      memory: 256Mi
  • line 199, ensure tempoQuery.securityContext is set
  securityContext:
     capabilities:
       drop:
       - ALL
  • line 209, ensure securityContext for containers is set
# -- securityContext for container
securityContext:
  fsGroup: 1001
  runAsGroup: 1001
  runAsNonRoot: true
  runAsUser: 1001
  • line 223, ensure serviceAccount.imagePullSecrets contains private-registry pull secret for IronBank images
  # -- Image pull secrets for the service account
  imagePullSecrets:
    - name: private-registry
  • line 245, ensure persistence is enabled and size is increased to 15Gi
persistence:
  enabled: true
  # storageClassName: local-path
  accessModes:
    - ReadWriteOnce
  size: 15Gi
  • line 253, ensure podAnnotations includes istio inbound ports
podAnnotations:
  traffic.sidecar.istio.io/includeInboundPorts: "16687,16686,3100"
  • EOF, add default bigbang.dev hostname and addditional Big Bang values

chart/templates/service.yaml

Added protocols to each port name (i.e. tcp, http, etc)

  • line 35, ensure name is http-tempo-prom-metrics
  • line 39, ensure name is http-jaeger-metrics
  • line 42, ensure name is http-tempo-query-jaeger-ui
  • line 46, ensure name is udp-tempo-jaeger-thrift-compact
  • line 50, ensure name is udp-tempo-jaeger-thrift-binary
  • line 54, ensure name is http-tempo-jaeger-thrift-http
  • line 62, ensure name is tcp-tempo-zipkin
  • line 66, ensure name is tcp-tempo-otlp-legacy
  • line 70, ensure name is http-tempo-otlp-http-legacy
  • line 78, ensure name is http-tempo-otlp-http
  • line 82, ensure name is tcp-tempo-opencensus

chart/templates/servicemonitor.yaml

Modified ports to match naming convention with http- prefix

  • line 26, ensure port is http-tempo-prom-metrics
  • line 40, ensure port is http-jaeger-metrics

chart/templates/statefulset.yaml

  • line 79-83, add in envFrom section to the tempo container 
            {{- if and .Values.objectStorage.access_key_id .Values.objectStorage.secret_access_key }}
        envFrom:
        - secretRef:
            name: tempo-object-storage
        {{- end }}

chart/templates/bigbang/*💣

  • Add Big Bang network Policies as applicable
  • Add VirtualService for tempo-query UI access
  • Add openTelemetry collector deployment/configurations

chart/tests/*💣

  • Add cypress testing configuration and tests
  • Add scripts for testing

Testing new Tempo Version💣

  • Deploy tempo as a part of BigBang with istio and monitoring enabled, but with jaeger DISabled
  • Visit https://tracing.bigbang.dev and ensure Services are listed and traces are being rendered
  • Check the logs for the tempo pod and container and ensure traceIDs are getting sent over from the istio mesh
  • Visit https://grafana.bigbang.dev > Login > Gear icon > Data Sources > Tempo > click Test datasource at the bottom
Last update: 2023-03-24 by Ryan Garcia