How to upgrade the Tempo Package chart📜

Navigate to the upstream chart repo and folder and find the tag that corresponds with the new chart version for this update
- Check the upstream release notes for upgrade notices.
Checkout the renovate/ironbank branch
From the root of the repo run kpt pkg update chart@<tag> --strategy alpha-git-patch, where tag is found in step 1 (Tempo ref: tempo-<tag>)
- Run a KPT package update
```
kpt pkg update chart@tempo-<tag> --strategy alpha-git-patch
```
- Restore all BigBang added templates and tests:
```
git checkout chart/templates/bigbang/
git checkout chart/tests/
git checkout chart/templates/tests
```
- Follow the Modifications made to upstream section of this document for a list of changes per file to be aware of, for how Big Bang differs from upstream.
Modify the version in Chart.yaml and append -bb.0 to the chart version from upstream. See Update main chart section of this document.

Update dependencies and binaries using helm dependency update ./chart

If needed, log into registry1

helm registry login https://registry1.dso.mil -u ${registry1.username}
helm registry logout https://registry1.dso.mil

Pull assets and commit the binaries as well as the Chart.lock file that was generated.

export HELM_EXPERIMENTAL_OCI=1
helm dependency update ./chart

Then log out.

helm registry logout https://registry1.dso.mil

Update CHANGELOG.md adding an entry for the new version and noting all changes in a list (at minimum should include - Updated <chart or dependency> to x.x.x).
Generate the README.md updates by following the guide in gluon.
Push up your changes, add upgrade notices if applicable, validate that CI passes.
- If there are any failures, follow the information in the pipeline to make the necessary updates.
- Add the debug label to the MR for more detailed information.
- Reach out to the CODEOWNERS if needed.
Follow the Testing new Tempo Version section of this document for manual testing.

Update main chart📜

chart/Chart.yaml

Update tempo version and appVersion
Ensure Big Bang version suffix is appended to chart version
```
version: $VERSION-bb.0
```

Ensure gluon dependencies and annotations are present and up to date

dependencies:
  - name: gluon
    version: $GLUON_VERSION
    repository: oci://registry1.dso.mil/bigbang
annotations:
  bigbang.dev/applicationVersions: |
    - Tempo: $TEMPO_VERSION
    - Tempo Query: $TEMPO_VERSION
  helm.sh/images: |
    - name: tempo
      image: registry1.dso.mil/ironbank/opensource/grafana/tempo:$TEMPO_VERSION
    - name: tempo-query
      image: registry1.dso.mil/ironbank/opensource/grafana/tempo-query:$TEMPO_VERSION

Modifications made to upstream📜

chart/values.yaml

line 14, update tempo.repository to pull hardened images from registry1

  # -- Docker image repository
  repository: registry1.dso.mil/ironbank/opensource/grafana/tempo

line 29, ensure tempo.resources requests and limits are set

  resources:
    limits:
      cpu: 500m
      memory: 4Gi
    requests:
      cpu: 500m
      memory: 4Gi

line 46, ensure tempo.ingester values are set

  ingester:
    trace_idle_period: 10s
    max_block_bytes: 1_000_000
    max_block_duration: 5m

line 54, ensure tempo.retention is set to 336h
```
  retention: 336h # 2 weeks retention
```
line 97, ensure tempo.receivers contains values for zipkin
```
    zipkin:
      endpoint: 0.0.0.0:9411
```

line 106, ensure tempo.securityContext is set

  securityContext:
     capabilities:
       drop:
       - ALL

line 165, update tempoQuery.repository to pull hardened images from registry1

  # -- Docker image repository
  repository: registry1.dso.mil/ironbank/opensource/grafana/tempo

line 180, ensure tempoQuery.resources requests and limits are set

  # -- Resource for query container
  resources:
    limits:
      cpu: 300m
      memory: 256Mi
    requests:
      cpu: 300m
      memory: 256Mi

line 181, ensure tempoQuery.enabled is true

Note: this upstream commit disabled tempo-query by default in the chart. Evidently this is because tempo-query was always meant as a shim between Tempo and Grafana, but it hasn’t been necessary since 7.5.0, as Grafana is capable of querying Tempo directly now.

Currently, Big Bang uses tempo-query for Cypress testing and users may expect a basic web interface for Tempo without Grafana (Tempo has non natively, only a HTTP API). This may be changed in an upcoming release, but we will keep utilizing tempo-query for the benefits of the interface.

  enabled: true

line 199, ensure tempoQuery.securityContext is set

  securityContext:
     capabilities:
       drop:
       - ALL

line 209, ensure securityContext for containers is set

# -- securityContext for container
securityContext:
  fsGroup: 1001
  runAsGroup: 1001
  runAsNonRoot: true
  runAsUser: 1001

line 223, ensure serviceAccount.imagePullSecrets contains private-registry pull secret for IronBank images

  # -- Image pull secrets for the service account
  imagePullSecrets:
    - name: private-registry

line 245, ensure persistence is enabled and size is increased to 15Gi

persistence:
  enabled: true
  # storageClassName: local-path
  accessModes:
    - ReadWriteOnce
  size: 15Gi

line 253, ensure podAnnotations includes istio inbound ports

podAnnotations:
  traffic.sidecar.istio.io/includeInboundPorts: "16687,16686,3100"

line 262, ensure serviceAccount.automountServiceAccountToken is set to false This helps maintain our NSA hardening guide-compliance
```
  automountServiceAccountToken: false
```
EOF, add default bigbang.dev hostname and addditional Big Bang values

chart/templates/service.yaml

Added protocols to each port name (i.e. tcp, http, etc)

line 35, ensure name is http-tempo-prom-metrics
line 39, ensure name is http-jaeger-metrics
line 42, ensure name is http-tempo-query-jaeger-ui
line 46, ensure name is udp-tempo-jaeger-thrift-compact
line 50, ensure name is udp-tempo-jaeger-thrift-binary
line 54, ensure name is http-tempo-jaeger-thrift-http
line 62, ensure name is tcp-tempo-zipkin
line 66, ensure name is tcp-tempo-otlp-legacy
line 70, ensure name is http-tempo-otlp-http-legacy
line 78, ensure name is http-tempo-otlp-http
line 82, ensure name is tcp-tempo-opencensus

chart/templates/servicemonitor.yaml

Modified ports to match naming convention with http- prefix

line 26, ensure port is http-tempo-prom-metrics
line 40, ensure port is http-jaeger-metrics

chart/templates/statefulset.yaml

line 79-83, add in envFrom section to the tempo container

{{- if and .Values.objectStorage.access_key_id .Values.objectStorage.secret_access_key }}
envFrom:
- secretRef:
    name: tempo-object-storage
{{- end }}

chart/templates/bigbang/*📜

Add Big Bang network Policies as applicable
Add VirtualService for tempo-query UI access
Add openTelemetry collector deployment/configurations

chart/tests/*📜

Add cypress testing configuration and tests
Add scripts for testing

Testing new Tempo Version📜

NOTE: For these testing steps it is good to do them on both a clean install and an upgrade. For clean install, point Loki to your branch. For an upgrade do an install with Loki pointing to the latest tag, then perform a helm upgrade with Loki pointing to your branch.

You will want to install with: - Tempo, monitoring and Istio packages enabled - Jaeger disabled

overrides/tempo.yaml

domain: bigbang.dev

flux:
  interval: 1m
  rollback:
    cleanupOnFail: false

clusterAuditor:
  enabled: false

gatekeeper:
  enabled: false

istioOperator:
  enabled: true

istio:
  enabled: true

monitoring:
  enabled: true

loki:
  enabled: false

promtail:
  enabled: false

tempo:
  enabled: true
  git:
    tag: null
    branch: "renovate/ironbank"

jaeger:
  enabled: false

Visit https://tracing.bigbang.dev
Ensure Services are listed and traces are being rendered
Check the logs for the tempo pod and container and ensure traceIDs are getting sent over from the istio mesh
Visit https://grafana.bigbang.dev and login with default credentials
Search for Data Sources -> click Tempo -> click Save & Test datasource at the bottom

When in doubt with any testing or upgrade steps, reach out to the CODEOWNERS for assistance.

Last update: 2024-02-14 by Blane Staskiewicz