How to upgrade the Tempo Package chart📜

Navigate to the upstream chart repo and folder and find the tag that corresponds with the new chart version for this update
- Check the upstream release notes for upgrade notices.
Checkout the renovate/ironbank branch
From the root of the repo run kpt pkg update chart@<tag> --strategy alpha-git-patch, where tag is found in step 1 (Tempo ref: tempo-<tag>)
- Run a KPT package update
```
kpt pkg update chart@tempo-<tag> --strategy alpha-git-patch
```
- Restore all BigBang added templates and tests:
```
git checkout chart/templates/bigbang/
git checkout chart/tests/
git checkout chart/templates/tests
```
- Follow the Modifications made to upstream section of this document for a list of changes per file to be aware of, for how Big Bang differs from upstream.
Modify the version in Chart.yaml and append -bb.0 to the chart version from upstream. See Update main chart section of this document.

Update dependencies and binaries using helm dependency update ./chart

If needed, log into registry1

helm registry login https://registry1.dso.mil -u ${registry1.username}
helm registry logout https://registry1.dso.mil

Pull assets and commit the binaries as well as the Chart.lock file that was generated.

export HELM_EXPERIMENTAL_OCI=1
helm dependency update ./chart

Then log out.

helm registry logout https://registry1.dso.mil

Update the dashboards in the files/dashboards directory manually by replacing them with the upstream dashboards here
Update CHANGELOG.md adding an entry for the new version and noting all changes in a list (at minimum should include - Updated <chart or dependency> to x.x.x).
Generate the README.md updates by following the guide in gluon.
Push up your changes, add upgrade notices if applicable, validate that CI passes.
- If there are any failures, follow the information in the pipeline to make the necessary updates.
- Add the debug label to the MR for more detailed information.
- Reach out to the CODEOWNERS if needed.
(Optional, only required if package changes are expected to have cascading effects on bigbang umbrella chart) As part of your MR that modifies bigbang packages, you should modify the bigbang bigbang/tests/test-values.yaml against your branch for the CI/CD MR testing by enabling your packages.
- To do this, at a minimum, you will need to follow the instructions at bigbang/docs/developer/test-package-against-bb.md with changes for Tempo enabled (the below is a reference, actual changes could be more depending on what changes where made to Tempo in the package MR).

test-values.yaml 📜

sso:           
  url: https://login.dso.mil/auth/realms/baby-yoda

tempo:
  enabled: true
  git:
    tag: null
    branch: "renovate/ironbank"
  values:
    istio:
      hardened:
        enabled: true
    ## Begin optional block for testing metrics_generator/dashboards:
    tempo:
      metricsGenerator:
        enabled: true
      values:
    ## You have to overwrite the entire config to not break the templating of the configmap for
    ## Tempo, just to add a few arguments to the metrics_generator.
    ## NOTE: This is NOT necessary for a production deployment. This is simply for faster dev testing
    ## of tempo metrics generation.
    config: |
        memberlist:
          cluster_label: "{{ .Release.Name }}.{{ .Release.Namespace }}"
        multitenancy_enabled: {{ .Values.tempo.multitenancyEnabled }}
        usage_report:
          reporting_enabled: {{ .Values.tempo.reportingEnabled }}
        compactor:
          compaction:
            block_retention: {{ .Values.tempo.retention }}
        distributor:
          receivers:
            {{- toYaml .Values.tempo.receivers | nindent 8 }}
        ingester:
          {{- toYaml .Values.tempo.ingester | nindent 6 }}
        server:
          {{- toYaml .Values.tempo.server | nindent 6 }}
        storage:
          {{- toYaml .Values.tempo.storage | nindent 6 }}
        querier:
          {{- toYaml .Values.tempo.querier | nindent 6 }}
        query_frontend:
          {{- toYaml .Values.tempo.queryFrontend | nindent 6 }}
        overrides:
          {{- toYaml .Values.tempo.overrides | nindent 6 }}
          {{- if .Values.tempo.metricsGenerator.enabled }}
        metrics_generator:
              storage:
                path: "/tmp/tempo"
                remote_write:
                  - url: {{ .Values.tempo.metricsGenerator.remoteWriteUrl }}
                wal:
                remote_write_flush_deadline: 6s
                remote_write_add_org_id_header: true
              traces_storage:
                path: "/tmp/traces"
              registry:
                collection_interval: 3s
                external_labels: {}
                stale_duration: 15m
          {{- end }}
    ## End optional block for metrics generator testing

  ### Additional components of Tempo should be changed to reflect testing changes introduced in the package MR

Follow the Testing new Tempo Version section of this document for manual testing.

Update main chart📜

`chart/Chart.yaml`📜

Update tempo version and appVersion
Ensure Big Bang version suffix is appended to chart version
```
version: $VERSION-bb.0
```

Ensure gluon dependencies and annotations are present and up to date

dependencies:
  - name: gluon
    version: $GLUON_VERSION
    repository: oci://registry1.dso.mil/bigbang
annotations:
  bigbang.dev/applicationVersions: |
    - Tempo: $TEMPO_VERSION
    - Tempo Query: $TEMPO_VERSION
  helm.sh/images: |
    - name: tempo
      image: registry1.dso.mil/ironbank/opensource/grafana/tempo:$TEMPO_VERSION
    - name: tempo-query
      image: registry1.dso.mil/ironbank/opensource/grafana/tempo-query:$TEMPO_VERSION

Modifications made to upstream📜

`chart/templates/_ports.tpl`📜

Update the port definition for tempo-prom-metrics by adding appProtocol: http . This helps istio know how to proxy this port, since the port name doesn’t contain http.
```
- name: tempo-prom-metrics
  port: 3100
  protocol: TCP
  targetPort: 3100
  appProtocol: http
```

`chart/values.yaml`📜

Update tempo.repository to pull hardened images from registry1

  # -- Docker image repository
  repository: registry1.dso.mil/ironbank/opensource/grafana/tempo

Ensure tempo.resources requests and limits are set

  resources:
    limits:
      cpu: 500m
      memory: 4Gi
    requests:
      cpu: 500m
      memory: 4Gi

Ensure tempo.reportingEnabled is set to false
```
  reportingEnabled: false
```

Ensure tempo.ingester values are set

  ingester:
    trace_idle_period: 10s
    max_block_bytes: 1_000_000
    max_block_duration: 5m

Ensure tempo.retention is set to 336h
```
  retention: 336h # 2 weeks retention
```

Ensure tempo.receivers contains values for zipkin

    zipkin:
      endpoint: 0.0.0.0:9411

Ensure tempo.securityContext is set

  securityContext:
     capabilities:
       drop:
       - ALL

Update tempoQuery.repository to pull hardened images from registry1

  # -- Docker image repository
  repository: registry1.dso.mil/ironbank/opensource/grafana/tempo-query

Ensure tempoQuery.enabled is false

Note: this upstream commit disabled tempo-query by default in the chart. Evidently this is because tempo-query was always meant as a shim between Tempo and Grafana, but it hasn’t been necessary since 7.5.0, as Grafana is capable of querying Tempo directly now.

Previously, Big Bang used tempo-query for Cypress testing and to provide users a basic web interface for Tempo without Grafana (Tempo offers only an HTTP API natively). This was changed after breaking changes to the tempo-query deployment model were introduced in version 2.6.0. Other tempoQuery configuration settings remain below, in the event users wish to deploy it with the associated Iron Bank container image.

  enabled: false

Ensure tempoQuery.resources requests and limits are set

  # -- Resource for query container
  resources:
    limits:
      cpu: 300m
      memory: 256Mi
    requests:
      cpu: 300m
      memory: 256Mi

Ensure tempoQuery.securityContext is set

  securityContext:
     capabilities:
       drop:
       - ALL

Ensure securityContext for containers is set

# -- securityContext for container
securityContext:
  fsGroup: 1001
  runAsGroup: 1001
  runAsNonRoot: true
  runAsUser: 1001

Ensure serviceAccount.imagePullSecrets contains private-registry pull secret for IronBank images

  # -- Image pull secrets for the service account
  imagePullSecrets:
    - name: private-registry

Ensure serviceAccount.automountServiceAccountToken is set to false This helps maintain our NSA hardening guide-compliance
```
  automountServiceAccountToken: false
```

Ensure serviceMonitor has scheme and tlsConfig values shown below:

serviceMonitor:
  enabled: false
  interval: ""
  additionalLabels: {}
  annotations: {}
  scheme: ""
  tlsConfig: {}
  # scrapeTimeout: 10s

Ensure persistence is enabled and size is increased to 15Gi

persistence:
  enabled: true
  # -- Enable StatefulSetAutoDeletePVC feature
  enableStatefulSetAutoDeletePVC: false
  # storageClassName: local-path
  accessModes:
    - ReadWriteOnce
  size: 15Gi

Ensure podAnnotations includes istio inbound ports

podAnnotations:
  traffic.sidecar.istio.io/includeInboundPorts: "3100,4317,4318"

EOF, add default dev.bigbang.mil hostname and additional Big Bang values

# -- Domain used for BigBang created exposed services
domain: dev.bigbang.mil
# -- Toggle istio integration. Intended to be controlled via BigBang passthrough of istio package status
istio:
  enabled: false
  # -- Default peer authentication values
  hardened:
    enabled: false
    outboundTrafficPolicyMode: "REGISTRY_ONLY"
    customServiceEntries: []
      # - name: "allow-google"
      #   enabled: true
      #   spec:
      #     hosts:
      #       - google.com
      #     location: MESH_EXTERNAL
      #     ports:
      #       - number: 443
      #         protocol: TLS
      #         name: https
      #     resolution: DNS
    customAuthorizationPolicies: []
    # - name: "allow-nothing"
    #   enabled: true
    #   spec: {}
  mtls:
    # -- STRICT = Allow only mutual TLS traffic,
    # PERMISSIVE = Allow both plain text and mutual TLS traffic
    mode: STRICT
  # -- Tempo-Query specific VirtualService values
  tempoQuery:
    # -- Toggle VirtualService creation
    enabled: true
    annotations: {}
    labels: {}
    gateways:
      - istio-system/main
    hosts:
      - tracing.{{ .Values.domain }}

objectStorage:
  # -- AWS access_key_id for External ObjectStorage configuration
  access_key_id: ""
  # -- AWS secret_access_key for External ObjectStorage configuration
  secret_access_key: ""

# -- Toggle for BigBang specific NetworkPolicies.
# If disabled no NetworkPolicies will be installed with package
# ref: https://kubernetes.io/docs/concepts/services-networking/network-policies/
networkPolicies:
  enabled: false
  # -- Istio IngressGateway labels for VirtualService external routing to app UI
  ingressLabels:
    app: istio-ingressgateway
    istio: ingressgateway
  # -- Use `kubectl cluster-info` and then resolve to IP for kube-api.
  # Review value description in BigBang README.md
  controlPlaneCidr: 0.0.0.0/0
  additionalPolicies: []

# -- Toggle monitoring integration. Intended to be controlled via BigBang passthrough of monitoring package status
monitoring:
  enabled: false

# -- SSO toggle. Intended to be controlled via BigBang passthrough, only affects network/auth policies.
sso:
  enabled: false

bbtests:
  enabled: false
  cypress:
    artifacts: true
    envs:
      cypress_url: 'http://{{ template "tempo.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local:16686'
      cypress_tempo_datasource: 'http://{{ template "tempo.fullname" . }}.{{ .Release.Namespace }}.svc:3100'
      cypress_check_datasource: "false"
      cypress_grafana_url: "http://monitoring-grafana.monitoring.svc.cluster.local"
    resources:
      requests:
        cpu: "1"
        memory: "1Gi"
      limits:
        cpu: "2"
        memory: "2Gi"
  scripts:
    enabled: true
    image: registry1.dso.mil/ironbank/big-bang/base:2.1.0
    envs:
      TEMPO_METRICS_URL: 'http://{{ template "tempo.fullname" . }}.{{ .Release.Namespace }}.svc:3100'

# -- Toggle or openshift specific config
openshift: false

`chart/templates/servicemonitor.yaml`📜

removed port: jaeger-metrics definition

`chart/templates/statefulset.yaml`📜

Update templating to include tpl for spec.template.metadata.labels

{{- with .Values.podLabels }}
  {{- tpl (toYaml . | nindent 8) $ }}
{{- end }}

Add in envFrom section to the tempo container

{{- if and .Values.objectStorage.access_key_id .Values.objectStorage.secret_access_key }}
envFrom:
- secretRef:
    name: tempo-object-storage
{{- end }}

Testing new Tempo Version📜

NOTE: For these testing steps it is good to do them on both a clean install and an upgrade. For clean install, point Tempo to your branch. For an upgrade do an install with Tempo pointing to the latest tag, then perform a helm upgrade with Tempo pointing to your branch.

You will want to install with: - Tempo, monitoring and Istio packages enabled - Jaeger disabled

overrides/tempo.yaml

flux:
  interval: 1m
  rollback:
    cleanupOnFail: false

gatekeeper:
  enabled: false

monitoring:
  enabled: true

loki:
  enabled: true

alloy:
  enabled: true
  alloyLogs:
    enabled: true

grafana:
  enabled: true

tempo:
  enabled: true
  git:
    tag: null
    branch: <test-branch>

- Visit Kiali and login with a generated token - Check the ‘Applications’, ‘Workloads’, and ‘Services’ views for Tempo resources (they should be healthy) - Note: if no resources are appearing, make sure the ‘Tempo’ namespace is selected in each view - Visit Grafana and login with default credentials - Connections -> Data Sources -> click Tempo -> click Save & Test datasource at the bottom - Navigate to Explore -> Select “Tempo” as the data source -> Enter a TraceQL query like {resource.service.name="tempo-tempo.tempo"} -> Run query

When in doubt with any testing or upgrade steps, reach out to the CODEOWNERS for assistance.

How to upgrade the Tempo Package chart📜

test-values.yaml📜

Update main chart📜

chart/Chart.yaml📜

Modifications made to upstream📜

chart/templates/_ports.tpl📜

chart/values.yaml📜

chart/templates/servicemonitor.yaml📜

chart/templates/statefulset.yaml📜