How to upgrade the Tempo Package chart📜

Navigate to the upstream chart repo and folder and find the tag that corresponds with the new chart version for this update
- Check the upstream release notes for upgrade notices.
Checkout the renovate/ironbank branch
From the root of the repo run kpt pkg update chart@<tag> --strategy alpha-git-patch, where tag is found in step 1 (Tempo ref: tempo-<tag>)
- Run a KPT package update
```
kpt pkg update chart@tempo-<tag> --strategy alpha-git-patch
```
- Restore all BigBang added templates and tests:
```
git checkout chart/templates/bigbang/
git checkout chart/tests/
git checkout chart/templates/tests
```
- Follow the Modifications made to upstream section of this document for a list of changes per file to be aware of, for how Big Bang differs from upstream.
Modify the version in Chart.yaml and append -bb.0 to the chart version from upstream. See Update main chart section of this document.

Update dependencies and binaries using helm dependency update ./chart

If needed, log into registry1

helm registry login https://registry1.dso.mil -u ${registry1.username}
helm registry logout https://registry1.dso.mil

Pull assets and commit the binaries as well as the Chart.lock file that was generated.

export HELM_EXPERIMENTAL_OCI=1
helm dependency update ./chart

Then log out.

helm registry logout https://registry1.dso.mil

Update CHANGELOG.md adding an entry for the new version and noting all changes in a list (at minimum should include - Updated <chart or dependency> to x.x.x).
Generate the README.md updates by following the guide in gluon.
Push up your changes, add upgrade notices if applicable, validate that CI passes.
- If there are any failures, follow the information in the pipeline to make the necessary updates.
- Add the debug label to the MR for more detailed information.
- Reach out to the CODEOWNERS if needed.
As part of your MR that modifies bigbang packages, you should modify the bigbang bigbang/tests/test-values.yaml against your branch for the CI/CD MR testing by enabling your packages.
- To do this, at a minimum, you will need to follow the instructions at bigbang/docs/developer/test-package-against-bb.md with changes for Tempo enabled (the below is a reference, actual changes could be more depending on what changes where made to Tempo in the pakcage MR).

test-values.yaml 📜

```yaml
tempo:
  enabled: true
  git:
    tag: null
    branch: <my-package-branch-that-needs-testing>
  values:
    istio:
      hardened:
        enabled: true
  ### Additional compononents of Tempo should be changed to reflect testing changes introduced in the package MR
```

Follow the Testing new Tempo Version section of this document for manual testing.

Update main chart📜

chart/Chart.yaml

Update tempo version and appVersion
Ensure Big Bang version suffix is appended to chart version
```
version: $VERSION-bb.0
```

Ensure gluon dependencies and annotations are present and up to date

dependencies:
  - name: gluon
    version: $GLUON_VERSION
    repository: oci://registry1.dso.mil/bigbang
annotations:
  bigbang.dev/applicationVersions: |
    - Tempo: $TEMPO_VERSION
    - Tempo Query: $TEMPO_VERSION
  helm.sh/images: |
    - name: tempo
      image: registry1.dso.mil/ironbank/opensource/grafana/tempo:$TEMPO_VERSION
    - name: tempo-query
      image: registry1.dso.mil/ironbank/opensource/grafana/tempo-query:$TEMPO_VERSION

Modifications made to upstream📜

chart/values.yaml

line 21, update tempo.repository to pull hardened images from registry1

  # -- Docker image repository
  repository: registry1.dso.mil/ironbank/opensource/grafana/tempo

line 35, ensure tempo.resources requests and limits are set

  resources:
    limits:
      cpu: 500m
      memory: 4Gi
    requests:
      cpu: 500m
      memory: 4Gi

line 46, ensure tempo.reportingEnabled is set to false
```
  reportingEnabled: false
```

line 52, ensure tempo.ingester values are set

  ingester:
    trace_idle_period: 10s
    max_block_bytes: 1_000_000
    max_block_duration: 5m

line 60, ensure tempo.retention is set to 336h
```
  retention: 336h # 2 weeks retention
```
line 103, ensure tempo.receivers contains values for zipkin
```
    zipkin:
      endpoint: 0.0.0.0:9411
```

line 112, ensure tempo.securityContext is set

  securityContext:
     capabilities:
       drop:
       - ALL

line 171, update tempoQuery.repository to pull hardened images from registry1

  # -- Docker image repository
  repository: registry1.dso.mil/ironbank/opensource/grafana/tempo-query

line 184, ensure tempoQuery.enabled is true

Note: this upstream commit disabled tempo-query by default in the chart. Evidently this is because tempo-query was always meant as a shim between Tempo and Grafana, but it hasn’t been necessary since 7.5.0, as Grafana is capable of querying Tempo directly now.

Currently, Big Bang uses tempo-query for Cypress testing and users may expect a basic web interface for Tempo without Grafana (Tempo has non natively, only a HTTP API). This may be changed in an upcoming release, but we will keep utilizing tempo-query for the benefits of the interface.

  enabled: true

line 230, ensure tempoQuery.resources requests and limits are set

  # -- Resource for query container
  resources:
    limits:
      cpu: 300m
      memory: 256Mi
    requests:
      cpu: 300m
      memory: 256Mi

line 248, ensure tempoQuery.securityContext is set

  securityContext:
     capabilities:
       drop:
       - ALL

line 259, ensure securityContext for containers is set

# -- securityContext for container
securityContext:
  fsGroup: 1001
  runAsGroup: 1001
  runAsNonRoot: true
  runAsUser: 1001

line 272, ensure serviceAccount.imagePullSecrets contains private-registry pull secret for IronBank images

  # -- Image pull secrets for the service account
  imagePullSecrets:
    - name: private-registry

line 278, ensure serviceAccount.automountServiceAccountToken is set to false This helps maintain our NSA hardening guide-compliance
```
  automountServiceAccountToken: false
```

line 286, ensure serviceAccount has scheme and tlsConfig values shown below:

serviceMonitor:
  enabled: false
  interval: ""
  additionalLabels: {}
  annotations: {}
  scheme: ""
  tlsConfig: {}
  # scrapeTimeout: 10s

line 295, ensure persistence is enabled and size is increased to 15Gi

persistence:
  enabled: true
  # storageClassName: local-path
  accessModes:
    - ReadWriteOnce
  size: 15Gi

line 303, ensure podAnnotations includes istio inbound ports

podAnnotations:
  traffic.sidecar.istio.io/includeInboundPorts: "16687,16686,3100"

EOF, add default dev.bigbang.mil hostname and addditional Big Bang values

# -- Domain used for BigBang created exposed services
domain: dev.bigbang.mil
# -- Toggle istio integration. Intended to be controlled via BigBang passthrough of istio package status
istio:
  enabled: false
  # -- Default peer authentication values
  hardened:
    enabled: false
    outboundTrafficPolicyMode: "REGISTRY_ONLY"
    customServiceEntries: []
      # - name: "allow-google"
      #   enabled: true
      #   spec:
      #     hosts:
      #       - google.com
      #     location: MESH_EXTERNAL
      #     ports:
      #       - number: 443
      #         protocol: TLS
      #         name: https
      #     resolution: DNS
    customAuthorizationPolicies: []
    # - name: "allow-nothing"
    #   enabled: true
    #   spec: {}
    tempo:
        enabled: false
        namespaces:
        - tempo
        principals:
        - cluster.local/ns/tempo/sa/tempo-tempo  
  mtls:
    # -- STRICT = Allow only mutual TLS traffic,
    # PERMISSIVE = Allow both plain text and mutual TLS traffic
    mode: STRICT
  # -- Tempo-Query specific VirtualService values
  tempoQuery:
    # -- Toggle VirtualService creation
    enabled: true
    annotations: {}
    labels: {}
    gateways:
      - istio-system/main
    hosts:
      - tracing.{{ .Values.domain }}

objectStorage:
  # -- AWS access_key_id for External ObjectStorage configuration
  access_key_id: ""
  # -- AWS secret_access_key for External ObjectStorage configuration
  secret_access_key: ""

# -- Toggle for BigBang specific NetworkPolicies.
# If disabled no NetworkPolicies will be installed with package
# ref: https://kubernetes.io/docs/concepts/services-networking/network-policies/
networkPolicies:
  enabled: false
  # -- Istio IngressGateway labels for VirtualService external routing to app UI
  ingressLabels:
    app: istio-ingressgateway
    istio: ingressgateway
  # -- Use `kubectl cluster-info` and then resolve to IP for kube-api.
  # Review value description in BigBang README.md
  controlPlaneCidr: 0.0.0.0/0
  additionalPolicies: []

# -- Toggle monitoring integration. Intended to be controlled via BigBang passthrough of monitoring package status
monitoring:
  enabled: false

# -- SSO toggle. Intended to be controlled via BigBang passthrough, only affects network/auth policies.
sso:
  enabled: false

bbtests:
  enabled: false
  cypress:
    artifacts: true
    envs:
      cypress_url: 'http://{{ template "tempo.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local:16686'
      cypress_tempo_datasource: 'http://{{ template "tempo.fullname" . }}.{{ .Release.Namespace }}.svc:3100'
      cypress_check_datasource: "false"
      cypress_grafana_url: "http://monitoring-grafana.monitoring.svc.cluster.local"
    resources:
      requests:
        cpu: "1"
        memory: "1Gi"
      limits:
        cpu: "2"
        memory: "2Gi"
  scripts:
    enabled: true
    image: registry1.dso.mil/ironbank/big-bang/base:2.1.0
    envs:
      TEMPO_METRICS_URL: 'http://{{ template "tempo.fullname" . }}.{{ .Release.Namespace }}.svc:3100'

# -- Toggle or openshift specific config
openshift: false

chart/templates/service.yaml

Added protocols to each port name (i.e. tcp, http, etc)

line 35, ensure name is http-tempo-prom-metrics
line 39, ensure name is http-jaeger-metrics
line 42, ensure name is http-tempo-query-jaeger-ui
line 46, ensure name is udp-tempo-jaeger-thrift-compact
line 50, ensure name is udp-tempo-jaeger-thrift-binary
line 54, ensure name is http-tempo-jaeger-thrift-http
line 62, ensure name is tcp-tempo-zipkin
line 66, ensure name is tcp-tempo-otlp-legacy
line 70, ensure name is http-tempo-otlp-http-legacy
line 78, ensure name is http-tempo-otlp-http
line 82, ensure name is tcp-tempo-opencensus

chart/templates/servicemonitor.yaml

Modified ports to match naming convention with http- prefix

line 26, ensure port is http-tempo-prom-metrics

line 33-39, ensure this section is added for http-tempo-prom-metrics:

      {{- if .Values.serviceMonitor.scheme }}
      scheme: {{ .Values.serviceMonitor.scheme }}
      {{- end }}
      {{- if .Values.serviceMonitor.tlsConfig }}
      tlsConfig:
        {{- toYaml .Values.serviceMonitor.tlsConfig | nindent 8 }}
      {{- end }}

line 40, ensure port is http-jaeger-metrics

line 47-53, ensure this section is added for http-jaeger-metrics:

      {{- if .Values.serviceMonitor.scheme }}
      scheme: {{ .Values.serviceMonitor.scheme }}
      {{- end }}
      {{- if .Values.serviceMonitor.tlsConfig }}
      tlsConfig:
        {{- toYaml .Values.serviceMonitor.tlsConfig | nindent 8 }}
      {{- end }}

chart/templates/statefulset.yaml

line 95-99, add in envFrom section to the tempo container

{{- if and .Values.objectStorage.access_key_id .Values.objectStorage.secret_access_key }}
envFrom:
- secretRef:
    name: tempo-object-storage
{{- end }}

chart/templates/authorization-policies/*📜

Add the Istio Authorization Policies

chart/templates/bigbang/*📜

Add Big Bang network Policies as applicable
Add VirtualService for tempo-query UI access
Add openTelemetry collector deployment/configurations

chart/tests/*📜

Add cypress testing configuration and tests
Add scripts for testing

Testing new Tempo Version📜

NOTE: For these testing steps it is good to do them on both a clean install and an upgrade. For clean install, point Loki to your branch. For an upgrade do an install with Loki pointing to the latest tag, then perform a helm upgrade with Loki pointing to your branch.

You will want to install with: - Tempo, monitoring and Istio packages enabled - Jaeger disabled

overrides/tempo.yaml

flux:
  interval: 1m
  rollback:
    cleanupOnFail: false

istio:
  enabled: true

clusterAuditor:
  enabled: false

gatekeeper:
  enabled: false

istioOperator:
  enabled: true

monitoring:
  enabled: true
  values:
    istio:
      enabled: true
      hardened:
        enabled: true

loki:
  enabled: false

promtail:
  enabled: false

grafana:
  enabled: true
  values:
    istio:
      enabled: true
      hardened:
        enabled: true


tempo:
  enabled: true
  git:
    tag: null
    branch: "renovate/ironbank"
  values:
    istio:
      enabled: true
      hardened:
        enabled: true

jaeger:
  enabled: false

kyverno:
  enabled: true

kyvernoPolicies:
  enabled: true
kyvernoPolicies:
  values:
    exclude:
      any:
      # Allows k3d load balancer to bypass policies.
      - resources:
          namespaces:
          - istio-system
          names:
          - svclb-*
    policies:
      restrict-host-path-mount-pv:
        parameters:
          allow:
          - /var/lib/rancher/k3s/storage/pvc-*

Visit https://tracing.dev.bigbang.mil
Ensure Services are listed and traces are being rendered
Check the logs for the tempo pod and container and ensure traceIDs are getting sent over from the istio mesh
Visit https://grafana.dev.bigbang.mil and login with default credentials
Search for Data Sources -> click Tempo -> click Save & Test datasource at the bottom

When in doubt with any testing or upgrade steps, reach out to the CODEOWNERS for assistance.

Last update: 2024-07-29 by Kevin Scheunemann