Skip to content

How to upgrade the Thanos Package chart

All steps except 7 and 10 are automated by Renovate. The steps below are kept for reference and manual reproduction.

  1. Navigate to the upstream chart repo and folder and identify the released chart version.

    • Bitnami no longer publishes the Thanos chart to the OCI location previously used by this package.
    • The source Chart.yaml may contain a placeholder such as 17.x.x; replace it with the released version before running Helm.
    • Upgrades now require vendoring the unpacked upstream Thanos chart in chart/charts/thanos/.
    • Check the upstream release notes for upgrade notices.

  2. Checkout the working branch for the update, typically renovate/ironbank.

  3. Set shell variables for the old and new versions, then download and vendor the upstream chart source. Replace the placeholders with the actual versions (e.g. NEW_VERSION=17.6.0, OLD_VERSION=17.5.0):

    export NEW_VERSION=<new-version>
export OLD_VERSION=<old-version>
    • Pull the Bitnami charts source archive:
    curl -L https://github.com/bitnami/charts/archive/refs/tags/thanos/${NEW_VERSION}.tar.gz -o /tmp/bitnami-charts-main.tar.gz
    • Extract the archive:
    rm -rf /tmp/bitnami-charts-main
mkdir -p /tmp/bitnami-charts-main
tar -xzf /tmp/bitnami-charts-main.tar.gz -C /tmp/bitnami-charts-main
    • Replace the vendored Thanos chart source. The archive extracts to a directory named charts-thanos-${NEW_VERSION}, not charts-main:
    rm -rf chart/charts/thanos
cp -R /tmp/bitnami-charts-main/charts-thanos-${NEW_VERSION}/bitnami/thanos chart/charts/thanos
    • Resolve the upstream chart’s dependencies. This updates its Chart.lock and populates its charts/ directory from the dependencies declared by the new upstream version:
    helm dependency update ./chart/charts/thanos

  4. Update chart/Chart.yaml.

    • Set the upstream aliased dependency version to the new chart version.
    - name: thanos
  alias: upstream
  version: "<new-version>"
    • Update the package chart version and append -bb.0 to the new Big Bang package version. See Update main chart section of this document.

  5. Regenerate the package dependencies and chart/Chart.lock:

    helm dependency update ./chart

    Chart.lock is generated by Helm and must not be edited manually. Confirm that its thanos entry has the new version and an empty repository value.

  6. Package the resolved upstream chart and remove the unpacked source:

    helm package ./chart/charts/thanos --destination ./chart/charts
rm -rf chart/charts/thanos
[ "${OLD_VERSION}" != "${NEW_VERSION}" ] && rm -f chart/charts/thanos-${OLD_VERSION}.tgz

    Run helm dependency update ./chart before this step. It requires the unpacked chart/charts/thanos/ directory to generate Chart.lock. The final repository state should contain chart/charts/thanos-<new-version>.tgz, not the unpacked directory.

  7. Validate that the package still renders with the vendored chart archive:

    helm template thanos chart > /tmp/thanos-render.yaml
    • Clean up temp files and shell variables:
    rm -rf /tmp/bitnami-charts-main /tmp/bitnami-charts-main.tar.gz
unset NEW_VERSION
unset OLD_VERSION

  8. Update CHANGELOG.md adding an entry for the new version and noting all changes in a list (at minimum should include - Updated <chart or dependency> to x.x.x).

  9. Generate the README.md updates by following the guide in gluon.

  10. Push up your changes, add upgrade notices if applicable, validate that CI passes.

    • If there are any failures, follow the information in the pipeline to make the necessary updates.

    • Add the debug label to the MR for more detailed information.

    • Reach out to the CODEOWNERS if needed.

Testing a new Thanos version

NOTE: For these testing steps it is good to do them on both a clean install and an upgrade. For clean install, point Thanos to your branch. For an upgrade do an install with Thanos pointing to the latest tag, then perform a helm upgrade with Thanos pointing to your branch.

Because Thanos aggregates data, it makes sense to integrate Thanos with Prometheus, Garage, and Grafana. Two test suites run via helm test thanos -n thanos: - Cypress tests verify the Thanos UI: home page load, simple PromQL query, and Prometheus sidecar registration (/targets page shows monitoring targets). See chart/tests/cypress/. - Script test verifies the Garage S3 data path: authenticated PUT/GET/SHA-256-verify/DELETE via AWS SigV4 directly against garage:3900. See chart/tests/scripts/test-s3.py.

Before deploying, ensure the external dependency stack is running so that Garage is available on the k3d network. Pass --external-dependencies when creating your k3d cluster:

k3d-dev.sh --external-dependencies

This starts PostgreSQL, Garage, and Valkey alongside the cluster. Garage runs at garage:3900 on the k3d-dependencies network (172.30.0.11) with the default test credentials. The thanos bucket is included in the default bucket list and is pre-created automatically.

If the cluster already exists, re-start the dep stack via the interactive menu: select “Re-create external dependencies”.

You will want to install with:

  • Thanos, Monitoring, Grafana, and Istio packages

dev-overrides.yaml

Refer to the ./dev-overrides.yaml test overrides when deploying thanos in local testing. Reminder to set the addons.thanos.git to either the testing branch or tag.

Manual Testing Instructions

When in doubt with any testing or upgrade steps, reach out to the CODEOWNERS for assistance.

automountServiceAccountToken

The mutating Kyverno policy named update-automountserviceaccounttokens is leveraged to harden all ServiceAccounts in this package with automountServiceAccountToken: false. This policy is configured by namespace in the Big Bang umbrella chart repository at chart/templates/kyverno-policies/values.yaml.

This policy revokes access to the K8s API for Pods utilizing said ServiceAccounts. If a Pod truly requires access to the K8s API (for app functionality), the Pod is added to the pods: array of the same mutating policy. This grants the Pod access to the API, and creates a Kyverno PolicyException to prevent an alert.