Skip to content

How to upgrade the Thanos Package chart

Change to Steps 3-6 can be automated from the repository root: because scripts/vendor-helm-deps.sh does not check out a branch.

./scripts/vendor-helm-deps.sh
Step 1 should have been done by the renovate bot.

  1. Navigate to the upstream chart repo and folder and identify the released chart version.

    • Bitnami no longer publishes the Thanos chart to the OCI location previously used by this package.
    • The source Chart.yaml may contain a placeholder such as 17.x.x; replace it with the released version before running Helm.
    • Upgrades now require vendoring the unpacked upstream Thanos chart in chart/charts/thanos/.
    • Check the upstream release notes for upgrade notices.

  2. Checkout the working branch for the update, typically renovate/ironbank.

  3. Download the current upstream chart source from GitHub with matching release tag and vendor it. For example https://github.com/bitnami/charts/archive/refs/tags/thanos/<new-version>.tar.gz, so the vendored source matches Chart.yaml.

    • Pull the Bitnami charts source archive:
    curl -L https://github.com/bitnami/charts/archive/refs/tags/thanos/<new-version>.tar.gz -o /tmp/bitnami-charts-main.tar.gz
    • Extract the upstream bitnami/thanos chart:
    rm -rf /tmp/bitnami-charts-main
mkdir -p /tmp/bitnami-charts-main
tar -xzf /tmp/bitnami-charts-main.tar.gz -C /tmp/bitnami-charts-main
    • Replace the vendored Thanos chart source:
    rm -rf chart/charts/thanos
cp -R /tmp/bitnami-charts-main/charts-main/bitnami/thanos chart/charts/thanos
    • Resolve the upstream chart’s dependencies. This updates its Chart.lock and populates its charts/ directory from the dependencies declared by the new upstream version:
    helm dependency update ./chart/charts/thanos

  4. Update chart/Chart.yaml.

    • Set the upstream aliased dependency version to the new chart version.
    - name: thanos
  alias: upstream
  version: "<new-version>"
    • Update the package chart version and append -bb.0 to the new Big Bang package version. See Update main chart section of this document.

  5. Regenerate the package dependencies and chart/Chart.lock:

    helm dependency update ./chart

    Chart.lock is generated by Helm and must not be edited manually. Confirm that its thanos entry has the new version and an empty repository value.

  6. Package the resolved upstream chart and remove the unpacked source:

    helm package ./chart/charts/thanos --destination ./chart/charts
rm -rf chart/charts/thanos
rm -f chart/charts/thanos-<old-version>.tgz

    Run helm dependency update ./chart before this step. It requires the unpacked chart/charts/thanos/ directory to generate Chart.lock. The final repository state should contain chart/charts/thanos-<new-version>.tgz, not the unpacked directory.

  7. Validate that the package still renders with the vendored chart archive:

    helm template thanos chart > /tmp/thanos-render.yaml

  8. Update CHANGELOG.md adding an entry for the new version and noting all changes in a list (at minimum should include - Updated <chart or dependency> to x.x.x).

  9. Generate the README.md updates by following the guide in gluon.

  10. Push up your changes, add upgrade notices if applicable, validate that CI passes.

    • If there are any failures, follow the information in the pipeline to make the necessary updates.

    • Add the debug label to the MR for more detailed information.

    • Reach out to the CODEOWNERS if needed.

Testing a new Thanos version

NOTE: For these testing steps it is good to do them on both a clean install and an upgrade. For clean install, point Thanos to your branch. For an upgrade do an install with Thanos pointing to the latest tag, then perform a helm upgrade with Thanos pointing to your branch.

Because Thanos aggregates data, it makes sense to integrate Thanos with Prometheus, MiniIO, and Grafana. The cypress tests will verify datasources are enabled for the monitoring.prometheus-sidecar and an s3 objectstore datasource is registered. See the values.yaml and bigbang test-values.yaml for configuration settings.

You will want to install with:

  • Thanos, Monitoring, Grafana and Istio packages and passing in test-values.yaml

overrides/thanos.yaml

flux:
  interval: 1m
  rollback:
    cleanupOnFail: false

networkPolicies:
  enabled: true

grafana:
  enabled: true

istioCRDs:
  enabled: true

istiod:
  enabled: true
  values:
    hardened:
      enabled: true

monitoring:
  enabled: true
  values:
    prometheus:
      prometheusSpec:
        replicas: 3
    istio:
      enabled: true
      hardened:
        enabled: true

addons:
  thanos:
    enabled: true
    git:
      tag: null
      branch: renovate/ironbank
    values:
      minio:
        enabled: true
      storegateway:
        enabled: true
      upstream:
        objstoreConfig: |-
          type: s3
          config:
            bucket: "thanos"
            endpoint: "minio.thanos.svc.cluster.local:80"
            access_key: "minio"
            secret_key: "minio123"
            insecure: true
            trace:
              enable: true
        storegateway:
          enabled: true
        compactor:
          enabled: true
        bucketweb:
          enabled: true
  minioOperator:
    enabled: true

When in doubt with any testing or upgrade steps, reach out to the CODEOWNERS for assistance.

automountServiceAccountToken

The mutating Kyverno policy named update-automountserviceaccounttokens is leveraged to harden all ServiceAccounts in this package with automountServiceAccountToken: false. This policy is configured by namespace in the Big Bang umbrella chart repository at chart/templates/kyverno-policies/values.yaml.

This policy revokes access to the K8s API for Pods utilizing said ServiceAccounts. If a Pod truly requires access to the K8s API (for app functionality), the Pod is added to the pods: array of the same mutating policy. This grants the Pod access to the API, and creates a Kyverno PolicyException to prevent an alert.