Skip to content

How to upgrade the Thanos Package chart📜

  1. Navigate to the upstream chart repo and folder and identify the new chart version from bitnami/thanos/Chart.yaml.

    • Bitnami no longer publishes the Thanos chart to the OCI location previously used by this package.
    • Upgrades now require manually vendoring the upstream Thanos chart tarball into chart/charts/.
    • Check the upstream release notes for upgrade notices.

  2. Checkout the working branch for the update, typically renovate/ironbank.

  3. Download the current upstream chart source from GitHub and build a vendored tarball.

    • Pull the Bitnami charts source archive:
    curl -L https://github.com/bitnami/charts/archive/refs/heads/main.tar.gz -o /tmp/bitnami-charts-main.tar.gz
    • Extract the upstream bitnami/thanos chart:
    mkdir -p /tmp/bitnami-charts-main
tar -xzf /tmp/bitnami-charts-main.tar.gz -C /tmp/bitnami-charts-main
    • Extract the currently vendored Thanos chart so you can reuse its bundled charts/ subcharts:
    mkdir -p /tmp/thanos-current
tar -xzf chart/charts/thanos-<old-version>.tgz -C /tmp/thanos-current
    • Assemble and package the new vendored chart:
    rm -rf /tmp/thanos-next
mkdir -p /tmp/thanos-next
cp -R /tmp/bitnami-charts-main/charts-main/bitnami/thanos /tmp/thanos-next/
cp -R /tmp/thanos-current/thanos/charts /tmp/thanos-next/thanos/
helm package /tmp/thanos-next/thanos --destination /tmp/thanos-next/out
    • Replace the vendored tarball in this repo:
    cp /tmp/thanos-next/out/thanos-<new-version>.tgz chart/charts/
rm -f chart/charts/thanos-<old-version>.tgz

  4. Update chart/Chart.yaml.

    • Set the upstream aliased dependency version to the new chart version.
    • Keep the dependency pointed at the vendored local tarball directory:
    - name: thanos
  alias: upstream
  version: "<new-version>"
  repository: file://./charts
    • Update the package chart version and append -bb.0 to the new Big Bang package version. See Update main chart section of this document.

  5. Update chart/Chart.lock.

    • Update the thanos dependency entry to match the new version and repository: file://./charts.
    • Do not run helm dependency update ./chart for the upstream Thanos dependency anymore; the upstream OCI source is no longer available.

  6. Validate that the package still renders with the vendored tarball:

    helm template thanos chart >/tmp/thanos-render.yaml

  7. Update CHANGELOG.md adding an entry for the new version and noting all changes in a list (at minimum should include - Updated <chart or dependency> to x.x.x).

  8. Generate the README.md updates by following the guide in gluon.

  9. Push up your changes, add upgrade notices if applicable, validate that CI passes.

    • If there are any failures, follow the information in the pipeline to make the necessary updates.

    • Add the debug label to the MR for more detailed information.

    • Reach out to the CODEOWNERS if needed.

  10. Follow the Testing a new Thanos version section of this document for manual testing.

  11. As part of your MR that modifies bigbang packages, you should modify the bigbang bigbang/tests/test-values.yaml against your branch for the CI/CD MR testing by enabling your packages.

    • To do this, at a minimum, you will need to follow the instructions at bigbang/docs/developer/test-package-against-bb.md with changes for Thanos enabled (the below is a reference, actual changes could be more depending on what changes were made to Thanos in the package MR).

test-values.yaml📜

thanos:
  enabled: true
  git:
    tag: null
    branch: renovate/ironbank
  values:
    istio:
      hardened:
        enabled: true
  ### Additional components of Thanos should be changed to reflect testing changes introduced in the package MR

automountServiceAccountToken📜

The mutating Kyverno policy named update-automountserviceaccounttokens is leveraged to harden all ServiceAccounts in this package with automountServiceAccountToken: false. This policy is configured by namespace in the Big Bang umbrella chart repository at chart/templates/kyverno-policies/values.yaml.

This policy revokes access to the K8s API for Pods utilizing said ServiceAccounts. If a Pod truly requires access to the K8s API (for app functionality), the Pod is added to the pods: array of the same mutating policy. This grants the Pod access to the API, and creates a Kyverno PolicyException to prevent an alert.

Testing a new Thanos version📜

NOTE: For these testing steps it is good to do them on both a clean install and an upgrade. For clean install, point Thanos to your branch. For an upgrade do an install with Thanos pointing to the latest tag, then perform a helm upgrade with Thanos pointing to your branch.

Because Thanos aggregates data, it makes sense to integrate Thanos with Prometheus, MiniIO, and Grafana. The cypress tests will verify datasources are enabled for the monitoring.prometheus-sidecar and an s3 objectstore datasource is registered. See the values.yaml and bigbang test-values.yaml for configuration settings.

You will want to install with:

  • Thanos, Monitoring, Grafana and Istio packages and passing in test-values.yaml

overrides/thanos.yaml

flux:
  interval: 1m
  rollback:
    cleanupOnFail: false

networkPolicies:
  enabled: true

grafana:
  enabled: true

istioCRDs:
  enabled: true

istiod:
  enabled: true
  values:
    hardened:
      enabled: true

monitoring:
  enabled: true
  values:
    prometheus:
      prometheusSpec:
        replicas: 3
    istio:
      enabled: true
      hardened:
        enabled: true

addons:
  thanos:
    enabled: true
    git:
      tag: null
      branch: thanos-127/convert-to-passthrough
    values:
      minio:
        enabled: true
      storegateway:
        enabled: true
      upstream:
        objstoreConfig: |-
          type: s3
          config:
            bucket: "thanos"
            endpoint: "minio.thanos.svc.cluster.local:80"
            access_key: "minio"
            secret_key: "minio123"
            insecure: true
            trace:
              enable: true
        storegateway:
          enabled: true
        compactor:
          enabled: true
        bucketweb:
          enabled: true
  minioOperator:
    enabled: true

When in doubt with any testing or upgrade steps, reach out to the CODEOWNERS for assistance.