Release Notes - 2.48.0📜
Please see our documentation page for more information on how to consume and deploy BigBang. This release was primarily tested on Kubernetes 1.30.3 (RKE2).
Upgrade Notices📜
- BigBang - MR:
- This upgrades flux to version
flux version 2.5.1. Upgrade your local flux binary if needed.
- This upgrades flux to version
- Mimir - MR:
- mimir-distributed has deprecated the
nginxservice and related values in favor ofgateway. - All remote_write scraper configurations not included in Big Bang should be updated to point to
mimir-gatewayinstead of nginx. - Important- The upgrade steps below are applicable only when MinIO is enabled for Mimir. As a reminder, MinIO is only supported for development environments. If your environment is not using Big Bang’s MinIO for Mimir, you can ignore the following upgrade notice. If Big Bang’s MinIO is enabled for Mimir (
minio-tenant.enabledto true), remember to test upgrades and maintain proper backups. - Upgrading from
5.5.1-bb.4involves changes to Tenant immutable server field that requires 1) themimir-mimir-minio-tenanttenant to be deleted, 2) remove any finalizers protecting the persistent volumes (pv) that were used by themimir-mimir-minio-tenanttenant, and 3) delete the persistent volumes and persistent volume claims (pvc) used by themimir-mimir-minio-tenanttenant to free up the persistent volumes for the new MinIO tenant before upgrading the release. Due to the recreation of the tenant, any historical metric data in the MinIO tenant buckets, pre-upgrade, will be lost as part of the tenant redeployment. - The Mimir
upgradeJob.enabledfield in values.yaml can deploy apre-upgradejobthat automates the required tenant, pv, and pvc deletion without any additional steps by using a helmpre-upgradehook, so no additional actions is required. It should be noted, 1) the buckets containing historical metrics will be replaced so historical metrics collected will no longer available and 2) a brief outage is expected during upgrade while themimir-mimir-minio-tenanttenant is being rolled out. - However, if you would rather manually complete the pre-upgrade steps listed above prior to upgrade, then you would need to set the
upgradeJob.enabled=falsein thevalues.yamland follow steps outlined below. The below command assumes that the Mimir package is deployed in the default Big Bangmimirnamespace for Mimir, one should look to confirm the namespace of their Mimir deployment: - 1. Deleting the
mimir-mimir-minio-tenantTenantkubectl delete tenant mimir-mimir-minio-tenant -n mimir - 2. Remove any Finalizers for the Persistent Volumes that was used by the Tenant
kubectl patch pv <name of the pv used by the Minio Tenant> --type=json -p '[{"op": "remove", "path": "/metadata/finalizers"}]' - 3. Delete the Persistent Volumes that was used by the Tenant
kubectl delete pv <name of the pv used by the Minio Tenant> - 4. Delete the Persistent Volumes Claims that was used by the Tenant
kubectl delete pvc <name of the pvc used by the Minio Tenant> -n mimir - Once the resources have been deleted, you can upgrade the release
- mimir-distributed has deprecated the
- Alloy - MR:
- The 2.0 release of the k8smonitoring Helm chart includes major changes from the 1.x version. Many of the features have been rearranged to be organized around features, rather than data types (e.g. metrics, logs, etc.). This will require looking at Migration guide from k8smonitoring and possibly using their migration tool available at https://grafana.github.io/k8smonitoringhelmmigrator/ if you have any custom configuration or flags set in the values.yaml.
- Operatorless-Istio - !4906 Operatorless Istio Upgrade Notice -This release includes pre-release packages for some new operatorless Istio components. These packages are in an alpha state and will be subject to rapid changes as Big Bang transitions to operatorless Istio. In the coming days, look for a blog post outlining Big Bang’s strategy for migrating away from the Istio operator, what that will mean for Big Bang consumers, and a timeline for when these changes will be generally available. As always, we thank you for supporting Big Bang.
Upgrades from previous releases📜
If coming from a version pre-2.47.0, note the additional upgrade notices in any release in between. The BB team doesn’t test/guarantee upgrades from anything pre-2.47.0.
Packages📜
| Package | Type | Package Version | BB Version |
|---|---|---|---|
 Alloy |
Addon | 1.5.1 |
2.0.4-bb.0 🔗 |
| Anchore Enterprise | Addon | 5.13.1 |
3.3.2-bb.0 |
| Argocd |
Addon | 2.14.2 |
7.8.2-bb.0 🔗 |
| Authservice | Addon | 1.0.4 |
1.0.4-bb.1 |
| Cluster Auditor | Core | 0.0.7 |
1.5.0-bb.22 |
| Eck Operator | Core | 2.16.1 |
2.16.1-bb.0 |
| Elasticsearch Kibana |
Core | Kibana 8.17.3 Elasticsearch 8.17.3 |
1.27.0-bb.0 🔗 |
| External Secrets |
Addon | 0.14.3 |
0.14.3-bb.0 🔗 |
| Fluentbit |
Core | 3.2.7 |
0.48.6-bb.0 🔗 |
| Fortify | Addon | 24.4.2.0009 |
1.1.2320154-bb.22 |
| Gatekeeper | Core | 3.18.2 |
3.18.2-bb.0 |
| Gitlab |
Addon | 17.9.1 |
8.9.1-bb.0 🔗 |
| Gitlab Runner | Addon | 17.8.0 |
0.73.0-bb.0 |
| Grafana |
Core | 11.5.2 |
8.10.1-bb.0 🔗 |
| Haproxy | Addon | 2.2.33 |
1.19.3-bb.10 |
| Harbor | Addon | 2.11.0 |
1.16.1-bb.0 |
| Holocron |
Addon | 3.3.2 |
1.0.13 🔗 |
| Istio Controlplane |
Core | Istio 1.23.5 Tetrate Istio Distro 1.23.5 |
1.23.5-bb.0 🔗 |
| Istio Operator |
Core | Istio Operator 1.23.5 Tetrate Istio Distro Operator 1.23.5 |
1.23.5-bb.0 🔗 |
| Jaeger | Core | 1.62.0 |
2.57.0-bb.2 |
| Keycloak | Addon | 25.0.6 |
2.5.1-bb.6 |
| Kiali | Core | 2.5.0 |
2.5.0-bb.0 |
| Kyverno |
Core | 1.13.4 |
3.3.6-bb.0 🔗 |
| Kyverno Policies | Core | 3.3.4 |
3.3.4-bb.1 |
| Kyverno Reporter |
Core | 3.0.0 |
3.0.1-bb.2 🔗 |
| Loki |
Core | 3.3.2 |
6.25.1-bb.1 🔗 |
| Mattermost |
Addon | 10.5.1 |
10.5.1-bb.2 🔗 |
| Mattermost Operator | Addon | 1.22.1 |
1.22.1-bb.1 |
| Metrics Server | Addon | 0.7.2 |
3.12.2-bb.1 |
Mimir  |
Addon | '2.14.2' |
5.5.1-bb.6 🔗 |
| Minio |
Addon | RELEASE.2025-01-20T14-49-07Z |
7.0.0-bb.2 🔗 |
| Minio Operator | Addon | 7.0.0 |
7.0.0-bb.1 |
| Monitoring | Core | Prometheus 3.1.0 Grafana 11.4.0 Alertmanager 0.27.0 |
67.11.0-bb.2 |
| Neuvector | Core | 5.4.1 |
2.8.3-bb.1 |
| Nexus | Addon | 3.75.0-06 |
75.0.0-bb.2 |
| Promtail |
Core | 3.4.2 |
6.16.6-bb.2 🔗 |
| Sonarqube | Addon | 10.7.0-community |
10.7.0-bb.0 |
| Tempo |
Core | Tempo 2.7.1 Tempo Query 2.7.1 |
1.18.2-bb.0 🔗 |
| Thanos |
Addon | 0.37.2 |
15.9.1-bb.1 🔗 |
| Twistlock | Core | 33.01.137 |
0.18.0-bb.1 |
| Vault |
Addon | 1.18.5 |
0.29.1-bb.8 🔗 |
| Velero | Addon | 1.15.2 |
8.3.0-bb.0 |
| Wrapper | Core | N / A | 0.4.12 |
Changes in 2.48.0📜
Big Bang MRs📜
- !4906: Operatorless Istio with CORE packages only SKIP UPGRADE DEBUG
- !5844: Adds parameter to flux install to make ns configurable
- !5789: Update Flux
- !5791: remove deprecated, unmaintained utilities image
Alloy📜
- !5753: Migrate alloy from monitoring namespace to its own namespace
- !5802: alloy update to 2.0.4-bb.0
# Changelog Updates
## [2.0.4-bb.0] - 2025-02-06
### Changed
- k8s-monitoring updated from 1.6.18 to 2.0.4
- Added Network Policy to allow Alloy ingress to Alloy Deployment
- Migrated the values.yaml from 1.x.x to 2.0.4
Argocd📜
- !5801: argocd update to 7.8.2-bb.0
# Changelog Updates
## [7.8.2-bb.0] - 2025-02-20
### Changed
- Update ironbank/big-bang/argocd v2.14.1 -> v2.14.2
- Update registry1.dso.mil/ironbank/big-bang/argocd v2.14.1 -> v2.14.2
Elasticsearch Kibana📜
# Changelog Updates
## [1.27.0-bb.0] - 2025-03-05
### Changed
- ironbank/elastic/elasticsearch/elasticsearch updated from 8.17.2 to 8.17.3
- ironbank/elastic/kibana/kibana updated from 8.17.2 to 8.17.3
- prometheus-elasticsearch-exporter helm chart updated from 6.6.0 to 6.6.1
- ironbank/opensource/kubernetes/kubectl updated from v1.30.9 to v1.30.10
## [1.26.0-bb.0] - 2025-02-12
### Changed
- ironbank/elastic/elasticsearch/elasticsearch updated from 8.17.1 to 8.17.2
- ironbank/elastic/kibana/kibana updated from 8.17.1 to 8.17.2
External Secrets📜
# Changelog Updates
## [0.14.3-bb.0] - 2025-02-28
### Changed
- Updated registry1.dso.mil/ironbank/opensource/external-secrets/external-secrets (source) v0.14.2 -> v0.14.3
## [0.14.2-bb.0] - 2025-02-19
### Changed
- Updated registry1.dso.mil/ironbank/opensource/external-secrets/external-secrets (source) v0.14.1 -> v0.14.2
- Updated registry1.dso.mil/ironbank/opensource/kubernetes/kubectl (source) v1.30.9 -> v1.30.10
Fluentbit📜
- !5847: fluentbit update to 0.48.6-bb.0
# Changelog Updates
## [0.48.6-bb.0] - 2025-02-25
### Changed
- ironbank/opensource/fluent/fluent-bit updated from 3.2.6 to 3.2.7
Gitlab📜
- !5842: gitlab update to 8.9.1-bb.0
- !5821: gitlab update to 8.8.2-bb.0
- !5800: gitlab update to 8.8.1-bb.2
# Changelog Updates
## [8.9.1-bb.0] - 2025-02-28
### Changed
- ironbank/gitlab/gitlab/gitlab-webservice (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/certificates (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/gitaly (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-base (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-container-registry (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-exporter (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-mailroom (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-pages (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-shell (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-sidekiq (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-toolbox (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-webservice (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-workhorse (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/gitlab/gitlab/kubectl (source) 17.8.2 -> 17.9.1
- registry1.dso.mil/ironbank/opensource/postgres/postgresql (source) 14.16 -> 14.17
## [8.8.2-bb.0] - 2025-02-24
### Changed
- ironbank/gitlab/gitlab/gitlab-webservice 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/certificates 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/gitaly 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-base 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-container-registry 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-exporter 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-mailroom 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-pages 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-shell 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-sidekiq 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-toolbox 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-webservice 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-workhorse 17.8.1 -> 17.8.2
- registry1.dso.mil/ironbank/gitlab/gitlab/kubectl 17.8.1 -> 17.8.2
Grafana📜
- !5834: grafana update to 8.10.1-bb.0
# Changelog Updates
## [8.10.1-bb.0] - 2025-02-21
### Changed
- ironbank/big-bang/grafana/grafana-plugins updated from 11.5.1 to 11.5.2
- ironbank/kiwigrid/k8s-sidecar updated from 1.29.1 to 1.30.0
Holocron📜
- !5851: SKIP UPGRADE holocron update to 1.0.13
# Changelog Updates
## [1.0.13] - 2025-03-04
### Updated
- Updated gluon chart 0.5.0 => 0.5.14
- Updated postgres chart ^12.0.0 => 13.2.2
- Updated postgres image ironbank/opensource/postgres/postgresql:15.5 => ironbank/opensource/postgres/postgresql:17.4
Istio Controlplane📜
- !5811: istio update to 1.23.5-bb.0
# Changelog Updates
## [1.23.5-bb.0] - 2025-02-19
### Changed
- ironbank/opensource/istio/install-cni updated from 1.23.4 to 1.23.5
- ironbank/opensource/istio/pilot updated from 1.23.4 to 1.23.5
- ironbank/opensource/istio/proxyv2 updated from 1.23.4 to 1.23.5
- ironbank/opensource/kubernetes/kubectl updated from v1.30.8 to v1.30.10
- ironbank/tetrate/istio/install-cni updated from 1.23.4 to 1.23.5
- ironbank/tetrate/istio/pilot updated from 1.23.4 to 1.23.5
- ironbank/tetrate/istio/proxyv2 updated from 1.23.4 to 1.23.5
- Updated Gluon to v0.5.14
Istio Operator📜
- !5810: istioOperator update to 1.23.5-bb.0
# Changelog Updates
## [1.23.5-bb.0] - 2025-02-19
### Changed
- Updated registry1.dso.mil/ironbank/opensource/istio/operator from 1.23.4 to 1.23.5
- Updated registry1.dso.mil/ironbank/tetrate/istio/operator from 1.23.4-tetratefips-v0 to 1.23.5-tetratefips-v0
Kyverno📜
- !5805: kyverno update to 3.3.6-bb.0
# Changelog Updates
## [3.3.6-bb.0] - 2025-02-17
### Changed
- Updated upstream chart from `3.3.4` to `3.3.6` and app version from `v1.13.2` to `v1.13.4`
- Updated `background-controller`, `cleanup-controller`, `reports-controller`, `kyverno`, `kyvernocli`, `kyvernopre` from `v1.13.2` to `v1.13.4`
- Updated `kubectl` from `v1.30.6` to `v1.30.10`
Kyverno Reporter📜
- !5803: kyvernoReporter update to 3.0.1-bb.2
# Changelog Updates
## [3.0.1-bb.2] - 2025-02-18
### Updated
- Updated Big Bang NPs to use a common selector label to match all pods
## [3.0.1-bb.1] - 2025-02-11
### Updated
- Added support for istio Operatorless network policy values
## [3.0.1-bb.0] - 2025-02-10
### Changed
- Updated upstream chart reference from `2.24.2` ---> `3.0.1`
- Updated image from `registry1.dso.mil/ironbank/opensource/kyverno/policy-reporter:2.20.2` ----> `registry1.dso.mil/ironbank/opensource/kyverno/policy-reporter:3.0.0`
- Updated `gluon` package dependency version from `0.5.4` ---> `0.5.14`
Loki📜
- !5836: loki update to 6.25.1-bb.1
# Changelog Updates
## [6.25.1-bb.1] - 2025-02-26
### Changed
- Modified `loki-canary` values to use registry1 image
- Modified `enterprise-logs-provisioner` values to use registry1 image
- Modified `loki-helm-test` values to use registry1 image
Mattermost📜
# Changelog Updates
## [10.5.1-bb.2] - 2025-03-03
### Added
- added namespace-labels to test/dependencies.yaml
## [10.5.1-bb.1] - 2025-02-27
### Changed
- ironbank/opensource/kubernetes/kubectl updated from v1.30.9 to v1.30.10
- ironbank/opensource/postgres/postgresql updated from 17.2 to 17.4
## [10.5.1-bb.0] - 2025-02-25
### Changed
- Updated mattermost to 10.5.1
## [10.4.2-bb.1] - 2025-01-24
### Changed
- minio-instance updated from 6.0.4 to 7.0.0
- ironbank/opensource/minio/operator-sidecar updated from v6.0.2 to v7.0.0
- Updated registry1.dso.mil/ironbank/opensource/minio/mc to RELEASE.2025-01-17T23-25-50Z
Mimir📜
- !5837: mimir update to 5.5.1-bb.6
# Changelog Updates
## [5.5.1-bb.6] - 2025-02-26
### Changed
- Added pre-upgrade job to remove MinIO Tenant Pool prior to upgrade
- Set `.Values.nginx.enabled` to `false` as this is deprecated in favor of `gateway`
## [5.5.1-bb.5] - 2025-02-14
### Changed
- MinIO Tenant Pool from 4 to 1 server
Minio📜
- !5804: minio update to 7.0.0-bb.2
# Changelog Updates
## [7.0.0-bb.2] - 2025-02-07
### Changed
- Add filler value to test-wait-job
Promtail📜
- !5819: promtail update to 6.16.6-bb.2
# Changelog Updates
## [6.16.6-bb.2] - 2025-02-19
### Updated
- Update promtail from `v3.3.2` -> `v3.4.2`
Tempo📜
- !5843: tempo update to 1.18.2-bb.0
# Changelog Updates
## [1.18.2-bb.0] - 2025-02-28
### Updated
- Synchronized with upstream chart version 1.18.2
- Updated tempo: `2.7.0` -> `2.7.1`
- Updated tempo-query `2.7.0` -> `2.7.1`
Thanos📜
# Changelog Updates
## [15.9.1-bb.1] - 2025-02-12
### Upgraded
- Upgraded gluon from `0.5.12` -> `0.5.14`
- Upgraded minio-instance from `6.0.4-bb.5` -> `7.0.0-bb.1`
- Upgraded common from `2.29.0` -> `2.29.1`
Vault📜
- !5835: vault update to 0.29.1-bb.8
- !5833: vault update to 0.29.1-bb.7
- !5832: vault update to 0.29.1-bb.6
# Changelog Updates
## [0.29.1-bb.8] - 2025-02-27
### Changed
- registry1.dso.mil/ironbank/hashicorp/vault-csi-provider v1.6.0 -> 1.5.0
## [0.29.1-bb.7] - 2025-02-27
### Changed
- registry1.dso.mil/ironbank/hashicorp/vault (source) 1.18.4 -> 1.18.5
## [0.29.1-bb.6] - 2025-02-26
### Changed
- Added logic to init container script to check if vault is alsready initialized
Known Issues📜
- Kiali - ISSUE
- On Kubernetes 1.29+, the kiali operator may fail with a 404 while running the kiali-deploy playbook if the cluster returns the
flowcontrol.apiserver.k8s.io/v1beta2api version (no longer served as of v1.29).
In this case, removing the invalid api version should resolve the issue and allow the kiali operator to run successfully.
$ kubectl delete apiservices.apiregistration.k8s.io v1beta2.flowcontrol.apiserver.k8s.io
-
- If you have a manually created gitlab-rails secret, your upgrade may fail with:
$ Errno::EBUSY: Device or resource busy @ rb_file_s_rename - (/srv/gitlab/config/secrets.yml, /srv/gitlab/config/secrets.yml.orig.1738013281) - In order to resolve this you will likely need to manually generate the other 3 secrets described here
- if you see in Prometheus the Error scraping target for the Gitlab-exporter, mentioned in the upgrade notices please read the following :
Steps to Resolve
1. **Verify Service Monitor Configuration** Use the following command to check if the `fallbackScrapeProtocol` line is present: `kubectl -n gitlab get servicemonitor gitlab-gitlab-exporter -o yaml` If the `fallbackScrapeProtocol: PrometheusText1.0.0` is missing, proceed with the next steps. 2. **Update Service Monitor** First, export the current service monitor configuration: `kubectl -n gitlab get servicemonitor gitlab-gitlab-exporter -o yaml > servicemonitor_gitlab_exporter.yaml` Then, delete the existing service monitor: `kubectl -n gitlab delete servicemonitor gitlab-gitlab-exporter` 3. **Redeploy or Update the Helm Release** Redeploy BigBang or force a redeployment of the Helm release. This should ensure that the `fallbackScrapeProtocol: PrometheusText1.0.0` is included, resolving the Prometheus scraping error. - If you have a manually created gitlab-rails secret, your upgrade may fail with:
-
- there is currently a bug within the prometheus Kyverno Reporter (serviceMonitor/kyverno-reporter/policy-reporter-monitoring/0) target
Workaround Steps
- set both the default-istio-system and policy-reporter-default peerauthentication’s to PERMISSIVE
kubectl edit peerauthentication default-istio-system -n istio-system kubectl edit peerauthentication policy-reporter-default -n kyverno-reporter
- set both the default-istio-system and policy-reporter-default peerauthentication’s to PERMISSIVE
- there is currently a bug within the prometheus Kyverno Reporter (serviceMonitor/kyverno-reporter/policy-reporter-monitoring/0) target
Workaround Steps
Helpful Links📜
As always, we welcome and appreciate feedback from our community of users. Please feel free to:
- Open issues here
- Join our chat
- Check out the documentation for guidance on how to get started
Future📜
Don’t see your feature and/or bug fix? Check out our epics for estimates on when you can expect things to drop, and as always, feel free to comment or create issues if you have questions, comments, or concerns.