Release Notes - 3.11.0📜
Please see our documentation page for more information on how to consume and deploy BigBang.\ This release was primarily tested on Kubernetes 1.33.5 (EKS).
Deprecations📜
HAProxy📜
- Big Bang team is planning to deprecate support for the HAProxy package in BigBang. If your team/project relies on the big bang HAProxy package, please let us know, either on the Big Bang community Slack, or P1 Mattermost or by creating an issue in the big bang backlog. This will help us ensure we account for all user needs going forward. If we do not receive any feedback, we will assume the software is no longer used, and will proceed with our plan to deprecate support and updates. As of this announcement, plans are to remove HAProxy from the Big Bang umbrella in an upcoming release.
Upcoming Documentation Restructure - Feedback Requested📜
We’re preparing a significant reorganization of the Big Bang documentation to improve navigability and maintainability. This is currently available for review in MR #6636.
What’s Changing📜
This restructure represents the first phase of our documentation improvement initiative, focusing primarily on reorganization to make content easier to locate and navigate. Key changes include:
- File relocations: Documentation files have been moved to more logical locations within the structure
- Asset path updates: Content previously under
/docs/assets/has been relocated to/docs/reference/ - Link updates: Internal documentation links have been updated to reflect the new structure
- Binary Files: Binary files have been relocated to bb-static
Upgrade Notices📜
BigBang - MR📜
/scripts/install_flux.sh will now use REGISTRY1_USERNAME and REGISTRY1_TOKEN if they are available, unless the flags are set.
BigBang - MR📜
Mimir Monitoring Improvements: Mimir will now enable ServiceMonitors and MetaMonitoring Grafana dashboards when applicable Big Bang services are enabled.
BigBang - MR📜
The Endpoint field is no longer required as part of the loki.objectStorage schema. Users can remove the Endpoint field and setup Loki to S3 connection with IRSA
BigBang - MR📜
Kubernetes 1.32 is now required as 1.31 reached EOL.
BigBang - MR📜
Upgrade your Flux CLI. This MR contains the following updates to Flux controllers:
| Package | Update | Change |
|---|---|---|
| registry1.dso.mil/ironbank/fluxcd/helm-controller (source) | minor | v1.3.0 -> v1.4.2 |
| registry1.dso.mil/ironbank/fluxcd/kustomize-controller (source) | minor | v1.6.1 -> v1.7.1 |
| registry1.dso.mil/ironbank/fluxcd/notification-controller (source) | minor | v1.6.0 -> v1.7.3 |
| registry1.dso.mil/ironbank/fluxcd/source-controller (source) | minor | v1.6.2 -> v1.7.2 |
Mattermost - MR📜
The internal PostgreSQL has been updated to release 18 which requires a dump/restore using pg_dumpall or using pg_upgrade for migrating data to this new, major release. Platform One does not support an internal postgres database for production deployments. This option should only be used for development or CI pipelines. Documentation and other changes can be found here: https://www.postgresql.org/docs/18/release-18.html#RELEASE-18-MIGRATION
Monitoring - MR📜
All upstream chart value overrides in ./chart/values.yaml will need to be shifted under the upstream key.
For example:
Previous Values:
prometheus:
image:
repository: ironbank/opensource/prometheus/prometheus
pullPolicy: Always
tag: v3.0.0
Current Values:
upstream:
prometheus:
image:
repository: ironbank/opensource/prometheus/prometheus
pullPolicy: Always
tag: v3.0.0
Please note, this change does not apply to Big Bang-added top-level keys, including: istio, networkPolicies, openshift, upgradeCrds.
Tempo - MR📜
Upgrades from < 1.21.1-bb.3
If you are upgrading from a Tempo package version earlier than 1.21.1-bb.3, you must also account for the breaking changes introduced in 1.21.1-bb.3. Please review the 1.21.1-bb.3 Upgrade Notices.
In summary:
- Passthrough refactor: all package-specific values that were previously under tempo: must be moved under tempo.values.upstream:. Big Bang–specific top-level keys such as domain, sso, istio, networkPolicies, bbtests, and waitJob remain at the top level and are not moved.
Example:
Previous values:
tempo:
repository: registry1.dso.mil/ironbank/opensource/grafana/tempo
tag: 2.7.2
pullPolicy: IfNotPresent
tempo:
values:
upstream:
repository: registry1.dso.mil/ironbank/opensource/grafana/tempo
tag: 2.7.2
pullPolicy: IfNotPresent
upgradeJob.enabled=true deploys a Helm pre-upgrade job that deletes and recreates the tempo-tempo StatefulSet to handle immutable field changes. A brief outage is expected while the tempo-tempo pods roll.
Upgrades to 1.23.3-bb.0 – Tempo Port Change (3100 -> 3200)
The pod annotation traffic.sidecar.istio.io/includeInboundPorts has been removed from the tempo-tempo pods and requires a pod restart to take effect. When upgrading directly from <1.21.1-bb.3 to 1.23.3-bb.0, the upgradeJob introduced in 1.21.1-bb.3 (if enabled) should also take care of restarting the Tempo pods so they pick up the annotation changes and new port configuration in the 1.23.3-bb.0 release.
Operators should verify that the tempo-tempo pods have been restarted; if not, restart the pods for the Tempo StatefulSet (for example, with kubectl rollout restart statefulset tempo-tempo -n tempo) before considering the upgrade complete.
Tempo - MR📜
There are two major upgrade notices 1) passthrough refactor and 2) upgrade job for redeploying immutable statefulset.
Upgrade Job for Immutable Statefulset: As part of this passthrough pattern refactor mentioned above, changes to the tempo package, upgrading to 1.21.1-bb.3 could involve changes to immutable fields that requires the tempo-tempo statefulset to be deleted before upgrading the release.
The Tempo upgradeJob field in values.yaml can deploy a pre-upgrade job that automates the required statefulset deletion without any additional steps by using a helm pre-upgrade hook, so no additional actions is required. It should be noted, a brief outage is expected during upgrade while the tempo-tempo statefulset is being rolled out.
However, if you would rather manually delete the tempo-tempo statefulset prior to upgrade, then you would need to set the upgradeJob.enabled=false in the values.yaml and follow steps outlined below. The below command assumes that the Tempo package is deployed in the default Big Bang tempo namespace and tempo-tempo statefulset name for Tempo, one should look to confirm the namespace and name of their Tempo statefulset:
kubectl delete statefulset -l app.kubernetes.io/instance=tempo-tempo -n tempo
Once the resources have been deleted, you can upgrade the release.
Harbor - MR📜
-
This release includes major postgresql version change in harbor Helm chart. Due to these changes in postgersql version, this can cause the
helm upgradecommand to fail. If you are using the not recommended way of the database provided with the Harbor helm chart you will need to dump database , reinstall the harbor helm chart with the new postgresql version and then restore the database. The recommended approachfor productionis always to use an external database for harbor. -
The OCI url for redis-bb has been changed to include maintained. Ensure any other references are updated, as new artifacts are no longer published to the previous location.
Upgrades from previous releases📜
If coming from a version pre-3.10.0, note the additional upgrade notices in any release in between. The BB team doesn’t test/guarantee upgrades from anything pre-3.10.0.
Packages📜
| Package | Type | Package Version | BB Version |
|---|---|---|---|
 Alloy |
Core | v1.10.0 |
3.2.1-bb.5 🔗 |
| Anchore Enterprise | Addon | 5.20.2 |
3.14.2-bb.4 |
| Argocd | Addon | v3.1.8 |
9.0.3-bb.0 |
| Authservice | Addon | 1.1.1 |
1.1.1-bb.3 |
Backstage  |
Addon | 1.0.10 |
2.5.3-bb.1 |
| Bbctl | Core | 2.0.0 |
3.0.1-bb.0 |
| Eck Operator | Core | 3.1.0 |
3.1.0-bb.1 |
| Elasticsearch Kibana | Core | Kibana: 9.1.4 Elasticsearch: 9.1.4 |
1.32.0-bb.0 |
| External Secrets Operator |
Addon | 0.20.4 |
0.20.4-bb.0 🔗 |
| Fluentbit | Core | 4.0.8 |
0.52.0-bb.0 |
| Fortify | Addon | 25.2.1.0010 |
1.1.2320154-bb.37 |
| Gatekeeper |
Core | v3.20.1 |
3.20.1-bb.0 🔗 |
| Gitlab | Addon | 18.5.1 |
9.5.1-bb.0 |
| Gitlab Runner |
Addon | v18.5.0 |
0.82.0-bb.1 🔗 |
| Grafana | Core | 12.2.0 |
10.0.0-bb.1 |
| Haproxy | Addon | v2.2.33 |
1.19.3-bb.10 |
| Harbor |
Addon | 2.14.0 |
1.18.0-bb.6 🔗 |
| Headlamp |
Addon | 0.36.0 |
0.36.0-bb.5 🔗 |
| Istio Cni | Core | 1.27.3 |
1.27.3-bb.0 |
| Istio Crds | Core | 1.27.3 |
1.27.3-bb.0 |
| Istio Gateway | Core | 1.27.3 |
1.27.3-bb.0 |
| Istiod | Core | 1.27.3 |
1.27.3-bb.0 |
| Keycloak |
Addon | 26.4.1 |
7.1.4-bb.2 🔗 |
| Kiali | Core | 2.17.0 |
2.17.0-bb.1 |
| Kyverno | Core | v1.15.2 |
3.5.2-bb.0 |
| Kyverno Policies | Core | 3.3.4 |
3.3.4-bb.15 |
| Kyverno Reporter | Core | 3.5.0 |
3.5.0-bb.3 |
| Loki | Core | 3.5.1 |
6.30.1-bb.6 |
| Mattermost |
Addon | 11.0.4 |
11.0.4-bb.2 🔗 |
| Mattermost Operator |
Addon | 1.25.2 |
1.25.2-bb.1 🔗 |
| Metrics Server |
Addon | v0.8.0 |
3.13.0-bb.3 🔗 |
| Mimir |
Addon | 2.17.1 |
5.8.0-bb.3 🔗 |
| Minio |
Addon | RELEASE.2025-10-15T17-29-55Z |
7.1.1-bb.15 🔗 |
| Minio Operator |
Addon | v7.1.1 |
7.1.1-bb.3 🔗 |
| Monitoring |
Core | Prometheus: 3.4.2 Grafana: 12.0.2 Alertmanager: 0.28.1 |
75.6.1-bb.10 🔗 |
| Neuvector |
Core | 5.4.7 |
2.8.9-bb.0 🔗 |
| Nexus Repository Manager | Addon | 3.84.0-03 |
84.0.0-bb.1 |
| Prometheus Operator Crds | Core | 21.0.0 |
23.0.0-bb.0 |
| Sonarqube | Addon | 25.10.0.114319-community |
2025.5.0-bb.1 |
| Tempo |
Core | Tempo: 2.8.2 Tempo Query: 2.8.2 |
1.23.3-bb.0 🔗 |
| Thanos | Addon | v0.39.2 |
17.2.2-bb.0 |
| Twistlock |
Core | 34.02.133 |
0.23.0-bb.2 🔗 |
| Vault | Addon | 1.20.4 |
0.31.0-bb.1 |
| Velero |
Addon | 1.17.0 |
11.1.1-bb.0 🔗 |
| Wrapper | Core | 0.4.15 |
0.4.15 |
Changes in 3.11.0📜
Big Bang MRs📜
- !7063 SKIP SUMMARY CHECK bump neuvector back with ib controller image fix working
- !7062 SKIP SUMMARY CHECK SKIP UPGRADE Revert “Updated neuvector git tag”
- !7054 tests(monitoring): add unit tests for webhook netpol
- !7048 fix(monitoring): add netpol for kube api ingress to fix webhooks
- !7047 update helm unittest docs
- !7046 feat: Resolve “Have install flux respect the same variables as k3d dev”
- !7042 feat(mimir metamonitoring): Enable Mimir built-in Dashboards + ServiceMonitors
- !7041 remove superflouous end statement
- !7036 Remove Endpoint as Loki S3 ObjectStore required field
- !7033 chore: Resolve “Require K8s 1.32”
- !7027 feat(scripts): add script that templates all package resources
- !7022 Enable kyverno policy exceptions in test-values
- !7010 adjust grafana flux auth secret
- !6911 Update Flux
- !6794 Resolve “Document images v2 functionality”
Alloy📜
- !7052: alloy update to 3.2.1-bb.5
# Changelog Updates ## [3.2.1-bb.5] (2025-11-06) ### Fixed - Update cypress tests to run with generous retry
External Secrets Operator📜
- !7055: externalSecrets update to 0.20.4-bb.0
- !7032: externalSecrets update to 0.20.3-bb.2
# Changelog Updates ## [0.20.4-bb.0] - 2025-11-04 ### Updated - Updated registry1.dso.mil/ironbank/opensource/external-secrets/external-secrets v0.20.3 -> v0.20.4 - Updated gluon 0.9.5 -> 0.9.6 ## [0.20.3-bb.2] - 2025-10-30 ### Updated - Removed waitJob image from values.yaml ## [0.20.3-bb.1] - 2025-10-22 ### Updated - Updated waitJob values and format
Gatekeeper📜
- !7045: gatekeeper update to 3.20.1-bb.0
# Changelog Updates ## [3.20.1-bb.0] (2025-11-03) ### Changed - gluon 0.9.0 -> 0.9.6 - registry1.dso.mil/ironbank/opensource/kubernetes/kubectl v1.32.8 -> v1.33 - registry1.dso.mil/ironbank/opensource/openpolicyagent/gatekeeper v3.20.0 -> v3.20.1 ### Changed - gluon 0.5.19 -> 0.9.0
Gitlab Runner📜
- !7016: gitlabRunner update to 0.82.0-bb.1
# Changelog Updates ## [0.82.0-bb.1] (2025-10-28) ### Changed - ironbank/gitlab/gitlab-runner/gitlab-runner (source) v18.4.0 -> v18.5.0 - ironbank/gitlab/gitlab-runner/gitlab-runner-helper (source) v18.4.0 -> v18.5.0 - registry1.dso.mil/ironbank/gitlab/gitlab-runner/gitlab-runner v18.4.0 -> v18.5.0 - registry1.dso.mil/ironbank/gitlab/gitlab-runner/gitlab-runner-helper v18.4.0 -> v18.5.0
Harbor📜
- !7050: harbor update to 1.18.0-bb.6
- !7025: SKIP UPGRADE CHECK harbor update to 1.18.0-bb.5
# Changelog Updates ## [1.18.0-bb.6] (2025-11-06) ### Changed - gluon 0.9.5 -> 0.9.6 - registry1.dso.mil/ironbank/opensource/nginx/nginx 1.29.2 -> 1.29.3 - postgresql 18.1.1 -> 18.1.4 ## [1.18.0-bb.5] (2025-10-21) ### Changed - postgresql 18.0.17 -> 18.1.1 - registry1.dso.mil/ironbank/opensource/postgres/postgresql 17.6 -> 18.0 - redis 22.0.7-bb.1 -> 23.1.1-bb.0
Headlamp📜
- !7030: headlamp update to 0.36.0-bb.5
# Changelog Updates ## [0.36.0-bb.5] (2025-10-30) ### Updated - Updated gluon 0.9.5 -> 0.9.6
Keycloak📜
- !7037: keycloak update to 7.1.4-bb.2
# Changelog Updates ## [7.1.4-bb.2] - 2025-10-28 ### Added - Added bb-common netpol integration
Mattermost📜
- !7061: mattermost update to 11.0.4-bb.2
- !7024: mattermost update to 11.0.4-bb.0
# Changelog Updates ## [11.0.4-bb.2] (2025-11-06) ### Updated - Updated waitjob image format ## [11.0.4-bb.1] (2025-10-31) ### Updated - Updated Hardcoded ssl disable in db-credentials secret - Updated registry1.dso.mil/ironbank/opensource/minio/minio:RELEASE.2025-09-07T16-13-09Z to registry1.dso.mil/ironbank/opensource/minio/minio:RELEASE.2025-10-15T17-29-55Z ## [11.0.4-bb.0] (2025-10-30) ### Updated - Updated registry1.dso.mil/ironbank/opensource/postgres/postgresql (source) 17.6 -> 18.0 - Updated registry1.dso.mil/ironbank/opensource/minio/minio (source) RELEASE.2025-09-07T16-13-09Z -> RELEASE.2025-10-15T17-29-55Z - Updated mattermost image tag to 11.0.4 - Updated gluon updated from 0.9.5 -> 0.9.6
Mattermost Operator📜
- !7039: mattermostOperator update to 1.25.2-bb.1
# Changelog Updates ## [1.25.2-bb.1] (2025-11-03) ### Changed - Updated the allow-egress-api network policy template to allow setting vpcCidr
Metrics Server📜
- !7007: metricsServer update to 3.13.0-bb.3
# Changelog Updates ## [3.13.0-bb.3] (2025-10-27) ### Changed - gluon 0.7.0 -> 0.9.5
Mimir📜
- !7023: mimir update to 5.8.0-bb.3
- !7006: mimir update to 5.8.0-bb.2
# Changelog Updates ## [5.8.0-bb.3] (2025-10-24) ### Changed - Convert NetworkPolicies to bb-common format. - Updated gluon to 0.9.5 ## [5.8.0-bb.2] (2025-10-17) ### Changed - Modified NetworkPolicy and AuthorizationPolicy to allow alloy-metrics connectivity to Mimir - Increased default Alloy limits
Minio📜
- !7060: minio update to 7.1.1-bb.15
- !7029: minio update to 7.1.1-bb.13
# Changelog Updates ## [7.1.1-bb.15] - 2025-11-10 ### Updated - Updated gluon from 0.9.5 -> 0.9.6 - Updated bbtest.scripts.image -> registry1.dso.mil/ironbank/big-bang/devops-tester:1.0 ## [7.1.1-bb.14] - 2025-11-10 ### Updated - Removed kubectl image from values and chart.yaml to update MiniO waitJob ## [7.1.1-bb.13] - 2025-10-23 ### Updated - Updated registry1.dso.mil/ironbank/opensource/minio/minio (source) RELEASE.2025-09-07T16-13-09Z -> RELEASE.2025-10-15T17-29-55Z
Minio Operator📜
- !7044: Minio operator post passthrough cleanup
# Changelog Updates ## [7.1.1-bb.3] (2025-11-06) ### Changed - Removed outdated env vars
Monitoring📜
- !6915: monitoring update to 75.6.1-bb.10
- !6875: monitoring update to 75.6.1-bb.8
# Changelog Updates ## [75.6.1-bb.10] (2025-10-06) ### Fixed - Fixed multi-replica prometheus AuthorizationPolicy ## [75.6.1-bb.9] (2025-10-01) ### Updated - Updated gluon 0.8.0 -> 0.9.2 ## [75.6.1-bb.8] (2025-09-30) ### Fixed - Fixed Image list in chart.yaml for airgap/zarf customers ## [75.6.1-bb.7] (2025-09-29) ### Updated - Adjusted helmignore preventing package release - Updated prometheus-snmp-exporter to 8.0.0 ## [75.6.1-bb.6] (2025-09-25) ### Updated - Updated monitoring to passthrough pattern ### Changed - set `.Values.upstream.prometheus-node-exporter.hostNetwork` and `.Values.upstream.prometheus-node-exporter.hostPID` to `false` in order to resolve OPA violations with prometheus node exporter daemonset
Neuvector📜
- !7040: neuvector update to 2.8.9-bb.0
# Changelog Updates ## [2.8.9-bb.0] (2025-11-04) ### Changed - registry1.dso.mil/ironbank/neuvector/neuvector/controller 5.4.6 -> 5.4.7 - registry1.dso.mil/ironbank/neuvector/neuvector/enforcer 5.4.6 -> 5.4.7 - registry1.dso.mil/ironbank/neuvector/neuvector/manager 5.4.6 -> 5.4.7 - registry1.dso.mil/ironbank/opensource/neuvector/registry-adapter 0.1.9 -> 0.2.1 - Updated gluon subchart v0.9.0 -> v0.9.6
Tempo📜
- !7034: tempo update to 1.23.3-bb.0
- !6812: tempo update to 1.21.1-bb.3 (Passthrough Pattern Refactor)
# Changelog Updates ## [1.23.3-bb.0] (2025-10-09) ### Updated - Update tempo chart tag from 1.21.1 to 1.23.3 - Changed tempo port from `3100` to `3200` - Removed the podAnnotation/traffic.sidecar.istio.io/includeInboundPorts ## [1.21.1-bb.3] (2025-10-05) ### Changed - Refactor Chart to Passthrough Pattern - Added an upgrade job for automated upgrade - Added a configmap to help provide meta data on helm chart version - gluon updated from 0.5.19 to 0.9.2 - Update the tempo and tempoQuery from 2.7.2 to 2.8.2
Twistlock📜
- !7028: twistlock update to 0.23.0-bb.2
# Changelog Updates ## [0.23.0-bb.2] (2025-10.29) ### Changed - gluon updated from 0.9.2 to 0.9.6
Velero📜
- !7058: velero update to 11.1.1-bb.0
# Changelog Updates ## [11.1.1-bb.0] - 2025-11-03 ### Updated - Update velero 10.0.7 -> 11.1.1 - Update gluon 0.9.5 -> 0.9.6 - Update ironbank/opensource/nginx/nginx 1.29.2 -> 1.29.3 - Update registry1.dso.mil/ironbank/opensource/nginx/nginx 1.29.2 -> 1.29.3 - Update registry1.dso.mil/ironbank/opensource/velero/velero v1.16.1 -> v1.17.0
Known Issues📜
- bbctl Dashboards
- CRON job output longer than 16kb will be split into multiple log entries when using the dockerd CRI causing invalid JSON structures to be imported into Loki. Use containerd as the CRI to ensure long log lines are parsed correctly
- bbctl-violations-dashboard / bbctl-all-logs-dashboard(Violations Logs)
- These items will not populate if you have too large of a kubernets cluster with too many violations. There is a limit to the amount of data that can be processed. If you hit this limit and need the information, you can still use the
bbctl violationscommand to obtain the data.
- These items will not populate if you have too large of a kubernets cluster with too many violations. There is a limit to the amount of data that can be processed. If you hit this limit and need the information, you can still use the
- Headlamp
- An issue with the flux plugin being able to load certain menu items has been identified. This appears to be an issue with the javascript code used to create the plugin.
- Menu items having an issue:
- Kustomizations
- HelmReleases
- ImageAutomations
- Notifications
- Attempting to login using OIDC will create a login ‘loop’. See upstream issue for further information.
Helpful Links📜
As always, we welcome and appreciate feedback from our community of users. Please feel free to:
- Open issues here
- Join our Mattermost channel
- Join our Slack
- Check out the documentation for guidance on how to get started
Future📜
Don’t see your feature and/or bug fix? Check out our epics for estimates on when you can expect things to drop, and as always, feel free to comment or create issues if you have questions, comments, or concerns.