Troubleshoot a Big Bang deployment¶

Use this guide when a Big Bang deployment, upgrade, or package is not healthy.

Big Bang is deployed declaratively through Flux. Find the first resource that is not ready and fix that failure before troubleshooting downstream resources. An earlier reconciliation failure can cause multiple downstream symptoms.

Find the Failing Layer¶

flowchart TD
    A[Big Bang deployment has a problem] --> B{Is Flux healthy?}

    B -->|No| C[Troubleshoot Flux]
    B -->|Yes| D{Is environment configuration ready?}

    D -->|No| E[Troubleshoot source or Kustomization]
    D -->|Yes| F{Is the Big Bang HelmRelease ready?}

    F -->|No| G[Troubleshoot Big Bang HelmRelease]
    F -->|Yes| H{Are package HelmReleases ready?}

    H -->|No| I[Troubleshoot affected package]
    H -->|Yes| J{Are package workloads healthy?}

    J -->|No| K[Troubleshoot workload or component]
    J -->|Yes| L[Investigate application-specific symptoms]

Work from the highest failing layer downward. Avoid troubleshooting downstream symptoms while an earlier dependency is not ready.

Check Deployment Status¶

Start by checking Flux and listing resources that are not ready:

flux check
flux get all -A --status-selector ready=false

If you need the complete reconciliation status:

flux get sources all -A
flux get kustomizations -A
flux get helmreleases -A

For a resource that is not ready, inspect its conditions, message, and recent events before moving to the next layer.

Failing layer	Continue with
Flux controller or health check	Flux
Source or Kustomization	Environment configuration
Big Bang `HelmRelease`	Big Bang HelmRelease
Package `HelmRelease` or workload	Package or workload
Upgrade	Upgrade problems
Healthy deployment with performance problems	Performance troubleshooting

Troubleshoot by Layer¶

Flux¶

If flux check reports unhealthy controllers, resolve the Flux problem before troubleshooting Big Bang resources.

For controller health, logs, source failures, and other Flux-specific problems, see the Flux troubleshooting guide.

Environment Configuration¶

Check the sources and Kustomizations that manage the environment:

flux get sources all -A
flux get kustomizations -A

If a Kustomization is not ready, inspect it:

kubectl describe kustomization <kustomization-name> -n <namespace>

Use the resource conditions and messages to identify source access, manifest build or apply, decryption, or dependency failures.

For detailed troubleshooting, see the Flux troubleshooting guide and Flux Kustomization documentation. For Git repository authentication, certificate, or source-access failures, see Flux GitRepository documentation.

Big Bang HelmRelease¶

If the Flux resources that manage the environment are ready, check the Big Bang umbrella HelmRelease:

flux get helmrelease bigbang -n bigbang
kubectl describe helmrelease bigbang -n bigbang

Use its conditions, reason, message, and events to identify failures involving Big Bang values, chart or source configuration, dependencies, or install and upgrade reconciliation.

Correct the declarative configuration that caused the failure. For detailed HelmRelease behavior and remediation, see the Flux HelmRelease documentation.

Reconcile after Correcting the Cause¶

Flux reconciles resources automatically. If you changed values managed by an environment Kustomization, reconcile that Kustomization first so it applies the updated configuration. Use the name and namespace of the Kustomization that owns those values:

flux reconcile kustomization <kustomization-name> -n <namespace> --with-source

After the environment Kustomization finishes successfully, reconcile the Big Bang HelmRelease:

flux reconcile helmrelease bigbang -n bigbang --with-source

The HelmRelease --with-source option reconciles its chart source; it does not apply changed values from the environment repository. See the Flux Kustomization command and HelmRelease command references.

Verify the release:

flux get helmrelease bigbang -n bigbang

Full Reset¶

Use a full reset only after the failure cannot be corrected through declarative configuration. A full reset is a destructive recovery action that can remove Big Bang-managed workloads and resources. Before proceeding, confirm backup and persistent-data requirements and identify the Kustomization that will recreate the Big Bang HelmRelease.

Delete the Big Bang HelmRelease:

kubectl delete helmrelease bigbang -n bigbang

If namespaces remain stuck in a terminating state, use the remove-ns-finalizer.sh script to remove the remaining finalizers.

After deletion completes, reconcile the Kustomization that manages the Big Bang HelmRelease:

flux reconcile kustomization <kustomization-name> -n <namespace> --with-source

Verify that the deployment is recreated successfully:

flux get helmrelease bigbang -n bigbang

Warning: Removing namespace finalizers forces deletion to complete. Use the script only for namespaces that remain stuck during the full reset.

Package or workload¶

After the Big Bang HelmRelease is ready, check the downstream package releases:

flux get helmreleases -A

Inspect a package that is not ready:

kubectl describe helmrelease <package-name> -n <namespace>

If the package release is ready but the application is not functioning, inspect its workloads:

kubectl get pods -n <namespace>
kubectl describe pod <pod-name> -n <namespace>

Use the reported conditions and events to choose the next step.

For package-specific configuration and troubleshooting, see the Big Bang package documentation. Select the affected package for its Big Bang integration guide and package repository.

Problem	Continue with
Pod is `Pending`, fails to start, or repeatedly restarts	Kubernetes application debugging
`ErrImagePull` or `ImagePullBackOff`	Inspect pod events, image references, and pull credentials. See Big Bang registry credentials and Kubernetes private registry authentication. For Registry1 `unauthorized: authentication required` errors, see Iron Bank robot-account guidance.
DNS failure	Kubernetes DNS debugging
Service has no reachable backend	Kubernetes Service debugging
NetworkPolicy may be blocking traffic	Big Bang Network Policies
Istio routing, mesh, TLS, or authorization problem	Istio diagnostic tools
Big Bang authorization problem	Big Bang Authorization Policies
Big Bang route problem	Big Bang Routes
Kyverno admission or policy failure	Kyverno troubleshooting and Big Bang Kyverno Policies exceptions. For other admission controllers, consult the documentation for the controller identified in the error.
Insufficient CPU or memory	Kubernetes resource management
Storage or PersistentVolumeClaim problem	Kubernetes persistent volumes
Pod scheduling failure	Kubernetes scheduling
Kubernetes resources are healthy but the application is not	Consult the affected package’s Big Bang integration guide or package repository.
Deployment is healthy but slow or resource constrained	Performance troubleshooting

Check Service Connectivity¶

If the problem involves Service connectivity, confirm that the Service has the expected EndpointSlices:

kubectl get svc <service-name> -n <namespace>
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service-name>

If the Service is expected to select workloads but has no endpoints, verify its selector and backing workloads.

For CNI, load balancer, or infrastructure failures, use the documentation for the components deployed in your environment.

For cluster networking problems, see Kubernetes Network Plugins and the documentation for your installed CNI.

For external traffic or load-balancer problems, see Kubernetes Service debugging and the documentation for your cluster’s load-balancer implementation.

Warning: Do not disable security policies, add broad allow rules, or manually modify Flux-managed workloads as a standard troubleshooting step. Correct the declarative configuration that caused the failure.

Upgrade Problems¶

Use the same top-down troubleshooting workflow on this page to find the first resource that is not ready.

For an upgrade failure, also review:

Big Bang release notes and upgrade notices for the relevant upgrade path.
The affected package’s changelog.
Required configuration or value changes.

Correct the desired configuration rather than using a direct helm upgrade, manually editing Flux-managed resources, or performing a direct Helm rollback as the first response.

For upgrade procedures, supported upgrade paths, and post-upgrade verification, see Upgrades.

Verify Recovery¶

After correcting the failure, check for resources that are still not ready:

flux get all -A --status-selector ready=false

Then verify the affected workloads:

kubectl get pods -n <namespace>

Verify that the original symptom is also resolved. A successful reconciliation does not necessarily mean that the application is functioning correctly.

Escalate the Issue¶

If you cannot resolve the problem, collect information about the first failing layer.

Include:

Big Bang version
Affected package and version, if applicable
First Flux resource that is not ready
Resource condition and error message
Relevant events or logs
Recent configuration or upgrade changes
Steps to reproduce the problem

Useful starting commands are:

flux get all -A --status-selector ready=false
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

Warning: Review diagnostic output before sharing it. Logs, events, and resource information can contain environment-specific or sensitive information.