Troubleshooting📜

Big Bang can take a long time to run. After making changes, it could take 10-15 minutes to take effect. Use the sync.sh script to speed this up.

Big Bang is configured to retry failed package installations and upgrades. Before concluding you have a failure, make sure you allow Big Bang to attempt to resolve dependencies and retry.

Iron Bank authentication📜

Symptom	Cause	Resolution
Despite entering correct credentials, get `unauthorized: authentication required` from Iron Bank	Using a non-robot account with an expired token	Login with the non-robot account manually at `registry1.dso.mil`, then retry. For production, contact the Iron Bank team to obtain a robot account and update pull credentials to use it in your environment

Flux install📜

Helpful debugging commands:

# Get the status
kubectl get pods -n flux-system

# Get the logs
kubectl get events -n flux-system

Symptom	Cause	Resolution
Install script timed and pods are still pulling the image	Slow connection to docker registry	Adjust `--timeout` value in `flux install` to wait longer
Pod status is `ImagePullBackOff` or `ErrImagePull`	Bad registry, version, or credentials	Fix the `--registry`, `--version`, or `--image-pull secret` options or use the `./scripts/install_flux.sh` script for pulling from Iron Bank

Git Repository📜

Helpful debugging commands:

# Get the status
kubectl get gitrepositories -A

# Get the logs
kubectl get events --field-selector involvedObject.kind=GitRepository -A

Symptom	Cause	Resolution
`unable to clone ... error: authentication required`	Pull credentials for Git invalid or not provided	Add credentials to a `Secret` and reference it in `GitRepository.spec.secretRef.name`. If possible, encrypt the secret and include it in the Kustomization deployment for your environment.
`auth secret error: Secret ... not found`	`GitRepository` is trying to use credentials but cannot find the `Secret`	Make sure the secret exists and is in the same namespace as the `GitRepository` resource. If possible, encrypt the secret and include it in the Kustomization deployment for your environment.
`unable to clone ... error: repository not found`	Invalid Git url	Fix url for Git repository and redeploy
`unable to clone ... error: couldn't find remote ref`	Invalid branch or tag	Fix branch or tag for Git repository and redeploy

ConfigMap or Secrets📜

Symptom	Cause	Resolution
`ConfigMap` or `Secret` does not exist	GitRepository or Kustomization failed. Namespace was incorrect.	Use GitRepository and Kustomization sections to troubleshoot. Use `kubectl get secrets,configmaps -A` to verify resource was not in the wrong Namespace.

Helm Release📜

Helpful debugging commands:

# Get the status
kubectl get hr -A

# Get the logs
kubectl get events --field-selector involvedObject.kind=HelmRelease -A

# Describe the HelmRelease to get more information
kubectl describe hr <NAME> -n bigbang

# Get all logs/events for a specific HelmRelease object
flux logs --kind=HelmRelease --namespace bigbang --name <NAME>

Symptom	Cause	Resolution
`Reconciliation in Progress`	This is normal and indicates flux is currently applying updates	Wait
`dependency ... is not ready`	This is normal and indicates flux is currently waiting on another resource to complete	Wait
`Error: YAML parse error on ...`	Syntax error in helm chart	Use `helm template` to narrow down the problem. Fix it and commit to Git
`Helm install failed: failed to create resource ... unable to create new content in namespace because it is being terminated`	This seems to happen when a re-deploy of Big Bang occurs to early after a Big Bang delete.	Try to remove the namespace using `kubectl get ns <stuck namespace> -o json \| jq '.spec.finalizers = []' \| kubectl replace --raw "/api/v1/namespaces/$NS/finalize" -f`. If this does not work, a cluster restart may be necessary.
`Error: failed to download ...`	Path to Helm chart is incorrect	Find the HelmRelease configuration and update `spec.path` to the correct path of the helm chart
`Helm uninstall failed: uninstall: Release not loaded: ____: release: not found`	Helm install failed because of an error and a rollback/uninstall is attempted but release has not been installed.	Describe the HelmRelease in question or use flux to get the logs to get more info abut why it failed to install.
`reconciliation failed: Helm rollback failed: an error occurred while cleaning up resources. original rollback error: no XXXX with the name "XXXX" found: unable to cleanup resources: object not found, skipping delete`	This error happens when an upgrade fails and flux attempts a rollback but there are templates that have been renamed/removed.	Describe the HelmRelease in question or use flux to get the logs to get more info abut why exactly the upgrade failed.

Kustomization📜

Helpful debugging commands:

# Get the status
kubectl get kustomizations -A

# Get the logs
kubectl get events --field-selector involvedObject.kind=Kustomization -A

Symptom	Cause	Resolution
`kustomization path not found`	`spec.path` in Kustomization resource in is incorrect	Fix `spec.path` and redeploy
`Source not found`	`spec.sourceRef` in Kustomization resource is incorrect	Fix `spec.sourceRef` to point to repository resource and redeploy
`decryption secret error: Secret ... not found`	SOPS private key secret is missing or misconfigured	Check `decryption` settings in the Kustomization resource to make sure `secretRef` is pointing to the correct secret. Make sure the `Secret` holding the private key is deployed in the cluster.
`kustomize build failed: json: unknown field`	There is a syntax error with the kustomization files.	Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem. Fix the error and push to Git.
`evalsymlink failure ... no such file or directory`	A reference to a file in `kustomization.yaml` is incorrect	Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem. Fix the error and push to Git.
`Error: accumulating resources ...`	A reference to a base is incorrect	Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem.Review the `bases:` section for correct paths to find the error. Fix the error and push to Git.
`Error fetchingref: fatal: couldn't find remote ref ...`	The branch, tag, or sha used for a remote base is incorrect	Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem. It is likely the remote reference to the Big Bang’s Kustomize in the `base` folder. Review the `bases:` section for correct paths to find the error. Fix the error and push to Git.
`Error: merging from generator ...`	Kustomize is trying to merge with a resource that is non-existent. This is usually due to naming the merging `ConfigMap` or `Secret` incorrectly compared to a base `ConfigMap` or `Secret`.	Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem. Look for the keyword `merge` in the `kustomization.yaml` files and verify the `name` is correctly set.

Packages📜

Helpful debugging commands:

# Get the status
kubectl get deployments,po -n <namespace of package>

# Get the logs
kubectl get events --field-selector involvedObject.kind=Deployment -n <namespace of package>
kubectl get events --field-selector involvedObject.kind=Pod -n <namespace of package>

Last update: 2022-08-18 by Micah Nagel