Deployment💣
GitOps💣
Big Bang follows a GitOps approach to deployment. All configuration changes will be pulled and reconciled with what is stored in the Git repository. The only exception to this is the initial manifests (e.g. dev.yaml) which points to the Git repository and path.
Installation💣
-
Before pushing changes to Git, validate all configuration is syntactically correct.
# If everything is successful, YAML should be output kustomize build ./dev -
If you have not already done so, push configuration changes to Git
git push -
Validate the Kubernetes context is correct
# This should match the environment you intend to deploy kubectl config current-context -
Deploy the Big Bang manifest to the cluster
kubectl apply -f dev.yaml
Upgrade💣
All changes to the Big Bang cluster should be made through Git. After changes are pushed, Big Bang will automatically reconcile the difference with the cluster.
It may take Big Bang up to 10 minutes to recognize your changes and start to deploy them. This is based on the
intervalvalue set for polling. You can force Big Bang to immediately check for changes by running the sync.sh script.
Changes to values can be tested in each environment using the named folders to override values and/or point to specific repo branches or tags. After testing, the changes can be placed into the ./base folder if the change is shared between all environments.
Monitor💣
The following commands will help you monitor the progress of the Big Bang deployment. Review the flowchart, if needed, to understand the progression. Use the Troubleshooting Guide if you have failures.
-
Verify Flux is running
kubectl get deploy -n flux-system # All resources should be in the 'Ready' state NAME READY UP-TO-DATE AVAILABLE AGE source-controller 1/1 1 1 106s kustomize-controller 1/1 1 1 106s notification-controller 1/1 1 1 105s helm-controller 1/1 1 1 106s -
Verify the environment was pulled from the Git repo
kubectl get gitrepository -A # `environment-repo`: STATUS should be True NAMESPACE NAME URL READY STATUS AGE bigbang environment-repo https://repo1.dso.mil/platform-one/big-bang/customers/template.git True Fetched revision: main/185e252f4452d897531ab0314adc7a189562be31 2m7s -
Verify the environment Kustomization properly worked
kubectl get kustomizations -A # `environment`: READY should be True NAMESPACE NAME READY STATUS AGE bigbang environment True Applied revision: main/185e252f4452d897531ab0314adc7a189562be31 6m41s -
Verify the ConfigMaps were deployed
kubectl get configmap -l kustomize.toolkit.fluxcd.io/namespace -A # 'common' and 'environment' should exist NAMESPACE NAME DATA AGE bigbang common-cch6942dk9 1 19m bigbang environment-d2tgb27f56 1 19m -
Verify the Secrets were deployed
kubectl get secrets -l kustomize.toolkit.fluxcd.io/namespace -A # 'common-bb' and 'environment-bb' should exist NAMESPACE NAME TYPE DATA AGE bigbang common-bb-kc5t8dbdfh Opaque 1 18m bigbang environment-bb-mhddkt46bd Opaque 1 18m -
Verify the Big Bang Helm Chart was pulled
kubectl get gitrepositories -A # 'bigbang' READY should be True NAME URL READY STATUS AGE bigbang https://repo1.dso.mil/platform-one/big-bang/bigbang.git True Fetched revision: master/8a4a1ddd0c9edf316f5362680cf2921baf0c3451 25m -
Verify the Big Bang Helm Chart was deployed
kubectl get hr -A # 'bigbang' READY should be True NAMESPACE NAME READY STATUS AGE bigbang bigbang True Release reconciliation succeeded 28m -
Verify Big Bang package Helm charts are pulled
kubectl get gitrepository -A # The Git repository holding the Helm charts for each package can be seen in the URL column. # The STATUS column shows the branch and tag of the revision being used. NAMESPACE NAME URL READY STATUS AGE bigbang bigbang https://repo1.dso.mil/platform-one/big-bang/bigbang.git True Fetched revision: master/3a44686520152e576a8c2c6f264876efff497c4b 8m25s bigbang logging https://repo1.dso.mil/platform-one/big-bang/apps/core/logging.git True Fetched revision: release-v0.2.x/9cfe1e14c12098464ee89eb877614f781cd78fb7 8m23s bigbang certmanager https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/cert-manager.git True Fetched revision: release-v1.0.x/1247135baf145dcfad4a4a02ef679c48fb76d9fb 8m23s bigbang istio https://repo1.dso.mil/platform-one/big-bang/apps/core/servicemesh.git True Fetched revision: chart-release/2b02a51b7950ce21bac26403fa25d09e7e3f86c3 8m23s bigbang twistlock https://repo1.dso.mil/platform-one/big-bang/apps/security-tools/twistlock.git True Fetched revision: chart-release/faf038197291915713e0f213a4e35991e72f73f6 8m23s bigbang gatekeeper https://repo1.dso.mil/platform-one/big-bang/apps/core/policy.git True Fetched revision: chart-release/1a5f32c8e7f672c3b5937b604e5f38eaa08ce246 8m23s bigbang monitoring https://repo1.dso.mil/platform-one/big-bang/apps/core/monitoring.git True Fetched revision: release-v0.2.x/ca60bedcc106b95beb0bf9ccdc6e0e759e6fd6bf 8m23s bigbang cluster-auditor https://repo1.dso.mil/platform-one/big-bang/apps/core/cluster-auditor.git True Fetched revision: chart-release/598c35670db0cbdb3a48063b2d558965afe73185 8m23s -
Verify the packages get deployed
# Use watch since it take a long time to deploy watch kubectl get hr,deployments,po -A # Flux will not attempt to deploy a package until its dependencies are ready # All Helm Release resources and Pods Every 2.0s: kubectl get hr,deployments,po -A localhost: Mon Nov 9 10:14:56 2020 NAMESPACE NAME READY STATUS AGE bigbang helmrelease.helm.toolkit.fluxcd.io/bigbang True Release reconciliation succeeded 64s bigbang helmrelease.helm.toolkit.fluxcd.io/gatekeeper True Release reconciliation succeeded 62s bigbang helmrelease.helm.toolkit.fluxcd.io/eck-operator False dependency 'bigbang/gatekeeper' is not ready 62s bigbang helmrelease.helm.toolkit.fluxcd.io/istio-operator Unknown Reconciliation in progress 62s bigbang helmrelease.helm.toolkit.fluxcd.io/istio False dependency 'bigbang/istio-operator' is not ready 62s bigbang helmrelease.helm.toolkit.fluxcd.io/efk False dependency 'bigbang/eck-operator' is not ready 62s bigbang helmrelease.helm.toolkit.fluxcd.io/logging-operator False dependency 'bigbang/gatekeeper' is not ready 62s bigbang helmrelease.helm.toolkit.fluxcd.io/twistlock False dependency 'bigbang/gatekeeper' is not ready 62s bigbang helmrelease.helm.toolkit.fluxcd.io/cluster-auditor-policies False dependency 'bigbang/gatekeeper' is not ready 62s bigbang helmrelease.helm.toolkit.fluxcd.io/cluster-auditor False dependency 'bigbang/gatekeeper' is not ready 62s bigbang helmrelease.helm.toolkit.fluxcd.io/certmanager True Release reconciliation succeeded 62s bigbang helmrelease.helm.toolkit.fluxcd.io/monitoring False dependency 'bigbang/gatekeeper' is not ready 62s NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE kube-system deployment.apps/local-path-provisioner 1/1 1 1 4m48s kube-system deployment.apps/coredns 1/1 1 1 4m48s flux-system deployment.apps/helm-controller 1/1 1 1 4m6s flux-system deployment.apps/notification-controller 1/1 1 1 4m6s flux-system deployment.apps/source-controller 1/1 1 1 4m7s flux-system deployment.apps/kustomize-controller 1/1 1 1 4m7s gatekeeper-system deployment.apps/gatekeeper-controller-manager 1/1 1 1 2m8s gatekeeper-system deployment.apps/gatekeeper-audit 1/1 1 1 2m8s istio-operator deployment.apps/istio-operator 0/1 1 0 8s NAMESPACE NAME READY STATUS RESTARTS AGE kube-system pod/local-path-provisioner-6d59f47c7-s6rln 1/1 Running 0 4m36s kube-system pod/coredns-7944c66d8d-flk4p 1/1 Running 0 4m36s flux-system pod/helm-controller-578cdbcd8b-tjzs7 1/1 Running 0 4m6s flux-system pod/notification-controller-7c59d85f77-92ckv 1/1 Running 0 4m6s flux-system pod/source-controller-7d6f889df9-f888j 1/1 Running 0 4m7s flux-system pod/kustomize-controller-5cfb78859c-n85xn 1/1 Running 0 4m6s gatekeeper-system pod/gatekeeper-controller-manager-5b9cf6c85d-cqd8t 1/1 Running 0 2m8s gatekeeper-system pod/gatekeeper-audit-7db49c54d5-pwzwh 1/1 Running 0 2m8s istio-operator pod/istio-operator-79f966cfc-rjhhc 0/1 ContainerCreating 0 8s -
Wait until all Helm Releases, Deployments, and Pods are Ready. Be patient, this can take 15-30 minutes.
The Git repositories are monitored periodically (default is 10m) for changes. If a change is detected, the configuration will be reconciled using Flux. The monitoring techniques above can be used to monitor the reconciliation.