Skip to content

How to Upgrade Velero Package chart📜

  1. Checkout the branch that renovate created. This branch will have the image tag updates and typically some other necessary version changes that you will want. You can either work off of this branch or branch off of it.
  2. Navigate to the upstream chart repo and folder and find the tag that corresponds to the version of this chart. Start with the newest chart version to make sure we get the latest patches. For changes to velero, the changelog is located here.
  3. From the root of the repo run kpt pkg update chart@velero-X.Y.Z --strategy alpha-git-patch replacing X.Y.Z with the version tag from step 1 (ex. velero-8.4.0). You may be prompted to resolve some conflicts - choose what makes sense (if there are BB additions/changes keep them, if there are upstream additions/changes keep them).
  4. Modify the version in Chart.yaml - you will want to append -bb.0 to the chart version from upstream.
  5. Update binaries. Pull assets and commit the binaries as well as the Chart.lock file that was generated by running: helm dependency update ./chart
  6. Update CHANGELOG.md adding an entry for the new version and noting all changes (at minimum should include Updated Velero chart to x.x.x and Updated image versions to latest in IB.
  7. Generate the README.md updates by following the guide in gluon.
  8. Push up your changes, validate that CI passes in the renovate MR (or create an MR if necessary). If there are any failures follow the information in the pipeline to make the necessary updates and reach out to the team if needed.
  9. Perform the steps below for manual testing. CI provides a good set of basic smoke tests but it is beneficial to run some additional checks.

Testing new Velero version📜

Cluster setup📜

⚠️ Always make sure your local bigbang repo is current before deploying. 1. Export your Ironbank/Harbor credentials (this can be done in your ~/.bashrc or ~/.zshrc file if desired). These specific variables are expected by the k3d-dev.sh script when deploying metallb, and are referenced in other commands for consistency: 

export REGISTRY_USERNAME='<your_username>'
export REGISTRY_PASSWORD='<your_password>'
1. Export the path to your local bigbang repo (without a trailing /):

⚠️ Note that wrapping your file path in quotes when exporting will break expansion of `~`.
```
export BIGBANG_REPO_DIR=<absolute_path_to_local_bigbang_repo>
```
e.g.
```
export BIGBANG_REPO_DIR=~/repos/bigbang
```

  1. Run the k3d_dev.sh script to deploy a dev cluster (-a flag required if deploying a local Keycloak):

    "${BIGBANG_REPO_DIR}/docs/assets/scripts/developer/k3d-dev.sh"

  2. Export your kubeconfig:

    export KUBECONFIG=~/.kube/<your_kubeconfig_file>
    e.g. 
    export KUBECONFIG=~/.kube/Sam.Sarnowski-dev-config
    1. Deploy flux to your cluster: 
    "${BIGBANG_REPO_DIR}/scripts/install_flux.sh -u ${REGISTRY_USERNAME} -p ${REGISTRY_PASSWORD}"

    Deploy Bigbang📜

⚠️ Note that testing against your local branch or tag is only possible if you edit the overrides file to point to your changes.

From the root of this repo, run the following deploy command :

helm upgrade -i bigbang ${BIGBANG_REPO_DIR}/chart/ -n bigbang --create-namespace \
--set registryCredentials.username=${REGISTRY_USERNAME} --set registryCredentials.password=${REGISTRY_PASSWORD} \
-f https://repo1.dso.mil/big-bang/bigbang/-/raw/master/tests/test-values.yaml \
-f https://repo1.dso.mil/big-bang/bigbang/-/raw/master/chart/ingress-certs.yaml \
-f https://repo1.dso.mil/big-bang/bigbang/-/raw/master/docs/assets/configs/example/dev-sso-values.yaml \
-f docs/dev-overrides/minimal.yaml \
-f docs/dev-overrides/velero-testing.yaml

Validation/Testing Steps📜

Setting up📜

  • The above instructions will bring Velero up quickly, by disabling most other bigbang packages aside from Istio and the Monitoring stack. ib_creds.yaml refers to wherever your registryCredentials key is saved.
  • Once everything comes up, log in to Minio using the credentials above (minio/minio123 by default) and create a Bucket called velero123 (or whatever you set under addons.velero.values.configuration.backupStorageLocation[].bucket). This ensures that Veleros backups will have a destination.
  • If you encounter an issue where Velero has no deployed releases, you may need to roll back any helm releases and Velero resources before redeploying.

Backing up a cluster📜

Velero can be interacted with using its pod:

% kubectl get pods -n velero
NAME                            READY   STATUS    RESTARTS   AGE
velero-velero-579bd4f68-6wl4p   2/2     Running   0          3m40s

% kubectl exec -it velero-velero-579bd4f68-6wl4p -n velero -- bash
bash-5.1$ velero backup create goodbackup
bash-5.1$ velero backup create monitoringbackup --include-namespaces=monitoring

These backup files will be delivered to Minio, or whatever BackupStorageLocation was chosen. If these backups are not being placed in the Minio bucket, ensure that it was created and named correctly. You can also use velero backup describe <backupname> to troubleshoot issues. Velero Backup Documentation can help with taking more fine-grained backups, and Velero Restore Documentation can assist in restoring backups to the same or different clusters.

Big Bang Integration Testing📜

As part of your MR that modifies bigbang packages, you should modify the bigbang bigbang/tests/test-values.yaml against your branch for the CI/CD MR testing by enabling your packages.

To do this, at a minimum, you will need to follow the instructions at bigbang/docs/developer/test-package-against-bb.md with changes for Velero enabled (the below is a reference, actual changes could be more depending on what changes where made to Velero in the pakcage MR).

```yaml
addons:
  velero:
    enabled: true
    git:
      tag: null
      branch: <my-package-branch-that-needs-testing>
    values:
      istio:
        hardened:
          enabled: true
  ### Additional compononents of Velero should be changed to reflect testing changes introduced in the package MR
```

Modifications made to upstream📜

This is a high-level list of modifications that Big Bang has made to the upstream helm chart. You can use this as as cross-check to make sure that no modifications were lost during the upgrade process.

chart/values.yaml📜

  • Added values for istio, networkPolicies, and bbtests
  • Changed image to default to Ironbank image
  • Set default podSecurityContext, containerSecurityContext, and imagePullSecrets
  • Added commented out values for serviceMonitor.scheme and serviceMonitor.tlsConfig
  • Added values for aws, csi, and azure under the new plugins section

chart/templates/deployment.yaml📜

  • Added CA bundle conditionally as ENV/volume mount
  • Changed initContainers configuration to merge .values.plugins with the initContainers configs

chart/templates/cert-secret.yaml📜

  • Added file/secret for CA cert

chart/templates/upgrade-crds/upgrade-crds.yaml📜

  • Added resources for containers

chart/templates/bigbang/📜

  • Network policies added
  • Istio mTLS (peerauthentication) resources added
  • Grafana dashboard added
  • CSI Snapshot class added

automountServiceAccountToken📜

The mutating Kyverno policy named update-automountserviceaccounttokens is leveraged to harden all ServiceAccounts in this package with automountServiceAccountToken: false. This policy is configured by namespace in the Big Bang umbrella chart repository at chart/templates/kyverno-policies/values.yaml.

This policy revokes access to the K8s API for Pods utilizing said ServiceAccounts. If a Pod truly requires access to the K8s API (for app functionality), the Pod is added to the pods: array of the same mutating policy. This grants the Pod access to the API, and creates a Kyverno PolicyException to prevent an alert.