Skip to content

BB Test Templates

Currently two testing types are supported through the library, scripts (for CLI testing - think bash, python, etc) and cypress (for UI testing). These two test types are described below along with examples of how to implement them. NOTE: If your package can be interacted with via a UI and a CLI both test types should be included. By default UI tests run before CLI tests due to the way the library is written, but this can be overridden as described below.

Pipelines will automatically run these tests, but if you wish to run them locally you will need to install the package with the test values, then run helm test {{HELMRELEASE_NAME}} -n {{HELMRELEASE_NAMESPACE}}, replacing the variables with the proper values for your package (you can check the helmrelease name and namespace with helm ls -A).

You will need to set the bbtests.enabled toggle to true in your test-values so that the tests are deployed.

bbtests:
  enabled: true

Cypress

To include the Helm test templates for Cypress you will need to create a file called test-ui.yaml under chart/templates/tests which includes the content below:

{{- include "gluon.tests.cypress-configmap.base" .}}
---
{{- include "gluon.tests.cypress-runner.base" .}}

This will work for a “base” install, but if you want to override anything (example below shows how to add labels), you can include different templates and create package specific templates with the overrides:

{{- include "gluon.tests.cypress-configmap.overrides" (list . "mattermost-test.cypress-configmap") }}
{{- define "mattermost-test.cypress-configmap" }}
metadata:
  labels:
    {{ include "mattermost.labels" . | nindent 4 }}
{{- end }}
---
{{- include "gluon.tests.cypress-runner.overrides" (list . "mattermost-test.cypress-runner") -}}
{{- define "mattermost-test.cypress-runner" -}}
metadata:
  labels:
    {{ include "mattermost.labels" . | nindent 4 }}
{{- end }}

The second step to implementing these tests is to set the bbtests.enabled value in /tests/test-values.yaml to true: 

networkPolicies:
  enabled: true

imagePullSecrets:
  - name: private-registry

console:
  persistence:
    size: 5Gi
  localVolumeUpgrade: true

bbtests:
  enabled: true

Next, include values that would be needed for the tests in your chart/values.yaml file in the bbtest: section. There are several values you will want to consider (all values are optional):

  • bbtests.cypress.artifacts: This should be set to true in almost all cases so that artifacts are exported from the helm test pods and available as artifacts in the pipeline.
  • bbtests.cypress.exports: This should be set to true if you want to export values from your Cypress tests to be used in your script tests. Files should be written to the subdirectory exports and will then be available at that same subdirectory from the script run location in the script runner.
  • bbtests.cypress.secretEnvs: This is where you should put any configuration that your tests need from secrets that are created by the helm chart (common examples are passwords) - Helm templating is supported here with proper syntax (braces must be wrapped in quotes, any quotes inside braces must be escaped)
  • bbtests.cypress.envs: This is where any configuration that is not in pre-existing secrets should go (common examples are the service name to hit and usernames) - Helm templating is also supported here with the same restrictions of syntax. While you may be able to place ENVs into your cypress.json, that will not support Helm templating, which is the major benefit of using values for ENVs.
  • bbtests.cypress.additionalVolumes: This defines additional volumes for the cypress testing pod that is not one of the pre-existing testing volumes. This takes normal Kubernetes volume configurations as yamls. This supports helm templating for these values.
  • bbtest.cypress.additionalVolumeMounts: This defines additional volumes to mount into the main cypress container itself. This takes normal Kubernetes volumeMount configuration in yaml. This supports Helm templating for values.
  • bbtest.cypress.resources: This defines requests and limits to set in the main cypress container itself. This takes normal Kubernetes resources configuration in yaml. This supports Helm templating for values.
  • bbtests.istio.hosts: This defines istio hostnames to add to the cypress test pods /etc/hosts for resolution. It should be defined as a list and supports helm templating. Note this will also apply to any script tests.
  • bbtests.enableImagePullSecrets: This defines whether it should use the default private-registry secret and should be overridden in the umbrella template for big bang integrated packages. It defaults to true if left unset and should not be needed at the package level. This same value is applied to both the cypress and scripts pods.
  • bbtests.seccompProfile: This is empty by default, but can be overridden by end users to pass through to the security context of both the cypress and scripts test pods (i.e. bbtests.seccompProfile.type: RuntimeDefault).

A sample is included below:

bbtests:
  # -- Toggle bbtests on/off for CI/Dev
  enabled: false
  cypress:
    # -- Toggle creation of cypress artifacts
    artifacts: true
    # -- Set envs for use in cypress tests
    envs:
      cypress_baseUrl: "http://{{ .Release.Name }}-console.{{ .Release.Namespace }}.svc.cluster.local:8081"
  scripts:
    # -- Image to use for script tests
    image: registry1.dso.mil/ironbank/stedolan/jq:1.6
    # -- Set envs for use in script tests
    envs:
      twistlock_host: "https://{{ .Release.Name }}-console.{{ .Release.Namespace }}.svc.cluster.local:8083"
      desired_version: "{{ .Values.console.image.tag }}"

NOTE: Environment variables must be prefixed with cypress_ to be available to Cypress.

To set up the package test directory chart/tests:

  • Install test dependencies using npm install cypress. This will generate a package.json and a package-lock.json in the /chart/tests folder.

  • Add all *.cy.js tests to the chart/tests/cypress/e2e folder.

Any cypress tests should be written following cypress best practices and functionally to test the UI components of a package.

Your final directory structure and files should look like this:

Directory Structure

.
├── chart
│   ├── Chart.yaml (which includes the library dependency)
│   ├── templates
│   │   └── tests
│   │       └── test.yaml (which uses the library templates)
│   └── tests
│       ├── cypress
│       │   ├── videos (contains your test video files)
│       │   ├── screenshots (contains your test screenshots files)
│       │   ├── e2e (contains your test spec files)
│       │       └── test-health.cy.js 
│       ├── package.json
│       ├── package-lock.json
└── tests
    └── test-values.yaml (where you enable bbtests)

Cypress exports

Cypress “exports” should be enabled if you want to pass information from your Cypress test to your script test. Cypress documentation covers many scenarios for how to use the writeFile function to create plaintext, JSON, and other file types for different needs. To make sure that files are properly passed, they need to be written to the exports subdirectory. See the below example as a way of writing the value of an input field to a file for availability in your script. One important option to know about is the append option as shown below:

cy.writeFile('exports/envs-from-cypress.env', 'export MY_TOKEN=', { flag: 'a+' })
cy.get('input[id="token"]').invoke('val').then(token => cy.writeFile('exports/envs-from-cypress.env', token + '\n', { flag: 'a+' }))

These files will be available in any scripts you run at the same file structure (under the subdirectory exports). Note that nothing is done with the files, so your script determines how you extract and use the files. The example above shows writing out an export “command” so that the file can be easily sourced, but you could write out a yaml or json file to parse instead (or any other thing you want to do with the file).

Cypress artifacts

Cypress artifacts are written to the host running the k3d cluster at /cypress. This is done via hostPath mounts from the runner -> k3d host -> helm test pod. When gatekeeper is running, exceptions will need to be added to its configuration to allow this. See “Add gatekeeper exceptions” in this document.

Scripts

To include the Helm test templates for script based tests you will need to make a file under chart/templates/tests which includes the content below:

{{- include "gluon.tests.script-configmap.base" .}}
---
{{- include "gluon.tests.script-runner.base" .}}

This will work for a “base” install, but if you want to override anything (example below shows how to add labels), you can include different templates and create package specific templates with the overrides:

{{- include "gluon.tests.script-configmap.overrides" (list . "mattermost-test.script-configmap") }}
{{- define "mattermost-test.script-configmap" }}
metadata:
  labels:
    {{ include "mattermost.labels" . | nindent 4 }}
{{- end }}
---
{{- include "gluon.tests.script-runner.overrides" (list . "mattermost-test.script-runner") -}}
{{- define "mattermost-test.script-runner" -}}
metadata:
  labels:
    {{ include "mattermost.labels" . | nindent 4 }}
{{- end }}

The second step to implementing these tests will be including values in your test-values.yaml file that would be needed for the tests. There are several values you will want to consider, only the image value is required:

  • bbtests.scripts.image: This is the image name that should be used to run your script. This should be a small image with the CLI tools needed (for example, to test Minio you would want an image with Minio CLI). Ironbank images are not required but are preferred if one exists with the CLI tool you need installed. This field supports Helm templating if you want to grab an image being used elsewhere in your values (ex: image: "{{ .Values.my.spec.to.image }}").
  • bbtests.scripts.secretEnvs: This is where you should put any configuration that your tests need from secrets that are created by the helm chart (common examples are passwords) - Helm templating is supported here with proper syntax (braces must be wrapped in quotes, any quotes inside braces must be escaped)
  • bbtests.scripts.envs: This is where any configuration that is not in pre-existing secrets should go (common examples are the service name to hit and standard usernames) - Helm templating is also supported here with the same restrictions of syntax.
  • bbtests.scripts.additionalVolumes: This defines additional volumes for the cypress testing pod that is not one of the pre-existing testing volumes. This takes normal Kubernetes volume configurations as yamls. This supports helm templating for these values.
  • bbtest.scripts.additionalVolumeMounts: This defines additional volumes to mount into the main cypress container itself. This takes normal Kubernetes volumeMount configuration in yaml. This supports Helm templating for values.
  • bbtest.scripts.resources: This defines requests and limits to set in the main script container itself. This takes normal Kubernetes resources configuration in yaml. This supports Helm templating for values.
  • bbtests.istio.hosts: This defines istio hostnames to add to the script test pods /etc/hosts for resolution. It should be defined as a list and supports helm templating. Note this will also apply to any cypress tests.
  • bbtests.scripts.permissions.apiGroups: This defines the API groups that the scriptrunner should have access to. This should be a list of API groups.
  • bbtests.scripts.permissions.resources: This defines the resources that the scriptrunner should have access to. This should be a list of resources.
  • bbtests.scripts.permissions.verbs: This (optional) list defines the verbs that can be taken upon the resources the scriptrunner has access to. If not provided, this defaults to ["get", "list", "delete", "create"].

A sample is included below:

bbtests:
  enabled: true
  scripts:
    image: "{{ .Values.mcImage }}"
    permissions:
      apiGroups:
        - ""
      resources:
        - configmaps
      verbs:
        - create
        - delete
        - list
        - get
    additionalVolumeMounts:
      - name: "{{ .Chart.Name }}-example"
        mountPath: /example
      - name: "{{ .Chart.Name }}-example-config"
        mountPath: /otherexample
        subpath: something
    additionalVolumes:
      - name: "{{ .Chart.Name }}-example"
        emptyDir: {}
      - name: "{{ .Chart.Name }}-example-config"
        configMap:
          name: "{{ .Chart.Name }}-example-config"
    envs:
      MINIO_PORT: "{{ .Values.service.port }}"
      MINIO_HOST: '{{ include "minio.serviceName" . }}'
    secretEnvs:
      - name: SECRET_KEY
        valueFrom:
          secretKeyRef:
            name: "{{ .Values.minioRootCreds }}"
            key: secretkey
      - name: ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: "{{ .Values.minioRootCreds }}"
            key: accesskey
    resources:
      requests:
        cpu: "1"
        memory: "1Gi"
      limits:
        cpu: "1"
        memory: "1Gi"
  istio:
    hosts:
      - "minio.{{ .Values.hostname }}"

Finally, any script files (the helm template will run any and all files in the scripts folder) must be placed in the directory chart/tests/scripts/. Your test script(s) should run the CLI tool(s) and perform any other necessary operations to functionally test the package. Make sure to call out the proper shell / executable to run your script (for example call out that bash should be used with #!/bin/bash as the first script line). If you need any additional files for running the scripts they should be placed under subfolders so that the pipeline does not try to run them as scripts. Scripts can be expected to run in sequential order based on file name (files beginning with 0-9, then A-Z, then a-z).

As a reminder, if you had any cypress “exports” those will be accessible via the exports directory for usage in your script (provided you toggled bbtests.cypress.exports on).

An example is provided below for Minio:

#!/bin/bash
set -ex
source exports/envs-from-cypress.env
mc config host add bigbang http://${MINIO_HOST}:${MINIO_PORT} ${ACCESS_KEY} ${SECRET_KEY}
# cleanup from previous runs
mc rb bigbang/foobar --force || true
mc mb bigbang/foobar
mc ls bigbang/foobar
base64 /dev/urandom | head -c 10000000 > /tmp/file.txt
md5sum /tmp/file.txt > /tmp/filesig
mc cp /tmp/file.txt bigbang/foobar/file.txt
mc ls bigbang/foobar/file.txt
mc cp bigbang/foobar/file.txt /tmp/file.txt
mc rb bigbang/foobar --force
md5sum -c /tmp/filesig

Your final directory structure and files should look like this:

|-- chart
|  |-- Chart.yaml (which includes the library dependency)
|  |-- tests
|  |  `-- scripts
|  |    `-- mytest.sh
|  `-- templates
|     `-- tests
|        `-- test-ui.yaml (which uses the library templates)
`-- tests
   `-- test-values.yaml (where you enable bbtests)

Add Gatekeeper exceptions

Most bigbang clusters will have gatekeeper enabled. When running in a bigbang integration test, these exceptions are required so that the artifacts generated by cypress tests (screenshots and videos) can be exported after the test is complete.

Ensure this is in place in your bigbang test-values.yaml in the gatekeeper section under gatekeeper.values.violations:

      volumeTypes:
        parameters:
          excludedResources:
          # Add all cypress test pods here to allow hostpath cypress mount
          - namespace/pod-name
      allowedHostFilesystem:
        parameters:
          excludedResources:
          # Add all cypress test pods here to allow hostpath cypress mount
          - namespace/pod-name