Skip to content


Currently this is in proof of concept mode, so play around with this to get an idea of it.

This work was quickly developed to entertain certain paths for image packaging and deployment.

Image Packaging / DeploymentπŸ“œ - Proof of concept script for image packaging

  • Dependencies
  • docker - The docker CLI tool
  • images.txt - A list of all requires airgap images
  • jq - The jq CLI tool
  • Deliverables
  • registry:package.tar.gz - Modified registry:2 container loaded with airgap images
    • NOTE - registry:2 vs harbor vs anything else is trivial, we can use whatever we want
    • Packaged images are loaded and retrievable immediately upon container start
    • /var/lib/registry-package is created and populated with images
    • /etc/docker/registry/config.yml is templated to use new registry folder
    • This is due to the fact that /var/lib/registry is a docker volume - Proof of concept script for image deployment

  • Dependencies
  • docker - The docker CLI tool
  • registry:package.tar.gz - Modified registry:2 container loaded with airgap images
  • Deliverables
  • Running registry container with airgap images deployed and retrievable

Hack commands:

  • curl -sX GET http://localhost:5000/v2/_catalog | jq -r .
  • Verify the catalog of a local running registry container

Repository Packaging / DeploymentπŸ“œ

Airgap Deployment is a form of deployment which does not have any direct connection to the Internet or external network during cluster setup or runtime. During installation, bigbang requires certain images and git repositories for installation. Since we will be installing in internet-disconnected environment, we need to perform extra steps to make sure these resources are available.

Requirements and PrerequisitesπŸ“œ

General PrerequisitesπŸ“œ

  • A kubernetes cluster with container mirroring support. There is a section below that covers mirroring in more detail with examples for supported clusters.
  • BigBang(BB) release artifacts.
  • Utility Server.

Package Specific PrerequisitesπŸ“œ

Elastic (Logging)πŸ“œ

Elastic requires a larger number of memory map areas than some OSes support by default. This can be change at startup with a cloud config or later using sysctl.

MIME-Version: 1.0
    Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

    Content-Type: text/x-shellscript; charset="us-ascii"

    # Set the vm.max_map_count to 262144. 
    # Required for Elastic to run correctly without OOM errors.
    sysctl -w vm.max_map_count=262144

Utility ServerπŸ“œ

Utility Server is an internet-disconnected server that will host the private registry and git server that are required to deploy bigbang. It should include these command-line tools below;

  • docker: for running docker registry.
  • registry:2 image
  • openssl for self-signed certificate.
  • curl: For troubleshooting registry.
  • git: for setup git server.

Git ServerπŸ“œ

As part of BB release, we provide repositories.tar.gz which contains all the git repositories that BB depend on for deployment. You have two options for serving up these packages for Flux.

Option OneπŸ“œ

You can follow the process below to setup git with repositories.tar.gz on the Utility Server.

  • Create Git user and SSH key
sudo useradd --create-home --shell /bin/bash git
ssh-keygen  -b 4096 -t rsa -f ~/.ssh/identity -q -N ""
  • Create .SSH folder for git user
sudo su - git
mkdir -p .ssh && chmod 700 .ssh/
touch ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
  • Add client ssh key to git user authorized_keys
sudo su
cat /[client-public-key-path]/ >> /home/git/.ssh/authorized_keys
  • Extract repositories.tar.gz to git user home directory
sudo tar -xvf repositories.tar.gz --directory /home/git/
  • Add Hostname alias
PRIVATEIP=$( curl )
sudo sed -i -e '1i'$PRIVATEIP'   ''\' /etc/hosts
sudo sed -i -e '1i'$PRIVATEIP'   'host.k3d.internal'\' /etc/hosts #only for k3d
  • To test the client key;
GIT_SSH_COMMAND='ssh -i /[client-private-key-path] -o IdentitiesOnly=yes' git clone git@[hostname/IP]:/home/git/repos/[sample-repo]

#For example;
GIT_SSH_COMMAND='ssh -i ~/.ssh/identity -o IdentitiesOnly=yes' git clone git@host.k3d.internal:/home/git/repos/bigbang 
#checkout release branch
git checkout 1.3.0

Option TwoπŸ“œ

There are some cases where you do not have access to or cannot create an ssh user on the utility server. It is possible to run an ssh git server on a non-standard port using Docker.

  • Create an SSH key
ssh-keygen  -b 4096 -t rsa -f ./identity -q -N ""
  • Extract repositories.tar.gz to your working directory
sudo tar -xvf repositories.tar.gz
  • Start the provided Docker image (TODO: move this to an IB image when ready)
docker run -d -p 4001:22 -v ${PWD}/ -v ${PWD}/repos:/home/git servicesengineering/gitshim:0.0.1

You will now be able to test by checking out some of the code.

GIT_SSH_COMMAND='ssh -i /[client-private-key-path] -o IdentitiesOnly=yes' git clone git@[hostname/IP]:[PORT]/home/git/repos/[sample-repo]

# For example;
GIT_SSH_COMMAND='ssh -i ~/.ssh/identity -o IdentitiesOnly=yes' git clone git@host.k3d.internal:[PORT]/home/git/repos/bigbang 
# Check out release branch
git checkout 1.3.0

Private RegistryπŸ“œ

Images needed to run BB in your cluster is packaged as part of the release in images.tar.gz. You can see the list of required images in images.txt. In our airgap environment, we need to setup a registry that our cluster can pull required images from or an existing cluster where we can copy images from images.tar.gz into.

Set UpπŸ“œ

To setup the registry, we will be using registry:2 to run a private registry with self-signed certificate.

  • First, untar images.tar.gz;
tar -xvf images.tar.gz -C .
  • SCP registry:2 tar file
docker save -o registry2.tar registry:2
docker save -o k3s.tar rancher/k3s:v1.20.5-rc1-k3s1 #check release matching version
scp registry2.tar k3s.tar ubuntu@hostname:~ #modify according to your environment
docker load -i registry2.tar #on your registry server
docker load -i k3s.tar
$ chmod +x && sudo ./

Required information:
Enter bit size for certs (Ex. 4096): 4096
Enter number of days to sign the certs with (Ex. 3650): 3650
Enter the 'Country' for the cert (Ex. US): US
Enter the 'State' for the cert (Ex. CO): CO
Enter the 'Location' for the cert (Ex. ColoradoSprings): ColoradoSprings
Enter the 'Organization' for the cert (Ex. PlatformOne): PlatformOne
Enter the 'Organizational Unit' for the cert (Ex. Bigbang): BigBang
Enter the 'Common Name' for the cert (Must be a FQDN (at least one period character) E.g.
Enter the 'Subject Alternative Name' for the cert(E.g.

Generating certs ...
mkdir: cannot create directory β€˜certs’: File exists
Generating RSA private key, 4096 bit long modulus
e is 65537 (0x10001)
Generating RSA private key, 4096 bit long modulus
e is 65537 (0x10001)
Signature ok
Getting CA Private Key

Launching our private registry ...

Installation finished ...


To see images in the registry;

curl -k

A folder is created with TLS certs that we are going to supply to our k8s cluster when pulling from the registry.

You can ensure the images are now loaded in the registry;

 curl -k 


The images specified as part of the helm charts in BB are expected to be sourced from hence this registry needs to be mirrored to the one setup above. To reduce the amount of work needed on the developer part, we will be taking advantage of container mirroring which is supported by containerd as well as cri-o. Check if your container runtime supports this as it is required for smooth developer experience when deploying BB. You should also check documentation on how your cluster supports passing these configuration to the runtime. For example, TKG and RKE2 support such configuration for containerd below to enable and .

​You need to also configure your cluster with appropriate registry TLS. Please consult your cluster documentation on how to configure this.

If you need to handle mirroring manually, there is an example Ansible script provided that will update the containerd mirroring and restart the container runtimes for each node in your inventory. (copy-containerd-config.yaml)

Konvoy ClusterπŸ“œ

Modify the cluster.yaml file and apply. More details can be found on the D2iQ Konvoy documentation.

kind: ClusterConfiguration
    - server:
      username: "myuser"
      password: "mypassword"
      default: true
    - server:
      username: "myuser"
      password: "mypassword"
      default: true

TKG ClusterπŸ“œ

      - path: /etc/containerd/config.toml
        content: |
          version = 2
              sandbox_image = ""
                    endpoint = [""]
                    endpoint = [""]

RKE2 clusterπŸ“œ

      ca_file: "/etc/ssl/certs/registry1.pem"

Installing Big BangπŸ“œ

cd bigbang

Install flux

Install Flux 2 into the cluster using the provided artifacts. These are located in the scripts section of the Big Bang repository.

kubectl apply -f ./scripts/deploy/flux.yaml

After Flux is up and running you are ready to deploy Big Bang. We will do this using Helm. To first check to see if Flux is ready you can do.

You can watch to see if Flux is reconciling the projects by watching the progress.

watch kubectl get all -n flux-system

We need a namespace for our preparations and eventually for Big Bang to deploy into.

kubectl create ns bigbang

Installing Big Bang in an air gap environment currently uses the Helm charts from the Big Bang Repo.

All changes are modified in the custom values.yaml file. Modify as needed and replace IP.

Change the hostname for the installation. It is currently set to the development domain:

# -- Domain used for BigBang created exposed services, can be overridden by individual packages.

Add your registry URL. This will be the IP address or URL of the utility server or the registry in which you have loaded all of the Big Bang images (note: it is possible that your registry doesn’t have a username or password, there will be ignored for insecure registries.):

# -- Single set of registry credentials used to pull all images deployed by BigBang.
  username: "asdfasdfasdf"
  password: "asdfasdfasdfasdfasdf"
  email: ""

For your Git repository you have two options for setting up the credentials.

Option 1: Use an existing secret.

cd ~/.ssh
ssh-keygen  -b 4096 -t rsa -f ~/.ssh/identity -q -N ""
ssh-keyscan  <YOUR GIT URL HERE> ./known_hosts

kubectl create secret generic -n bigbang ssh-credentials \
    --from-file=./identity \
    --from-file=./ \

In the above example we created a new set of keys to use, you could also use an existing set of keys. These are just SSH keys, so any SSH key pair should work. The second command is going to create a known hosts file. There is no way to answer yes to the unknown hosts prompt, this alleviates that need.

Once we have our private key, public key and the known hosts file, we place all of those into the secret using kubectl. This creates a BASE64 encoded secret of these values. !!! It is VERY important that the names of the files match above. So if you are using your own keypair change the names. Kubernetes uses the names of the files to create the keys inside of the secret.

If you want to create your secret and store in the Kubernetes format you can add the -o yaml –dry-run to the above command to get that output.

kubectl create secret generic ssh-credentials \
    --from-file=./identity \
    --from-file=./ \
    --from-file=./known_hosts \
    -o yaml --dry-run

Once your secret is created you can add that value to the values.yaml that we were modifying above.

  # -- Existing secret to use for git credentials, must be in the appropriate format:
  existingSecret: "ssh-credentials"

** Note that we substituted the name of the secret from the example to the secret created above. This value is arbitrary, so if you created your secret with a different name use that name instead.

Option 2: Put the values of your ssh keys directly in the values.yaml file.

You can also elect to just put the key values and the known hosts directly into the chart’s values.yaml file.

ssh-keygen -q -N "" -f ./identity
ssh-keyscan <YOUR GIT URL HERE> ./known_hosts

cat identity
cat known_hosts

Take the values from each of these files and place in the correct fields in the values.yaml.

    # -- SSH git credentials, privateKey, publicKey, and knownHosts must be provided
    privateKey: |
      -----BEGIN RSA PRIVATE KEY-----
      -----END RSA PRIVATE KEY-----
    publicKey: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwbpgqyoMLrvbxdnv/yiKqdBrDc6RCe8wGvO9mPF3/+wSiDFVz4X1ipbgUh3MfnvA2olcgLiIxsTRAZ8r7XI6hiJeSGLrq2123kU36fg94ye8pwIle0VaZ7hb/1icST/exs06AAw+WEyFOy53MuJe0d3e$"
    knownHosts: " ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPFZzQ6BmaswdhT8UWD5a/VYmZYrGv1qD3T+euf/gFjkPkeySYRIyM+Kg/UdHCHVBzc4aaFdBDmugHimZ4lbWpE="

** Note the above values are all examples and are intentionally not operational keys.

Then install Big Bang using Helm.

    helm upgrade -i bigbang chart -n bigbang --create-namespace -f values.yaml
    watch kubectl get gitrepositories,kustomizations,hr,po -A

** Note that the –create-namespace isn’t needed if you created it earlier, but it doesn’t hurt anything.

You should see the different projects configure working through their reconciliation starting with “gatekeeper”.

Using 3rd Party PackagesπŸ“œ

The third party guide assumes that you already have or are planning to install Big Bang Core.

Package your Git repositoryπŸ“œ

Packaging your repository from Git

git clone --no-checkout && tar -zcvf kafka-repo.tar.gz kafka

This creates a tar of a full git repo without a checkout. After you have placed this git repo in its destination you can get the files to view by doing.

git checkout

Package your registry imagesπŸ“œ

Package image

docker save -o image-name.tar image-name:image-version

Unpack the image on your utility server

tar -xvf image-name.tar

Move the image to the location of your other images.

Restart your local registry and it should pick up the new image.

cd ./var/lib/registry
docker run -p 25000:5000 -v $(pwd):/var/lib/registry registry:2
# verify the registry mounted correctly
curl http://localhost:25000/v2/_catalog -k
# a list of Big Bang images should be displayed, if not check the volume mount of the registry

Configure ./synker.yaml


    # Hostname of the destination registry to push to
    # Port of the destination registry to push to
    port: 5000

If you are using runtime mirroring the new image should be available at the original location on your cluster.

Last update: 2023-10-02 by Christopher O'Connell