Skip to content

Development k3d Cluster Automation💣

NOTE: This script does not install Flux or deploy Big Bang. You must handle those deployments after your k3d dev cluster is ready.

The instance will automatically terminate 8 hours after creation.

Install and Configure Dependencies💣

  1. Install aws cli
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
# sudo apt install unzip -y
unzip awscliv2.zip
sudo ./aws/install
rm -rf aws
rm awscliv2.zip
aws --version
  1. Configure aws cli
aws configure
# aws_access_key_id - The AWS access key part of your credentials
# aws_secret_access_key - The AWS secret access key part of your credentials
# region - us-gov-west-1
# output - json

# Verify configuration
aws configure list

  1. Install jq
    Follow jq installation instructions for your workstation operating system.
    https://stedolan.github.io/jq/download/

  2. Mac users will need to install the GNU version of the sed command.
    https://medium.com/@bramblexu/install-gnu-sed-on-mac-os-and-set-it-as-default-7c17ef1b8f64

Usage💣

The default with no options specified is to use the EC2 public IP for the k3d cluster and the security group.

./docs/assets/scripts/developer/k3d-dev.sh -h
AWS User Name: your.name
Usage:
k3d-dev.sh -b -p -m -d -h

 -b   use big M5 instance. Default is t3.2xlarge
 -p   use private IP for security group and k3d cluster
 -m   create k3d cluster with metalLB
 -d   destroy related AWS resources
 -h   output help
After running the script, it is recommended that you save the instructions outputted.

Export KUBECONFIG💣

In order to use kubectl commands, export your KUBECONFIG

export KUBECONFIG=~/.kube/your.name-dev-config

Update IP💣

To access packages via the web, your IP needs to be updated. Open up /etc/hosts and update the IP corresponding to <package>.bigbang.dev

sudo vi /etc/hosts

If the <package>.bigbang.dev field does not exist, create <package>.bigbang.dev for each package you are using, or plan to open on the web, then apply the cluster IP to the front of that list

Additional Step for WSL users💣

Using WSL requires users to update both their Windows Hosts File with the cluster IP as well as updating the /etc/hosts file on WSL.

PowerToys - It is recommended to install the PowerToys application to update your Windows Hosts File using the Host File Editor.

  • After opening PowerToys, navigate to Host File Editor and update the IP field for <package>.bigbang.dev
  • If the <package>.bigbang.dev field does not exist, create <package>.bigbang.dev for each package you are using, or plan to open on the web, then apply the cluster IP

Alternative to using PowerToys:

  1. Open Notepad or another text editor like Notepad++

  2. In the text editor, select File > Open and open the HOST file location at C:\Windows\System32\drivers\etc\.

  3. Select Text Documents (*txt) in the bottom-right of the Open window and change it to All Files.

  4. When files appear in the folder, double click hosts to open it.

  5. Edit the HOSTS file and update the IP field for <package>.bigbang.dev

    5a. If the <package>.bigbang.dev field does not exist, create <package>.bigbang.dev for each package you are using, or plan to open on the web, then apply the cluster IP

  6. Save your changes

Install FluxCD💣

The Big Bang product is tightly coupled with the GitOps tool FluxCD. Before you can deploy Big Bang you must deploy FluxCD on your k8s cluster. To guarantee that you are using the version of FluxCD that is compatible with the version of Big Bang that you are deploying use the Big Bang provided script ./scripts/install_flux.sh. You will need your Iron Bank pull credentials and command line access to the k8s cluster from your workstation.

./scripts/install_flux.sh -u <your-user-name> -p <your-password>

Deploy Bigbang💣

From the bigbang directory deploy Big Bang via helm 

helm upgrade -i bigbang chart/ -n bigbang --create-namespace --set registryCredentials.username=XXXXX --set registryCredentials.password='XXXXX' -f chart/values.yaml

Overrides can be supplemented by adding references to the specific yaml file, the right-most values file will take highest precedence: 

-f ../other-overrides.yaml

It is recommended to create a registry-values.yaml file in an overrides directory, located outside of the bigbang directory, that contains your credentials instead of having to --set them each time.

registry-values.yaml💣

registryCredentials:
  registry: registry1.dso.mil
  username: "XXXXXXXXXXX"
  password: "XXXXXXXXXXX"
  email: "XXXXXXXXXXX"

Using this overrides file:

helm upgrade -i bigbang ./chart -n bigbang --create-namespace -f ../overrides/registry-values.yaml

To use a different AWS profile, VPC, or AMI💣

The ./docs/assets/scripts/developer/k3d-dev.sh script uses the default AWS profile and defaults the VPC id and AMI id. To change this default behavior you can export environment variables before running the script.

Run the script with a specific AWS profile by first exporting the AWS_PROFILE environment variable: 

export AWS_PROFILE=my-aws-profile
To run the script with a specific VPC: 
export VPC_ID=vpc-XXXXXXXXXXXX
To run the script with a specific AMI: 
export AMI_ID=ami-XXXXXXXXXX
To specify a specific Kubernetes version (the eligible choices are here): 
export K3S_IMAGE_TAG=sometag

You can also run these inline with the script, without exporting them to your environment. Example:

K3S_IMAGE_TAG=v1.24.12-k3s1 ./docs/assets/scripts/developer/k3d-dev.sh

Troubleshooting💣

  1. If you are on a Mac insure that you have GNU sed command installed. Otherwise you will see this error and the kubeconfig will not be updated with the IP from the instance.
copy kubeconfig
config                         100% 3019    72.9KB/s   00:00
sed: 1: "...": extra characters at the end of p command

  1. If you get a failure from the script study and correct the error. Then run script with “-d” option to clean up resources. Then re-run your original command.

  2. Occasionally a ssh command will fail because of connection problems. If this happens the script will fail with “unexpected EOF”. Simply try again. Run the script with -d to clean up resources. Then re-run your original command.

Termination and Teardown💣

Termination💣

Run the k3d-dev.sh script with the -d flag

./docs/assets/scripts/developer/k3d-dev.sh -d

Teardown💣

  1. Delete the bigbang namespace

    helm delete bigbang -n bigbang

    1a. If problems occur during tear where other namespaces do not get deleted, run the remove-ns-finalizer.sh script for each namespace having issues

    ./scripts/remove-ns-finalizer.sh <namespace>
    2. Remove bigbang

    kubectl delete ns bigbang