Gitlab Runner¶

Table Of Contents¶

Gitlab Runner
Table Of Contents

Application Overview¶

GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with GitLab CI/CD, the open-source continuous integration service included with GitLab that coordinates the jobs. The gitlab-runner pod is deployed via a Helm chart into its own dedicated gitlab-runner namespace for improved security and isolation.

Note: If using the autoRegister.enabled=true feature (default when enabling via Big Bang umbrella values), Kyverno is required. A Kyverno policy (sync-gitlab-runner-secret) is used to automatically copy the necessary registration token secret from the gitlab namespace to the gitlab-runner namespace. See the Global Shared Gitlab Runner section for more details. For direct package installs with autoRegister.enabled=true, set upstream.runners.secret to the synced token Secret name, such as gitlab-gitlab-runner-secret. The Big Bang umbrella chart sets this value automatically when it enables auto-registration.

Daily Application Use¶

Gitlab Runner is configured to run as a globally shared runner. Any repository in the gitlab installation will have this runner available for CI/CD pipelines. This runner automatically registers with the gitlab installation upon installation.

This globally shared gitlab-runner pod can be seen in any repository CI/CD Settings page, for example: https://code.bigbang.dev/GROUP/PROJECT/-/settings/ci_cd

The runner uses the IronBank RedHat UBI8 image to run jobs.

Integrations¶

Prometheus¶

Prerequisites¶

Monitoring stack or Prometheus

Configuration Items¶

Setting metrics.enabled to true will expose the gitlab-runner /metrics endpoint to the cluster using port 9252. If using the monitoring stack with BigBang, it will detect that gitlab-runner is enabled and automatically create a service monitor to scrape the endpoint with Prometheus.

List of metrics gathered¶

The exposed information includes:

Runner business logic metrics (e.g., the number of currently running jobs)
Go-specific process metrics (garbage collection stats, goroutines, memstats, etc.)
General process metrics (memory usage, CPU usage, file descriptor usage, etc.)
Build version information

Reference: GitLab Runner monitoring

Verification¶

To verify that the metrics are working, open a browser to your prometheus endpoint (e.g. https://prometheus.bigbang.dev). Under Status, select Targets. You should find monitoring/gitlab-gitlab-runner/0 (if using Big Bang) or the name of your Service Monitor as a target and the State should be UP.

Elastic¶

Pre-requisites¶

Monitoring stack is deployed
ECK stack deployed

Configuration Items¶

Login to Kibana
Username: elastic
Password: obtain with kubectl get secret logging-ek-es-elastic-user -n logging -o json | jq -r .data.elastic | base64 -d
Create Index by selecting Management > Stack Management from the left menu, then click Kibana > Index patterns. In the Create Index patterns enter logstash-* and click Create index pattern. In the the next step, click on the time filed dropdown and select @timestamp. Finally, create Create index pattern
To use the index, click Kibana > Discovery on the side menu. In the KQL entry box, enter kubernetes.labels.release : "gitlab-runner" and press enter.

Global Shared Gitlab Runner¶

On a default installation, there will be a single pod loaded in the format gitlab-runner-gitlab-runner-# in the gitlab-runner namespace as shown below.

# Example from gitlab-runner namespace
NAME                                          READY   STATUS    RESTARTS   AGE
gitlab-runner-gitlab-runner-7d48f9cc9c-abcde  1/1     Running   0          42m

Important Note on Namespace and Auto-Registration: As mentioned in the Application Overview, GitLab Runner resides in its own gitlab-runner namespace, separate from the main GitLab components in the gitlab namespace. This enhances security.

For the runner to automatically register with GitLab (autoRegister.enabled=true), it needs access to the registration token stored in the gitlab-gitlab-runner-secret within the gitlab namespace. Since Kubernetes namespaces provide isolation, direct access is blocked.

Big Bang solves this using a Kyverno GeneratingPolicy named sync-gitlab-runner-secret. This policy automatically:

Detects the gitlab-runner namespace.
Clones the gitlab-gitlab-runner-secret from the gitlab namespace into the gitlab-runner namespace.
Keeps the cloned secret synchronized with the original.

This ensures the runner in the gitlab-runner namespace can securely obtain the token needed for registration without compromising namespace isolation. Kyverno must be installed and functional for auto-registration to work.

Useful Queries¶

kubernetes.pod_name : "gitlab-runner-gitlab-runner-#" to get logs from a specific # pod
kubernetes.container_name : "gitlab-runner-gitlab-runner" to get logs from a specific container
kubernetes.labels.release : "gitlab-runner" to get logs from all gitlab runners

Network Policies¶

By default, Gitlab Runner egress for runner jobs is locked down within its dedicated gitlab-runner namespace. You can dynamically create additional network policies with a values override. For example:

networkPolicies:
  additionalPolicies:
  # expected use case is to open egress for runner jobs
  # This is a dev example policy spec and CIDR 0.0.0.0/0 is unsafe for operational environments
  # requests to controlPlane should also be blocked in an operational policy
  - name: egress-runner-jobs
    spec: 
      podSelector: {}
      policyTypes:
      - Egress
      egress:
      - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
            # Block requests to AWS cloud metadata IP
            - 169.254.169.254/32
            # Block requests to controlPlane if CIDR not 0.0.0.0/0
            - "{{ $.Values.networkPolicies.controlPlaneCidr }}"

Note: By default, the Big Bang Gitlab Runner package is configured to pull kubernetes executor images from registry1.dso.mil. Also, the gitlab-runner pod will require egress to the kube-apiserver in order to create pods for CI jobs. The networkPolicy templates handle creation of the necessary policy.

Troubleshooting Tips¶

The gitlab-runner is configured to clone the repository from the gitlab webservice via the “clone_url” setting. This setting typically points to the GitLab instance’s external URL or internal service name (e.g., gitlab-webservice.gitlab.svc...). The runner also uses this endpoint (or the one specified in gitlabUrl) to register with the GitLab API.
If the GitLab instance (webservice) is behind HTTPS ingress and uses a certificate not trusted by the runner’s environment (e.g., a self-signed or private CA), certificate issues can occur during cloning or registration. In such cases, the certsSecretName value in values.yaml can be populated with the name of a secret containing the necessary CA certificate (ca.crt) and optionally client certificates (tls.crt, tls.key). The runner will mount these certificates to establish trust.
Registration Failure: If the runner fails to register (often seen as errors in the runner pod logs about failing to connect or register), and you have autoRegister.enabled=true, verify the following:
Kyverno is installed and running correctly in the cluster.
The Kyverno GeneratingPolicy named sync-gitlab-runner-secret exists and is active.
Check if the gitlab-gitlab-runner-secret has been successfully created/synced into the gitlab-runner namespace (kubectl get secret gitlab-gitlab-runner-secret -n gitlab-runner). If this secret is missing, the runner cannot get the token to register. Investigate Kyverno policy status and logs.