Skip to content

Gitlab RunnerπŸ’£

Table Of ContentsπŸ’£

Application OverviewπŸ’£

GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with GitLab CI/CD, the open-source continuous integration service included with GitLab that coordinates the jobs. The gitlab-runner pod is deployed in the gitlab namespace via a Helm chart.

Daily Application UseπŸ’£

Gitlab Runner is configured to run as a globally shared runner. Any repository in the gitlab installation will have this runner available for CI/CD pipelines. This runner automatically registers with the gitlab installation upon installation.

This globally shared gitlab-runner pod can be seen in any repository CI/CD Settings page eg: https://code.bigbang.dev/GROUP/PROJECT/-/settings/ci_cd

The runner uses the IronBank RedHat UBI8 image to run jobs.

IntegrationsπŸ’£

PrometheusπŸ’£

PrerequisitesπŸ’£
  • Monitoring stack or Prometheus
Configuration ItemsπŸ’£

Setting metrics.enabled to true will expose the gitlab-runner /metrics endpoint to the cluster using port 9252. If using the monitoring stack with BigBang, it will detect that gitlab-runner is enabled and automatically create a service monitor to scrape the endpoint with Prometheus.

List of metrics gatheredπŸ’£

The exposed information includes:

  • Runner business logic metrics (e.g., the number of currently running jobs)
  • Go-specific process metrics (garbage collection stats, goroutines, memstats, etc.)
  • General process metrics (memory usage, CPU usage, file descriptor usage, etc.)
  • Build version information

Reference: GitLab Runner monitoring

VerificationπŸ’£

To verify that the metrics are working, open a browser to your prometheus endpoint (e.g. https://prometheus.bigbang.dev). Under Status, select Targets. You should find monitoring/gitlab-gitlab-runner/0 (if using Big Bang) or the name of your Service Monitor as a target and the State should be UP.

ElasticπŸ’£

Pre-requisitesπŸ’£
  • Monitoring stack is deployed
  • ECK stack deployed
Configuration ItemsπŸ’£
  • Login to Kibana
  • Username: elastic
  • Password: obtain with kubectl get secret logging-ek-es-elastic-user -n logging -o json | jq -r .data.elastic | base64 -d
  • Create Index by selecting Management > Stack Management from the left menu, then click Kibana > Index patterns. In the Create Index patterns enter logstash-* and click Create index pattern. In the the next step, click on the time filed dropdown and select @timestamp. Finally, create Create index pattern
  • To use the index, click Kibana > Discovery on the side menu. In the KQL entry box, enter kubernetes.labels.release : "gitlab-runner" and press enter.

Global Shared Gitlab RunnerπŸ’£

On a default installation, there will be a single pod loaded in the format gitlab-runner-gitlab-runner-# in the gitlab namespace as shown below.

NAME                                          READY   STATUS    RESTARTS   AGE
gitlab-runner-gitlab-runner-858b5c6796-s694b  1/1     Running   0          156m

Useful QueriesπŸ’£

  • kubernetes.pod_name : "gitlab-runner-gitlab-runner-#" to get logs from a specific # pod
  • kubernetes.container_name : "gitlab-runner-gitlab-runner" to get logs from a specific container
  • kubernetes.labels.release : "gitlab-runner" to get logs from all gitlab runners

Network PoliciesπŸ’£

By default, Gitlab Runner will inherit the network policies from the Gitlab namespace. Until a long-term solution is implemented that works for all Platform One teams, Gitlab Runner users may manually create their own network policies for the Gitlab Runner pods. For example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: runner-allow-egress
  namespace: gitlab
spec:
  policyTypes:
    - Egress
  podSelector:
    matchLabels:
      app: gitlab-runner-gitlab-runner
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            # ONLY Block requests to AWS metadata IP
            except:
              - 169.254.169.254/32

Note: By default, the Big Bang Gitlab Runner package is configured to pull kubernetes executor images from registry1.dso.mil. Also, the gitlab-runner pod will require egress to the kube-apiserver in order to create pods for CI jobs.

Troubleshooting TipsπŸ’£

The gitlab-runner is configured to clone the repository from the gitlab webservice via the β€œclone_url” setting so the runner clones from the same endpoint it uses to register with the gitlab webservice API which is the gitlab-webservice kubernetes service in the gitlab namespace. If gitlab webservice is behind HTTPS ingress then within the values.yaml file β€œcertSecretName” can be populated with certificate and key files if there are certificate issues when cloning.


Last update: 2021-10-15 by Micah Nagel