Skip to content

Monitoring💣

Overview💣

Monitoring in Bigbang is deployed using the upstream chart kube-prometheus-stack

Installs the kube-prometheus stack, a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

graph LR
 subgraph "Monitoring"
   alertmanagerpods("AlertManager Pod(s)") --> monitoringpods("Monitoring Pod(s)")
   alertmanagerservice{{AlertManager Service}} --> alertmanagerpods("AlertManager Pod(s)")
   alertmanagersvcmonitor("Service Monitor") --"Metrics Port"--> alertmanagerservice
   Prometheus --> alertmanagersvcmonitor("Service Monitor")
   grafanapods("Grafana Pod(s)") --> monitoringpods("Monitoring Pod(s)")
   grafanaservice{{Grafana Service}} --> grafanapods("Grafana Pod(s)")
   grafanasvcmonitor("Service Monitor") --"Metrics Port"--> grafanaservice
   Prometheus --> grafanasvcmonitor("Service Monitor")
   nodeexporterpods("Node-Exporter Pod(s)") --> monitoringpods("Monitoring Pod(s)")
   nodeexporterservice{{Node-Exporter Service}} -->  nodeexporterpods("Node-Exporter Pod(s)")
   nodeexportersvcmonitor("Service Monitor") --"Metrics Port"-->  nodeexporterservice
   Prometheus --> nodeexportersvcmonitor("Service Monitor")
   kubestatemetricspods("Kube-State-Metrics Pod(s)") --> monitoringpods("Monitoring Pod(s)")
   kubestatemetricsservice{{Kube-State-Metrics Service}} -->  kubestatemetricspods("Kube-State-Metrics Pod(s)")
   kubestatemetricssvcmonitor("Service Monitor") --"Metrics Port"-->  kubestatemetricsservice
   Prometheus --> kubestatemetricssvcmonitor("Service Monitor")
   Prometheus --> prometheussvcmonitor("Service Monitor")
   prometheussvcmonitor("Service Monitor") --"Metrics Port"--> prmetheussservice{{Prometheus Service}}
   prmetheussservice{{Prometheus Service}} --> Prometheus
   PromOperator ---|Manages/Creates| Prometheus
   VirtualServices --"App Port"--> alertmanagerservice
   VirtualServices --"App Port"--> grafanaservice
   VirtualServices --"App Port"--> Prometheus
 end
 subgraph "Logging"
   monitoringpods("Monitoring Pod(s)") ---|Logs|fluent(Fluentbit) --> logging-ek-es-http
   logging-ek-es-http{{Elastic Service<br />logging-ek-es-http}} --> elastic[(Elastic Storage)]
 end
 subgraph "Istio-system (Ingress)"
   ig(Ingress Gateway, Gateway) --> VirtualServices
 end

Big Bang Touchpoints💣

UI💣

Alertmanager, Prometheus and Grafana within the monitoring Package have UIs that are accessible and configurable. By default they are externally available behind an Istio installation.

Storage💣

Alertmanager💣

Persistent storage values for Alert Manager can be set/modified in the Big Bang chart:

monitoring:
  values:
    alertmanager:
      alertmanagerSpec:
        storage:
          volumeClaimTemplate:
            spec:
                storageClassName:
                accessModes: ["ReadWriteOnce"]
                resources:
                  requests:
                    storage: 50Gi
              selector: {}

Prometheus-Operator💣

Persistent storage values for Prometheus-Operator can be set/modified in the Big Bang chart:

monitoring:
  values:
    prometheus:
      prometheusSpec:
        storageSpec:
          volumeClaimTemplate:
            spec:
                storageClassName:
                accessModes: ["ReadWriteOnce"]
                resources:
                  requests:
                    storage: 50Gi
              selector: {}

Grafana💣

Persistent storage values for Grafana can be set/modified in the Big Bang chart:

monitoring:
  values:
    grafana:
      persistence:
        type: pvc
        enabled: false
        # storageClassName: default
        accessModes:
          - ReadWriteOnce
        size: 10Gi
        # annotations: {}
        finalizers:
          - kubernetes.io/pvc-protection
        # selectorLabels: {}
        # subPath: ""
        # existingClaim:

Logging💣

Within the kube-prometheus-stack chart, you can customize both the LogFormat and LogLevel for the following components: Note: within Big Bang, logs are captured by fluentbit and shipped to elastic by default.

Prometheus-Operator💣

LogFormat and LogLevel can be set for Prometheus-Operator via the following values in the Big Bang chart:

monitoring:
  values:
    prometheusOperator:
      logFormat: logfmt
      logLevel: info

Prometheus💣

LogFormat and LogLevel can be set for Prometheus via the following values in the Big Bang chart:

monitoring:
  values:
    prometheus:
      prometheusSpec:
        logFormat: logfmt
        logLevel: info

Alertmanager💣

LogFormat and LogLevel can be set for Alertmanager via the following values in the Big Bang chart:

monitoring:
  values:
    alertmanager:
      alertmanagerSpec:
        logFormat: logfmt
        logLevel: info

Grafana💣

LogLevel can be set for Grafana via the following values in the Big Bang chart:

monitoring:
  values:
    grafana:
      grafana.ini:
        log:
          mode: console

Single Sign on (SSO)💣

SSO can be configured for monitoring through Authservice, more info is included in the following documentation: Monitoring SSO Integration

Monitoring💣

Monitoring deployment has serviceMonitors enabled for

  • core-dns
  • kube-api-server
  • kube-controller-manager
  • kube-dns
  • kube-etcd
  • kube-proxy
  • kube-scheduler
  • kube-state-metrics
  • kubelet
  • node-exporter
  • alert manager
  • grafana
  • prometheus
  • prometheus-operator
  • node-exporter

Note: Other packages are responsible for deploying Service Monitors for their components as needed.

HA💣

Support for Prometheus and other apps within the package are being researched and section will be updated:

Alertmanager💣

High Availability can be accomplished by increasing the number of replicas for the deployment of Alertmanager;

monitoring:
  values:
    alertmanager:
      alertmanagerSpec:
        replicas: 3

Grafana💣

High Availability can be accomplished by increasing the number of replicas for the deployment of Grafana and configuring an external database connection (postgresql/mysql) so users and dashboard information can be centrally located for the replicas to have a source of truth. See Grafana’s upstream documentation

monitoring:
  values:
    grafana:
      replicas: 3
      grafana.ini:
        ...
    database:
      type: [postgres|mysql]
      host: external-db:5432
      name: grafana
      user: ""
      password: ""

Dependency Packages💣

When deploying BigBang, monitoring depends on gatekeeper/kyverno and istio being installed prior.

  {{- if or .Values.gatekeeper.enabled .Values.istio.enabled .Values.kyvernopolicies.enabled }}
  dependsOn:
  {{- if .Values.istio.enabled }}
    - name: istio
      namespace: {{ .Release.Namespace }}
  {{- end }}
  {{- if .Values.gatekeeper.enabled }}
    - name: gatekeeper
      namespace: {{ .Release.Namespace }}
  {{- end }}
  {{- if .Values.kyvernopolicies.enabled }}
    - name: kyvernopolicies
      namespace: {{ .Release.Namespace }}
  {{- end }}
  {{- end }}

Last update: 2022-11-04 by Ryan Garcia