Skip to content

Metrics w/ Prometheus

Prometheus💣

Prometheus is a time series database optimized for storing metrics instead of a push-based database where it waits for an entity to push data into it. Prometheus functions as a pull-based database, gathering and storing data. It collects data by “scraping” Metric Endpoints and periodically curls web pages hosting Prometheus metrics.

Prometheus Operator Components💣

  • Prometheus Operator Pod:
    • Reads the desired state of Custom Resources (CRs) and make them a reality.
  • Prometheus CR
    • –spawns–> Prometheus Statefulset.
  • AlertManager CR
    • –spawns–> AlertManager Statefulset.
  • ServiceMonitors and PrometheusRules
    • Store snippets of configuration for Prometheus and AlertManager. The Prometheus Operator Pod configures Prometheus & AlertManager based on the contents of these CRs (lab).
  • Prometheus
    • A database optimized for metrics. It’s config is read in from a secret, Prom Op pod dynamically edits the secret based on contents of ServiceMonitors.
  • AlertManager
    • Can send alerts based on metric thresholds.

  • Grafana

    • Shows dashboards and graphs of Prometheus metrics.

  • In addition to Prometheus, Grafana, AlertManager, Prometheus Operator, and its Custom Resources, the following are also part of the Prometheus Operator Stack.

  • The Grafana deployment has a sidecar container, so ConfigMaps that follow a convention can automatically import Dashboards into Grafana. (Lab.)
  • Kube-state-metrics deployment makes some additional Kubernetes metrics available to Prometheus.
  • Prometheus Node Exporter Daemonset makes node level metrics available to Prometheus.

Example Prometheus CR💣

The Prometheus Operator Pod spins up Prometheus Server Pods based on the Prometheus Custom Resource:

Notice Prometheus is a recognized Kubernetes object type. It stores the desired state/configuration, showing a replica config snippet as an example.

kubectl get prometheus --all-namespaces
NAMESPACE NAME AGE
monitoring k8s 3d5h

kubectl get prometheus k8s -n=monitoring -o yaml | grep replicas
replicas: 2

kubectl get pod -n=monitoring -l=app=prometheus
NAME              READY  STATUS   RESTARTS  AGE 
prometheus-k8s-0  3/3    Running  1         3d5h 
prometheus-k8s-1  3/3    Running  1         3d5h

PromQL💣

What is PromQL: Prometheus has it’s own query language called PromQL.

Basic Examples of what PromQL looks like: kube_pod_container_status_last_terminated_reason kube_pod_container_status_last_terminated_reason{reason=”Completed”} (To narrow down the metrics, you can utilize tags for filtering. )

What is PromQL used for: * Creating Metric Based Alerts in AlertManager. * Creating Graphs in Grafana based on Prometheus Metrics. * You’d only need to know if you were creating or customizing a Grafana Dashboard or setting up Alerts. The best way to learn it is to explore PromQL entries used to generate dashboards.

GitOps💣

The Prometheus Operator Stack is GitOps friendly:

  • ServiceMonitors can be used to configure Prometheus’s Metric Collection.
  • PrometheusRules can be used to configure Automated Alerts.
  • ConfigMaps that follow a convention can be used to automatically import Dashboards into Grafana.

These tasks can be done by creating a YAML and doing a kubectl apply or uploading said YAML to a Git repo and letting GitOps operator/software bot kubectl apply on your behalf.