Skip to content

Metrics w/ Prometheus

Prometheus📜

Prometheus is a time series database optimized for storing metrics. Prometheus functions as a pull-based database, gathering and storing data. It collects data by “scraping” Metric Endpoints and periodically curls web pages hosting Prometheus metrics.

Prometheus Operator Components📜

  • Prometheus Operator Pod:
    • Reads the desired state of Custom Resources (CRs) and make them a reality.
  • Prometheus CR
    • –spawns–> Prometheus Statefulset.
  • AlertManager CR
    • –spawns–> AlertManager Statefulset.
  • ServiceMonitors and PrometheusRules
    • Store snippets of configuration for Prometheus and AlertManager. The Prometheus Operator Pod configures Prometheus & AlertManager based on the contents of these CRs (lab).
  • Prometheus
    • A database optimized for metrics. It’s config is read in from a secret, Prom Op pod dynamically edits the secret based on contents of ServiceMonitors.
  • AlertManager
    • Can send alerts based on metric thresholds.

  • Grafana

    • Shows dashboards and graphs of Prometheus metrics.

  • In addition to Prometheus, Grafana, AlertManager, Prometheus Operator, and its Custom Resources, the following are also part of the Prometheus Operator Stack.

  • The Grafana deployment has a sidecar container, so ConfigMaps that follow a convention can automatically import Dashboards into Grafana. (Lab.)
  • Kube-state-metrics deployment makes some additional Kubernetes metrics available to Prometheus.
  • Prometheus Node Exporter Daemonset makes node level metrics available to Prometheus.

Example Prometheus CR📜

The Prometheus Operator Pod spins up Prometheus Server Pods based on the Prometheus Custom Resource:

Notice Prometheus is a recognized Kubernetes object type. It stores the desired state/configuration, showing a replica config snippet as an example.

kubectl get prometheus --all-namespaces
NAMESPACE NAME AGE
monitoring k8s 3d5h

kubectl get prometheus k8s -n=monitoring -o yaml | grep replicas
replicas: 2

kubectl get pod -n=monitoring -l=app=prometheus
NAME              READY  STATUS   RESTARTS  AGE 
prometheus-k8s-0  3/3    Running  1         3d5h 
prometheus-k8s-1  3/3    Running  1         3d5h

PromQL📜

What is PromQL: Prometheus has it’s own query language called PromQL.

Basic Examples of what PromQL looks like: kube_pod_container_status_last_terminated_reason kube_pod_container_status_last_terminated_reason{reason=”Completed”} (To narrow down the metrics, you can utilize tags for filtering. )

What is PromQL used for: * Creating Metric Based Alerts in AlertManager. * Creating Graphs in Grafana based on Prometheus Metrics. * You’d only need to know if you were creating or customizing a Grafana Dashboard or setting up Alerts. The best way to learn it is to explore PromQL entries used to generate dashboards.

GitOps📜

The Prometheus Operator Stack is GitOps friendly:

  • ServiceMonitors can be used to configure Prometheus’s Metric Collection.
  • PrometheusRules can be used to configure Automated Alerts.
  • ConfigMaps that follow a convention can be used to automatically import Dashboards into Grafana.

These tasks can be done by creating a YAML and doing a kubectl apply or uploading said YAML to a Git repo and letting GitOps operator/software bot kubectl apply on your behalf.

Quiz Questions📜

What is Prometheus?

Prometheus is a time series database optimized for storing metrics. Prometheus functions as a pull-based database, gathering and storing data. It collects data by “scraping” Metric Endpoints and periodically curls web pages hosting Prometheus metrics.

What package could be used with Prometheus in order to display graphs and dashboards of Prometheus metrics?

Grafana can be used to show dashboards and graphs of Prometheus metrics. The Grafana deployment has a sidecar container, so ConfigMaps that follow a convention can automatically import Dashboards into Grafana

Does Prometheus have it’s own query language?

Yes, PromQL is a query language used for Prometheus