Prometheus💣
Prometheus is a time series database optimized for storing metrics instead of a push-based database where it waits for an entity to push data into it. Prometheus functions as a pull-based database, gathering and storing data. It collects data by “scraping” Metric Endpoints and periodically curls web pages hosting Prometheus metrics.
Prometheus Operator Components💣
- Prometheus Operator Pod:
- Reads the desired state of Custom Resources (CRs) and make them a reality.
- Prometheus CR
- –spawns–> Prometheus Statefulset.
- AlertManager CR
- –spawns–> AlertManager Statefulset.
- ServiceMonitors and PrometheusRules
- Store snippets of configuration for Prometheus and AlertManager. The Prometheus Operator Pod configures Prometheus & AlertManager based on the contents of these CRs (lab).
- Prometheus
- A database optimized for metrics. It’s config is read in from a secret, Prom Op pod dynamically edits the secret based on contents of ServiceMonitors.
- AlertManager
- Can send alerts based on metric thresholds.
-
Grafana
- Shows dashboards and graphs of Prometheus metrics.
-
In addition to Prometheus, Grafana, AlertManager, Prometheus Operator, and its Custom Resources, the following are also part of the Prometheus Operator Stack.
- The Grafana deployment has a sidecar container, so ConfigMaps that follow a convention can automatically import Dashboards into Grafana. (Lab.)
- Kube-state-metrics deployment makes some additional Kubernetes metrics available to Prometheus.
- Prometheus Node Exporter Daemonset makes node level metrics available to Prometheus.
Example Prometheus CR💣
The Prometheus Operator Pod spins up Prometheus Server Pods based on the Prometheus Custom Resource:
(Notice Prometheus is a recognized Kubernetes object type. It stores the desired state/configuration, showing a replica config snippet as an example.💣
)
kubectl get prometheus --all-namespaces
NAMESPACE NAME AGE
monitoring k8s 3d5h
kubectl get prometheus k8s -n=monitoring -o yaml | grep replicas
replicas: 2
kubectl get pod -n=monitoring -l=app=prometheus
NAME READY STATUS RESTARTS AGE
prometheus-k8s-0 3/3 Running 1 3d5h
prometheus-k8s-1 3/3 Running 1 3d5h
PromQL💣
What is PromQL: Prometheus has it’s own query language called PromQL.
Basic Examples of what PromQL looks like: kube_pod_container_status_last_terminated_reason kube_pod_container_status_last_terminated_reason{reason=”Completed”} (To narrow down the metrics, you can utilize tags for filtering. )
What is PromQL used for: * Creating Metric Based Alerts in AlertManager. * Creating Graphs in Grafana based on Prometheus Metrics. * You’d only need to know if you were creating or customizing a Grafana Dashboard or setting up Alerts. The best way to learn it is to explore PromQL entries used to generate dashboards.
GitOps💣
The Prometheus Operator Stack is GitOps friendly:
- ServiceMonitors can be used to configure Prometheus’s Metric Collection.
- PrometheusRules can be used to configure Automated Alerts.
- ConfigMaps that follow a convention can be used to automatically import Dashboards into Grafana.
These tasks can be done by creating a YAML and doing a kubectl
apply or uploading said YAML to a Git repo and letting GitOps operator/software bot kubectl
apply on your behalf.