Skip to content

Node Affinity & Anti-Affinity with Monitoring📜

Affinity is exposed through values options for Monitoring. If you want to schedule your pods to deploy on specific nodes you can do that through multiple values. Additional info is provided below as well to help in configuring this.

It is good to have a basic knowledge of node affinity and available options to you before customizing in this way - the upstream kubernetes documentation has a good walkthrough of this.

Values for NodeSelector📜

The nodeSelector value can be set at multiple levels (alertmanager, prometheusOperator, prometheus, grafana, kube-state-metrics, and the node-exporter) to do basic node selection for deployments. See the below example for an example to schedule pods to only nodes with the label node-type equal to monitoring:

alertmanager:
  alertmanagerSpec:
    nodeSelector:
      node-type: monitoring

prometheus:
  prometheusSpec:
    nodeSelector:
      node-type: monitoring

prometheusOperator:
  nodeSelector:
    node-type: monitoring

grafana:
  nodeSelector:
    node-type: monitoring

kube-state-metrics:
  nodeSelector:
    node-type: monitoring

prometheus-node-exporter:
  nodeSelector:
    node-type: monitoring

Values for Affinity📜

Affinity values for prometheus can be specified at all the same levels. The format to include follows what you’d specify at a pod level. See the examples below to see the correct values for scheduling pods to only nodes with the label node-type equal to monitoring:

alertmanager:
  alertmanagerSpec:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: node-type
              operator: In
              values:
              - monitoring

prometheus:
  prometheusSpec:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: node-type
              operator: In
              values:
              - monitoring

prometheusOperator:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-type
            operator: In
            values:
            - monitoring

grafana:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-type
            operator: In
            values:
            - monitoring

kube-state-metrics:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-type
            operator: In
            values:
            - monitoring

prometheus-node-exporter:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-type
            operator: In
            values:
            - monitoring

Values for Anti-Affinity📜

Anti-affinity is also configurable for all the same levels. Alertmanager and prometheus provide simple “soft”/”hard” configuration to configure those pods to not deploy on the same nodes as other alertmanager/prometheus pods. The other 4 pieces of monitoring require a manual configuration of anti-affinity (following the upstream pattern). For the “manual” anti-affinity examples are provided for keeping pods from scheduling on nodes that have pods labeled dont-schedule-with: <name>:

alertmanager:
  alertmanagerSpec:
    podAntiAffinity: "hard"

prometheus:
  prometheusSpec:
    podAntiAffinity: "hard"

prometheusOperator:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - topologyKey: kubernetes.io/hostname
        labelSelector:
          matchLabels:
            dont-schedule-with: prometheusOperator

grafana:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - topologyKey: kubernetes.io/hostname
        labelSelector:
          matchLabels:
            dont-schedule-with: grafana

kube-state-metrics:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - topologyKey: kubernetes.io/hostname
        labelSelector:
          matchLabels:
            dont-schedule-with: kube-state-metrics

prometheus-node-exporter:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - topologyKey: kubernetes.io/hostname
        labelSelector:
          matchLabels:
            dont-schedule-with: prometheus-node-exporter