Node Affinity & Anti-Affinity with Monitoringπ£
Affinity is exposed through values options for Monitoring. If you want to schedule your pods to deploy on specific nodes you can do that through multiple values. Additional info is provided below as well to help in configuring this.
It is good to have a basic knowledge of node affinity and available options to you before customizing in this way - the upstream kubernetes documentation has a good walkthrough of this.
Values for NodeSelectorπ£
The nodeSelector
value can be set at multiple levels (alertmanager, prometheusOperator, prometheus, grafana, kube-state-metrics, and the node-exporter) to do basic node selection for deployments. See the below example for an example to schedule pods to only nodes with the label node-type
equal to monitoring
:
alertmanager:
alertmanagerSpec:
nodeSelector:
node-type: monitoring
prometheus:
prometheusSpec:
nodeSelector:
node-type: monitoring
prometheusOperator:
nodeSelector:
node-type: monitoring
grafana:
nodeSelector:
node-type: monitoring
kube-state-metrics:
nodeSelector:
node-type: monitoring
prometheus-node-exporter:
nodeSelector:
node-type: monitoring
Values for Affinityπ£
Affinity values for prometheus can be specified at all the same levels. The format to include follows what youβd specify at a pod level. See the examples below to see the correct values for scheduling pods to only nodes with the label node-type
equal to monitoring
:
alertmanager:
alertmanagerSpec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-type
operator: In
values:
- monitoring
prometheus:
prometheusSpec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-type
operator: In
values:
- monitoring
prometheusOperator:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-type
operator: In
values:
- monitoring
grafana:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-type
operator: In
values:
- monitoring
kube-state-metrics:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-type
operator: In
values:
- monitoring
prometheus-node-exporter:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-type
operator: In
values:
- monitoring
Values for Anti-Affinityπ£
Anti-affinity is also configurable for all the same levels. Alertmanager and prometheus provide simple βsoftβ/βhardβ configuration to configure those pods to not deploy on the same nodes as other alertmanager/prometheus pods. The other 4 pieces of monitoring require a manual configuration of anti-affinity (following the upstream pattern). For the βmanualβ anti-affinity examples are provided for keeping pods from scheduling on nodes that have pods labeled dont-schedule-with: <name>
:
alertmanager:
alertmanagerSpec:
podAntiAffinity: "hard"
prometheus:
prometheusSpec:
podAntiAffinity: "hard"
prometheusOperator:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
dont-schedule-with: prometheusOperator
grafana:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
dont-schedule-with: grafana
kube-state-metrics:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
dont-schedule-with: kube-state-metrics
prometheus-node-exporter:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
dont-schedule-with: prometheus-node-exporter