Alertmanager💣
Alertmanager Overview💣
Alertmanager is the frontend used for sending and managing alerts for a prometheus-stack installation. Out of the box this package installs a single replica of both prometheus and alertmanager which are pre-configured to sync. Alerts are evaluated and generated within Prometheus and then posted to the REST API for Alertmanager to manage based on it’s configuration.
Upstream documentation can be rather sparse about exactly what’s required to configure Alertmanager and what’s recommended. This doc will go over setting up SMTP & webhook alerts destined for email inboxes and Mattermost (or slack) installations.
SMTP Alert Configuration💣
It is recommended to keep the default ‘null’ receiver as that is where you can route non-important alerts like the built in Watchdog alert which only exists to show the end-user that the communication between Prometheus & Alertmanager is working as expected. Here is a link to the full list of options available for the email_config section within alertmanager.
alertmanager:
config:
global:
resolve_timeout: 5m
route:
# Keeping the default of 'job' but 'alertname' is also used often
group_by: ['namespace']
group_wait: 30s
group_interval: 5m
# repeat_interval upped from default of 12h
repeat_interval: 24h
# receiver updated from 'null' to 'smtp-email' so it becomes the default
receiver: 'smtp-email'
routes:
- matchers:
alertname: "InfoInhibitor|Watchdog"
receiver: 'null'
receivers:
- name: 'null'
# Adding a new receiver below the default 'null' receiver
- name: 'smtp-email'
email_configs:
- to: `it@example.org, ops.example.org`
from: no-reply@example.org
smarthost: smtp.office365.org:587
auth_username:
auth_identity:
auth_password:
require_tls: true
send_resolved: true
templates:
- '/etc/alertmanager/config/*.tmpl'
Chat Webhook Alert Configuration💣
Alertmanager contains built in support for a few chat apps: slack, telegram & wechat. The included slack configuration can be used to send to Mattermost Incoming Webhooks within an installation. Just like the example above we’ll want to keep the existing receiver and route for the ‘null’ receiver so that the built in Watchdog alert isn’t sent to your new receiver. Here is a link to the full list of options available for the slack_config section within alertmanager, however to send to a mattermost incoming webhook, only the value show below is needed.
alertmanager:
config:
global:
resolve_timeout: 5m
route:
# Keeping the default of 'job' but 'alertname' is also used often
group_by: ["job"]
group_wait: 30s
group_interval: 5m
# repeat_interval upped from default of 12h
repeat_interval: 24h
# receiver updated from 'null' to 'mattermost' so it becomes the default
receiver: "mattermost"
routes:
- match:
alertname: Watchdog
receiver: "null"
receivers:
- name: "null"
# Adding a new receiver below the default 'null' receiver
- name: "mattermost"
slack_configs:
- api_url: https://chat.domain.url/hooks/AAAAAAAAAAAAAAA
templates:
- "/etc/alertmanager/config/*.tmpl"