Prometheus tasks#

Monitor a service#

If the prometheus.node_exporter state file applies to the target, then Node Exporter is served on port 7231 using the HTTPS scheme and a self-signed certificate. Only connections from the Prometheus server are allowed.

  1. Add a job to salt/prometheus/files/conf-prometheus.yml.

  2. Deploy the Prometheus service.

  3. Check that the job is “UP” on Prometheus’ Targets page.

Note

To test the Node Exporter endpoint from the Prometheus server, replace SUBDOMAIN with the target’s subdomain, and PASSWORD with the URL-encoded value of the prometheus.node_exporter.password variable in the pillar/private/prometheus_client.sls file:

cd ~prometheus-client
curl -v --cacert node_exporter.pem https://prom:PASSWORD@SUBDOMAIN.open-contracting.org:7231/metrics

Test Alert Manager#

SSH into alertmanager.prometheus.open-contracting.org as the root user, and run:

curl --json '[{"labels":{"alertname":"TestAlert"}}]' localhost:9095/api/v1/alerts

Reference: StackOverflow

Upgrade Prometheus#

We set the version numbers of the Prometheus software in the pillar/prometheus_client.sls and pillar/prometheus_server.sls files:

  • prometheus.prometheus.version

  • prometheus.alertmanager.version

  • prometheus.node_exporter.version

Our practice is to upgrade annually to an LTS version. We can upgrade sooner if there is a release with a bugfix or feature that we want.

Setup#

  1. Access the changelogs for the Server, Alert Manager and Node Exporter (they follow semantic versioning).

  2. Check whether any breaking changes are relevant to us. In particular, check whether newer versions work with data from older versions (the server and alert manager store data on disk).

Deploy#

Once you’re ready to upgrade, as with other deployment tasks, do the setup tasks before (and the cleanup tasks after) the steps below.

  1. Change the version numbers in the pillar/prometheus_client.sls and pillar/prometheus_server.sls files. (To test locally, you can use to a virtual machine.)

  2. If you’re upgrading the server and/or alert manager, deploy the prometheus target.

  3. If you’re upgrading the node exporter, deploy all targets.