Prometheus

Monitor a service

If the prometheus.node_exporter state file applies to the target, then Node Exporter is served on port 7231 using the HTTPS scheme and a self-signed certificate. Only connections from the Prometheus server are allowed.

  1. Add a job to salt/prometheus/files/conf-prometheus.yml.

  2. Deploy the Prometheus service.

  3. Check that the job is “UP” on Prometheus’ Targets page.

Note

To test the Node Exporter endpoint from the Prometheus server, replace SUBDOMAIN with the target’s subdomain, and PASSWORD with the URL-encoded value of the prometheus.node_exporter.password variable in the pillar/private/prometheus_client.sls file:

cd ~prometheus-client
curl -v --cacert node_exporter.pem https://prom:PASSWORD@SUBDOMAIN.open-contracting.org:7231/metrics

Test Alert Manager

SSH into alertmanager.prometheus.open-contracting.org as the root user, and run:

curl --json '[{"labels":{"alertname":"TestAlert"}}]' localhost:9095/api/v1/alerts

Reference: GitHub issue

Upgrade Prometheus

We set the version numbers of the Prometheus software in the pillar/prometheus_client.sls and pillar/prometheus_server.sls files:

  • prometheus.prometheus.version

  • prometheus.alertmanager.version

  • prometheus.node_exporter.version

Our practice is to upgrade annually to an LTS version. We can upgrade sooner if there is a release with a bugfix or feature that we want.

Setup

  1. Access the changelogs for the Server, Alert Manager and Node Exporter (they follow semantic versioning).

  2. Check whether any breaking changes are relevant to us. In particular, check whether newer versions work with data from older versions (the server and alert manager store data on disk).

Deploy

Once you’re ready to upgrade, as with other deployment tasks, do the setup tasks before the steps below.

  1. Change the version numbers in the pillar/prometheus_client.sls and pillar/prometheus_server.sls files. (To test locally, you can use to a virtual machine.)

  2. If you’re upgrading the server and/or alert manager, deploy the prometheus target.

  3. If you’re upgrading the node exporter, deploy all targets.

Migrate from an old server

  1. Stop Prometheus, replace the /home/prometheus-server/data/ directory, and start Prometheus. For example:

    systemctl stop prometheus-server
rm -rf /home/prometheus-server/data/*
rsync -avz ocp99:/home/prometheus-server/data/ /home/prometheus-server/data/
systemctl start prometheus-server

  2. Update the IP addresses in the pillar/prometheus_client.sls file, and deploy to all services