Reference

Communicating during downtime

Please see the protocol for planned and unplanned downtime.

Monitoring

Prometheus

Servers are monitored by Prometheus. Salt is used to:

  • Install a Node Exporter service on each server, to export hardware and OS metrics like disk space used, memory used, etc.
  • Set up a Prometheus server to collect metrics from all servers, and to email alerts if metrics are out of bounds

Read the user guide to learn how to use Prometheus.

DMARC Analyzer

OCP’s DMARC policy (dig TXT _dmarc.open-contracting.org) sends aggregate and forensic reports to DMARC Analyzer.

Sentry

Application errors are reported to Sentry, which notifies individual email addresses. All Salt-managed, OCP-authored services report errors to Sentry.

See the Software Development Handbook for administrative access management.

Hosting

All servers (not services) are managed by Dogsbody Technology (sysadmin@dogsbody.com). Servers are hosted by:

  • Linode for the Helpdesk CRM
    • Network status: The relevant systems are: Regions: EU-West (London), Backups: EU-West (London) Backups.
    • Access: The ‘opencontractingpartnership’ and ‘opencontracting-dogsbody’ users have full access.
    • Backups: It is configured to have one daily backup and two weekly backups. Dogsbody also configured daily and weekly backups to Google Cloud Platform.
  • Hetzner for Kingfisher
  • Bytemark for all others
    • Network status
    • Access: The ‘opendataservices’ and ‘opencon-tech’ users have secondary access to the ‘opencontracting’ account.
    • Backups: It is configured to have one weekly backup (see Create a server).

Unmanaged services are:

Administrative access

The staff of the following organizations can have administrative roles:

The ssh.root lists in Pillar files and the ssh.admin list in the pillar/common.sls file give people access to servers. All people should belong to the above organizations.

Root access

Server owners (OCP) and server managers (Dogsbody) should have root access to all servers. Otherwise, only developers who are reasonably expected to deploy to a server should have root access to that server.

If a developer did not deploy (and was not granted root access) to a server within the last six months, their root access to that server should be revoked.

If a developer intends to deploy to a server, anyone with root access can grant that developer root access to that server.

Root access should be routinely reviewed.

Redash

There should be a minimum of two admin members from OCP, and at most two from CDS and ODS.

Redmine

There should be a minimum of two Administrator roles from OCP, and at most two from CDS and ODS.