Reference

Communicating during downtime

Please see the protocol for planned and unplanned downtime.

Monitoring

Prometheus

Servers are monitored by Prometheus. Salt is used to configure Prometheus monitoring on each server, and to set up a Prometheus server to to collect metrics from these servers. Node Exporter is installed on each server to export hardware and OS metrics like disk space used, memory used, etc.

Read the user guide to learn how to use Prometheus.

DMARC Analyzer

OCP’s DMARC policy (dig TXT _dmarc.open-contracting.org) sends aggregate and forensic reports to DMARC Analyzer.

Sentry

Application errors are reported to Sentry, which notifies individual email addresses. All Salt-managed, OCP-authored services report errors to Sentry.

There should be at most one member with the Admin role from each of ODS and CDS, and at least one member with the Owner role from OCP.

Hosting

All servers (not services) are managed by Dogsbody Technology (sysadmin@dogsbody.com). Servers are hosted by:

  • Linode for the Helpdesk CRM
    • Network status: The relevant systems are: Regions: EU-West (London), Backups: EU-West (London) Backups.
    • Access: The ‘opencontractingpartnership’ and ‘opencontracting-dogsbody’ users have full access. The ‘opencontracting’ user has limited access.
    • Backups: It is configured to have one daily backup and two weekly backups. Dogsbody also configured daily and weekly backups to Google Cloud Platform.
  • Hetzner for Kingfisher
  • Bytemark for all others
    • Network status
    • Access: The ‘opendataservices’ and ‘opencon-tech’ users have secondary access to the ‘opencontracting’ account.
    • Backups: It is configured to have one weekly backup (see Create a server).
  • GitHub Pages for the Extension Explorer

Administrative access

The staff of the following organizations can have administrative roles:

The files referenced by ssh_auth.present states give people access to servers. All people should belong to the above organizations.

Root access

Server owners (OCP) and server managers (Dogsbody) should have root access to all servers. Otherwise, only developers who are reasonably expected to deploy to a server should have root access to that server.

If a developer did not deploy (and was not granted root access) to a server within the last six months, their root access to that server should be revoked.

If a developer intends to deploy to a server, anyone with root access can grant that developer root access to that server.

Root access should be routinely reviewed.

Redash

There should be at most two users in the admin group from each of OCP and ODS.

Redmine

There should be at most two users with the Administrator role from each of OCP and ODS.