Troubleshoot downtime¶

This page describes diagnostics and solutions.

Inaccessible server¶

If you can’t SSH into the server:

Have you port knocked? (if the server isn’t using Docker)
Is the external firewall blocking you?
Is the server’s firewall blocking you?
Does any website or endpoint respond?
- This confirms whether the issue affects the whole server or is limited to one service.
How recent is the Grafana data?
- This confirms whether outbound networking is working.
Is resource usage high in Grafana?
- Maxed CPU or memory usage can affect the performance of the SSH service.

Solution: Force system reboot

Linode

Log into Linode
Click the server label
Click the … > Reboot menu item

Hetzner

Log into Hetzner Robot
Click the Server left-hand menu item
Click the Server ID
Click the Reset tab
Power off the server
1. Check Press power button of server
2. Click the Send button
After a minute, if the server hasn’t responded to the button press:
1. Check Long power button press
2. Click the Send button
Wait until the Current status is “Powered off”
Power on the server
1. Check Press power button of server
2. Click the Send button

System resources¶

Solutions to high resource utilization vary, from application updates to configuration changes.

Dashboard¶

See Prometheus. You typically:

Select the server
Narrow the time to the outage
Check system resources over a longer timeframe, to compare to normal utilization

CPU¶

View processes sorted by CPU utilization:
```
top
```

Memory¶

View processes sorted by memory utilization:
```
top # then press M
```
View total memory and used memory:
```
free -h
```

Disk space¶

View total disk space:
```
df -h
```
View disk usage of a directory and its contents:
```
du -h --max-depth=1 /path/to/directory
```

System services¶

List failed services:
```
systemctl --failed
```
Tip

Run this after a reboot, to double-check all services have started successfully.
Get one service’s status, for example:
```
systemctl status apache2.service
```

Get one service’s log, for example:

journalctl -u apache2.service
journalctl -u apache2.service --since "2 days ago"

Read logs from /var/log directly, notably:
- System log: /var/log/syslog
- Apache: /var/log/apache2
- Nginx: /var/log/nginx
- Docker containers: /var/log/docker-custom

Solution: Restart the service

For example:

systemctl restart apache2.service

If it still fails, see: Configuration changes

Docker applications¶

List all containers:
```
docker ps -a
```

List all containers for a Docker Compose file, for example:

sudo -u deployer docker compose -f /data/deploy/cove-ocds/docker-compose.yaml ps -a

Read container logs in /var/log/docker-custom/

Solution: Restart the containers

If newer images fix the diagnosed issue, pull all images:

sudo -u deployer docker compose -f /data/deploy/cove-ocds/docker-compose.yaml pull

If a service was removed, the network was reconfigured, a bind-mounted configuration file changed, or things are otherwise in a bad state, stop and remove containers and networks:

sudo -u deployer docker compose -f /data/deploy/cove-ocds/docker-compose.yaml down

Then, restart all containers for a Docker Compose file:

sudo -u deployer docker compose -f /data/deploy/cove-ocds/docker-compose.yaml up -d

Configuration changes¶

Changes can cause downtime. Two common examples:

A typo in an updated configuration file prevents the service from starting
An updated service fails to start due to now-obsolete or missing configuration options

To diagnose:

Read recent commits to the deploy repository
Read recent software patches and changes:
```
less /var/log/apt/history.log
```
List logged-in users, who might have made changes directly:
```
w
```
List recent users:
```
last
```
List recent commands:
```
history
```
Note

If a user is currently logged in, their history will not be written yet.

Solution: Revert the changes

If a solution is unknown, or is taking too long, revert to the previous working version.

List available versions and install a specific version, for example:
```
apt-cache policy apache2
apt-get install apache2:1.2.3
```

DOS attacks¶

Denial-of-service (DOS) attacks can cause downtime.

Heavy traffic is not necessarily a DOS attack; web crawlers can inadvertently overwhelm a server. Our response depends on the legitimacy of the traffic and its impact.

You can identify a DOS attack by reading the logs and seeing a single IP address send many requests at high frequency.

Attention

Most web traffic is proxied through Cloudflare. Before blocking an IP, check that it’s not a Cloudflare IP.

Solution: Block an IP address in iptables

Note

iptables is disabled on servers running Docker.

Edit /home/sysadmin-tools/firewall-settings.local, for example:

DENYALL_IPV4="192.0.2.1 192.0.2.4"
DENYALL_IPV6="2001:db8::/64"

Update the iptables rules:
```
/home/sysadmin-tools/bin/firewall.sh
```

Solution: Block an IP address in Cloudflare

Block an IP address from accessing any server on the entire domain, for example:

Log into Cloudflare
Select the “Open Contracting” account
Click the Domains > Overview menu item
Click the domain (e.g. “open-contracting.org”)
Click the Security > Security rules menu item
Click the Create rule button in the Custom rules panel
1. Rule name: Block 192.0.2.1
2. When incoming requests match…
  1. Field: IP Source Address
  2. Value: 192.0.2.1
3. Then take action…: Block
4. Click the Deploy button

DDOS attacks¶

A distributed denial-of-service (DDOS) attack is harder to mitigate, because traffic originates from multiple IP addresses.

Solution: Cloudflare Under Attack mode

This affects the entire domain.

Log into Cloudflare
Select the “Open Contracting” account
Click the Domains > Overview menu item
Click the domain (e.g. “open-contracting.org”)
Toggle Under Attack Mode under Quick Actions

Service provider incidents¶

Check the service provider’s status page

Linode physical host outage

If an issue is limited to one physical host (e.g. hardware issue), the status page will not update. Instead, Linode will contact us via a support ticket that explains how Linode is responding and sets expectations for resolution time.

Solution: Open a support ticket

Our alerts often fire before the service provider updates its status page. If it nonetheless seems like a service provider issue, open an urgent support ticket with the service provider.

Solution: Rebuild the server

In the rare event of a total loss, create a new server and recover from backups:

Create a new server and hostname, using the same OS version as the previous server
Update any configurations in the deploy repository that use the old hostname or IP address
Recover from backups
Preview and test the services on the server, by updating your /etc/hosts file
Update DNS records