Troubleshoot

Increase verbosity

./run.py --log-level=trace TARGET FUNCTION

Salt hangs inexplicably

If you haven’t previously connected to a server using SSH, then ./run.py will log a TRACE-level message like:

The authenticity of host 'example.com (101.2.3.4)' can't be established.
ECDSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

You can also experience this issue if anyone changes the hostnames in the salt-config/roster file.

Unless you increase verbosity, you won’t see this message, and ./run.py will appear to hang.

To fix this, simply connect to the server using SSH, for example:

curl --silent --connect-timeout 1 standard.open-contracting.org:8255 || true
ssh root@standard.open-contracting.org

Then, re-run the ./run.py command.

Note

It is not recommended to use the -i (--ignore-host-keys) option, as this disables strict host key checking, allowing for man-in-the-middle attacks.

Watch Salt activity

If you want to check whether a deployment is simply slow or actually stalled, perform these steps:

  1. Find the server’s IP or fully-qualified domain name in the roster:

    cat salt-config/roster
    
  2. Open a secondary terminal to connect to the server as root, for example:

    curl --silent --connect-timeout 1 standard.open-contracting.org:8255 || true
    ssh root@standard.open-contracting.org
    
  3. Watch the processes on the server:

    watch -n 1 pstree
    
  4. Look at the lines below these:

|-sshd-+-sshd---bash---watch
|      |-sshd---bash---watch---watch---sh---pstree

Then, once the deployment is done:

  1. Stop watching the processes, e.g. with Ctrl-C
  2. Disconnect from the server, e.g. with Ctrl-D

Restart a service

Services should restart normally. To manually restart a service, run, for example:

./run.py TARGET service.restart uwsgi

Check git revision

To check which branch is deployed, run, for example:

./run.py covid19 git.current_branch /home/covid19admin/covid19admin

To check which commit is deployed, run, for example:

./run.py covid19 git.revision /home/covid19admin/covid19admin