Troubleshoot¶

Increase verbosity¶

./run.py --log-level=trace TARGET FUNCTION

Salt hangs inexplicably¶

If you haven’t previously connected to a server using SSH, then ./run.py will log a TRACE-level message like:

The authenticity of host 'example.com (101.2.3.4)' can't be established.
ECDSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

You can also experience this issue if anyone changes the hostnames in the salt-config/roster file.

Unless you increase verbosity, you won’t see this message, and ./run.py will appear to hang.

To fix this, simply connect to the server.

Then, re-run the ./run.py command.

Note

It is not recommended to use the -i (--ignore-host-keys) option, as this disables strict host key checking, allowing for man-in-the-middle attacks.

Watch Salt activity¶

If you want to check whether a deployment is simply slow or actually stalled, perform these steps:

Find the server’s IP or fully-qualified domain name in the roster:
```
cat salt-config/roster
```
Open a secondary terminal to connect to the server as the root user.
Watch the processes on the server:
```
watch -n 1 pstree
```
Look at the lines below these:

|-sshd-+-sshd---bash---watch
|      |-sshd---bash---watch---watch---sh---pstree

Then, once the deployment is done:

Stop watching the processes, e.g. with Ctrl-C.
Disconnect from the server, e.g. with Ctrl-D.

Restart a service¶

Services should restart normally. To manually restart a service, run, for example:

./run.py TARGET service.restart uwsgi

If a new configuration isn’t taking effect, check the service’s status on the server:

systemctl status uwsgi

Note

During deployment, uWSGI reloads rather than restarts. However, deleted environment variables are not unset during reload. To remove a variable from the environment, you must restart uWSGI.

Check git revision¶

To check which branch is deployed, run, for example:

./run.py covid19 git.current_branch /home/covid19admin/covid19admin

To check which commit is deployed, run, for example:

./run.py covid19 git.revision /home/covid19admin/covid19admin