./run.py --log-level=trace TARGET FUNCTION
Salt hangs inexplicably#
If you haven’t previously connected to a server using SSH, then
./run.py will log a
TRACE-level message like:
The authenticity of host 'example.com (126.96.36.199)' can't be established. ECDSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. Are you sure you want to continue connecting (yes/no/[fingerprint])?
You can also experience this issue if anyone changes the hostnames in the
Unless you increase verbosity, you won’t see this message, and
./run.py will appear to hang.
To fix this, simply connect to the server.
Then, re-run the
It is not recommended to use the
--ignore-host-keys) option, as this disables strict host key checking, allowing for man-in-the-middle attacks.
Watch Salt activity#
If you want to check whether a deployment is simply slow or actually stalled, perform these steps:
Find the server’s IP or fully-qualified domain name in the roster:
Open a secondary terminal to connect to the server as the
Watch the processes on the server:
watch -n 1 pstree
Look at the lines below these:
|-sshd-+-sshd---bash---watch | |-sshd---bash---watch---watch---sh---pstree
Then, once the deployment is done:
Stop watching the processes, e.g. with
Disconnect from the server, e.g. with
Restart a service#
Services should restart normally. To manually restart a service, run, for example:
./run.py TARGET service.restart uwsgi
If a new configuration isn’t taking effect, check the service’s status on the server:
systemctl status uwsgi
During deployment, uWSGI reloads rather than restarts. However, deleted environment variables are not unset during reload. To remove a variable from the environment, you must restart uWSGI.
Check git revision#
To check which branch is deployed, run, for example:
./run.py covid19 git.current_branch /home/covid19admin/covid19admin
To check which commit is deployed, run, for example:
./run.py covid19 git.revision /home/covid19admin/covid19admin