Maintain a server

Perform a long-running operation

If an operation will take a long time to run, run it in a terminal multiplexer (tmux), in case you lose your connection to the server. To open a session in tmux, use this command, replacing initials-task-description with your initials and a short description of your task. By including your initials, it is easy for others to determine to whom the session belongs – especially if you forget to close it.

tmux new -s initials-task-description

If you lose your connection to the server, re-attach to your session with:

tmux attach-session -t initials-task-description

To manually detach from a session, press Ctrl-b, release both keys, then press d.

If you forget the name of your session, list all sessions with:

tmux ls

Clean root user directory

  1. Run:

    salt-ssh '*' cmd.run 'ls'
    
  2. Leave any post.install.log files

  3. Delete any index.html* files

    • These are created when a developer runs wget commands to e.g. test proxy settings.

Upgrade packages

All servers use the unattended-upgrades package for security updates.

To show the upgradable packages, run:

salt-ssh '*' pkg.list_upgrades

Auto-remove packages

To show the packages that were automatically installed and are no longer required:

salt-ssh 'ocds-docs-staging' pkg.autoremove list_only=True

To remove these, run:

salt-ssh 'ocds-docs-staging' pkg.autoremove purge=True

To show the packages that were removed but not purged, run:

salt-ssh '*' pkg.list_pkgs removed=True

Upgrade Ubuntu

To determine the current releases, run:

salt-ssh '*' cmd.run 'lsb_release -a'

To check the long term support of the releases, consult the Ubuntu documentation.

Check mail

Find mailboxes with mail across servers:

salt-ssh '*' cmd.run 'find /var/mail -type f -not -size 0'

Connect to a server, for example:

ssh root@process.kingfisher.open-contracting.org

Open the mailbox:

mail -f /var/mail/root

You might see a lot of repeat messages.

Here are common commands:

  • number: open that message
  • h: show a screen of messages
  • z: go to the next screen
  • d 5-10: delete the messages 5 through 10
  • d *: delete all messages
  • q: save changes and exit
  • x: exit without saving changes

In most cases, all messages can be ignored and deleted. Relevant messages might include:

Failed cron jobs
Try to correct the failure
Failed attempts to use sudo
If the attempt is not attributable to a team member, discuss security measures

Check that no messages were saved:

ls ~/mbox