Update server configurations

1. Update private templates

If you add, remove or rename a file or variable in pillar/private or salt/private, replicate the changes in pillar/private-templates or salt/private-templates.

This allows others to use this repository to, for example, deploy Kingfisher to their own servers.

2. Test changes

To preview what is going to change, use test=True, for example:

salt-ssh 'ocds-docs-live' state.apply test=True

To preview changes to a Pillar file, run, for example:

salt-ssh 'ocds-docs-live' pillar.items

To compare Jinja2 output after refactoring but before committing, use script/diff to compare a full state or one SLS file, for example:

./script/diff ocds-docs-staging
./script/diff ocds-docs-staging ocds-docs-common

If you get the error, An Exception occurred while executing state.show_highstate: 'list' object has no attribute 'values', run state.apply test=True as above. You might have conflicting IDs.

Using a testing virtual host

To test changes to the Apache files for the OCDS Documentation (for example, to test new redirects or proxy settings):

  1. Make changes inside {% if testing %} blocks in the config files
  2. Deploy the OCDS Documentation
  3. To test manually, visit the testing version of the live website or staging website
  4. To test automatically, run (using the fish shell):
pip install -r requirements.txt
env FQDN=testing.live.standard.open-contracting.org pytest

Update the tests if you changed the behavior of the Apache files.

Once satisfied, move the changes outside {% if testing  %} blocks. After deployment, the tests should pass if FQDN is omitted or set to standard.open-contracting.org.

Using a virtual machine

  1. Create a virtual machine

  2. Get the virtual machine’s IP address

    • If using VirtualBox, run (replacing VMNAME):

      VBoxManage guestproperty get VMNAME "/VirtualBox/GuestInfo/Net/0/V4/IP"
      
  3. Update the relevant target in salt-config/roster to point to the virtual machine’s IP address

  4. In the relevant Pillar file, change https to no, if certbot is used to enable HTTPS

  5. Edit /etc/hosts to map the virtual machine’s IP address to the service’s hostname

  6. Deploy to the virtual machine and test

Note that Python errors that occur on the virtual machine might still be reported to Sentry. The server_name tag in any error reports is expected to be different, but the error reports might still confuse other developers who don’t know to check that tag.

3. Review code

For context, for other repositories, work is done on a branch and tested on a local machine before a pull request is made, which is then tested on Travis CI, reviewed and approved before merging.

However, for this repository, in some cases, it’s impossible to test changes to server configurations, for example: if SSL certificates are involved (because certbot can’t verify a virtual machine), or if external services like Travis are involved. In other cases, it’s too much effort to setup a test environment in which to test changes.

In such cases, the same process is followed as in other repositories, but without the benefit of tests.

In entirely uncontroversial or time-sensitive cases, work is done on the master branch, deployed to servers, and committed to the master branch once successful. In cases where the changes require trial and error, the general approach is discussed in a GitHub issue, and then work is done on the master branch as above. Developers can always request informal reviews from colleagues.

Take extra care when making larger changes or when making changes to higher-priority apps.

Change server name

If the virtual host uses HTTPS, you will need to acquire SSL certificates for the new server name and remove the SSL certificates for the old server name.

  1. Change the ServerName

  2. In the relevant Pillar file, change https to certonly

  3. Deploy the app

  4. In the relevant Pillar file, change https to force or both

  5. Remove the old SSL certificates, for example:

    salt-ssh 'ocds-docs-staging' file.remove /etc/letsencrypt/live/dev.standard.open-contracting.org
    

To check for old SSL certificates that were previously not removed, run:

salt-ssh '*' cmd.run 'ls /etc/letsencrypt/live'

Remove content

If you delete a file, service, package, user, authorized key, Apache module, or virtual host from a file, it will not be removed from the server. To remove it, after you deploy:

Delete an authorized key

  1. Cut it from salt/private/authorized_keys/root_to_add

  2. Paste it into salt/private/authorized_keys/root_to_remove

  3. Run:

    salt-ssh '*' state.sls_id root_authorized_keys_add core
    salt-ssh '*' state.sls_id root_authorized_keys_remove core
    
  4. Delete it from salt/private/authorized_keys/root_to_remove

Delete a file

Run, for example:

salt-ssh 'ocds-docs-staging' file.remove /path/to/file_to_remove

Delete a service

Stop and disable the service.

To stop and disable the icinga2 service on the ocds-docs-staging target, for example:

salt-ssh 'ocds-docs-staging' service.stop icinga2
salt-ssh 'ocds-docs-staging' service.disable icinga2

If you deleted the uwsgi service, also run, for example:

salt-ssh 'cove-live-ocds-2' file.remove /etc/uwsgi/apps-available/cove.ini
salt-ssh 'cove-live-ocds-2' file.remove /etc/uwsgi/apps-enabled/cove.ini

Delete a package

Remove a package and its configuration files, and remove any of its dependencies that are no longer needed.

To scrub Icinga-related packages from the ocds-docs-staging target, for example:

salt-ssh 'ocds-docs-staging' pkg.purge icinga2,nagios-plugins,nagios-plugins-contrib
salt-ssh 'ocds-docs-staging' pkg.autoremove list_only=True
salt-ssh 'ocds-docs-staging' pkg.autoremove purge=True

Then, login to the server and check for and delete any remaining packages, files or directories relating to the package, for example:

dpkg -l | grep icinga
dpkg -l | grep nagios
ls /etc/icinga2
ls /usr/lib/nagios

Delete an Apache module

  1. Add a temporary Salt ID, for example:

    headers:
        apache_module.disabled
    
  2. Deploy the relevant target, for example:

    salt-ssh 'toucan' state.apply
    
  3. Remove the temporary salt ID

Delete a virtual host

Run, for example:

salt-ssh 'cove-ocds-live-2' file.remove /etc/apache2/sites-enabled/cove.conf
salt-ssh 'cove-ocds-live-2' file.remove /etc/apache2/sites-available/cove.conf
salt-ssh 'cove-ocds-live-2' file.remove /etc/apache2/sites-available/cove.conf.include

You might also delete the SSL certificates as when changing server name.

Troubleshoot

Gotchas in Pillar files:

  • If unquoted, yes, no, true and false are parsed as booleans. Use quotes to parse as strings.

  • A blank value is parsed as None. Use the empty string '' to parse as a string.

  • Below, if a is equal to an empty string, then b will be None:

    {% set extracontext %}
    b: {{ a }}
    {% endset %}
    

    Instead, surround it in quotes:

    {% set extracontext %}
    b: "{{ a }}"
    {% endset %}
    

Check history

The files in this repository were originally in the opendataservices-deploy repository. You can browse that repository from before the change (August 5, 2019). That repository was itself re-organized at different times. You can browse before moving content from *.conf to *.conf.include (June 5, 2019).

Track upstream

The files in this repository were originally in the opendataservices-deploy repository. Some common files might have improvements in the original repository. To check for updates, run:

git clone git@github.com:OpenDataServices/opendataservices-deploy.git
cd opendataservices-deploy
git log --name-status setup_for_non_root.sh updateToMaster.sh Saltfile pillar/common_pillar.sls salt-config/master salt/apache.sls salt/apache/000-default.conf salt/apache/000-default.conf.include salt/apache/_common.conf salt/apache/cove.conf salt/apache/cove.conf.include salt/apache/prometheus-client.conf salt/apache/prometheus-client.conf.include salt/apache/robots_dev.txt salt/apt/10periodic salt/apt/50unattended-upgrades salt/core.sls salt/cove.sls salt/letsencrypt.sls salt/lib.sls salt/nginx/redash salt/prometheus-client-apache.sls salt/prometheus-client/prometheus-node-exporter.service salt/system/ocdskingfisher_motd salt/uwsgi.sls salt/uwsgi/cove.ini
  • setup_for_non_root.sh corresponds to script/setup
  • updateToMaster.sh corresponds to script/update
  • salt-config/roster, pillar/top.sls and salt/top.sls are common files, but are unlikely to contain improvements

This repository has all improvements up to September 30, 2019.