Create a server

A server is created either when a service is moving to a new server, or when a service is being introduced.

As with other deployment tasks, do the setup tasks before (and the cleanup tasks after) the steps below.

1. Create the new server

Create the server via the host’s interface.

Bytemark

  1. Login to Bytemark

  2. Click Servers and Add a cloud server

    1. Enter a Name

    2. Select a Group indicating the environment (production or development)

    3. Set Location to “York”

    4. Set Server Resources

      Note

      If you drag the slider to the minimum, the disc size will be changed to 5 GiB, which is too little.

    5. Set Operating system to “Ubuntu 18.04 (LTS)”

    6. Check Enable backups

    7. Set Take a backup every to 7 days

    8. Set Starting on to the following Thursday at a random time before 10:00 UTC

    9. Set Root user has to “SSH key (+ Password)” and enter your public SSH key

      Note

      This adds your public key to /root/.ssh/authorized_keys.

    10. Click Add this server

  3. Wait for the server to boot (a few minutes)

  4. Click Info and copy the Hostname/SSH

Note

This step creates a /root/post.install.log file, which logs the packages installed in addition to the operating system.

Hetzner

Note

Hetzner dedicated servers are physical servers, and are commissioned to order. Pay attention to any wait times displayed, as some servers may not be available for several days.

  1. Go to Hetzner

  2. Click the Dedicated menu to browser for a suitable server

  3. Click the Order button for the chosen server

    1. Set Server Location (no issues to date with the lowest price option)

    2. Set Operating System to “Ubuntu 18.04 LTS - minimal”.

      Note

      If Ubuntu isn’t an option, you will need to Install Ubuntu after these steps.

    3. Set Drives as needed

    4. Click the Order Now button

    5. In the Server Login Details panel, set Type to “Public key” and enter your public SSH key

      Note

      This adds your public key to /root/.ssh/authorized_keys.

    6. Click the Save button

    7. Review the order and click the Checkout button

    8. If prompted, login using OCP’s credentials

    9. Check the “I have read your Terms and Conditions as well as your Privacy Policy and I agree to them.” box

    10. Click the Order in obligation button

  4. Wait to be notified via email that the server is ready.

Install Ubuntu

If Ubuntu wasn’t an option, follow these steps to install Ubuntu:

  1. Activate and load the Rescue System, if not already loaded.

  2. Connect to the server as the root user using the password provided when activating the Rescue System.

  3. Test the server hardware:

    1. Test the drives. The SMART values to check vary depending on the drive manufacturer. Ask a colleague if you need help.

      smartctl -t long /dev/<device>
      smartctl -a /dev/<device>
      
    2. Test the hardware RAID controller, if there is one. The software to do so varies depending on the RAID controller. Ask a colleague if you need help.

  4. Run the pre-installed Hetzner OS installer (see documentation) and accept the defaults, unless stated otherwise below:

    installimage
    
    1. Select “Ubuntu 18.04 - minimal”

    2. The installer opens a configuration file.

      1. Set DRIVE1, DRIVE2, etc. to the drives you want to use (see documentation). You can identify drives with the smartctl command. If you ordered two large drives for a server that already includes two small drives, you might only set the large drives. For example:

        DRIVE1 /dev/sdb
        DRIVE2 /dev/sdd
        
      2. Set SWRAIDLEVEL 1

      3. Set the hostname (see more under 2. Create DNS records). For example:

        HOSTNAME ocp##.open-contracting.org
        
      4. Create partitions. Set the swap partition size according to the comments in swap.sls. For example:

        PART swap swap 16G
        PART /boot ext2 1G
        PART / ext4 all
        
    3. Press F2 to save

    4. Confirm that you want to overwrite the drives, when prompted

  5. Reboot the server:

    reboot
    

2. Create DNS records

Hostnames follow the format ocp##.open-contracting.org (ocp01, ocp02, etc.). Increment the number by 1 for each new server, to ensure the hostname is unique and used only once. To determine the greatest number, refer to GoDaddy and the salt-config/roster file.

  1. Login to GoDaddy
  2. If access was delegated, open Delegate Access and click the Access Now button
  3. Open DNS Management for open-contracting.org
  4. Add an A record for the hostname:
    1. Click ADD
    2. Select “A” from the Type dropdown
    3. Enter the hostname in Host (ocp42, for example)
    4. Enter the IPv4 address in Points to
    5. Leave TTL at the 1 Hour default
    6. Click the Save button
  5. If the server has an IPv6 address, add an AAAA record for the hostname:
    1. Click ADD
    2. Select “AAAA” from the Type dropdown
    3. Enter the hostname in Host (ocp42, for example)
    4. Enter the IPv6 address in Points to
    5. Leave TTL at the 1 Hour default
    6. Click the Save button

3. Apply core changes

  1. Connect to the server as the root user using SSH, and change its password, using the passwd command. Use a strong password, and save it to OCP’s LastPass account.

    Note

    The root password is needed if you can’t login via SSH (for example, due to a broken configuration). For Bytemark, open the panel, click the server’s Console button, and login.

  2. Add a target to the salt-config/roster file in this repository. Name the target after the service.

    • If the service is moving to a new server, you can use the old target’s name for the new target, and add a -old suffix to the old target’s name.
    • If the service is an instance of CoVE, add a cove- prefix.
    • If the environment is development, add a -dev suffix.
    • Do not include an integer suffix in the target name.

    Note

    If the DNS records have not yet propagated, you can temporarily use the server’s IP address instead of its hostname in the roster.

  3. Run the onboarding and core state files, which upgrade all packages, configure the hostname, and apply the base configuration. Replace TARGET and ocpXX:

    salt-ssh --log-level=trace TARGET state.apply 'onboarding,core*' pillar='{"host_id": "ocpXX"}'
    

    Note

    This step takes 3-4 minutes, so --log-level=trace is used to show activity.

  4. Reboot the server:

    ./run.py TARGET system.reboot
    

Note

The hostname configured in this step and the DNS records created in the previous step are relevant to:

  • verify that an email message has a legitimate source (for example, from cron jobs)
  • communicate between servers (for example, for database replication)
  • identify servers in human-readable way

As such, DNS records that match the hostname must be maintained, until the server is decommissioned.

4. Deploy the service

  1. If the service is being introduced, add the target to the salt/top.sls and pillar/top.sls files, and include any new state or Pillar files you authored for the service.
  2. If the service is moving to the new server, update occurrences of the old server’s hostname and IP address. (In some cases described in the next step, you’ll need to deploy the related services.)
  3. Deploy the service.

Some IDs might fail (#156):

  • uwsgi, using the service.running function. If so, run:

    ./run.py TARGET service.restart uwsgi
    

5. Migrate from the old server

  1. Check mail for the root user
  2. Check the user directory of the root user

For Django application servers:

  1. Copy the media directory and the db.sqlite3 file from the app’s directory
  2. Check mail for the app user
  3. Check the user directory of the app user
  4. Optionally, copy the Apache and uWSGI log files

For OCDS documentation servers:

  1. Copy the /home/ocds-docs/web directory
  2. Update the IP addresses in the pillar/cove.sls file, and deploy the cove-* services
  3. Optionally, copy the Apache log files

For Kingfisher servers (instructions are incomplete):

  1. Update the IP addresses in the pillar/tinyproxy.sls file, and deploy the docs service

For Redash servers, see Redash tasks.

If the server runs a database like PostgreSQL or Elasticsearch, copy the database.

6. Update external services

  1. Add the server to Prometheus
  2. Add (or update) the service’s DNS entries in GoDaddy, for example:
    1. Click ADD
    2. Select “CNAME” from the Type dropdown
    3. Enter the public hostname in Host (standard, for example)
    4. Enter the internal hostname in Points to (ocp42.open-contracting.org, for example)
    5. Leave TTL at the 1 Hour default
    6. Click the Save button
  3. Add (or update) the service’s row in the Health of software products and services spreadsheet
  4. Add (or update) managed passwords, if appropriate
  5. Contact Dogsbody Technology Ltd to set up maintenance (see readme)
  6. Delete the old server

If the service is being introduced:

  1. Add its error monitor to Sentry
  2. Add the embed code for Fathom Analytics, if appropriate

If the service uses a new domain name:

  1. Add the domain to Google Webmaster Central
  2. Add the domain to Google Search Console