Create a server#

A server is created either when a service is moving to a new server, or when a service is being introduced.

As with other deployment tasks, do the setup tasks before (and the cleanup tasks after) the steps below.

1. Create the new server#

Create the server via the host’s interface.

Linode#

  1. Log into Linode

  2. Click Create Linode

    1. Set Images to “Ubuntu 20.04 LTS”

    2. Set Region to London UK

    3. Select a Linode Plan

    4. Set Linode Label to the server’s FQDN (e.g. ocp12.open-contracting.org)

    5. Set Add Tags to either Production or Development

    6. Set Root Password to a strong password

    7. Check Backups

    8. Click Create Linode and wait a few minutes for the server to power on

  3. From the Linodes list:

    1. Click on the label for the new server

    2. Click Power Off and wait for the server to power off

    3. On the Storage tab:

      1. Resize the “Swap Image” disk to the appropriate size

        The swap size should be at most 200% of RAM and:

        • If RAM is less than 2 GB: at least 100% of RAM

        • If RAM is less than 32 GB: at least 50% of RAM

        • Otherwise, at least 16 GB or 25% of RAM, whichever is greater

        Note

        If the swap image is too small, a swap file is configured by Salt.

      2. Rename the “Swap Image” disk to “### MB Swap Image”

      3. Resize the “Ubuntu 20.04 LTS Disk” disk to the desired size (recommended minimum 20 GB / 20480 MB)

    4. On the Configurations tab:

      1. Click Edit for the “My Ubuntu 20.04 LTS Disk Profile” (or similar) config

      2. Uncheck Auto-configure networking

      3. Click Save Changes

    5. Click Power On

    6. Copy SSH Access to your clipboard

  4. Open a support ticket with Linode to assign our IPv6 block to the new server.

    Hello,

    Please assign our IPv6 /64 block (2a01:7e00:e000:02cc::/64) to the server ocp##.open-contracting.org.

    Thank you,

    Note

    Linode can take a day to close the ticket. In the meantime, proceed with the instructions below. Once the ticket is closed, assign a specific address within the /64 block in the network configuration.

  5. If using Docker, configure an external firewall.

Hetzner#

Note

Hetzner dedicated servers are physical servers, and are commissioned to order. Pay attention to any wait times displayed, as some servers may not be available for several days.

  1. Go to Hetzner

  2. Click the Dedicated menu to browser for a suitable server

  3. Check the Server Auction for a comparable server

  4. Click the Order button for the chosen server

    1. Set Server Location (no issues to date with the lowest price option)

    2. Set Operating System to “Ubuntu 18.04 LTS - minimal”

      Note

      If Ubuntu isn’t an option, you will need to Install Ubuntu after these steps. Servers from the Server Auction are delivered in the Hetzner Rescue System.

    3. Set Drives as needed

    4. Click the Order Now button

    5. In the Server Login Details panel, set Type to “Public key” and enter your public SSH key

      Note

      This adds your public SSH key to /root/.ssh/authorized_keys.

    6. Click the Save button

    7. Review the order and click the Checkout button

    8. If prompted, login using OCP’s credentials

    9. Check the “I have read your Terms and Conditions as well as your Privacy Policy and I agree to them.” box

    10. Click the Order in obligation button

  5. Wait to be notified via email that the server is ready.

Install Ubuntu#

If Ubuntu wasn’t an option, follow these steps to install Ubuntu:

  1. Activate and load the Rescue System, if not already loaded.

  2. Connect to the server as the root user using the password provided when activating the Rescue System.

  3. Test the server hardware:

    1. Test the drives. The SMART values to check vary depending on the drive manufacturer. Ask a colleague if you need help.

      smartctl -t long /dev/<device>
      smartctl -a /dev/<device>
      
    2. Test the hardware RAID controller, if there is one. The software to do so varies depending on the RAID controller. Ask a colleague if you need help.

  4. Run the pre-installed Hetzner OS installer (see documentation) and accept the defaults, unless stated otherwise below:

    installimage
    
    1. Select “Ubuntu 18.04 - minimal”

    2. The installer opens a configuration file.

      1. Set DRIVE1, DRIVE2, etc. to the drives you want to use (see documentation). You can identify drives with the smartctl command. If you ordered two large drives for a server that already includes two small drives, you might only set the large drives. For example:

        DRIVE1 /dev/sdb
        DRIVE2 /dev/sdd
        
      2. Set SWRAIDLEVEL 1

      3. Set the hostname (see more under 2. Create DNS records). For example:

        HOSTNAME ocp##.open-contracting.org
        
      4. Create partitions. Set the swap partition size according to the comments in swap.sls. For example:

        PART swap swap 16G
        PART /boot ext2 1G
        PART / ext4 all
        
    3. Press F2 to save

    4. Confirm that you want to overwrite the drives, when prompted

  5. Reboot the server:

    reboot
    
  6. If using Docker, configure an external firewall.

Install Windows#

Reference:

2. Create DNS records#

Hostnames follow the format ocp##.open-contracting.org (ocp01, ocp02, etc.). Increment the number by 1 for each new server, to ensure the hostname is unique and used only once. To determine the greatest number, refer to GoDaddy and the salt-config/roster file.

  1. Login to GoDaddy

  2. If access was delegated, open Delegate Access and click the Access Now button

  3. Open DNS Management for open-contracting.org

  4. Add an A record for the hostname:

    1. Click ADD

    2. Select “A” from the Type dropdown

    3. Enter the hostname in Host (ocp42, for example)

    4. Enter the IPv4 address in Points to

    5. Leave TTL at the 1 Hour default

    6. Click the Save button

  5. If the server has an IPv6 /64 block, add an AAAA record for the hostname:

    1. Click ADD

    2. Select “AAAA” from the Type dropdown

    3. Enter the hostname in Host (ocp42, for example)

    4. Enter the IPv6 address in Points to (use 2 as the last group of digits)

    5. Leave TTL at the 1 Hour default

    6. Click the Save button

3. Apply core changes#

  1. Connect to the server as the root user using SSH, and change its password, using the passwd command. Use a strong password, and save it to OCP’s LastPass account.

    Note

    The root password is needed if you can’t login via SSH (for example, due to a broken configuration). For Bytemark, open the panel, click the server’s Console button, and login.

  2. Add a target to the salt-config/roster file in this repository. Name the target after the service.

    • If the service is moving to a new server, you can use the old target’s name for the new target, and add a -old suffix to the old target’s name.

    • If the service is an instance of CoVE, add a cove- prefix.

    • If the environment is development, add a -dev suffix.

    • Do not include an integer suffix in the target name.

    Note

    If the DNS records have not yet propagated, you can temporarily use the server’s IP address instead of its hostname in the roster.

  3. Configure networking.

  4. Run the onboarding and core state files, which upgrade all packages, configure the hostname and apply the base configuration.

    salt-ssh --log-level=trace TARGET state.apply 'onboarding,core*'
    

    Note

    This step takes 3-4 minutes, so --log-level=trace is used to show activity.

  5. Reboot the server:

    ./run.py TARGET system.reboot
    

Note

The hostname configured in this step and the DNS records created in the previous step are relevant to:

  • verify that an email message has a legitimate source (for example, from cron jobs)

  • communicate between servers (for example, for database replication)

  • identify servers in human-readable way

As such, DNS records that match the hostname must be maintained, until the server is decommissioned.

4. Deploy the service#

  1. If the service is being introduced, add the target to the salt/top.sls and pillar/top.sls files, and include any new state or Pillar files you authored for the service.

  2. If the service is moving to the new server, update occurrences of the old server’s hostname and IP address. (In some cases described in the next step, you’ll need to deploy the related services.)

  3. Deploy the service.

Some IDs might fail (#156):

  • uwsgi, using the service.running function. If so, run:

    ./run.py TARGET service.restart uwsgi
    

5. Migrate from the old server#

  1. Check mail for the root user and, if applicable, each app user

  2. Check the user directory of the root user and, if applicable, each app user

  3. If the server runs a database like PostgreSQL (pg_dump), MySQL (mysqldump) or Elasticsearch, copy the database

  4. If the server runs a web server like Apache or application server like uWSGI, optionally copy the log files

For Django application servers:

  1. Copy the media directory and the db.sqlite3 file from the app’s directory

For OCDS documentation servers:

  1. Copy the /home/ocds-docs/web directory

  2. Update the IP addresses in the pillar/cove.sls file, and deploy the cove-* services

For Redmine servers:

  1. Copy the /home/redmine/public_html/files directory

For Kingfisher servers (instructions are incomplete):

  1. Update the IP addresses in the pillar/tinyproxy.sls file, and deploy the docs service

For Redash servers, see Redash tasks.

6. Update external services#

  1. Add the server to Prometheus

  2. Add (or update) the service’s DNS entries in GoDaddy, for example:

    1. Click ADD

    2. Select “CNAME” from the Type dropdown

    3. Enter the public hostname in Host (standard, for example)

    4. Enter the internal hostname in Points to (ocp42.open-contracting.org, for example)

    5. Leave TTL at the 1 Hour default

    6. Click the Save button

  3. Add (or update) the service’s row in the Health of software products and services spreadsheet

  4. Add (or update) managed passwords, if appropriate

  5. Contact Dogsbody Technology Ltd to set up maintenance (see readme)

  6. Delete the old server

If the service is being introduced:

  1. Add its error monitor to Sentry

  2. Add the embed code for Fathom Analytics, if appropriate

If the service uses a new top-level domain name:

  1. Add the domain to Google Webmaster Central

  2. Add the domain to Google Search Console