Create a server#

A server is created either when a service is moving to a new server, or when a service is being introduced.

As with other deployment tasks, do the setup tasks before (and the cleanup tasks after) the steps below.

1. Create the new server#

Create the server via the host’s interface.

Linode#

  1. Log into Linode

  2. Click Create Linode

    1. Set Images to the latest Ubuntu LTS version

    2. Set Region to London UK

    3. Select a Linode Plan

    4. Set Linode Label to the server’s FQDN (e.g. ocp12.open-contracting.org)

    5. Set Add Tags to either Production or Development

    6. Set Root Password to a strong password

    7. Check Backups

    8. Click Create Linode and wait a few minutes for the server to power on

  3. From the Linodes list:

    1. Click on the label for the new server

    2. Click Power Off and wait for the server to power off

    3. On the Storage tab:

      1. Resize the “Swap Image” disk to the appropriate size

        The swap size should be at most 200% of RAM and:

        • If RAM is less than 2 GB: at least 100% of RAM

        • If RAM is less than 32 GB: at least 50% of RAM

        • Otherwise, at least 16 GB or 25% of RAM, whichever is greater

        Note

        If the swap image is too small, a swap file is configured by Salt.

      2. Rename the “Swap Image” disk to “### MB Swap Image”

      3. Resize the “Ubuntu ##.04 LTS Disk” disk to the desired size (recommended minimum 20 GB / 20480 MB)

    4. On the Configurations tab:

      1. Click Edit for the “My Ubuntu ##.04 LTS Disk Profile” (or similar) configuration

      2. Uncheck Auto-configure networking

      3. Click Save Changes

    5. Click Power On

    6. Copy SSH Access to your clipboard

  4. Open a support ticket with Linode to assign an IPv6 /64 block to the new server.

    Hello,

    Please assign an IPv6 /64 block to the server ocp##.open-contracting.org.

    Thank you,

    Note

    Linode can take a day to close the ticket. In the meantime, proceed with the instructions below. Once the ticket is closed, assign a specific address within the /64 block in the network configuration.

  5. If using Docker, configure an external firewall.

Hetzner#

Note

Hetzner dedicated servers are physical servers, and are commissioned to order. Pay attention to any wait times displayed, as some servers may not be available for several days.

  1. Go to Hetzner

  2. Click the Dedicated menu to browser for a suitable server

  3. Check the Server Auction for a comparable server

  4. Click the Order button for the chosen server

    1. Set Server Location (no issues to date with the lowest price option)

    2. Set Operating System to the latest Ubuntu LTS version

      Note

      If Ubuntu isn’t an option, you will need to Install Ubuntu after these steps. Servers from the Server Auction are delivered in the Hetzner Rescue System.

    3. Set Drives as needed

    4. Click the Order Now button

    5. In the Server Login Details panel, set Type to “Public key” and enter your public SSH key

      Note

      This adds your public SSH key to /root/.ssh/authorized_keys.

    6. Click the Save button

    7. Review the order and click the Checkout button

    8. If prompted, login using OCP’s credentials

    9. Check the “I have read your Terms and Conditions as well as your Privacy Policy and I agree to them.” box

    10. Click the Order in obligation button

  5. Wait to be notified via email that the server is ready.

Install Ubuntu#

If Ubuntu wasn’t an option, follow these steps to install Ubuntu:

  1. Activate and load the Rescue System, if not already loaded.

  2. Connect to the server as the root user using the password provided when activating the Rescue System.

  3. Test the server hardware:

    1. Test the drives. The SMART values to check vary depending on the drive manufacturer. Ask a colleague if you need help.

      smartctl -t long /dev/<device>
      smartctl -a /dev/<device>
      
    2. Test the hardware RAID controller, if there is one. The software to do so varies depending on the RAID controller. Ask a colleague if you need help.

  4. Run the pre-installed Hetzner OS installer (see documentation) and accept the defaults, unless stated otherwise below:

    installimage
    
    1. Select the latest Ubuntu LTS version.

    2. The installer opens a configuration file.

      1. Set DRIVE1, DRIVE2, etc. to the drives you want to use (see documentation). You can identify drives with the smartctl command. If you ordered two large drives for a server that already includes two small drives, you might only set the large drives. For example:

        DRIVE1 /dev/sdb
        DRIVE2 /dev/sdd
        
      2. Set SWRAIDLEVEL 1

      3. Set the hostname (see more under 2. Create DNS records). For example:

        HOSTNAME ocp##.open-contracting.org
        
      4. Create partitions. Set the swap partition size according to the comments in swap.sls. For example:

        PART swap swap 16G
        PART /boot ext2 1G
        PART / ext4 all
        
    3. Press F2 to save

    4. Confirm that you want to overwrite the drives, when prompted

  5. Reboot the server:

    reboot
    
  6. If using Docker, configure an external firewall.

Install Windows#

Reference:

2. Create DNS records#

Hostnames follow the format ocp##.open-contracting.org (ocp01, ocp02, etc.). Increment the number by 1 for each new server, to ensure the hostname is unique and used only once. To determine the greatest number, refer to GoDaddy and the salt-config/roster file.

  1. Login to GoDaddy

  2. If access was delegated, open Delegate Access and click the Access Now button

  3. Open DNS Management for open-contracting.org

  4. Add an A record for the hostname:

    1. Click ADD

    2. Select “A” from the Type dropdown

    3. Enter the hostname in Host (ocp42, for example)

    4. Enter the IPv4 address in Points to

    5. Set TTL to 1 Day

    6. Click the Save button

  5. If the server has an IPv6 /64 block, add an AAAA record for the hostname:

    1. Click ADD

    2. Select “AAAA” from the Type dropdown

    3. Enter the hostname in Host (ocp42, for example)

    4. Enter the IPv6 address in Points to (use 2 as the last group of digits)

    5. Set TTL to 1 Day

    6. Click the Save button

See also

DNS

Configure reverse DNS#

Linode#

  1. Log into Linode

  2. Select the new server

  3. On the Network tab:

    1. Click Edit RDNS for the IPv4 – Public address

    2. Set Reverse DNS to the server’s FQDN (e.g. ocp12.open-contracting.org)

    3. If the server has an IPv6 /64 block:

      1. Click Edit RDNS for the IPv6 – Range IP block

      2. Set Enter a domain name to the server’s FQDN (e.g. ocp12.open-contracting.org)

Hetzner#

  1. Log into Hetzner

  2. Select the new server

  3. On the IPs tab (default tab):

    1. Under IP addresses: heading, set Reverse DNS entry to the server’s FQDN (e.g. ocp12.open-contracting.org)

    2. If the server has an IPv6 /64 block:

      1. Under the Subnets: heading, click the symbol on the left

      2. Click the Add new Reverse DNS entry link

      3. Set Enter IP to the IPv6 address with 2 as the last group of digits

      4. Set Enter RDNS to the server’s FQDN (e.g. ocp12.open-contracting.org)

3. Apply core changes#

  1. Connect to the server as the root user using SSH, and change its password, using the passwd command. Use a strong password, and save it to OCP’s LastPass account.

    Note

    The root password is needed if you can’t login via SSH (for example, due to a broken configuration).

  2. Add a target to the salt-config/roster file in this repository. Name the target after the service.

    • If the service is moving to a new server, you can use the old target’s name for the new target, and add a -old suffix to the old target’s name.

    • If the service is an instance of CoVE, add a cove- prefix.

    • If the environment is development, add a -dev suffix.

    • Do not include an integer suffix in the target name.

    Note

    If the DNS records have not yet propagated, you can temporarily use the server’s IP address instead of its hostname in the roster.

  3. Configure networking.

  4. Run the onboarding and core state files, which upgrade all packages, configure the hostname and apply the base configuration.

    salt-ssh --log-level=trace TARGET state.apply 'onboarding,core*'
    

    Note

    This step takes 3-4 minutes, so --log-level=trace is used to show activity.

  5. Reboot the server:

    ./run.py TARGET system.reboot
    

Note

The hostname configured in this step and the DNS records created in the previous step are relevant to:

  • verify that an email message has a legitimate source (for example, from cron jobs)

  • communicate between servers (for example, for database replication)

  • identify servers in human-readable way

As such, DNS records that match the hostname must be maintained, until the server is decommissioned.

4. Deploy the service#

  1. If the service is being introduced, add the target to the salt/top.sls and pillar/top.sls files, and include any new state or Pillar files you authored for the service.

  2. If the service is moving to the new server, update occurrences of the old server’s hostname and IP address. (In some cases described in the next step, you’ll need to deploy the related services.)

  3. Deploy the service.

Some IDs might fail (#156):

  • uwsgi, using the service.running function. If so, run:

    ./run.py TARGET service.restart uwsgi
    

5. Migrate from the old server#

  1. Check mail for the root user and, if applicable, each app user

  2. Check the user directory of the root user and, if applicable, each app user

  3. If the server runs a database like PostgreSQL (pg_dump), MySQL (mysqldump) or Elasticsearch, copy the database

  4. If the server runs a web server like Apache or application server like uWSGI, optionally copy the log files

Data support server#

See Data support tasks.

Django applications#

  1. Copy the media directory and the db.sqlite3 file from the app’s directory

OCDS documentation#

  1. Copy the /home/ocds-docs/web directory. For example:

    rsync -avz ocp99:/home/ocds-docs/web/ /home/ocds-docs/web/
    
  2. Stop Elasticsearch, replace the /var/lib/elasticsearch/ directory, and start Elasticsearch. For example:

    systemctl stop elasticsearch
    rm -rf /var/lib/elasticsearch/*
    rsync -avz ocp99:/var/lib/elasticsearch/ /var/lib/elasticsearch/
    systemctl start elasticsearch
    
  3. Mark the elasticsearch package as held back:

    apt-mark hold elasticsearch
    

Prometheus#

  1. Stop Prometheus, replace the /home/prometheus-server/data/ directory, and start Prometheus. For example:

    systemctl stop prometheus-server
    rm -rf /home/prometheus-server/data/*
    rsync -avz ocp99:/home/prometheus-server/data/ /home/prometheus-server/data/
    systemctl start prometheus-server
    
  2. Update the IP addresses in the pillar/prometheus_client.sls file, and deploy to all services

Redash#

See Redash tasks.

Redmine#

  1. Copy the /home/redmine/public_html/files directory. For example:

    rsync -avz ocp99:/home/redmine/public_html/files/ /home/redmine/public_html/files/
    

6. Update external services#

  1. Add the server to Prometheus

  2. Add (or update) the service’s DNS entries in GoDaddy, for example:

    1. Click ADD

    2. Select “CNAME” from the Type dropdown

    3. Enter the public hostname in Host (standard, for example)

    4. Enter the internal hostname in Points to (ocp42.open-contracting.org, for example)

    5. Leave TTL at the 1 Hour default

    6. Click the Save button

    See also

    DNS

  3. Add (or update) the service’s row in the Health of software products and services spreadsheet

  4. Add (or update) managed passwords, if appropriate

  5. Contact Dogsbody Technology Ltd to set up maintenance (see readme)

  6. Delete the old server

If the service is being introduced:

  1. Add its error monitor to Sentry

  2. Add the embed code for Fathom Analytics, if appropriate

If the service uses a new top-level domain name:

  1. Add the domain to Google Search Console