Create a server

Note

Dogsbody Technology requires a lead time of six weeks for new servers. This means OCP typically creates new servers.

A server is created either when a service is moving to a new server, or when a service is being introduced.

As with other deployment tasks, do the setup tasks before (and the cleanup tasks after) the steps below.

1. Collect server requirements

• Number of CPUs

• GBs of RAM

• GBs of disk

• Whether Docker is used

• What DNS to configure (e.g. subdomain)

• What services to configure

2. Create the new server

Create the server via the host’s interface.

Linode

1. Log into Linode

2. Click Create Linode

   1. Set Images to the latest Ubuntu LTS version

   2. Set Region to London UK

   3. Select a Linode Plan

   4. Set Linode Label to the server’s FQDN (e.g. ocp42.open-contracting.org)

   5. Set Add Tags to either Production or Development

   6. Set Root Password to a strong password, and save it to OCP’s LastPass account

   7. Check Backups

   8. Click Create Linode and wait a few minutes for the server to power on

3. From the Linodes list:

   1. Click on the label for the new server

   2. Click Power Off and wait for the server to power off

   3. On the Storage tab:

      1. Resize the “Swap Image” disk to the appropriate size

         The swap size should be at most 200% of RAM and:

         • If RAM is less than 2 GB: at least 100% of RAM

         • If RAM is less than 32 GB: at least 50% of RAM

         • Otherwise, at least 16 GB or 25% of RAM, whichever is greater

         Note

         If the swap image is too small, a swap file is configured by Salt.

      2. Rename the “Swap Image” disk to “### MB Swap Image”

      3. Resize the “Ubuntu ##.04 LTS Disk” disk to the desired size (recommended minimum 20 GB / 20480 MB)

   4. On the Configurations tab:

      1. Click Edit for the “My Ubuntu ##.04 LTS Disk Profile” (or similar) configuration

      2. Uncheck Auto-configure networking (skip if configuring a non-OCP server)

      3. Click Save Changes

   5. Click Power On

   6. Copy SSH Access to your clipboard

4. Open a support ticket with Linode to assign an IPv6 /64 block to the new server.

   Hello,

   Please assign an IPv6 /64 block to the server ocp##.open-contracting.org.

   Thank you,

   Note

   Linode can take a day to close the ticket. In the meantime, proceed with the instructions below. Once the ticket is closed, assign a specific address within the /64 block in the network configuration.

5. If using Docker, configure an external firewall.

Hetzner Cloud

1. Go to the Hetzner Cloud Console

2. Click the Default project

3. Click the Add Server button

   1. Click the Falkenstein location

   2. Click the Ubuntu image

   3. Select a Type

   4. Click the Add SSH key button

      1. Enter your public SSH key in SSH key

      2. Enter your full name in Name

      3. Click the Add SSH key button

      Note

      This adds your public SSH key to /root/.ssh/authorized_keys.

   5. Check the Backups box

   6. Enter the hostname in Server name (ocp42, for example)

   7. Click the Create & Buy now button

4. If using Docker, configure an external firewall.

Hetzner Dedicated

Note

Hetzner dedicated servers are physical servers, and are commissioned to order. Pay attention to any wait times displayed, as some servers may not be available for several days.

1. Go to Hetzner

2. Click the Dedicated menu to browser for a suitable server

3. Check the Server Auction for a comparable server

4. Click the Order button for the chosen server

   1. Set Server Location (no issues to date with the lowest price option)

   2. Set Operating System to the latest Ubuntu LTS version

      Note

      If Ubuntu isn’t an option, you will need to install Ubuntu after these steps. Servers from the Server Auction are delivered in the Hetzner Rescue System.

   3. Set Drives as needed

   4. Click the Order Now button

   5. In the Server Login Details panel, set Type to “Public key” and enter your public SSH key

      Note

      This adds your public SSH key to /root/.ssh/authorized_keys.

   6. Click the Save button

   7. Review the order and click the Checkout button

   8. If prompted, login using OCP’s credentials

   9. Check the “I have read your Terms and Conditions as well as your Privacy Policy and I agree to them.” box

   10. Click the Order in obligation button

5. Wait to be notified via email that the server is ready.

Install Ubuntu

If Ubuntu wasn’t an option, follow these steps to install Ubuntu:

1. Activate and load the Rescue System, if not already loaded.

2. Connect to the server as the root user using the password provided when activating the Rescue System.

3. Test the server hardware:

   1. Test the drives. The SMART values to check vary depending on the drive manufacturer. Ask a colleague if you need help.

      smartctl -t long /dev/<device>
      smartctl -a /dev/<device>
      

   2. Test the hardware RAID controller, if there is one. The software to do so varies depending on the RAID controller. Ask a colleague if you need help.

4. Run the pre-installed Hetzner OS installer (see documentation) and accept the defaults, unless stated otherwise below:

   installimage
   

   1. Select the latest Ubuntu LTS version.

   2. The installer opens a configuration file.

      1. Set DRIVE1, DRIVE2, etc. to the drives you want to use (see documentation). You can identify drives with the smartctl command. If you ordered two large drives for a server that already includes two small drives, you might only set the large drives. For example:

         DRIVE1 /dev/sdb
         DRIVE2 /dev/sdd
         

      2. Set SWRAIDLEVEL 1

      3. Set the hostname (see more under 3. Create DNS records). For example:

         HOSTNAME ocp##.open-contracting.org
         

      4. Create partitions. Set the swap partition size according to the comments in swap.sls. For example:

         PART swap swap 16G
         PART /boot ext2 1G
         PART / ext4 all
         

   3. Press F2 to save

   4. Confirm that you want to overwrite the drives, when prompted

5. Reboot the server:

   reboot
   

6. If using Docker, configure an external firewall.

Install Windows

See also

• Windows Server 2019

• Installing Windows without KVM

Azure

See also

• Pricing calculator, to estimate costs

• Virtual machine series

• Virtual machine sizes naming conventions

1. Log into Azure

2. Click the Virtual machines icon

3. Click the Create menu

4. Click the Azure virtual machine menu item

   1. Set Subscription to “Microsoft Azure Sponsorship (4e98b5b1-1619-44be-a38e-90cdb8e4bc95)”

   2. Set Resource group to “default”

   3. Set Virtual machine name to the server’s FQDN (e.g. ocp42.open-contracting.org)

   4. Set Region to “(Europe) UK South” (or “(US) East US” or “(US) West US 2”)

   5. Leave Security type as Trusted launch virtual machines

   6. Set Image to the latest Ubuntu LTS version

   7. Set Size to an appropriate size (e.g. B2s) (Select No grouping when browsing)

   8. Set Authentication type to “Password”

   9. Set Username to “ocpadmin”

   10. Set Password to a strong password, and save it to OCP’s LastPass account

5. Click the Next : Disks > button

   1. Change OS disk size, if appropriate

      See also

      Expand virtual hard disks on a Linux VM

   2. Set OS disk type to Standard SSD (or Standard HDD in development)

   3. Add additional disks, if appropriate:

      1. Click the Create and attach a new disk link

      2. Click the Change size link

      3. Set Storage type to “Standard SSD”

      4. Click the desired size

      5. Click the OK button

6. Click the Next : Networking > button

   1. Set Virtual network to an appropriate name with a -vnet suffix (e.g. ocp42.open-contracting.org-vnet)

   2. Set Subnet to default (10.0.0.0/24)

   3. Set Public IP to the server’s FQDN (e.g. ocp42.open-contracting.org-ip)

   4. If not using Docker, set NIC network security group to None

   5. If using Docker, set NIC network security group to Advanced

      1. Click the Create new link for Configure network security group

      2. Set Name to the server’s FQDN with a -nsg suffix (e.g. ocp42.open-contracting.org-nsg)

      3. Click the + Add an inbound rule link, to produce rules matching the following:

         Source	Service	Destination port ranges	Protocol	Priority	Name
         Any	SSH	22	TCP	1000	default-allow-ssh
         Any	HTTP	80	TCP	1010	AllowAnyHTTPInbound
         Any	HTTPS	443	TCP	1020	AllowAnyHTTPSInbound
         Any	Custom	*	ICMP	1030	AllowAnyICMPInbound
         139.162.253.17/32	Custom	7231	TCP	1040	AllowPrometheusIPv4Inbound
         2a01:7e00::f03c:93ff:fe13:a12c/128	Custom	7231	TCP	1050	AllowPrometheusIPv6Inbound

      4. Click the OK button

7. Click the Next : Management > button

   1. Check the Enable backup box

   2. Set Recovery Services vault to “default-backups”

   3. Click the Create new link for Backup policy

   4. Set Policy name to “default-backups-daily”

   5. Set Frequency to “Daily”

   6. Set Instant restore to 1

8. Click the Next : Monitoring > button

9. Click the Next : Advanced > button

10. Click the Next : Tags > button

    1. Set Name to the first part of the server’s FQDN (e.g. ocp42)

11. Click the Next : Review + create > button

12. Click the Create button and wait a few minutes for the server to power on

3. Create DNS records

Hostnames follow the format ocp##.open-contracting.org (ocp01, ocp02, etc.). Determine the greatest number by referring to GoDaddy and the salt-config/roster file. Then, increment the number by 1 for the new server, to ensure the hostname is unique and used only once.

Add A and AAAA records

1. Login to GoDaddy

2. If access was delegated, open Delegate Access and click the Access Now button

3. Open DNS Management for open-contracting.org

4. Add an A record for the hostname:

   1. Click the Add New Record button

   2. Select “A” from the Type dropdown

   3. Enter the hostname in Host (ocp42, for example)

   4. Enter the IPv4 address in Points to

   5. Set TTL to 1 Day

   6. Click the Save button

5. If the server has an IPv6 /64 block, add an AAAA record for the hostname:

   1. Click the Add New Record button

   2. Select “AAAA” from the Type dropdown

   3. Enter the hostname in Host (ocp42, for example)

   4. Enter the IPv6 address in Points to (use 2 as the last group of digits)

   5. Set TTL to 1 Day

   6. Click the Save button

See also

DNS TTL standardization

Configure reverse DNS

Linode

1. Log into Linode

2. Select the new server

3. On the Network tab:

   1. Click Edit RDNS for the IPv4 – Public address

   2. Set Enter a domain name to the server’s FQDN (e.g. ocp42.open-contracting.org)

   3. Click the Save button

   4. If the server has an IPv6 /64 block:

      1. Click Edit RDNS for the IPv6 – Range IP block

      2. Set Enter a domain name to the server’s FQDN (e.g. ocp42.open-contracting.org)

      3. Click the Save button

Hetzner Cloud

1. Log into Hetzner Cloud Console

2. Click the Default project

3. On the Primary IPs tab:

   1. Click the … button for the server’s IPv4 address

   2. Click the Edit Reverse DNS menu item

   3. Set Reverse DNS to the server’s FQDN (e.g. ocp42.open-contracting.org)

   4. Click the Edit Reverse DNS button

   5. If the server has an IPv6 /64 block:

      1. Click the … button for the server’s IPv6 address

      2. Click the Edit Reverse DNS menu item

      3. Set the end of the IPv6 address to “::”

      4. Set Reverse DNS to the server’s FQDN (e.g. ocp42.open-contracting.org)

      5. Click the Edit Reverse DNS button

Hetzner Dedicated

1. Log into Hetzner Robot

2. Select the new server

3. On the IPs tab (default tab):

   1. Under IP addresses: heading, set Reverse DNS entry to the server’s FQDN (e.g. ocp42.open-contracting.org)

   2. If the server has an IPv6 /64 block:

      1. Under the Subnets: heading, click the ⊕ symbol on the left

      2. Click the Add new Reverse DNS entry link

      3. Set Enter IP to the IPv6 address with 2 as the last group of digits

      4. Set Enter RDNS to the server’s FQDN (e.g. ocp42.open-contracting.org)

      5. Click the Create button

Azure

1. Log into Azure

2. Select the new server

3. Click on the public IP address:

   1. Enter the hostname in DNS name label (optional) (ocp42, for example)

   2. Click the Save button (at the top)

4. Create an A record in GoDaddy for the configuration (e.g. ocp42..uksouth.cloudapp.azure.com)

4. Apply core changes

1. Connect to the server’s FQDN as the root user (ocpadmin user, if Azure) using SSH, to add it to your known hosts. Then, disconnect.

   Warning

   On macOS, run the ssh command with sudo.

   1. On Hetzner, change the root password, using the passwd command. Use a strong password, and save it to OCP’s LastPass account.

2. Add a target to the salt-config/roster file in this repository. Name the target after the service.

   • If the service is moving to a new server, use the old target’s name for the new target, and add a -old suffix to the old target’s name.

   • If the service is an instance of CoVE, add a cove- prefix.

   • If the environment is development, add a -dev suffix.

   • Do not include an integer suffix in the target name.

   Warning

   On Azure, add user: ocpadmin and sudo: true to the target’s data.

   Tip

   If DNS is not propagated, temporarily set host to the server’s IP address instead of its hostname.

3. Configure networking, adding the target to the pillar/top.sls file, if needed.

   Attention

   If using Docker, add docker: to the service’s Pillar file, to not configure a server-side firewall.

4. Run the onboarding and core state files (replace TARGET).

   salt-ssh --log-level=trace TARGET state.apply 'onboarding,core*'
   

   Note

   This step takes 3-4 minutes, so --log-level=trace is used to show activity.

   Tip

   If configuring a non-OCP server:

   1. Suffix -test to the target’s name in the salt-config/roster file

   2. Comment out the '*' section in the pillar/top.sls file

   3. If configuring Apache, edit the salt/apache/files/404.html file

   The service’s Pillar file needs system_contacts, network.domain, ssh.admin, locale, ntp and, preferably, maintenance sections.

5. Reboot the server:

   ./run.py TARGET system.reboot
   

Note

The hostname configured in this step and the DNS records created in the previous step are relevant to:

• verify that an email message has a legitimate source (for example, from cron jobs)

• communicate between servers (for example, for database replication)

• identify servers in human-readable way

As such, DNS records that match the hostname must be maintained, until the server is decommissioned.

5. Deploy the service

1. If the service is being introduced, add the target to the salt/top.sls and pillar/top.sls files, and include any new state or Pillar files you authored for the service.

2. If the service is moving to the new server, update occurrences of the old server’s hostname and IP address. (In some cases described in the next step, you’ll need to deploy the related services.)

3. Deploy the service.

Some IDs might fail (#156):

• uwsgi, using the service.running function. If so, run:

  ./run.py TARGET service.restart uwsgi
  

6. Migrate from the old server

1. Check mail for the root user and, if applicable, each app user

2. Check the user directory of the root user and, if applicable, each app user

3. If the server runs a database like PostgreSQL (pg_dump), MySQL (mysqldump) or Elasticsearch, copy the database

4. If the server runs a web server like Apache or application server like uWSGI, optionally copy the log files

Data support server

See Data support tasks.

Django applications

1. Copy the media directory and the db.sqlite3 file from the app’s directory

OCDS documentation

1. Copy the /home/ocds-docs/web directory. For example:

   rsync -avz ocp99:/home/ocds-docs/web/ /home/ocds-docs/web/
   

2. Stop Elasticsearch, replace the /var/lib/elasticsearch/ directory, and start Elasticsearch. For example:

   systemctl stop elasticsearch
   rm -rf /var/lib/elasticsearch/*
   rsync -avz ocp99:/var/lib/elasticsearch/ /var/lib/elasticsearch/
   systemctl start elasticsearch
   

3. Mark the elasticsearch package as held back:

   apt-mark hold elasticsearch
   

Prometheus

1. Stop Prometheus, replace the /home/prometheus-server/data/ directory, and start Prometheus. For example:

   systemctl stop prometheus-server
   rm -rf /home/prometheus-server/data/*
   rsync -avz ocp99:/home/prometheus-server/data/ /home/prometheus-server/data/
   systemctl start prometheus-server
   

2. Update the IP addresses in the pillar/prometheus_client.sls file, and deploy to all services

Redash

See Redash tasks.

7. Update external services

1. Add the server to Prometheus

2. Add (or update) the service’s DNS entries in GoDaddy, for example:

   1. Click the Add New Record button

   2. Select “CNAME” from the Type dropdown

   3. Enter the public hostname in Host (standard, for example)

   4. Enter the internal hostname in Points to (ocp42.open-contracting.org, for example)

   5. Leave TTL at the 1 Hour default

   6. Click the Save button

   See also

   DNS

3. Add (or update) the service’s row in the Health of software products and services spreadsheet

4. Add (or update) managed passwords, if appropriate

5. Contact Dogsbody Technology to set up maintenance (see readme)

6. Delete the old server

If the service is being introduced:

1. Add its error monitor to Sentry

2. Add the embed code for Fathom Analytics, if appropriate

If the service uses a new top-level domain name:

1. Add the domain to Google Search Console