Create a server

A server is created either when an existing service is moving to a new server, or when a new service is being introduced on its own server.

As with other deployment tasks, do the setup tasks before the steps below.

1. Collect server requirements

• Number of CPUs

• GBs of RAM

• GBs of disk

• Whether Docker is used

• What DNS to configure (e.g. subdomain)

• What services to configure

2. Create the new server

Create the server via the host’s interface.

Linode

1. Log into Linode

2. Click Create Linode

   1. Set Linux Distribution to the latest Ubuntu LTS version

   2. Set Region to London, UK (eu-west)

   3. Select a Linode Plan

   4. Set Linode Label to the server’s FQDN (e.g. ocp42.open-contracting.org)

   5. Set Add Tags to either Production or Development

   6. Set Root Password to a strong password, and save it to OCP’s LastPass account

   7. Check Backups

   8. Click Create Linode and wait a few minutes for the server to power on

3. From the Linodes list:

   1. Click on the label for the new server

   2. Click Power Off and wait for the server to power off

   3. On the Storage tab:

      1. Resize the “Ubuntu ##.04 LTS Disk” disk to the desired size (recommended minimum 20 GB / 20480 MB)

      2. Resize the “Swap Image” disk to the appropriate size

         The swap size should be at most 200% of RAM and:

         • If RAM is less than 2 GB: at least 100% of RAM

         • If RAM is less than 32 GB: at least 50% of RAM

         • Otherwise, at least 16 GB or 25% of RAM, whichever is greater

         Note

         If the swap image is too small, a swap file is configured by Salt.

      3. Rename the “Swap Image” disk to “### MB Swap Image”

   4. On the Configurations tab:

      1. Click Edit for the “My Ubuntu ##.04 LTS Disk Profile” (or similar) configuration

      2. Uncheck Auto-configure networking (skip if configuring a non-OCP server)

      3. Click Save Changes

   5. Click Power On

   6. Copy SSH Access to your clipboard

4. Open a support ticket with Linode to assign an IPv6 /64 block to the new server.

   Hello,

   Please assign an IPv6 /64 block to the server ocp##.open-contracting.org.

   Thank you,

   Note

   Linode can take a day to close the ticket. In the meantime, proceed with the instructions below. Once the ticket is closed, assign a specific address within the /64 block in the network configuration.

5. If using Docker, configure an external firewall.

Hetzner Cloud

1. Go to the Hetzner Cloud Console

2. Click the Default project

3. Click the Add Server button

   1. Click the Falkenstein location

   2. Click the Ubuntu image

   3. Select a Type

   4. Click the Add SSH key button

      1. Enter your public SSH key in SSH key

      2. Enter your full name in Name

      3. Click the Add SSH key button

      Note

      This adds your public SSH key to /root/.ssh/authorized_keys.

   5. Check the Backups box

   6. Set Server name to the first part of the server’s FQDN (ocp42, for example)

   7. Click the Create & Buy now button

4. If using Docker, configure an external firewall.

Hetzner Dedicated

Note

Hetzner dedicated servers are physical servers, and are commissioned to order. Pay attention to any wait times displayed, as some servers may not be available for several days.

1. Go to Hetzner

2. Click the Dedicated menu to browser for a suitable server

3. Check the Server Auction for a comparable server

4. Click the Order button for the chosen server

   1. Set Server Location (no issues to date with the lowest price option)

   2. Set Operating System to the latest Ubuntu LTS version

      Note

      If Ubuntu isn’t an option, you will need to install Ubuntu after these steps. Servers from the Server Auction are delivered in the Hetzner Rescue System.

   3. Set Drives as needed

   4. Click the Order Now button

   5. In the Server Login Details panel, set Type to “Public key” and enter your public SSH key

      Note

      This adds your public SSH key to /root/.ssh/authorized_keys.

   6. Click the Save button

   7. Review the order and click the Checkout button

   8. If prompted, login using OCP’s credentials

   9. Check the “I have read your Terms and Conditions as well as your Privacy Policy and I agree to them.” box

   10. Click the Order in obligation button

5. Wait to be notified via email that the server is ready.

Install Ubuntu

If Ubuntu wasn’t an option, follow these steps to install Ubuntu:

1. Activate and load the Rescue System, if not already loaded.

2. Connect to the server as the root user using the password provided when activating the Rescue System.

3. Test the server hardware:

   1. Test the drives. The SMART values to check vary depending on the drive manufacturer. Ask a colleague if you need help.

      smartctl -t long /dev/<device>
      smartctl -a /dev/<device>
      

   2. Test the hardware RAID controller, if there is one. The software to do so varies depending on the RAID controller. Ask a colleague if you need help.

4. Run the pre-installed Hetzner OS installer (see documentation) and accept the defaults, unless stated otherwise below:

   installimage
   

   1. Select the latest Ubuntu LTS version.

   2. The installer opens a configuration file.

      1. Set DRIVE1, DRIVE2, etc. to the drives you want to use (see documentation). You can identify drives with the smartctl command. If you ordered two large drives for a server that already includes two small drives, you might only set the large drives. For example:

         DRIVE1 /dev/sdb
         DRIVE2 /dev/sdd
         

      2. Set SWRAIDLEVEL 1

      3. Set the hostname (see more under 3. Create DNS records). For example:

         HOSTNAME ocp##.open-contracting.org
         

      4. Create partitions. Set the swap partition size according to the comments in swap.sls. For example:

         PART swap swap 16G
         PART /boot ext2 1G
         PART / ext4 all
         

   3. Press F2 to save

   4. Confirm that you want to overwrite the drives, when prompted

5. Reboot the server:

   reboot
   

6. If using Docker, configure an external firewall.

Install Windows

See also

• Windows Server 2019

• Installing Windows without KVM

Azure

See also

• Pricing calculator, to estimate costs

• Virtual machine series

• Virtual machine sizes naming conventions

1. Log into Azure

2. Click the Virtual machines icon

3. Click the Create menu

4. Click the Azure virtual machine menu item

   1. Set Subscription to “Microsoft Azure Sponsorship (4e98b5b1-1619-44be-a38e-90cdb8e4bc95)”

   2. Set Resource group to “default”

   3. Set Virtual machine name to the server’s FQDN (e.g. ocp42.open-contracting.org)

   4. Set Region to “(Europe) UK South” (or “(US) East US” or “(US) West US 2”)

   5. Leave Security type as Trusted launch virtual machines

   6. Set Image to the latest Ubuntu LTS version

   7. Set Size to an appropriate size (e.g. B2s) (Select No grouping when browsing)

   8. Set Authentication type to “Password”

   9. Set Username to “ocpadmin”

   10. Set Password to a strong password, and save it to OCP’s LastPass account

5. Click the Next : Disks > button

   1. Change OS disk size, if appropriate

      See also

      Expand virtual hard disks on a Linux VM

   2. Set OS disk type to Standard SSD (or Standard HDD in development)

   3. Add additional disks, if appropriate:

      1. Click the Create and attach a new disk link

      2. Click the Change size link

      3. Set Storage type to “Standard SSD”

      4. Click the desired size

      5. Click the OK button

6. Click the Next : Networking > button

   1. Set Virtual network to an appropriate name with a -vnet suffix (e.g. ocp42.open-contracting.org-vnet)

   2. Set Subnet to default (10.0.0.0/24)

   3. Set Public IP to the server’s FQDN with an -ip suffix (e.g. ocp42.open-contracting.org-ip)

   4. If not using Docker, set NIC network security group to None

   5. If using Docker, set NIC network security group to Advanced

      1. Click the Create new link for Configure network security group

      2. Set Name to the server’s FQDN with a -nsg suffix (e.g. ocp42.open-contracting.org-nsg)

      3. Click the + Add an inbound rule link, to produce rules matching the following:

         Source	Service	Destination port ranges	Protocol	Priority	Name
         Any	SSH	22	TCP	1000	default-allow-ssh
         Any	HTTP	80	TCP	1010	AllowAnyHTTPInbound
         Any	HTTPS	443	TCP	1020	AllowAnyHTTPSInbound
         Any	Custom	*	ICMP	1030	AllowAnyICMPInbound
         139.162.253.17/32	Custom	7231	TCP	1040	AllowPrometheusIPv4Inbound
         2a01:7e00::f03c:93ff:fe13:a12c/128	Custom	7231	TCP	1050	AllowPrometheusIPv6Inbound

      4. Click the OK button

7. Click the Next : Management > button

   1. Check the Enable backup box

   2. Set Recovery Services vault to “default-backups”

   3. Click the Create new link for Backup policy

   4. Set Policy name to “default-backups-daily”

   5. Set Frequency to “Daily”

   6. Set Instant restore to 1

8. Click the Next : Monitoring > button

9. Click the Next : Advanced > button

10. Click the Next : Tags > button

    1. Set Name to the first part of the server’s FQDN (e.g. ocp42)

11. Click the Next : Review + create > button

12. Click the Create button and wait a few minutes for the server to power on

3. Create DNS records

Hostnames follow the format ocp##.open-contracting.org (ocp01, ocp02, etc.). Determine the greatest number by referring to GoDaddy and the salt-config/roster file. Then, increment the number by 1 for the new server, to ensure the hostname is unique and used only once.

Add A, AAAA and SPF records

1. Login to GoDaddy

2. If access was delegated, open Delegate Access and click the Access Now button

3. Open DNS Management for open-contracting.org

4. Add an A record for the hostname:

   1. Click the Add New Record button

   2. Select “A” from the Type dropdown

   3. Enter the hostname in Name (ocp42, for example)

   4. Enter the IPv4 address in Value

   5. Set TTL to 1 Day

5. Add an SPF record for the hostname, because cron jobs send mail from this hostname:

   1. Click the Add More Records button

   2. Select “TXT” from the Type dropdown

   3. Enter the hostname in Name (ocp42, for example)

   4. Enter the SPF record in Value (v=spf1 a:ocp42.open-contracting.org -all, for example)

   5. Set TTL to 1 Hour

6. If the server has an IPv6 /64 block, add an AAAA record for the hostname:

   1. Get the server’s IPv6 address. Connect to the server and run:

      curl -6 https://icanhazip.com/
      

   2. Click the Add More Records button

   3. Select “AAAA” from the Type dropdown

   4. Enter the hostname in Name (ocp42, for example)

   5. Enter the IPv6 address in Value

   6. Set TTL to 1 Day

7. Click the Save button

See also

DNS TTL standardization

Configure reverse DNS

Linode

1. Log into Linode

2. Select the new server

3. On the Network tab:

   1. Click Edit RDNS for the IPv4 – Public address

   2. Set Enter a domain name to the server’s FQDN (e.g. ocp42.open-contracting.org)

   3. Click the Save button

   4. If the server has an IPv6 /64 block:

      1. Click Edit RDNS for the IPv6 – Range IP block

      2. Set Enter a domain name to the server’s FQDN (e.g. ocp42.open-contracting.org)

      3. Click the Save button

Hetzner Cloud

1. Log into Hetzner Cloud Console

2. Click the Default project

3. On the Primary IPs tab:

   1. Click the … button for the server’s IPv4 address

   2. Click the Edit Reverse DNS menu item

   3. Set Reverse DNS to the server’s FQDN (e.g. ocp42.open-contracting.org)

   4. Click the Edit Reverse DNS button

   5. If the server has an IPv6 /64 block:

      1. Get the server’s IPv6 address. Connect to the server and run:

         curl -6 https://icanhazip.com/
         

      2. Click the … button for the server’s IPv6 address

      3. Click the Edit Reverse DNS menu item

      4. Set the end of the IPv6 address

      5. Set Reverse DNS to the server’s FQDN (e.g. ocp42.open-contracting.org)

      6. Click the Edit Reverse DNS button

Hetzner Dedicated

1. Log into Hetzner Robot

2. Select the new server

3. On the IPs tab (default tab):

   1. Under IP addresses: heading, set Reverse DNS entry to the server’s FQDN (e.g. ocp42.open-contracting.org)

   2. If the server has an IPv6 /64 block:

      1. Get the server’s IPv6 address. Connect to the server and run:

         curl -6 https://icanhazip.com/
         

      2. Under the Subnets: heading, click the ⊕ symbol on the left

      3. Click the Add new Reverse DNS entry link

      4. Set Enter IP to the IPv6 address

      5. Set Enter RDNS to the server’s FQDN (e.g. ocp42.open-contracting.org)

      6. Click the Create button

Azure

1. Log into Azure

2. Select the new server

3. Click on the public IP address:

   1. Enter the first part of the server’s FQDN in DNS name label (optional) (ocp42, for example)

   2. Click the Save button (at the top)

4. Create an A record in GoDaddy for the configuration (e.g. ocp42.uksouth.cloudapp.azure.com)

4. Apply core changes

1. Connect to the server’s FQDN as the root user (ocpadmin user, if Azure) using SSH, to add it to your known hosts. Then, disconnect.

   Warning

   On macOS, run the ssh command with sudo.

   1. On Hetzner, change the root password, using the passwd command. Use a strong password, and save it to OCP’s LastPass account.

2. Add (or update) the target in the salt-config/roster file. Name the target after the service.

   • If the service is moving to a new server, update the host of the existing target, and add a history entry in the file.

   • If the environment is development, add a -dev suffix.

   • Do not include an integer suffix in the target name.

   Tip

   If DNS is not propagated, temporarily set host to the server’s IP address instead of its FQDN.

3. Configure networking, adding the target to the pillar/top.sls file, if needed.

   Attention

   If using Docker, add docker: to the server’s Pillar file, to not configure a server-side firewall.

4. Run the onboarding and core state files (replace TARGET).

   salt-ssh --log-level=trace TARGET state.apply 'onboarding,core*'
   

   Note

   This step takes 3-4 minutes, so --log-level=trace is used to show activity.

   Warning

   On Azure, add --user=ocpadmin and --sudo:

   salt-ssh --log-level=trace TARGET state.apply 'onboarding,core*' --user=ocpadmin --sudo
   

   These flags are not needed for subsequent deployments.

   Tip

   If you get the error message:

   ModuleNotFoundError: No module named 'backports'
   

   Try connecting to the server and running <https://github.com/saltstack/salt/issues/65360#issuecomment-1839127208>:

   pip3 install backports.ssl_match_hostname --break-system-packages
   

   Tip

   If configuring a non-OCP server:

   1. Suffix -test to the target’s name in the salt-config/roster file

   2. Comment out the '*' section in the pillar/top.sls file

   3. If configuring Apache, edit the salt/apache/files/404.html file

   The server’s Pillar file needs system_contacts, network.domain, ssh.admin, locale, ntp and, preferably, maintenance sections.

5. If a disk is larger than 100 GB, reduce reserve space.

6. Reboot the server:

   ./run.py TARGET system.reboot
   

Note

The hostname configured in this step and the DNS records created in the previous step are relevant to:

• verify that an email message has a legitimate source (for example, from cron jobs)

• communicate between servers (for example, for database replication)

• identify servers in human-readable way

As such, DNS records that match the hostname must be maintained, until the server is decommissioned.

5. Deploy the server

New service

1. Add a new target to the salt/top.sls and pillar/top.sls files

2. Deploy the server

3. Add the server to Prometheus

Existing service

1. On the old server:

   1. Check mail for the root user and, if applicable, each app user

   2. Check the user directory of the root user and, if applicable, each app user

2. Open a pull request with the networking changes from the previous step and:

   1. Update occurrences of the old server’s FQDN and IP address in this repository, including in the salt-config/roster and salt/prometheus/files/conf-prometheus.yml files

   2. Make any changes related to new operating system versions (PostgreSQL version, for example)

3. Prepare a migration plan (example), including:

   1. Notify relevant users of the change

   2. Reduce the TTLs of CNAME entries in GoDaddy

   3. If the server uses SSL certificates, copy the /etc/apache2/md directory

   4. If the server runs any Django applications (like CoVE), copy the media directory and the db.sqlite3 file from the app’s directory

   5. If the server runs a database like PostgreSQL (pg_dump), MySQL (mysqldump) or Elasticsearch, copy the database

   6. If the server runs a web server like Apache or application server like PHP-FPM, optionally copy the log files

   7. Test the new server, by manually changing the /etc/hosts file on your local machine

   8. Deploy the server

   9. Deploy the Prometheus service

   See also

   • Data registry

   • Data support

   • OCDS documentation

   • Prometheus

4. Once the pull request is approved, merge the pull request and implement the migration plan.

7. Update external services

1. Add (or update) the server’s DNS entries in GoDaddy, for example:

   1. Click the Add New Record button

   2. Select “CNAME” from the Type dropdown

   3. Enter the public hostname in Name (standard, for example)

   4. Enter the internal hostname in Value (ocp42.open-contracting.org, for example)

   5. Leave TTL at the 1 Hour default

   6. Click the Save button

   See also

   DNS

2. Add (or update) the service’s row in the Health of software products and services spreadsheet

3. Contact the relevant server manager to set up monitoring and maintenance

4. Delete the old server

If any services are being introduced:

1. Configure error monitoring with Sentry

2. Configure web analytics with Fathom Analytics, if appropriate

If any services use a new top-level domain name:

1. Add the domain to Google Search Console