Data registry

Migrate from an old server

Dependencies

Tinyproxy

1. Update the allowed IP addresses in the pillar/tinyproxy.sls file.

2. Deploy the docs service, when ready.

Update Salt configuration and halt jobs

1. Check that docker.uid in the server’s Pillar file matches the entry in the /etc/passwd file for the docker.user (deployer).

2. Change cron.present to cron.absent in the salt/registry/init.sls file.

3. Change cron.present to cron.absent in the salt/pelican/backend/init.sls file.

4. Comment out the postgres.backup section of the Pillar file.

5. Deploy the old server and the new server.

6. On the old server:

   1. Delete the /etc/cron.d/postgres_backups file.

   2. docker compose down all containers, except the web and static containers of the data-registry service.

7. Check that no crawls are running at https://collect.data.open-contracting.org/jobs.

   If a crawl is running, Django administrators can cancel jobs.

8. Check that no messages are enqueued at https://rabbitmq.data.open-contracting.org.

   If a job is running in Kingfisher Process, job owners can cancel jobs.

Filesystem

Copy these directories from the old server to the new server, using rsync -avz:

• /data/storage/exporter

• /data/storage/spoonbill

• /home/collect/scrapyd/dbs

• /home/collect/scrapyd/eggs

• /home/collect/scrapyd/logs

Databases

1. Copy the data_registry and spoonbill databases from the old server to the new server, using pg_dump.

2. Copy the exchange_rates rows from the old server to the new server.

Docker apps

Perform the same tasks as for Data support.

Restore Salt configuration and start jobs

1. Change cron.absent to cron.present in the salt/registry/init.sls file.

2. Change cron.absent to cron.present in the salt/pelican/backend/init.sls file.

3. Uncomment the postgres.backup section of the Pillar file.

4. Deploy the new server.